How Apache Spark lit up the tech world and outshone its big data brethren

The Register queries author Matei Zaharia on a decade of the project

Interview  Big data is no longer hailed as the “new oil.” It has gone out of fashion, both in terms of hype and because its foundational technology – Apache Hadoop – was surpassed by cloud-based blob storage such as AWS S3. However, a sister project born in the big data era has become more influential in the modern world of LLMs and internet-scale data systems.…