Databricks was started by the original creators of Apache Spark in 2013. The company's mission is to "accelerate the development of data science and machine learning." Databricks provides a cloud-based data platform that makes using Apache Spark for data engineering, data science, and machine learning easy. The company’s revenue was estimated at $425 million in 2021 and $1.2 billion in 2022 (170% growth).
Apache Spark was created in 2009 at the University of California, Berkeley AMPLab. It was developed as a response to the limitations of MapReduce, a popular big data processing framework at the time. MapReduce is a batch processing framework, meaning it must read all input data into memory before it can begin processing. This can be a significant bottleneck for large datasets. On the other hand, Spark is a general-purpose cluster computing system that can be used for batch and streaming processing. It uses a more in-memory approach to processing, which can significantly improve performance for large datasets. Spark is very well suited for analytical workloads, and its user base includes data engineers, data scientists, and machine learning engineers.
Spark is now one of the world's most popular big data processing frameworks, used by more than 22,000 organizations (Databricks is estimated to have more than 7,000 worldwide customers).
Similar to other enterprise software platforms, Databricks created a partner program to accelerate the adoption of its solutions. Below are some highlights of its service partner ecosystem.
As of Q2 2023, Databricks has approximately 455 service partners (consulting, implementation, managed services, etc). 328 partners (72%) are based in North America and Europe. Asia-based services partners account for 14% of total partners but a third of the partner headcount, implying these Asia partners are of a larger scale relative to other partners.
Looking at the data by partner headcount tiers, we find that 50% of Databricks services partners are small-scale (with fewer than 100 employees). 80% of all service partners have fewer than 1,000 employees, and these 361 partners account for only 1% of the total headcount tracked. It is estimated that more than 100,000 certified Databricks and Apache Spark professionals are worldwide. It seems boutique data engineering, analytics, and data science services companies see value in leveraging the Databricks platform for their customers.
Databricks is a key component of the modern data stack. Apache Spark's open-source nature makes the overall approach more accessible for a larger set of users, and the platform’s web-based interface, notebook environment, and library of pre-built machine learning models make it appealing to data science and data analytics end-users.
At Alten Capital we enjoy the technology services space and are fond of data / AI offerings. Please reach out to find ways to partner and scale your organization.