Databricks was started by the original creators of Apache Spark in 2013. The company's mission is to "accelerate the development of data science and machine learning." Databricks provides a cloud-based data platform that makes it easy to use Apache Spark for data engineering, data science, and machine learning. The company’s revenue was estimated to be $425 million in 2021 and $1.2 billion in 2022 (170% growth).
Apache Spark was created in 2009 at the University of California, Berkeley AMPLab. It was developed as a response to the limitations of MapReduce, a popular big data processing framework at the time. MapReduce is a batch processing framework, which means that it must read all of the input data into memory before it can begin processing. This can be a significant bottleneck for large datasets. Spark, on the other hand, is a general-purpose cluster computing system that can be used for both batch and streaming processing. It uses a more in-memory approach to processing, which can significantly improve performance for large datasets. Spark is very well suited for analytical workloads, and its user base includes data engineers, data scientists and machine learning engineers.
Spark is now one of the most popular big data processing frameworks in the world, used by more than 22,000 organizations (Databricks is estimated to have more than 7,000 worldwide customers).
Similar to other enterprise software platforms, Databricks created a partner program to accelerate adoption of their solutions. Below some highlights of their services partner ecosystem.
As of Q2 2023, Databricks has approximately 455 services partners (consulting, implementation, managed services, etc). 328 partners (72%) are based in North America and Europe. Asia-based services partners account for 14% of total partners, but a third of the partner headcount, implying these Asia partners are of a larger scale relative to other partners.
If we look at the data by partner headcount tiers we find that 50% of Databricks services partners are of small scale (have fewer than 100 employees). 80% of all services partners have fewer than 1,000 employees and these 361 partners account for only 1% of the total headcount tracked. It is estimated there are more than 100,000 certified Databricks and Apache Spark professionals worldwide. It seems boutique data engineering, analytics and data science services companies see value in leveraging the Databricks platform for their customers.
Databricks is a key component of the modern data stack. The open source nature of Apache Spark makes the overall approach more accessible for a larger set of users, and the platform’s web-based interface, notebook environment and library of pre-built machine learning models make it appealing for data science and data analytics end-users.
At Alten Capital we enjoy the technology services space and are fond of data / AI offerings. Please reach out to find ways to partner and scale your organization.