UPDATED 10:37 EDT / JUNE 14 2024

John Furrier, theCUBE Research, assessed data engineering innovations and Databricks' strategy at the Data + AI Summit 2024. AI

AnalystANGLE: Databricks’ bet on bottom-up data engineering gains momentum

In the eyes of theCUBE Research’s John Furrier (pictured), the bet being made is clear: Databricks Inc. believes the new infrastructure around data engineering will come from the bottom up rather than the top down.

The analysis came during the Data + AI Summit, an event that saw Databricks outline its strategic shifts in data engineering and interoperability. Conference attendees saw the company move the needle, according to Rob Strechay, principal analyst at theCUBE Research.

Rob Strechay, George Gilbert, John Furrier, theCUBE Research discuss data engineering innovations and what's next for Databricks at the Data + AI Summit 2024.

Rob Strechay, George Gilbert and John Furrier of theCUBE talk about what’s next for Databricks.

“Over 66% of the contributions to Spark are coming from outside of Databricks these days,” Strechay said. “That kind of shows that if they can get that to happen with Unity while they’ve open-sourced Unity Catalog and they can get some other momentum around that, I don’t know if it’s game over, but it definitely puts them in a really pole position.”

Furrier and Strechay were joined by their co-analyst George Gilbert in an analysis segment at the Data + AI Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed Databricks’ market position and the strategic importance of open data formats.

Data engineering: The latest moves in the Databricks vs. Snowflake rivalry

A big revelation out of the Data + AI Summit was the catalog layer, according to Gilbert. The Unity Catalog is a comprehensive operational and business catalog.

“Their aspiration is to govern your entire data estate,” Gilbert said. “By open sourcing it, Databricks is saying, ‘Forget the data catalog now. We’ve got that covered. You won’t have to care. The new platform is the catalog.’”

For months, theCUBE has been covering the rivalry been Databricks and Snowflake Inc. It poses an open question whether it gives Databricks an advantage if the company can force Snowflake to unplug compute from their data.

“I think that [Databricks is] taking an offensive strategic bet on plugging in through Unity Catalog and Delta Sharing that you can even go and read data out of Snowflake and then share it using Delta Sharing,” Strechay said.

Both Databricks and Snowflake are accelerating right now. But the rate of acceleration may be higher for Databricks, according to Strechay.

“I think they’re neck and neck at this point. And I think the acceleration has been, over the past year, looking at the ETR data in accounts that were shared … it was like a 50-50 split,” Strechay said. “They’re still accelerating in those accounts. In fact, their actual perception and the percent change has grown in those accounts.”

Databricks is also winning when it comes to those people who like to code, Stechay added. The company also understands that data has gravity but isn’t going to fight about where the data lives.

“It can be in your account. Longer-term, it could be in the Databricks account, and we’ll see how that works out, but that’s not their fight,” he said. “Their fight is, ‘We’re going to have the best compute engine, the best execution engine and best catalog to really bring the metadata to impact on top of that data.’ I think that’s their North Star right now, and I think that’s where they’re executing.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the Data + AI Summit:

Photo: SiliconANGLE

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU