Databricks Unveils New Innovations for its Industry Leading Data Lakehouse Platform
The Best Data Warehouse is the Lakehouse
Organizations like Amgen, AT&T, Northwestern Mutual and
Walgreens, are making the move to the lakehouse because of its ability to deliver analytics on both structured and unstructured data. Today, Databricks unveiled new data warehousing capabilities in its platform to further enhance analytics workloads:
Data Governance Highlighted as a Top Priority with Advanced Capability for Unity Catalog
Unity Catalog, generally available on AWS and Azure in the coming weeks, offers a centralized governance solution for all data and AI assets, with built-in search and discovery, automated lineage for all workloads, with performance and scalability for a lakehouse on any cloud. Also, Databricks introduceddata lineage for Unity Catalog earlier this month, significantly expanding data governance capabilities on the lakehouse and giving businesses a complete view of the entire data lifecycle. With data lineage, customers gain visibility into where data in their lakehouse came from, who created it and when, how it has been modified over time, how it's being used across data warehousing and data science workloads, and much more.
Enhanced Data Sharing Enabled By Databricks Marketplace and Cleanrooms
As the first marketplace for all data and AI, available in the coming months,
Databricks Marketplace provides an open marketplace to package and distribute data and analytics assets. Going beyond marketplaces that simply offer datasets, Databricks Marketplace enables data providers to securely package and monetize a host of assets such as data tables, files, machine learning models, notebooks and analytics dashboards. Data consumers can easily discover new data and AI assets, jumpstart their analysis and gain insights and value from data faster. For example, instead of acquiring access to a dataset and investing their own time to develop and maintain dashboards to report on it, they can choose to simply subscribe to pre-existing dashboards that already provide the necessary analytics. Databricks Marketplace is powered by Delta Sharing, allowing data providers to share their data without having to move or replicate the data from their cloud storage. This allows providers to deliver data to other clouds, tools, and platforms from a single source.
Databricks is also helping customers share and collaborate with data across organizational boundaries. Cleanrooms, available in the coming months, will provide a way to share and join data across organizations with a secure, hosted environment and no data replication required. In the context of media and advertising, for example, two companies may want to understand audience overlap and campaign reach. Existing clean room solutions have limitations, as they are commonly restricted to SQL tools and run the risk of data duplication across multiple platforms. With Cleanrooms, organizations can easily collaborate with customers and partners on any cloud and provide them the flexibility to run complex computations and workloads using both SQL and data science-based tools - including Python, R, and Scala - with consistent data privacy controls.
MLflow 2.0 Streamlines and Accelerates Production Machine Learning at Scale
Databricks continues to lead the way in MLOps innovation with the introduction of MLflow 2.0. Getting a machine learning pipeline into production requires setting up infrastructure, not just writing code. This can be difficult for new users and tedious for everyone at scale. MLflow Pipelines, made possible by MLflow 2.0, now handles the operational details for users. Instead of setting up orchestration of notebooks, users can simply define the elements of the pipeline in a configuration file and MLflow Pipelines manages execution automatically. Looking beyond MLflow, Databricks also added Serverless Model Endpoints to directly support production model hosting, as well as built-in Model Monitoring dashboards to help teams analyze the real-world model performance.
Delta Live Tables Includes Industry First Performance Optimizer for Data Engineering Pipelines
Delta Live Tables (DLT) is the first ETL framework to use a simple, declarative approach to building reliable data pipelines. Since its
launch earlier this year, Databricks continues to expand DLT with new capabilities including the introduction of a new performance optimization layer designed to speed up execution and reduce costs of ETL. Additionally, new Enhanced Autoscaling is purpose-built to intelligently scale resources with the fluctuations of streaming workloads, and Change Data Capture (CDC) for Slowly Changing Dimensions - Type 2, easily tracks every change in source data for both compliance and machine learning experimentation purposes.
To learn more about the Databricks Lakehouse Platform visit: https://databricks.com/product/data-lakehouse. Tune in virtually for more Data + AI Summit keynotes by registering here for the free, immersive online experience.
About Databricks
Databricks is the data and AI company. More than 7,000 organizations worldwide — including Comcast, Condé Nast, H&M, and over 40% of the Fortune 500 — rely on the Databricks Lakehouse Platform to unify their data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe. Founded by the original creators of Delta Lake, Apache Spark™, and MLflow, Databricks is on a mission to help data teams solve the world's toughest problems. To learn more, follow Databricks on Twitter, LinkedIn and Facebook.
Safe Harbor Statement
This information is provided to outline Databricks' general product direction and is for informational purposes only. Customers who purchase Databricks services should make their purchase decisions relying solely upon services, features, and functions that are currently available. Unreleased features or functionality described in forward-looking statements are subject to change at Databricks discretion and may not be delivered as planned or at all.
Contact: [email protected]
Logo - https://mma.prnewswire.com/media/1160675/Databricks_Logo.jpg
