Data Mesh Manager

Databricks Integration

Integration of Databricks with the Data Mesh Manager.

Databricks Connector

Databricks Connector is an open-source component that integrates Data Mesh Manager / Data Contract Manager with Databricks. It is based on the SDK and available as a Docker image. The source code can also be forked to implement custom integrations.

Features

  • Asset Synchronization: Sync tables and schemas of the Unity catalog to the Data Mesh Manager as Assets.
  • Access Management: Listen for AccessActivated and AccessDeactivated events in the Data Mesh Manager and grants access on Databricks to the data consumer.

Links

Databricks Asset Bundles

Databricks Asset Bundles (DABs) are the new format to bundle data products on Databricks. They contain all files necessary to run a data product on Databricks, including the code and the configuration. We've extended the standard open source template for a Databricks Asset Bundle with the Data Mesh Manager integration. Our extended template automatically registers the data product with its data contract at the Data Mesh Manager through the REST API.

# Create a new Databricks Asset Bundle using the Data Mesh Manager template
databricks bundle init https://github.com/datamesh-manager/databricks-bundle-python-template

Links