Because the Information Platform workforce at Databricks, we leverage our personal platform to supply an intuitive, composable, and complete Information and AI platform to inside information practitioners in order that they will safely analyze utilization and enhance our product and enterprise operations. As our firm matures, we’re particularly motivated to ascertain information governance to allow safe, compliant and cost-effective information operations. With hundreds of staff and tons of of groups analyzing information, we’ve to border and implement constant requirements to attain information governance at scale and continued compliance. We recognized Unity Catalog (UC), usually out there as of August 2022, as the inspiration for establishing commonplace governance practices and thus migrating 100% of our inside lakehouse to Unity Catalog turned a high firm precedence.
Why migrate to Unity Catalog to attain Information Governance?
Information migrations are HARD – and costly. So we requested ourselves: Can we obtain our governance targets with out migrating all the information to Unity Catalog?
We had been utilizing the default Hive Metastore (HMS) in Databricks to handle all of our tables. Constructing our personal information governance options from scratch on high of HMS could be a wasteful endeavor, setting us again a number of quarters. Unity Catalog, alternatively, supplied great worth out of the field:
- Any information on HMS was readable by anyone. UC securely helps fine-grained entry.
- HMS doesn’t present lineage or audit logs. Lineage help is essential to understanding information flows and empowering efficient information lifecycle administration. Together with audit logs, this gives observability about information adjustments and propagation.
- With higher integration with the in-product search characteristic, UC allows a greater expertise for customers to annotate and uncover high-quality information.
- Delta Sharing, question federation and catalog binding present efficient choices to create cross-region information meshes with out creating safety or compliance dangers.
Unity Catalog migration begins with a governance technique
At a excessive stage, we might go down one in all two paths:
- Elevate-and-shift: Copy all of the schemas and tables as is from legacy HMS to a UC catalog whereas giving all people learn entry to all information. This path is low stage of effort within the quick time period. Nevertheless, we threat bringing alongside outdated datasets and incoherent/dangerous practices motivated by HMS or natural development. The chance of needing a number of massive subsequent migrations to scrub in place could be excessive.
- Transformational: Selectively migrate datasets whereas establishing a core construction for information group in Unity Catalog. Whereas this path requires extra effort within the quick time period, it gives an important course-correction alternative. Subsequent rounds of incremental (smaller) clean-up could also be essential.
We selected the latter. It allowed us to put the groundwork to introduce future governance coverage whereas offering the requisite skeleton to construct round. We constructed infrastructure to allow paved paths that ensured clear information possession, naming conventions and intentional entry, versus opening entry to all staff by default.
One such instance is the catalog group technique we selected upfront:
| Catalog | Function | Governance |
|---|---|---|
| Customers | Particular person consumer areas (schemas) |
|
| Group | Collaborative areas for customers who work collectively |
|
| Integration | House for particular integration tasks throughout groups |
|
| Essential | Manufacturing surroundings. |
|
Challenges
Our inside information lake had develop into extra of a “information swamp” over time, as a result of beforehand highlighted lack of lineage and entry controls in HMS. We didn’t have solutions to three fundamental questions essential to any migration:
- Who owns desk foo?
- Are all of the tables upstream of foo already migrated to the brand new location?
- Who’re all of the downstream clients of desk foo that have to be up to date?
Now think about that lack of visibility on the scale of our information lake:
Now think about a four-person engineering workforce pulling this off with none devoted program administration help in 10 months.
Our Method
The migration can virtually be damaged down into 4 phases.
Section 1: Make a Plan, by Unlocking Lineage for HMS
We collaborated with the Unity Catalog and Discovery groups to construct information a lineage pipeline for HMS on inside Databricks workspaces. This allowed us to determine the next:
A. Who up to date a desk and when?
B. Who reads from a desk and when?
C. Whether or not the information was consumed through a dashboard, a question, a job or a pocket book?
A allowed us to deduce the almost certainly house owners of the tables. B and C helped set up the “blast radius” of an imminent migration i.e., who’re all of the downstream shoppers to inform and which of them are mission essential? Moreover, B allowed us to estimate how a lot “stale” information was mendacity round within the information lake that could possibly be merely ignored (and finally deleted) to simplify the migration.
This observability was essential in estimating the general migration effort, creating a sensible timeline for the corporate and informing what tooling, automation and governance insurance policies our workforce wanted to spend money on.
After proving its utility internally, we now present our clients a path to allow HMS Lineage for a restricted time frame to help with the migration to Unity Catalog. Discuss to your account consultant to allow it.
Section 2: Cease the Bleeding, by Imposing Information Retention
Lineage observability revealed two essential insights:
- There have been a ton of “stale” tables within the information lake, that had not been consumed shortly, and have been in all probability not price migrating
- The brand new desk creation price on HMS was pretty excessive. This needed to be introduced down considerably (virtually 0) for us to efficiently cutover to Unity Catalog finally and have a shot at a profitable migration.
These insights led us to spend money on information retention infrastructure upfront and roll out the next insurance policies, which turbo-charged our effort.
- Rubbish-Acquire Stale Information: This coverage, shipped proper out of the gates, deleted any HMS desk that wasn’t up to date for 30 days. We supplied groups with a grace interval to register exemptions. This tremendously decreased the dimensions of the “haystack” and allowed information practitioners to deal with information that truly mattered.
- No New Tables in HMS: 1 / 4 after the migration was underway and there was company-wide consciousness, we rolled out a coverage to forestall the creation of any new HMS tables. Whereas retaining the legacy metastore in examine, this measure successfully positioned a moratorium on information pipelines nonetheless on HMS as they might now not be prolonged or modified to supply new tables.
With these in place, we have been now not chasing a shifting goal.
Section 3: Distribute the work, utilizing Self-Serve Monitoring Instruments
Most organizations within the firm have a distinct cadence for planning, totally different processes for monitoring execution and totally different priorities and constraints. As a small information platform workforce, our aim was to attenuate coordination and empower groups to confidently estimate, coordinate, and monitor their OWN dataset migration efforts. To this finish, we turned the lineage observability information into executive-level dashboards, the place every workforce might perceive the excellent work on their plate, each as information producers and shoppers, ordered by significance. These allowed additional drill-downs to the supervisor and particular person contributor ranges. These have been up to date on a every day cadence for progress-tracking functions.
Moreover, the information was aggregated right into a leaderboard, permitting management to have visibility and apply strain when required. The worldwide monitoring dashboard additionally served the twin objective of a lookup desk the place shoppers might discover the areas of latest tables migrated to Unity Catalog.
The emphasis on managing the folks and course of dynamics of the Databricks group was an important success driver. Each group is totally different and tailoring your strategy to your organization is essential to your success.
Section 4: Deal with the Lengthy Tail, utilizing Automation
Successfully herding the lengthy tail is make or break for a migration with 2.5K information shoppers and over 50K consuming entities throughout each workforce of the corporate. Counting on information producers or our small platform workforce to trace and chase down all these shoppers to do their half by the deadline was a non-starter.
Beneath the moniker “Migration Wizard”, we constructed an information platform that allowed information producers to register the tables to be deleted or migrated to a catalog in Unity Catalog. Together with the desk paths (new and previous), producers supplied operational metadata just like the end-of-life (EOL) date for the legacy desk and how you can contact with questions or considerations.
The Migration Wizard would then:
- Leverage lineage to detect consumption and notify downstream groups. This focused strategy allowed groups to not should repeatedly inundate all people with information deprecation messages
- On EOL day, render a “gentle deletion” through lack of entry and purge the information every week later
- Auto-update DBSQL queries relying on the legacy information to learn from the brand new location
Thus with just a few strains of config, the information producer was successfully and confidently decoupled from the migration effort with out having to fret about downstream impression. Automation continued notifying clients and in addition supplied a swift repair for question breakage found after the deprecation set off was pulled.
Subsequently, the power to auto-update DBSQL and pocket book queries from legacy HMS tables to new UC alternate options has been added to the product to help our clients of their journey to Unity Catalog.
Sticking the Touchdown
In February 2024, we eliminated entry to Hive Metastore and began deleting all remaining legacy information. Given the quantity of communication and coordination, this probably disruptive change turned out to be clean. Our adjustments didn’t set off any incidents, and we have been in a position to declare “Success” quickly after.
We noticed fast value advantages as unowned jobs that failed as a result of adjustments might now be turned off. Dashboards silently deprecated now failed whereas incurring marginal compute value and could possibly be equally sunsetted.
A essential goal was to determine options to make migration to Unity Catalog simpler for Databricks clients. The Unity Catalog and different product groups gathered intensive actionable suggestions for product enhancements. The Information Platform workforce prototyped, proposed and architected numerous options that will probably be rolling out to clients shortly.
The Journey Continues
The transfer to Unity Catalog unshackled information practitioners, considerably lowering information sprawl and unlocking new options. For instance, the Advertising and marketing Analytics workforce noticed a 10x discount in tables managed through a lineage-enabled identification (and deletion) of deprecated datasets. Entry administration enhancements and lineage, alternatively, have enabled highly effective one-click entry obtainment paths and entry discount automation.
For extra on this, try our discuss on unified governance @ Information + AI Summit 2024. In future blogs on this collection, we may even dive deeper into governance selections. Keep tuned for extra about our journey to Information Governance!
We want to thank Vinod Marur, Sam Shah and Bruce Wong for his or her management and help and Product Engineering @ Databricks—particularly Unity Catalog and Information Discovery—for his or her continued partnership on this journey.
