A security lead at a utility client asked us a question we hear a version of on nearly every engagement: “Can you tell me right now which tables our trading desk’s AI agent is allowed to read?”
Nobody in the room could answer it. They had policies. Eleven pages, approved by their legal department, spelling out exactly how FERC’s Standards of Conduct wall off transmission data from the marketing and trading side of the business. Those eleven pages had no mechanical connection to the 38,000 tables sitting in their lakehouse.
That gap between a written requirement and an enforced control is where most enterprise AI programs stall. It is also where the money goes when things break.
Governance is what makes AI deployable
Gartner predicts organizations will abandon 60% of AI projects through 2026 for lack of AI-ready data. The security numbers point the same direction. IBM’s 2026 Cost of a Data Breach Report found that 21% of breached organizations had an incident involving an AI model or application, up from 13% a year earlier. Among those organizations, 92% lacked proper AI access controls. Only 40% of breached organizations applied access controls to AI models and data at all, and 68% had no AI governance policy in place.
The cost of getting it wrong is concentrated in highly regulated industries. Healthcare averaged $6.64 million per breach, financial services $6.29 million, and the U.S. average hit $11.5 million. Unapproved AI tools turned up in 43% of the incidents, and one in five breaches brought a regulatory fine.
Agents change the shape of this risk. A human analyst who queries a table they shouldn’t see is one incident, usually caught in review. An agent running under a service principal with a broad grant queries everything it can reach, quickly, and writes the results somewhere you didn’t plan for. The control has to sit on the data itself and hold at query time, whatever the agent asks for.
How access works in regulated industries
Access depends on what the data is, where it lives, and why you’re touching it. Encode those rules with roles and every intersection of job, geography, purpose, and status has to be pre-computed and provisioned as its own group. We’ve walked into environments with more than 4,000 groups, most of them created to handle one exception, none of them documented well enough to audit. Every new requirement multiplies the count.
- A pharmaceutical company where the same column is readable by a commercial analyst in the U.S., hidden from an analyst with the identical job title in the EU, and hidden from everyone while a trial is blinded.
- A utility where consumer identifiers are usable for billing and restricted for marketing analytics, and the line between the two moves with each customer’s consent state.
- A bank where account numbers are visible to fraud investigators on an open case and masked the rest of the time.
None of these examples map cleanly to “who you are.” The pharma example depends on geography and trial status. The utility example depends on purpose and a consent flag that changes without warning. The bank example depends on case assignment, which changes daily. RBAC has no native concept for this, so the only way to express it is to keep declaring roles: commercial_analyst_us, commercial_analyst_eu, commercial_analyst_us_unblinded, and so on for every new wrinkle.
The typical workaround usually results in the creation of multiple views designed to handle a multitude of audiences. Use a row filter for the EU restriction or a column mask for the blinded trial. Each view is an object that has to be created, granted, kept in sync with the underlying table’s schema, and re-pointed every time the source table changes. Multiply that by every table and every rule and you’re maintaining a shadow layer of views whose only job is to patch over what the access control model can’t express natively.
What Databricks ABAC in Unity Catalog changed
Attribute-based access control (ABAC) breaks the cycle by attaching the logic to the data itself. Tag a column as PII or country=EU and write a policy to evaluate conditions with has_tag() and has_tag_value(). The rule lives in one place, applies everywhere the tag appears, and updates automatically when the tag changes — no new group, no new view, no re-grant, no ongoing maintenance. When a new table lands with the right tags, the policy covers it immediately with no per-table configuration.
Databricks took governed tags (Databricks ABAC) to GA on April 2, 2026, and row filter and column mask policies to GA on April 28, 2026. With this change, Databricks solved enforcement. The remaining work is configuring it, and that’s where Lovelytics’ engagements focus.
Two steps, both done by hand today
With Databricks ABAC, getting from a written policy to an enforced control takes two steps, and right now, both are manual.
Step one: turn the requirement into a policy. Take a rule like “consumer data can’t be used for marketing without consent.” Someone has to decide what that means in practice — which tag marks the data, which masking function applies, which groups and service principals are exempt so pipelines don’t break. That’s a working session between legal, security, and data engineering, and it ends with a piece of SQL.
Step two: turn the policy into tags. The policy above only works if the right columns are actually tagged. Skip this step, and the policy does nothing. Worse, if affirmative tagging is on, an untagged column just gets hidden — including columns people need.
This second step is where programs stall. A policy checking for a “PII” tag is inert until every PII column in the estate has that tag on it.
The scale is the problem. Take an estate with 40,000 tables and 25 columns each — that’s a million column-level tagging decisions, and column tags in Unity Catalog don’t inherit from the table, so each one has to be applied individually. Even at a brisk 100 columns an hour, one steward needs 10,000 hours to get through it. Put five stewards on it full-time for a year and you still won’t finish, because new tables keep landing faster than you can tag the old ones.
And volume isn’t even the whole story:
- Custom classifications need custom work. Databricks catches obvious things like SSNs and credit card numbers automatically. Your own four-tier sensitivity scale or a contract-specific data-use restriction isn’t in that list — someone has to define and validate the rule for it.
- Tagging is itself a security control. Whoever can change a tag can change which policy applies to a table. That means every tag change needs to be logged: who did it, when, and why.
- The estate never holds still. New tables land daily across dev, test, prod, and DR. Tags have to reach all of them and stay in sync, or your lower environments quietly become the weak point.
Here is our recommendation
Five things we would recommend to a data executive starting this work, whether or not they call us.
- Write the taxonomy before you write any tags. Aim for 15 to 25 governed tags, each traceable to a named regulation, contract clause, or internal standard. If nobody can state the requirement behind a tag, that tag will be argued about for the life of the program.
- Default to hidden. Affirmative tagging, which Databricks documents as securing new tables by default with control tags, means a catalog-level policy masks or filters data until the required tags are present. New tables arrive protected. Sequence this carefully: switch it on before you’ve worked through the tagging backlog and you’ll break production reporting.
- Separate the people who write policies from the people who apply tags. Governance and security own the policies. Data stewards, who know what’s in the columns, own the tags.
- Have a model propose and a person approve. Give an LLM the column name, the full table definition, and a few sample values, and it will classify most columns correctly and explain its reasoning. Put a steward in front of the result with a review queue, capture every approval in an audit log, and only auto-apply patterns you’ve already validated by hand.
- Design for the second year. The tooling has to detect new objects, replicate a steward’s decision to identical columns elsewhere, push tags across environments, and reprocess tables that were empty on the first pass.
It is important to be mindful of a few items that cause challenges for teams: column tags don’t inherit from tables, so MATCH COLUMNS only sees column-level tags. Also, standard compute needs DBR 16.4 or later to read ABAC-protected tables. Time travel and clones fail on protected tables unless the principal is listed in the policy’s EXCEPT clause. Finally, a materialized view, whose pipeline run-as identity is subject to a mask, will permanently contain masked data.
Where Lovelytics comes in
Our data governance practice built an accelerator for the second translation, because our clients told us that performing it by hand was unacceptable and unsustainable.
A classification engine runs on serverless compute in the customer’s own workspace, calls a Databricks model serving endpoint, and evaluates every table and column against configurable rules: best-fit classification, scoring thresholds, custom tags, and Databricks governed tags. A Databricks App gives stewards a review queue where they edit, approve, and apply the results as Unity Catalog tags. Every operation lands in an audit table. Tags replicate across environments and to matching columns automatically. It runs entirely on Databricks-native services and extends Data Classification rather than replacing it. Nothing leaves the customer’s environment.
In the demo we showed at Data + AI Summit, tagging for access control ran roughly 100x faster than a steward working through the same columns by hand.
A typical implementation is an 8-week program followed by 4 weeks of support, saving thousands of hours of manual entry and review and accelerating compliance and mitigating data risk by months
