Data Governance for Public Agencies: The Playbook Before AI

Why governance has to precede AI, not follow it

Every agency conversation about AI eventually runs into the same wall: nobody can say with confidence what data feeds the system, who is allowed to touch it, or whether sensitive fields were scrubbed before a model ever saw them. That's not a model problem. It's a data management problem that existed long before anyone mentioned AI, and it doesn't go away because a vendor promises a chatbot.

NIST's AI Risk Management Framework (AI RMF 1.0) is explicit about this: the 'Map' and 'Measure' functions assume you already know your data's provenance, quality, and sensitivity classification. If you can't map your data, you can't map your AI risk. Agencies that skip straight to procurement usually end up retrofitting governance under audit pressure, which is slower and more expensive than doing it first.

This piece lays out the governance components that actually matter in practice: a working catalog, defensible lineage, least-privilege access control, disciplined PII/PHI handling, and a layered data architecture (commonly called the medallion pattern) that makes all of the above enforceable rather than aspirational.

Cataloging sensitive data: you can't protect what you haven't inventoried

A data catalog for a public agency is not a spreadsheet of table names. It's a living record of what each dataset contains, its sensitivity tier, its legal basis for collection, its retention schedule, and its owner. Without this, every access request and every AI use case becomes a manual investigation.

Classification should map to real regulatory categories your agency already answers to: PII under state privacy statutes and, where applicable, CCPA-style requirements; PHI under HIPAA if health data is in scope; and controlled unclassified information (CUI) under NIST SP 800-171 for federal contract data. Tag at the field level, not just the dataset level, because a single case-management table often mixes public program data with a Social Security number column that changes the entire handling requirement.

The payoff shows up fast: when someone asks 'can this dataset be used to train a model,' the catalog answers it in minutes instead of triggering a three-week legal review.

Lineage: proving where data came from and where it went

Lineage is the record of a data element's life: source system, every transformation applied, every system it was copied into, and every place it was ultimately consumed, including any AI or reporting pipeline. Agencies underestimate this until an incident, an audit, or a public records request forces the question 'where did this number come from,' and nobody can answer it without days of manual tracing.

Good lineage is captured automatically as part of the pipeline, not documented after the fact in a wiki that goes stale. Modern orchestration and transformation tools (dbt, Airflow, and similar) emit lineage metadata as a byproduct of running the pipeline, which means it stays accurate as the pipeline changes. That automation matters more than the specific tool choice.

Lineage is also what makes model risk manageable. If a model's output looks wrong, lineage lets you trace back through every transformation to find where the error entered, rather than treating the model as an unauditable black box.

Least-privilege access and the PII/PHI handling rules that make it real

Least-privilege access means every user, service account, and pipeline job gets the minimum permissions needed to do its job, nothing more, and access is reviewed on a schedule rather than granted once and forgotten. In practice this means role-based access control tied to job function, not individual named exceptions that accumulate over years.

For PII and PHI specifically, the handling rules need to be enforced in the pipeline, not just written in a policy document: masking or tokenizing sensitive fields before they reach analytics or development environments, encrypting data at rest and in transit consistent with NIST 800-171 control families, and logging every access to raw sensitive data with enough detail to support an audit trail.

A fail-closed posture is the practical test of whether this is real: if the access-control system can't verify a permission, does it deny by default, or does it quietly let the request through? Agencies that can answer 'deny by default' have working access control. Agencies that can't are relying on hope.

• Role-based access mapped to job function, reviewed quarterly at minimum
• Masking/tokenization applied before data reaches non-production environments
• Encryption at rest and in transit aligned to NIST 800-171 control families
• Access logging sufficient to reconstruct who touched what, and when
• Fail-closed defaults: unverifiable permission requests are denied, not passed through

The medallion pattern: raw, conformed, curated

The medallion architecture organizes data into three layers. Raw data lands exactly as it arrives from source systems, untouched, which preserves an auditable original copy. Conformed (sometimes called 'silver') data is cleaned, deduplicated, and standardized, with PII masked or tokenized as required. Curated ('gold') data is business-ready, aggregated for specific use cases like dashboards, reporting, or model training.

This layering matters for governance because each layer can carry different access rules. Raw data might be restricted to a small data engineering team with a documented need. Curated data, stripped of direct identifiers, can be opened up to a much wider set of analysts and, eventually, AI applications, without re-exposing sensitive fields every time someone needs a report.

It also solves a practical problem agencies hit constantly: when a downstream number looks wrong, you can isolate whether the issue is in the source system, the conforming logic, or the curation step, instead of re-auditing the entire pipeline end to end.

How VAERESOURCE builds this

We build governance into the pipeline itself rather than treating it as a document that sits next to the pipeline. That means catalog and classification metadata generated as part of ingestion, lineage captured automatically by the orchestration layer, access control that fails closed by default, and a medallion structure that keeps raw sensitive data isolated from the curated layer AI tools actually touch.

Every one of these controls is designed to be auditable after the fact and to keep a human in the loop for anything touching PII, PHI, or CUI: masking rules and access grants are reviewable, lineage is queryable, and nothing moves from raw to curated without a documented transformation a person can inspect. That's the foundation an AI initiative needs before it's worth building the AI part at all.