A scalable foundation for connected, discoverable, and standards-compliant public data.
Overview
We've developed a universal metadata infrastructure designed to unify and connect datasets across platforms—enabling automated discovery, enrichment, and redirection to authoritative data sources. This system supports government agencies, research institutions, and civic data initiatives in improving how public data is cataloged, accessed, and maintained.
The Challenge
Public data lives across dozens of platforms—each with its own metadata standards, publishing rules, and update cycles. This fragmentation creates blind spots for agencies trying to maintain visibility across datasets, and introduces risk around outdated or inconsistent information. Many organizations also face staffing or system constraints that limit their ability to scale metadata publishing, even when their data is valuable and already public.
Our Approach
We built a universal metadata ingestion and enrichment platform that connects directly to public data portals, APIs, and registries. It automatically harvests and normalizes metadata, monitors for changes, and synchronizes updates into a centralized catalog—without duplicating or replicating the source data.
The platform's metadata-first architecture ensures high-quality, discoverable records with minimal overhead. When a dataset changes at the source, the system detects the update, enriches the metadata, and refreshes client records—keeping everything in sync.
Key Capabilities
- Multi-source ingestion: Connects to a wide range of public data platforms, including Socrata, OpenGov, CKAN, and others
- Metadata enrichment: Harmonizes incoming metadata against DCAT, ISO 19115, and client-specific schemas
- Change detection: Monitors source systems for new or modified datasets and automatically updates client catalogs
- Optional dataset ingestion: Enables previewing, validation, or re-publishing of full datasets if needed
- Interoperability: Maintains alignment with key open data standards (DCAT, schema.org/DataCatalog, ISO)
Recent Work: HHS Open Data Modernization
Client: Major Federal Health Agency (via prime contractor)
Scope: Metadata enrichment, AI-assisted ingestion, and Socrata integration
In 2025, we supported the HHS Open Data Initiative in partnership with a federal contractor. This work focused on improving access to public health data through the HealthData.gov platform by deploying a scalable, standards-compliant metadata system.
Our team developed ingestion pipelines that aggregated content from key federal sources—including SAMHSA, NIH, and PubMed—automatically aligning metadata with DCAT 3.0 and the client's Socrata-based schema. LLM-based enrichment generated intelligent tags and keywords to improve discovery, searchability, and cross-domain linking.
The platform supports both metadata-only records (with direct links to original sources) and full dataset ingestion—giving agencies flexibility in how they publish while supporting transparency and outreach.
This work contributed to the HealthData.gov relaunch and advanced the goals of the Living HHS Open Data Plan, which emphasizes machine readability, open standards, and long-term data accessibility.
Impact
- Published 15,000+ machine-readable metadata records across federal health domains
- Enabled scalable, automated ingestion from multiple authoritative sources
- Ensured full compliance with DCAT 3.0 and HHS metadata standards
- Used LLMs to improve metadata quality, consistency, and findability
- Supported a major federal open data modernization effort with lasting value
Looking Ahead
The Universal Metadata Infrastructure is more than a single implementation—it's the blueprint for a metadata-driven public data ecosystem. By continuously syncing source systems into a standards-aligned, always-current catalog, it empowers any organization to improve data transparency, reduce duplication, and maintain confidence in the information they provide.
This work demonstrates how thoughtful, scalable engineering can help agencies manage large, complex data environments—while keeping discoverability, governance, and usability front and center.