Data Lake Architecture Pack

Published 2026-05-05, last updated 2026-05-05

The Metadata Trap in Modern Data Lakes

We see this pattern constantly. A team decides to build a data lake from scratch to handle complex, multiple data sources coming from databases, REST APIs, and streaming platforms. The initial design looks clean on a whiteboard. Bronze layers for raw ingestion, Silver for cleaning, Gold for analytics. But within weeks, the architecture collapses under its own weight because metadata is treated as an afterthought.

When you design a strategy for a data lake, the storage format is the easy part. The hard part is defining how metadata flows through the system. Without a metadata-first approach, you end up with schema drift that breaks downstream consumers, orphaned files that waste storage costs, and governance policies that exist only in documentation. You start building a GCP Data Platform Pack or a Data Warehouse Pack to patch the holes, but the root cause remains: the lakehouse lacks a unified governance layer.

We built the Data Lake Architecture Pack so you don't have to reverse-engineer medallion patterns every time you spin up a new environment. This skill defines the medallion architecture workflow, governance standards, and federated query patterns upfront. It forces the AI agent to respect the separation of concerns between raw ingestion, transformation, and consumption. Metadata isn't just tags; it's the foundation of data governance ^[2]. If your catalog doesn't track lineage across the medallion layers, you're not building a lakehouse; you're building a data swamp with extra steps.

The Hidden Cost of Ungoverned Medallion Layers

Ignoring governance in a medallion architecture doesn't just create technical debt; it creates business risk. When Bronze, Silver, and Gold layers operate in silos, you lose the ability to enforce row-level security or column masking consistently. For medallion architecture, you need to create schemas for bronze, silver, and gold layers within each catalog, ensuring that policies propagate correctly ^[4]. Without this, a junior engineer can accidentally expose PII in the Gold layer because the Silver transformation didn't apply the masking policy.

The downstream impact is severe. Data quality issues in Bronze propagate to Silver and explode in Gold, forcing analysts to rebuild dashboards and engineers to debug lineage that doesn't exist. You waste hours chasing data anomalies that could have been caught by a validator. If you're also managing Data Quality Pack requirements, the friction doubles when your lake architecture doesn't expose the necessary metadata for profiling and anomaly detection.

Furthermore, governance failures kill trust. If your platform can't answer "where did this column come from?" or "who has access to this dataset?", analytics teams stop using the lake. They spin up their own shadow pipelines. You end up maintaining two versions of the truth. A metadata-driven approach treats metadata as a first-class asset and picks a catalog people actually use, ensuring lineage, row filters, and masking are applied at the source ^[8]. Without this, your dbt Analytics Engineering Pack models will fail because the underlying data contracts are unstable.

The cost isn't just in hours. It's in compliance audits, delayed product launches, and the reputational damage of a data breach caused by misconfigured access controls. We've seen teams spend months refactoring their lake structure because they skipped the governance phase. Don't let that be you.

How a Platform Team Fixed Bronze, Silver, and Gold Governance

Imagine a platform team that inherits a legacy data lake with no clear layering. The Bronze layer contains raw JSON blobs from various APIs. The Silver layer has a mix of cleaned Parquet files and unstructured logs. The Gold layer is a mess of ad-hoc views that break whenever the source schema changes. The team decides to implement a proper medallion architecture to organize data pipelines by organizing data into distinct Bronze (raw), Silver (cleaned), and Gold (analytics) layers ^[3].

The first step is implementing a metadata-driven design. The medallion architecture is the proven way for modern data lakes to simplify data pipelines, governance, and reporting ^[5]. The team uses the Data Lake Architecture Pack to define the Iceberg DDLs for each layer. Bronze tables use append-only writes with strict schema validation. Silver tables enforce schema evolution and apply transformations that preserve lineage. Gold tables are optimized for query performance with partitioning and compaction.

Next, they tackle federation. Instead of copying data between systems, they configure Trino to query across Hive, Iceberg, and relational sources. This allows analysts to join Gold layer aggregates with external CRM data without moving a single byte. The medallion architecture 101 approach ensures that each layer has a clear purpose: Bronze for durability, Silver for quality, Gold for value ^[6].

The team also implements a security-first pattern where Bronze, Silver, and Gold each run as separate catalogs with distinct governance policies. This isolates risk and allows fine-grained access control. A secure medallion architecture pattern on Azure Databricks demonstrates how this separation prevents cross-layer contamination and enforces row-level security effectively ^[7]. By treating metadata as a first-class asset, the team ensures that every table in the lake has a clear owner, a defined retention policy, and a documented lineage path.

The result? The platform becomes self-service. Engineers can spin up new pipelines by referencing the pack's templates. Analysts can trust the Gold layer because governance is baked into the Silver transformation. The ETL Pipeline Pack integration ensures that extraction and loading follow the medallion standards, while the Multi-Tenant Knowledge Architecture Pack allows different business units to share the same lake without conflicting policies.

What Changes Once Governance and Federation Are Locked In

With the Data Lake Architecture Pack installed, your AI agent stops guessing and starts architecting. You get production-grade Apache Iceberg DDLs that handle schema evolution, partitioning, and compaction out of the box. The Trino federated connector configuration is ready to query across heterogeneous sources without data movement.

Governance is no longer a manual checklist. The governance policies JSON schema enforces row-level security, column masking, and access control aligned with medallion layer requirements. Every table created by the agent is validated against the canonical Iceberg metadata spec. If a required field like format-version or table-uuid is missing, the pipeline fails before it hits production.

You also get a validated project structure. The validate_lakehouse.sh script checks for required template files and verifies SQL syntax keywords. The iceberg-schema-validator.py script ensures metadata integrity. This automated validation catches errors that would otherwise take hours to debug. The worked example pipeline demonstrates how Bronze ingest, Silver transformation, Gold aggregation, and governance policy application work together in a declarative YAML format.

The result is a scalable analytics platform that handles structured and unstructured data sources with ease. You can integrate this with a Database Reliability Pack to monitor the health of your lakehouse infrastructure, ensuring high availability and performance. The medallion architecture simplifies data pipelines by enforcing clear boundaries between layers, making it easier to maintain and scale over time.

What's in the Data Lake Architecture Pack

• skill.md — Orchestrator skill file that defines the medallion architecture workflow, governance standards, and federated query patterns. References all templates, references, scripts, validators, and examples to guide the AI agent through end-to-end data lake design.
• templates/medallion-iceberg.sql — Production-grade Apache Iceberg DDL for Bronze, Silver, and Gold layers. Includes schema evolution, partitioning, compaction directives, and metadata queries aligned with Iceberg spec.
• templates/trino-federated-connector.yaml — Production-grade Trino connector configuration for federated queries across Hive, Iceberg, and relational sources. Includes catalog properties, security settings, and query routing rules.
• templates/governance-policies.json — JSON schema and policy definitions for row-level security, column masking, data retention, and access control aligned with medallion layer governance requirements.
• references/iceberg-metadata-spec.md — Embedded canonical knowledge from Apache Iceberg documentation. Covers table metadata structure, snapshots, statistics, manifest queries, and migration procedures.
• references/trino-federation-guide.md — Embedded canonical knowledge from Trino documentation. Covers federated query execution, native database querying via TABLE(query()), and identifier canonicalization.
• scripts/validate_lakehouse.sh — Executable bash script that validates the lakehouse project structure, checks for required template files, verifies SQL syntax keywords, and exits non-zero on failure.
• validators/iceberg-schema-validator.py — Executable Python script that validates Iceberg table metadata JSON against the canonical spec fields. Exits non-zero if required fields (format-version, table-uuid, location, etc.) are missing or malformed.
• examples/worked-medallion-pipeline.yaml — Worked example of a complete medallion pipeline definition. Demonstrates Bronze ingest, Silver transformation, Gold aggregation, and governance policy application in a declarative YAML format.

Stop Guessing. Start Architecting.

You don't need to reinvent the wheel for every data lake project. The medallion architecture is a proven pattern, but implementing it correctly requires discipline. Governance, federation, and metadata management are the hard parts. We've encoded that discipline into this skill.

Stop building data swamps. Start building scalable, governed lakehouses. Upgrade to Pro to install the Data Lake Architecture Pack and ship with confidence.