Data architecture
The organisational and platform layer above the processing engines: who owns data, how it moves, and how anyone finds and trusts it.
| # | File | Covers |
|---|---|---|
| 01 | Data mesh and federated architecture | domain ownership, data as a product, data contracts, federated query, when mesh is the wrong answer |
| 02 | ETL, orchestration and dbt | ELT vs ETL, CDC and watermarks, idempotent backfills, Airflow/Dagster/Temporal, dbt models and tests, data quality |
| 03 | Catalogue, governance and lineage | discovery, trust and impact analysis, DataHub/OpenMetadata/Glue, column-level lineage, OpenLineage, governance that engineers implement |
| 04 | Lambda and Kappa architectures | batch plus speed layer, the two-codebase cost, replay from a durable log, why the lakehouse dissolves the split |
| 05 | Snowflake, BigQuery, Databricks and Synapse | storage/compute separation, virtual warehouses vs slots, Unity Catalog namespace, dedicated vs serverless pools, the cost traps |
| 06 | BI and dashboarding tools | where the metric is defined, Power BI and Tableau and Superset, slow dashboards, warehouse cost |