🛢️ Data Engineering · Batch Pipelines

Evolve schemas without breaking consumers

Schema contracts, compatibility modes, and deprecation flows across pipeline boundaries.

advanced~40 minData EngineersAnalytics EngineersPlatform Data Teams

Steps

1Publish schema contracts per dataset with owner + compatibility mode (backward default)
2CI-validate producer changes against registered contracts
3Add columns nullable-first; never repurpose existing fields silently
4Rename via add-new → dual-write → migrate → drop-old, with telemetry at each step
5Notify consumer list (derived from query logs) before breaking changes
6Archive old schema versions; make lineage queryable

Common Pitfalls

▲"Temporary" column reuse changing meaning mid-history
▲Producers shipping struct changes on Friday evenings

Commands

Install with skills CLI

$ npx skills add aniruddhaadak80/skills --skill pipelines-batch-schema-evolution-contracts

Install globally

$ npx skills add aniruddhaadak80/skills --skill pipelines-batch-schema-evolution-contracts -g

Tags

#schema#contracts#governance#data-engineering#pipelines-batch

Install this skill

$ npx skills add aniruddhaadak80/skills --skill pipelines-batch-schema-evolution-contracts -a opencode

SKILL.md

# Evolve schemas without breaking consumers

> Schema contracts, compatibility modes, and deprecation flows across pipeline boundaries.

## When to Use This Skill

Schema contracts, compatibility modes, and deprecation flows across pipeline boundaries.

## Steps

1. Publish schema contracts per dataset with owner + compatibility mode (backward default)
2. CI-validate producer changes against registered contracts
3. Add columns nullable-first; never repurpose existing fields silently
4. Rename via add-new → dual-write → migrate → drop-old, with telemetry at each step
5. Notify consumer list (derived from query logs) before breaking changes
6. Archive old schema versions; make lineage queryable

## Common Pitfalls

- "Temporary" column reuse changing meaning mid-history
- Producers shipping struct changes on Friday evenings

View source on GitHub ↗

Idempotent stages, partitioned writes, and backfill strategies so failures heal instead of corrupt.

🛢️ Data Engineering·~35m

Open →

Test warehouse data like software (Snowflake)intermediate

Layered data quality tests that block bad data from reaching downstream consumers.

🛢️ Data Engineering·~30m

Open →

Test warehouse data like software (BigQuery)intermediate

Layered data quality tests that block bad data from reaching downstream consumers.

🛢️ Data Engineering·~30m

Open →