🛢️ Data Engineering · Batch Pipelines

Test warehouse data like software (Databricks/Delta)

Layered data quality tests that block bad data from reaching downstream consumers.

intermediate~30 minData EngineersAnalytics EngineersPlatform Data Teams

Steps

1Tier 0 volume/freshness: row counts within expected bands per partition
2Tier 1 structural: uniqueness of keys, nullability, referential integrity
3Tier 2 business invariants: revenue reconciles, statuses in enum sets
4Fail loudly at tier matching blast radius; warn below it
5Store test results historically to catch slow drift trends
6Review false-positive rate monthly; noisy tests get ignored
7Use Delta EXPECTATIONS for inline quality gates
8Track table properties for freshness metadata

Common Pitfalls

▲Tests only in BI layer where nobody sees red
▲100% pass-rate suites proving nothing was tested

Commands

Install with skills CLI

$ npx skills add aniruddhaadak80/skills --skill data-quality-tests-warehouse-databricks

Install globally

$ npx skills add aniruddhaadak80/skills --skill data-quality-tests-warehouse-databricks -g

Tags

#data-quality#testing#data-engineering#pipelines-batch

Install this skill

$ npx skills add aniruddhaadak80/skills --skill data-quality-tests-warehouse-databricks -a opencode

SKILL.md

# Test warehouse data like software (Databricks/Delta)

> Layered data quality tests that block bad data from reaching downstream consumers.

## When to Use This Skill

Layered data quality tests that block bad data from reaching downstream consumers.

## Steps

1. Tier 0 volume/freshness: row counts within expected bands per partition
2. Tier 1 structural: uniqueness of keys, nullability, referential integrity
3. Tier 2 business invariants: revenue reconciles, statuses in enum sets
4. Fail loudly at tier matching blast radius; warn below it
5. Store test results historically to catch slow drift trends
6. Review false-positive rate monthly; noisy tests get ignored
7. Use Delta EXPECTATIONS for inline quality gates
8. Track table properties for freshness metadata

## Common Pitfalls

- Tests only in BI layer where nobody sees red
- 100% pass-rate suites proving nothing was tested

View source on GitHub ↗

Idempotent stages, partitioned writes, and backfill strategies so failures heal instead of corrupt.

🛢️ Data Engineering·~35m

Open →

Evolve schemas without breaking consumersadvanced

Schema contracts, compatibility modes, and deprecation flows across pipeline boundaries.

🛢️ Data Engineering·~40m

Open →

Test warehouse data like software (Snowflake)intermediate

Layered data quality tests that block bad data from reaching downstream consumers.

🛢️ Data Engineering·~30m

Open →