Make batch pipelines rerunnable by design

Steps

1. 1Key every write by logical date/partition; overwrite partitions, never append blind
2. 2Design stages to be safely re-executable from any point of failure
3. 3Separate extraction, transformation, loading with checkpoints between
4. 4Backfill = same code, different parameter — prove it in staging first
5. 5Emit per-run metrics: rows in/out, duration, null-rates into a meta table
6. 6Alert on freshness SLA breach before stakeholders notice stale dashboards

Common Pitfalls

• ▲append-only loads duplicating on retry
• ▲One giant DAG task failing everything for one bad record

Commands

$ npx skills add aniruddhaadak80/skills --skill pipelines-batch-pipeline-idempotency

$ npx skills add aniruddhaadak80/skills --skill pipelines-batch-pipeline-idempotency -g

Tags