⚡AgentSkills
🛢️ Data Engineering · Lakehouse & Streaming

Blend streaming and batch where each belongs

Lambda-to-Kappa migrations, microbatch cadences, and consistency contracts across paths.

advanced~45 minData EngineersAnalytics EngineersPlatform Data Teams

Steps

  1. 1Audit which consumers actually need seconds-fresh vs minutes-fresh truth
  2. 2Consolidate logic into one stream processor where latency allows (Kappa direction)
  3. 3Keep batch for heavy history recomputes; document boundary contracts
  4. 4Unify schemas across paths or drift WILL create divergent truths
  5. 5Watermark strategy consistent between live and backfill processing
  6. 6Reconciliation job compares paths continuously; alert divergence early

Common Pitfalls

  • ▲Two codebases implementing subtly different business logic forever
  • ▲Backfills overwriting streaming results with stale semantics

Commands

Install with skills CLI
$ npx skills add aniruddhaadak80/skills --skill lakehouse-streaming-microbatch-streaming-hybrid
Install globally
$ npx skills add aniruddhaadak80/skills --skill lakehouse-streaming-microbatch-streaming-hybrid -g

Tags

#lambda#kappa#streaming#data-engineering#lakehouse-streaming

Related skills

Choose and operate open table formats deliberatelyadvanced

Iceberg/Delta/Hudi tradeoffs, compaction strategy, and catalog governance preventing metadata debt.

🛢️ Data Engineering·~40m
Open →