data-pipeline-engineering
Design reliable, observable, and maintainable data pipelines (batch and streaming). Use when building ETL/ELT, data ingestion, transformation, or analytics pipelines.
- License 许可
- MIT
- 最近更新
- 2026-07-22
- Stars
- ★ 7
- Forks
- 1
01 前言 Front Matter
- name
- data-pipeline-engineering
- description
- Design reliable, observable, and maintainable data pipelines (batch and streaming). Use when building ETL/ELT, data ingestion, transformation, or analytics pipelines.
03 SKILL.md 正文
Data Pipeline Engineering
Overview
Data pipelines are production systems. They need the same attention to reliability, testing, and observability as any other critical path.
When to Use
- Building or reviewing data ingestion and transformation pipelines
- Moving from ad-hoc scripts to production-grade data flows
- Improving reliability or cost of existing pipelines
Key Concerns
- Idempotency and exactly-once / at-least-once semantics
- Schema evolution and data contracts
- Late / out-of-order data
- Monitoring of freshness, volume, and quality
- Backfill and replay capabilities
- Cost of storage and compute
Principles
- Treat data pipelines as products with SLAs
- Make failures visible and recoverable
- Prefer declarative and testable transformations where possible
Verification
- Pipeline health (freshness, success rate, data quality) is monitored
- Reprocessing or backfill is possible without major heroics
- Schema changes are handled deliberately
04 同仓库更多 Skills
compliance-monitoring
Design compliance monitoring and testing programs that verify controls work in practice and surface issues before regulators do. Use when building monitoring plans, compliance testing, or continuous control monitoring.
complaints-handling
Operate complaints-handling processes that meet regulatory standards — logging, timely resolution, root-cause analysis, and fair outcomes. Use when designing or improving regulated complaints frameworks or analyzing complaint themes for conduct risk.
aml-kyc-basics
Apply AML/KYC fundamentals — customer due diligence, monitoring, suspicious activity escalation, and recordkeeping — for financial crime compliance. Use when designing onboarding controls, transaction monitoring concepts, or AML program components in financial services.
ai-model-compliance
Apply compliance controls to AI and automated decision systems — transparency, fairness, human oversight, documentation, and regulatory alignment. Use when deploying AI in regulated processes (credit, fraud, customer decisions) or responding to AI governance expectations.
data-quality-management
Manage data quality end-to-end — dimensions, rules, remediation, and prevention — beyond one-off cleaning scripts. Use when building sustained data quality programs or fixing systemic accuracy, completeness, or consistency problems.
data-lineage
Capture and use data lineage — where data comes from, how it transforms, and what depends on it — for impact analysis, debugging, and trust. Use when tracing metric origins, assessing change impact, or diagnosing broken pipelines.