[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M70JSD7V6TAA5YFAAGWWVPKY":3,"skill-same:01M70JSD7V6TAA5YFAAGWWVPKY":73},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":44,"linkOnly":45,"markdownContent":46,"markdownBody":47,"frontMatter":48,"refName":49,"commitSha":50,"skillDirectory":51,"forksCount":52,"repositoryHtmlUrl":53,"repositoryHomepageUrl":54,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M70JSD7V6TAA5YFAAGWWVPKY","airflow-dag-patterns",null,"Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.","shengdabai\u002FTony-Claude-Code-Skills","shengdabai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F116417419?v=4","01M70F6D2WY5PQV5A92BKYJQEW","Tony Sheng","01M70F6D3GAAAYZ5H4CC1G8VXM",3,"MIT","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fwshobson-agents\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fairflow-dag-patterns\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43],"airflow","patterns","airflow dag patterns","airflow dag","dag","scheduling","best practices","use","deployment","for","data","airflow dags","data pipelines","apache airflow","apache","operators","batch","when","workflows","testing","2026-11-10 10:12:30",false,"---\nname: airflow-dag-patterns\ndescription: Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.\n---\n\n# Apache Airflow DAG Patterns\n\nProduction-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.\n\n## When to Use This Skill\n\n- Creating data pipeline orchestration with Airflow\n- Designing DAG structures and dependencies\n- Implementing custom operators and sensors\n- Testing Airflow DAGs locally\n- Setting up Airflow in production\n- Debugging failed DAG runs\n\n## Core Concepts\n\n### 1. DAG Design Principles\n\n| Principle | Description |\n|-----------|-------------|\n| **Idempotent** | Running twice produces same result |\n| **Atomic** | Tasks succeed or fail completely |\n| **Incremental** | Process only new\u002Fchanged data |\n| **Observable** | Logs, metrics, alerts at every step |\n\n### 2. Task Dependencies\n\n```python\n# Linear\ntask1 >> task2 >> task3\n\n# Fan-out\ntask1 >> [task2, task3, task4]\n\n# Fan-in\n[task1, task2, task3] >> task4\n\n# Complex\ntask1 >> task2 >> task4\ntask1 >> task3 >> task4\n```\n\n## Quick Start\n\n```python\n# dags\u002Fexample_dag.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.operators.empty import EmptyOperator\n\ndefault_args = {\n    'owner': 'data-team',\n    'depends_on_past': False,\n    'email_on_failure': True,\n    'email_on_retry': False,\n    'retries': 3,\n    'retry_delay': timedelta(minutes=5),\n    'retry_exponential_backoff': True,\n    'max_retry_delay': timedelta(hours=1),\n}\n\nwith DAG(\n    dag_id='example_etl',\n    default_args=default_args,\n    description='Example ETL pipeline',\n    schedule='0 6 * * *',  # Daily at 6 AM\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'example'],\n    max_active_runs=1,\n) as dag:\n\n    start = EmptyOperator(task_id='start')\n\n    def extract_data(**context):\n        execution_date = context['ds']\n        # Extract logic here\n        return {'records': 1000}\n\n    extract = PythonOperator(\n        task_id='extract',\n        python_callable=extract_data,\n    )\n\n    end = EmptyOperator(task_id='end')\n\n    start >> extract >> end\n```\n\n## Patterns\n\n### Pattern 1: TaskFlow API (Airflow 2.0+)\n\n```python\n# dags\u002Ftaskflow_example.py\nfrom datetime import datetime\nfrom airflow.decorators import dag, task\nfrom airflow.models import Variable\n\n@dag(\n    dag_id='taskflow_etl',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'taskflow'],\n)\ndef taskflow_etl():\n    \"\"\"ETL pipeline using TaskFlow API\"\"\"\n\n    @task()\n    def extract(source: str) -> dict:\n        \"\"\"Extract data from source\"\"\"\n        import pandas as pd\n\n        df = pd.read_csv(f's3:\u002F\u002Fbucket\u002F{source}\u002F{{ ds }}.csv')\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def transform(extracted: dict) -> dict:\n        \"\"\"Transform extracted data\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(extracted['data'])\n        df['processed_at'] = datetime.now()\n        df = df.dropna()\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def load(transformed: dict, target: str):\n        \"\"\"Load data to target\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(transformed['data'])\n        df.to_parquet(f's3:\u002F\u002Fbucket\u002F{target}\u002F{{ ds }}.parquet')\n        return transformed['rows']\n\n    @task()\n    def notify(rows_loaded: int):\n        \"\"\"Send notification\"\"\"\n        print(f'Loaded {rows_loaded} rows')\n\n    # Define dependencies with XCom passing\n    extracted = extract(source='raw_data')\n    transformed = transform(extracted)\n    loaded = load(transformed, target='processed_data')\n    notify(loaded)\n\n# Instantiate the DAG\ntaskflow_etl()\n```\n\n### Pattern 2: Dynamic DAG Generation\n\n```python\n# dags\u002Fdynamic_dag_factory.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.models import Variable\nimport json\n\n# Configuration for multiple similar pipelines\nPIPELINE_CONFIGS = [\n    {'name': 'customers', 'schedule': '@daily', 'source': 's3:\u002F\u002Fraw\u002Fcustomers'},\n    {'name': 'orders', 'schedule': '@hourly', 'source': 's3:\u002F\u002Fraw\u002Forders'},\n    {'name': 'products', 'schedule': '@weekly', 'source': 's3:\u002F\u002Fraw\u002Fproducts'},\n]\n\ndef create_dag(config: dict) -> DAG:\n    \"\"\"Factory function to create DAGs from config\"\"\"\n\n    dag_id = f\"etl_{config['name']}\"\n\n    default_args = {\n        'owner': 'data-team',\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    }\n\n    dag = DAG(\n        dag_id=dag_id,\n        default_args=default_args,\n        schedule=config['schedule'],\n        start_date=datetime(2024, 1, 1),\n        catchup=False,\n        tags=['etl', 'dynamic', config['name']],\n    )\n\n    with dag:\n        def extract_fn(source, **context):\n            print(f\"Extracting from {source} for {context['ds']}\")\n\n        def transform_fn(**context):\n            print(f\"Transforming data for {context['ds']}\")\n\n        def load_fn(table_name, **context):\n            print(f\"Loading to {table_name} for {context['ds']}\")\n\n        extract = PythonOperator(\n            task_id='extract',\n            python_callable=extract_fn,\n            op_kwargs={'source': config['source']},\n        )\n\n        transform = PythonOperator(\n            task_id='transform',\n            python_callable=transform_fn,\n        )\n\n        load = PythonOperator(\n            task_id='load',\n            python_callable=load_fn,\n            op_kwargs={'table_name': config['name']},\n        )\n\n        extract >> transform >> load\n\n    return dag\n\n# Generate DAGs\nfor config in PIPELINE_CONFIGS:\n    globals()[f\"dag_{config['name']}\"] = create_dag(config)\n```\n\n### Pattern 3: Branching and Conditional Logic\n\n```python\n# dags\u002Fbranching_example.py\nfrom airflow.decorators import dag, task\nfrom airflow.operators.python import BranchPythonOperator\nfrom airflow.operators.empty import EmptyOperator\nfrom airflow.utils.trigger_rule import TriggerRule\n\n@dag(\n    dag_id='branching_pipeline',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n)\ndef branching_pipeline():\n\n    @task()\n    def check_data_quality() -> dict:\n        \"\"\"Check data quality and return metrics\"\"\"\n        quality_score = 0.95  # Simulated\n        return {'score': quality_score, 'rows': 10000}\n\n    def choose_branch(**context) -> str:\n        \"\"\"Determine which branch to execute\"\"\"\n        ti = context['ti']\n        metrics = ti.xcom_pull(task_ids='check_data_quality')\n\n        if metrics['score'] >= 0.9:\n            return 'high_quality_path'\n        elif metrics['score'] >= 0.7:\n            return 'medium_quality_path'\n        else:\n            return 'low_quality_path'\n\n    quality_check = check_data_quality()\n\n    branch = BranchPythonOperator(\n        task_id='branch',\n        python_callable=choose_branch,\n    )\n\n    high_quality = EmptyOperator(task_id='high_quality_path')\n    medium_quality = EmptyOperator(task_id='medium_quality_path')\n    low_quality = EmptyOperator(task_id='low_quality_path')\n\n    # Join point - runs after any branch completes\n    join = EmptyOperator(\n        task_id='join',\n        trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,\n    )\n\n    quality_check >> branch >> [high_quality, medium_quality, low_quality] >> join\n\nbranching_pipeline()\n```\n\n### Pattern 4: Sensors and External Dependencies\n\n```python\n# dags\u002Fsensor_patterns.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.sensors.filesystem import FileSensor\nfrom airflow.providers.amazon.aws.sensors.s3 import S3KeySensor\nfrom airflow.sensors.external_task import ExternalTaskSensor\nfrom airflow.operators.python import PythonOperator\n\nwith DAG(\n    dag_id='sensor_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n) as dag:\n\n    # Wait for file on S3\n    wait_for_file = S3KeySensor(\n        task_id='wait_for_s3_file',\n        bucket_name='data-lake',\n        bucket_key='raw\u002F{{ ds }}\u002Fdata.parquet',\n        aws_conn_id='aws_default',\n        timeout=60 * 60 * 2,  # 2 hours\n        poke_interval=60 * 5,  # Check every 5 minutes\n        mode='reschedule',  # Free up worker slot while waiting\n    )\n\n    # Wait for another DAG to complete\n    wait_for_upstream = ExternalTaskSensor(\n        task_id='wait_for_upstream_dag',\n        external_dag_id='upstream_etl',\n        external_task_id='final_task',\n        execution_date_fn=lambda dt: dt,  # Same execution date\n        timeout=60 * 60 * 3,\n        mode='reschedule',\n    )\n\n    # Custom sensor using @task.sensor decorator\n    @task.sensor(poke_interval=60, timeout=3600, mode='reschedule')\n    def wait_for_api() -> PokeReturnValue:\n        \"\"\"Custom sensor for API availability\"\"\"\n        import requests\n\n        response = requests.get('https:\u002F\u002Fapi.example.com\u002Fhealth')\n        is_done = response.status_code == 200\n\n        return PokeReturnValue(is_done=is_done, xcom_value=response.json())\n\n    api_ready = wait_for_api()\n\n    def process_data(**context):\n        api_result = context['ti'].xcom_pull(task_ids='wait_for_api')\n        print(f\"API returned: {api_result}\")\n\n    process = PythonOperator(\n        task_id='process',\n        python_callable=process_data,\n    )\n\n    [wait_for_file, wait_for_upstream, api_ready] >> process\n```\n\n### Pattern 5: Error Handling and Alerts\n\n```python\n# dags\u002Ferror_handling.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.utils.trigger_rule import TriggerRule\nfrom airflow.models import Variable\n\ndef task_failure_callback(context):\n    \"\"\"Callback on task failure\"\"\"\n    task_instance = context['task_instance']\n    exception = context.get('exception')\n\n    # Send to Slack\u002FPagerDuty\u002Fetc\n    message = f\"\"\"\n    Task Failed!\n    DAG: {task_instance.dag_id}\n    Task: {task_instance.task_id}\n    Execution Date: {context['ds']}\n    Error: {exception}\n    Log URL: {task_instance.log_url}\n    \"\"\"\n    # send_slack_alert(message)\n    print(message)\n\ndef dag_failure_callback(context):\n    \"\"\"Callback on DAG failure\"\"\"\n    # Aggregate failures, send summary\n    pass\n\nwith DAG(\n    dag_id='error_handling_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    on_failure_callback=dag_failure_callback,\n    default_args={\n        'on_failure_callback': task_failure_callback,\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    },\n) as dag:\n\n    def might_fail(**context):\n        import random\n        if random.random() \u003C 0.3:\n            raise ValueError(\"Random failure!\")\n        return \"Success\"\n\n    risky_task = PythonOperator(\n        task_id='risky_task',\n        python_callable=might_fail,\n    )\n\n    def cleanup(**context):\n        \"\"\"Cleanup runs regardless of upstream failures\"\"\"\n        print(\"Cleaning up...\")\n\n    cleanup_task = PythonOperator(\n        task_id='cleanup',\n        python_callable=cleanup,\n        trigger_rule=TriggerRule.ALL_DONE,  # Run even if upstream fails\n    )\n\n    def notify_success(**context):\n        \"\"\"Only runs if all upstream succeeded\"\"\"\n        print(\"All tasks succeeded!\")\n\n    success_notification = PythonOperator(\n        task_id='notify_success',\n        python_callable=notify_success,\n        trigger_rule=TriggerRule.ALL_SUCCESS,\n    )\n\n    risky_task >> [cleanup_task, success_notification]\n```\n\n### Pattern 6: Testing DAGs\n\n```python\n# tests\u002Ftest_dags.py\nimport pytest\nfrom datetime import datetime\nfrom airflow.models import DagBag\n\n@pytest.fixture\ndef dagbag():\n    return DagBag(dag_folder='dags\u002F', include_examples=False)\n\ndef test_dag_loaded(dagbag):\n    \"\"\"Test that all DAGs load without errors\"\"\"\n    assert len(dagbag.import_errors) == 0, f\"DAG import errors: {dagbag.import_errors}\"\n\ndef test_dag_structure(dagbag):\n    \"\"\"Test specific DAG structure\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    assert dag is not None\n    assert len(dag.tasks) == 3\n    assert dag.schedule_interval == '0 6 * * *'\n\ndef test_task_dependencies(dagbag):\n    \"\"\"Test task dependencies are correct\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    extract_task = dag.get_task('extract')\n    assert 'start' in [t.task_id for t in extract_task.upstream_list]\n    assert 'end' in [t.task_id for t in extract_task.downstream_list]\n\ndef test_dag_integrity(dagbag):\n    \"\"\"Test DAG has no cycles and is valid\"\"\"\n    for dag_id, dag in dagbag.dags.items():\n        assert dag.test_cycle() is None, f\"Cycle detected in {dag_id}\"\n\n# Test individual task logic\ndef test_extract_function():\n    \"\"\"Unit test for extract function\"\"\"\n    from dags.example_dag import extract_data\n\n    result = extract_data(ds='2024-01-01')\n    assert 'records' in result\n    assert isinstance(result['records'], int)\n```\n\n## Project Structure\n\n```\nairflow\u002F\n├── dags\u002F\n│   ├── __init__.py\n│   ├── common\u002F\n│   │   ├── __init__.py\n│   │   ├── operators.py    # Custom operators\n│   │   ├── sensors.py      # Custom sensors\n│   │   └── callbacks.py    # Alert callbacks\n│   ├── etl\u002F\n│   │   ├── customers.py\n│   │   └── orders.py\n│   └── ml\u002F\n│       └── training.py\n├── plugins\u002F\n│   └── custom_plugin.py\n├── tests\u002F\n│   ├── __init__.py\n│   ├── test_dags.py\n│   └── test_operators.py\n├── docker-compose.yml\n└── requirements.txt\n```\n\n## Best Practices\n\n### Do's\n- **Use TaskFlow API** - Cleaner code, automatic XCom\n- **Set timeouts** - Prevent zombie tasks\n- **Use `mode='reschedule'`** - For sensors, free up workers\n- **Test DAGs** - Unit tests and integration tests\n- **Idempotent tasks** - Safe to retry\n\n### Don'ts\n- **Don't use `depends_on_past=True`** - Creates bottlenecks\n- **Don't hardcode dates** - Use `{{ ds }}` macros\n- **Don't use global state** - Tasks should be stateless\n- **Don't skip catchup blindly** - Understand implications\n- **Don't put heavy logic in DAG file** - Import from modules\n\n## Resources\n\n- [Airflow Documentation](https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002F)\n- [Astronomer Guides](https:\u002F\u002Fdocs.astronomer.io\u002Flearn)\n- [TaskFlow API](https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow\u002Fstable\u002Ftutorial\u002Ftaskflow.html)\n","# Apache Airflow DAG Patterns\n\nProduction-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.\n\n## When to Use This Skill\n\n- Creating data pipeline orchestration with Airflow\n- Designing DAG structures and dependencies\n- Implementing custom operators and sensors\n- Testing Airflow DAGs locally\n- Setting up Airflow in production\n- Debugging failed DAG runs\n\n## Core Concepts\n\n### 1. DAG Design Principles\n\n| Principle | Description |\n|-----------|-------------|\n| **Idempotent** | Running twice produces same result |\n| **Atomic** | Tasks succeed or fail completely |\n| **Incremental** | Process only new\u002Fchanged data |\n| **Observable** | Logs, metrics, alerts at every step |\n\n### 2. Task Dependencies\n\n```python\n# Linear\ntask1 >> task2 >> task3\n\n# Fan-out\ntask1 >> [task2, task3, task4]\n\n# Fan-in\n[task1, task2, task3] >> task4\n\n# Complex\ntask1 >> task2 >> task4\ntask1 >> task3 >> task4\n```\n\n## Quick Start\n\n```python\n# dags\u002Fexample_dag.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.operators.empty import EmptyOperator\n\ndefault_args = {\n    'owner': 'data-team',\n    'depends_on_past': False,\n    'email_on_failure': True,\n    'email_on_retry': False,\n    'retries': 3,\n    'retry_delay': timedelta(minutes=5),\n    'retry_exponential_backoff': True,\n    'max_retry_delay': timedelta(hours=1),\n}\n\nwith DAG(\n    dag_id='example_etl',\n    default_args=default_args,\n    description='Example ETL pipeline',\n    schedule='0 6 * * *',  # Daily at 6 AM\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'example'],\n    max_active_runs=1,\n) as dag:\n\n    start = EmptyOperator(task_id='start')\n\n    def extract_data(**context):\n        execution_date = context['ds']\n        # Extract logic here\n        return {'records': 1000}\n\n    extract = PythonOperator(\n        task_id='extract',\n        python_callable=extract_data,\n    )\n\n    end = EmptyOperator(task_id='end')\n\n    start >> extract >> end\n```\n\n## Patterns\n\n### Pattern 1: TaskFlow API (Airflow 2.0+)\n\n```python\n# dags\u002Ftaskflow_example.py\nfrom datetime import datetime\nfrom airflow.decorators import dag, task\nfrom airflow.models import Variable\n\n@dag(\n    dag_id='taskflow_etl',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    tags=['etl', 'taskflow'],\n)\ndef taskflow_etl():\n    \"\"\"ETL pipeline using TaskFlow API\"\"\"\n\n    @task()\n    def extract(source: str) -> dict:\n        \"\"\"Extract data from source\"\"\"\n        import pandas as pd\n\n        df = pd.read_csv(f's3:\u002F\u002Fbucket\u002F{source}\u002F{{ ds }}.csv')\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def transform(extracted: dict) -> dict:\n        \"\"\"Transform extracted data\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(extracted['data'])\n        df['processed_at'] = datetime.now()\n        df = df.dropna()\n        return {'data': df.to_dict(), 'rows': len(df)}\n\n    @task()\n    def load(transformed: dict, target: str):\n        \"\"\"Load data to target\"\"\"\n        import pandas as pd\n\n        df = pd.DataFrame(transformed['data'])\n        df.to_parquet(f's3:\u002F\u002Fbucket\u002F{target}\u002F{{ ds }}.parquet')\n        return transformed['rows']\n\n    @task()\n    def notify(rows_loaded: int):\n        \"\"\"Send notification\"\"\"\n        print(f'Loaded {rows_loaded} rows')\n\n    # Define dependencies with XCom passing\n    extracted = extract(source='raw_data')\n    transformed = transform(extracted)\n    loaded = load(transformed, target='processed_data')\n    notify(loaded)\n\n# Instantiate the DAG\ntaskflow_etl()\n```\n\n### Pattern 2: Dynamic DAG Generation\n\n```python\n# dags\u002Fdynamic_dag_factory.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.models import Variable\nimport json\n\n# Configuration for multiple similar pipelines\nPIPELINE_CONFIGS = [\n    {'name': 'customers', 'schedule': '@daily', 'source': 's3:\u002F\u002Fraw\u002Fcustomers'},\n    {'name': 'orders', 'schedule': '@hourly', 'source': 's3:\u002F\u002Fraw\u002Forders'},\n    {'name': 'products', 'schedule': '@weekly', 'source': 's3:\u002F\u002Fraw\u002Fproducts'},\n]\n\ndef create_dag(config: dict) -> DAG:\n    \"\"\"Factory function to create DAGs from config\"\"\"\n\n    dag_id = f\"etl_{config['name']}\"\n\n    default_args = {\n        'owner': 'data-team',\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    }\n\n    dag = DAG(\n        dag_id=dag_id,\n        default_args=default_args,\n        schedule=config['schedule'],\n        start_date=datetime(2024, 1, 1),\n        catchup=False,\n        tags=['etl', 'dynamic', config['name']],\n    )\n\n    with dag:\n        def extract_fn(source, **context):\n            print(f\"Extracting from {source} for {context['ds']}\")\n\n        def transform_fn(**context):\n            print(f\"Transforming data for {context['ds']}\")\n\n        def load_fn(table_name, **context):\n            print(f\"Loading to {table_name} for {context['ds']}\")\n\n        extract = PythonOperator(\n            task_id='extract',\n            python_callable=extract_fn,\n            op_kwargs={'source': config['source']},\n        )\n\n        transform = PythonOperator(\n            task_id='transform',\n            python_callable=transform_fn,\n        )\n\n        load = PythonOperator(\n            task_id='load',\n            python_callable=load_fn,\n            op_kwargs={'table_name': config['name']},\n        )\n\n        extract >> transform >> load\n\n    return dag\n\n# Generate DAGs\nfor config in PIPELINE_CONFIGS:\n    globals()[f\"dag_{config['name']}\"] = create_dag(config)\n```\n\n### Pattern 3: Branching and Conditional Logic\n\n```python\n# dags\u002Fbranching_example.py\nfrom airflow.decorators import dag, task\nfrom airflow.operators.python import BranchPythonOperator\nfrom airflow.operators.empty import EmptyOperator\nfrom airflow.utils.trigger_rule import TriggerRule\n\n@dag(\n    dag_id='branching_pipeline',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n)\ndef branching_pipeline():\n\n    @task()\n    def check_data_quality() -> dict:\n        \"\"\"Check data quality and return metrics\"\"\"\n        quality_score = 0.95  # Simulated\n        return {'score': quality_score, 'rows': 10000}\n\n    def choose_branch(**context) -> str:\n        \"\"\"Determine which branch to execute\"\"\"\n        ti = context['ti']\n        metrics = ti.xcom_pull(task_ids='check_data_quality')\n\n        if metrics['score'] >= 0.9:\n            return 'high_quality_path'\n        elif metrics['score'] >= 0.7:\n            return 'medium_quality_path'\n        else:\n            return 'low_quality_path'\n\n    quality_check = check_data_quality()\n\n    branch = BranchPythonOperator(\n        task_id='branch',\n        python_callable=choose_branch,\n    )\n\n    high_quality = EmptyOperator(task_id='high_quality_path')\n    medium_quality = EmptyOperator(task_id='medium_quality_path')\n    low_quality = EmptyOperator(task_id='low_quality_path')\n\n    # Join point - runs after any branch completes\n    join = EmptyOperator(\n        task_id='join',\n        trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,\n    )\n\n    quality_check >> branch >> [high_quality, medium_quality, low_quality] >> join\n\nbranching_pipeline()\n```\n\n### Pattern 4: Sensors and External Dependencies\n\n```python\n# dags\u002Fsensor_patterns.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.sensors.filesystem import FileSensor\nfrom airflow.providers.amazon.aws.sensors.s3 import S3KeySensor\nfrom airflow.sensors.external_task import ExternalTaskSensor\nfrom airflow.operators.python import PythonOperator\n\nwith DAG(\n    dag_id='sensor_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n) as dag:\n\n    # Wait for file on S3\n    wait_for_file = S3KeySensor(\n        task_id='wait_for_s3_file',\n        bucket_name='data-lake',\n        bucket_key='raw\u002F{{ ds }}\u002Fdata.parquet',\n        aws_conn_id='aws_default',\n        timeout=60 * 60 * 2,  # 2 hours\n        poke_interval=60 * 5,  # Check every 5 minutes\n        mode='reschedule',  # Free up worker slot while waiting\n    )\n\n    # Wait for another DAG to complete\n    wait_for_upstream = ExternalTaskSensor(\n        task_id='wait_for_upstream_dag',\n        external_dag_id='upstream_etl',\n        external_task_id='final_task',\n        execution_date_fn=lambda dt: dt,  # Same execution date\n        timeout=60 * 60 * 3,\n        mode='reschedule',\n    )\n\n    # Custom sensor using @task.sensor decorator\n    @task.sensor(poke_interval=60, timeout=3600, mode='reschedule')\n    def wait_for_api() -> PokeReturnValue:\n        \"\"\"Custom sensor for API availability\"\"\"\n        import requests\n\n        response = requests.get('https:\u002F\u002Fapi.example.com\u002Fhealth')\n        is_done = response.status_code == 200\n\n        return PokeReturnValue(is_done=is_done, xcom_value=response.json())\n\n    api_ready = wait_for_api()\n\n    def process_data(**context):\n        api_result = context['ti'].xcom_pull(task_ids='wait_for_api')\n        print(f\"API returned: {api_result}\")\n\n    process = PythonOperator(\n        task_id='process',\n        python_callable=process_data,\n    )\n\n    [wait_for_file, wait_for_upstream, api_ready] >> process\n```\n\n### Pattern 5: Error Handling and Alerts\n\n```python\n# dags\u002Ferror_handling.py\nfrom datetime import datetime, timedelta\nfrom airflow import DAG\nfrom airflow.operators.python import PythonOperator\nfrom airflow.utils.trigger_rule import TriggerRule\nfrom airflow.models import Variable\n\ndef task_failure_callback(context):\n    \"\"\"Callback on task failure\"\"\"\n    task_instance = context['task_instance']\n    exception = context.get('exception')\n\n    # Send to Slack\u002FPagerDuty\u002Fetc\n    message = f\"\"\"\n    Task Failed!\n    DAG: {task_instance.dag_id}\n    Task: {task_instance.task_id}\n    Execution Date: {context['ds']}\n    Error: {exception}\n    Log URL: {task_instance.log_url}\n    \"\"\"\n    # send_slack_alert(message)\n    print(message)\n\ndef dag_failure_callback(context):\n    \"\"\"Callback on DAG failure\"\"\"\n    # Aggregate failures, send summary\n    pass\n\nwith DAG(\n    dag_id='error_handling_example',\n    schedule='@daily',\n    start_date=datetime(2024, 1, 1),\n    catchup=False,\n    on_failure_callback=dag_failure_callback,\n    default_args={\n        'on_failure_callback': task_failure_callback,\n        'retries': 3,\n        'retry_delay': timedelta(minutes=5),\n    },\n) as dag:\n\n    def might_fail(**context):\n        import random\n        if random.random() \u003C 0.3:\n            raise ValueError(\"Random failure!\")\n        return \"Success\"\n\n    risky_task = PythonOperator(\n        task_id='risky_task',\n        python_callable=might_fail,\n    )\n\n    def cleanup(**context):\n        \"\"\"Cleanup runs regardless of upstream failures\"\"\"\n        print(\"Cleaning up...\")\n\n    cleanup_task = PythonOperator(\n        task_id='cleanup',\n        python_callable=cleanup,\n        trigger_rule=TriggerRule.ALL_DONE,  # Run even if upstream fails\n    )\n\n    def notify_success(**context):\n        \"\"\"Only runs if all upstream succeeded\"\"\"\n        print(\"All tasks succeeded!\")\n\n    success_notification = PythonOperator(\n        task_id='notify_success',\n        python_callable=notify_success,\n        trigger_rule=TriggerRule.ALL_SUCCESS,\n    )\n\n    risky_task >> [cleanup_task, success_notification]\n```\n\n### Pattern 6: Testing DAGs\n\n```python\n# tests\u002Ftest_dags.py\nimport pytest\nfrom datetime import datetime\nfrom airflow.models import DagBag\n\n@pytest.fixture\ndef dagbag():\n    return DagBag(dag_folder='dags\u002F', include_examples=False)\n\ndef test_dag_loaded(dagbag):\n    \"\"\"Test that all DAGs load without errors\"\"\"\n    assert len(dagbag.import_errors) == 0, f\"DAG import errors: {dagbag.import_errors}\"\n\ndef test_dag_structure(dagbag):\n    \"\"\"Test specific DAG structure\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    assert dag is not None\n    assert len(dag.tasks) == 3\n    assert dag.schedule_interval == '0 6 * * *'\n\ndef test_task_dependencies(dagbag):\n    \"\"\"Test task dependencies are correct\"\"\"\n    dag = dagbag.get_dag('example_etl')\n\n    extract_task = dag.get_task('extract')\n    assert 'start' in [t.task_id for t in extract_task.upstream_list]\n    assert 'end' in [t.task_id for t in extract_task.downstream_list]\n\ndef test_dag_integrity(dagbag):\n    \"\"\"Test DAG has no cycles and is valid\"\"\"\n    for dag_id, dag in dagbag.dags.items():\n        assert dag.test_cycle() is None, f\"Cycle detected in {dag_id}\"\n\n# Test individual task logic\ndef test_extract_function():\n    \"\"\"Unit test for extract function\"\"\"\n    from dags.example_dag import extract_data\n\n    result = extract_data(ds='2024-01-01')\n    assert 'records' in result\n    assert isinstance(result['records'], int)\n```\n\n## Project Structure\n\n```\nairflow\u002F\n├── dags\u002F\n│   ├── __init__.py\n│   ├── common\u002F\n│   │   ├── __init__.py\n│   │   ├── operators.py    # Custom operators\n│   │   ├── sensors.py      # Custom sensors\n│   │   └── callbacks.py    # Alert callbacks\n│   ├── etl\u002F\n│   │   ├── customers.py\n│   │   └── orders.py\n│   └── ml\u002F\n│       └── training.py\n├── plugins\u002F\n│   └── custom_plugin.py\n├── tests\u002F\n│   ├── __init__.py\n│   ├── test_dags.py\n│   └── test_operators.py\n├── docker-compose.yml\n└── requirements.txt\n```\n\n## Best Practices\n\n### Do's\n- **Use TaskFlow API** - Cleaner code, automatic XCom\n- **Set timeouts** - Prevent zombie tasks\n- **Use `mode='reschedule'`** - For sensors, free up workers\n- **Test DAGs** - Unit tests and integration tests\n- **Idempotent tasks** - Safe to retry\n\n### Don'ts\n- **Don't use `depends_on_past=True`** - Creates bottlenecks\n- **Don't hardcode dates** - Use `{{ ds }}` macros\n- **Don't use global state** - Tasks should be stateless\n- **Don't skip catchup blindly** - Understand implications\n- **Don't put heavy logic in DAG file** - Import from modules\n\n## Resources\n\n- [Airflow Documentation](https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002F)\n- [Astronomer Guides](https:\u002F\u002Fdocs.astronomer.io\u002Flearn)\n- [TaskFlow API](https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow\u002Fstable\u002Ftutorial\u002Ftaskflow.html)\n",{"name":5,"description":7},"main","7850740bd837adb32f0a109ab2af82482196c6f2","skills\u002F_collections\u002Fwshobson-agents\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fairflow-dag-patterns",1,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills","https:\u002F\u002Fzturnsgo.com","https:\u002F\u002Fgithub.com\u002Fshengdabai","https:\u002F\u002Fzturnsgo.com\u002F","AI × education creator — producer, not consumer. Taught 6,000+ learners; shipping learning tools & daily essays with Claude Code, practicing @xiaolai's way 🥋","2026-11-01 21:28:23",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add shengdabai\u002FTony-Claude-Code-Skills --skill airflow-dag-patterns",{"tool":65,"command":66},"bunx","bunx skills add shengdabai\u002FTony-Claude-Code-Skills --skill airflow-dag-patterns",{"tool":68,"command":69},"pnpm","pnpm dlx skills add shengdabai\u002FTony-Claude-Code-Skills --skill airflow-dag-patterns","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002F_collections\u002Fwshobson-agents\u002Fplugins\u002Fdata-engineering\u002Fskills\u002Fairflow-dag-patterns\u002FSKILL.md",true,[],[74,83,91,99,107,115],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":78,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":82},"01M73H7XKBHE1ETQDZT94MF9HH","seedance-prompt-en","Write effective prompts for Jimeng Seedance 2.0 multimodal AI video generation. Use when users want to create video prompts using text, images, videos, and audio inputs with the @ reference system. Covers camera movements, effects replication, video extension, editing, music beat-matching, e-commerce ads, short dramas, and educational content.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fseedance-prompt-en\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":87,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":82},"01M73GRE9998Q1DZWC71N41JWG","onepage-pdf","Convert an HTML page into a single continuous-page PDF (one tall page, no pagination breaks), preserving desktop layout, backgrounds and selectable text. Use when the user asks to turn an HTML file\u002Freport\u002Fproposal into a \"single-page PDF\", \"long PDF\", \"不分页 PDF\", \"单页 PDF\", \"长图式 PDF\", or complains that an HTML-to-PDF export breaks into pages or cuts content. Supports optional string redaction with leak verification.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fonepage-pdf\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":82},"01M73GQG1KZDADVPV3MHAT1J5A","native-feel-cross-platform-desktop","Use when the user is designing, prototyping, or rewriting a desktop app that must run on multiple OSes (macOS + Windows, optionally Linux) AND feel indistinguishable from a native app to its users — fast launch, native windowing, native input handling, native materials. Trigger words include \"cross-platform desktop\", \"Electron alternative\", \"Tauri vs native\", \"WebView wrapper\", \"near-native performance\", \"Raycast architecture\", \"WebKit\u002FWebView2 quirks\", \"WKWebView\", \"system tray app\", \"global hotkey app\", \"launcher app\". Do NOT trigger this skill for pure web apps, pure mobile apps, or for greenfield projects that have no native-feel requirement.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fnative-feel-cross-platform-desktop\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":103,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":82},"01M73G87PDNSMKG2FHWCYER6N8","gstack","Router for the gstack skill suite. (gstack)","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fgstack\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":111,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":82},"01M73G3EAQWKCM8DB8D0N0QSVG","fireworks-tech-graph","Use when the user wants to create any technical diagram - architecture, data flow, flowchart, sequence, agent\u002Fmemory, or concept map - and export as SVG+PNG. Trigger on: \"画图\" \"帮我画\" \"生成图\" \"做个图\" \"架构图\" \"流程图\" \"可视化一下\" \"出图\" \"generate diagram\" \"draw diagram\" \"visualize\" or any system\u002Fflow description the user wants illustrated.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Ffireworks-tech-graph\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":82},"01M73G086AXAH9DW1H5G5N3CPB","dbs-video-extract","接收抖音、小红书、微信视频号等短视频链接或分享文案，查询作品／账号数据并提取语音文字稿，按作者和标题归档为 Markdown。用户希望完整解析短视频、取得数据或文字稿时使用；运行前检查 TikHub 与轻抖使用凭证，缺少时用新手能理解的方式引导购买和安全配置。","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fdbs-video-extract\u002FSKILL.md",[],[]]