[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACDYBTKYKDP9Z0BQKF7XMZ":3,"skill-same:01M6ACDYBTKYKDP9Z0BQKF7XMZ":64},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":36,"linkOnly":37,"markdownContent":38,"markdownBody":39,"frontMatter":40,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":44,"repositoryHtmlUrl":45,"repositoryHomepageUrl":46,"ownerHtmlUrl":47,"ownerBlogUrl":48,"ownerDescription":6,"updatedAt":49,"install":50,"resources":63},"01M6ACDYBTKYKDP9Z0BQKF7XMZ","yaml-development",null,"Guides YAML SDK development in Apache Beam, including environment setup, testing, and key concepts. Use when working with Beam YAML code in sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002F.","apache\u002Fbeam","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACCXED4SMY85J25QX8K9SY",8718,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fyaml-development\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,9,31,32,33,34,35],"development","yaml development","yaml","when","testing","sdks","concepts","environment setup","with","guides","use","environment","code","setup","sdk","python","beam","2026-11-10 10:12:30",false,"---\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: yaml-development\ndescription: Guides YAML SDK development in Apache Beam, including environment setup, testing, and key concepts. Use when working with Beam YAML code in sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002F.\n---\n\n# YAML Development in Apache Beam\n\n## Project Structure\n\n### Key Files in `sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002F`\n- `integration_tests.py` - Runs integration tests defined in YAML files or using testcontainers.\n- `main.py` - Entry point for running YAML pipelines from the command line.\n- `pipeline.schema.yaml` - JSON schema defining the valid structure for Beam YAML pipelines.\n- `standard_io.yaml` - Declarations of standard IO transforms and their mappings to providers.\n- `standard_providers.yaml` - Configuration for standard providers (e.g., Java expansion services).\n- `yaml_combine.py` - Implementations for aggregation and combining operations.\n- `yaml_io.py` - Mappings and logic for IO transforms (e.g., PubSub, BigQuery, Iceberg).\n- `yaml_join.py` - Implementations for join operations.\n- `yaml_mapping.py` - Implementations for mapping operations (e.g., `MapToFields`).\n- `yaml_provider.py` - Manages providers (Python, Java cross-language) that implement transforms.\n- `yaml_transform.py` - Core YAML expansion logic, parsing, and translation to Beam pipelines.\n\n## Environment Setup\nSince Beam YAML is implemented within the Python SDK, the environment setup is identical to Python development. Refer to the `python-development` skill for details on using `pyenv` and installing in editable mode (e.g., use `pip install -e sdks\u002Fpython[gcp,test]` from the root directory).\n\n## Running YAML Pipelines\n\nYou can run Beam YAML pipelines using the `main.py` script in the YAML directory.\n\n### Using `main.py` directly\n```bash\npython -m apache_beam.yaml.main --yaml_pipeline_file=\u002Fpath\u002Fto\u002Fpipeline.yaml [pipeline_options]\n```\n\n### Example: Running locally\n```bash\npython -m apache_beam.yaml.main \\\n  --yaml_pipeline_file=sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fexamples\u002Fsimple_filter.yaml \\\n  --runner=DirectRunner\n```\n\n### Example: Running on Dataflow\n```bash\npython -m apache_beam.yaml.main \\\n  --yaml_pipeline_file=sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fexamples\u002Fsimple_filter.yaml \\\n  --runner=DataflowRunner \\\n  --project=my-project \\\n  --region=us-central1 \\\n  --temp_location=gs:\u002F\u002Fmy-bucket\u002Ftemp\n```\n\n## Running Tests\n\n### Unit Tests\nBeam YAML has extensive unit tests covering parsing, expansion, and specific transforms.\n```bash\n# Run all tests in a file\npytest sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fyaml_transform_test.py\n\n# Run a specific test\npytest sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fyaml_transform_test.py::YamlTransformTest::test_simple_pipeline\n```\n\n### Integration Tests\nIntegration tests often spin up Docker containers (via `testcontainers`) for external services like MongoDB, Kafka, or databases.\n```bash\n# Run integration tests matching a specific keyword (e.g., mongodb)\npytest sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fintegration_tests.py -k mongodb\n```\n\n## Key Concepts\n\n### Providers\nBeam YAML uses \"providers\" to find implementations for transforms requested in the YAML file.\n- **Inline\u002FPython Providers**: Leverage Python functions or PTransforms directly.\n- **Java\u002FExternal Providers**: Use Beam's cross-language capabilities to invoke Java transforms via an expansion service.\n\n### Preprocessing\nBefore execution, a YAML pipeline is preprocessed to resolve schemas, match transforms to providers, and expand shorthand notations (like `chain` or `source`\u002F`sink` composites).\n\n## Common Issues\n\n### Cross-language Failures\nIf a test requires a Java transform, ensure that:\n1. Docker is running (if using testcontainers).\n2. The correct expansion service is available or can be started.\n3. Java environment is correctly configured (sometimes requires specific Java versions like Java 17\u002F21).\n\n### Schema Mismatches\nYAML relies heavily on Beam schemas. Ensure that fields produced by a transform match the fields expected by the next transform. Use explicit mapping if necessary.\n","# YAML Development in Apache Beam\n\n## Project Structure\n\n### Key Files in `sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002F`\n- `integration_tests.py` - Runs integration tests defined in YAML files or using testcontainers.\n- `main.py` - Entry point for running YAML pipelines from the command line.\n- `pipeline.schema.yaml` - JSON schema defining the valid structure for Beam YAML pipelines.\n- `standard_io.yaml` - Declarations of standard IO transforms and their mappings to providers.\n- `standard_providers.yaml` - Configuration for standard providers (e.g., Java expansion services).\n- `yaml_combine.py` - Implementations for aggregation and combining operations.\n- `yaml_io.py` - Mappings and logic for IO transforms (e.g., PubSub, BigQuery, Iceberg).\n- `yaml_join.py` - Implementations for join operations.\n- `yaml_mapping.py` - Implementations for mapping operations (e.g., `MapToFields`).\n- `yaml_provider.py` - Manages providers (Python, Java cross-language) that implement transforms.\n- `yaml_transform.py` - Core YAML expansion logic, parsing, and translation to Beam pipelines.\n\n## Environment Setup\nSince Beam YAML is implemented within the Python SDK, the environment setup is identical to Python development. Refer to the `python-development` skill for details on using `pyenv` and installing in editable mode (e.g., use `pip install -e sdks\u002Fpython[gcp,test]` from the root directory).\n\n## Running YAML Pipelines\n\nYou can run Beam YAML pipelines using the `main.py` script in the YAML directory.\n\n### Using `main.py` directly\n```bash\npython -m apache_beam.yaml.main --yaml_pipeline_file=\u002Fpath\u002Fto\u002Fpipeline.yaml [pipeline_options]\n```\n\n### Example: Running locally\n```bash\npython -m apache_beam.yaml.main \\\n  --yaml_pipeline_file=sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fexamples\u002Fsimple_filter.yaml \\\n  --runner=DirectRunner\n```\n\n### Example: Running on Dataflow\n```bash\npython -m apache_beam.yaml.main \\\n  --yaml_pipeline_file=sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fexamples\u002Fsimple_filter.yaml \\\n  --runner=DataflowRunner \\\n  --project=my-project \\\n  --region=us-central1 \\\n  --temp_location=gs:\u002F\u002Fmy-bucket\u002Ftemp\n```\n\n## Running Tests\n\n### Unit Tests\nBeam YAML has extensive unit tests covering parsing, expansion, and specific transforms.\n```bash\n# Run all tests in a file\npytest sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fyaml_transform_test.py\n\n# Run a specific test\npytest sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fyaml_transform_test.py::YamlTransformTest::test_simple_pipeline\n```\n\n### Integration Tests\nIntegration tests often spin up Docker containers (via `testcontainers`) for external services like MongoDB, Kafka, or databases.\n```bash\n# Run integration tests matching a specific keyword (e.g., mongodb)\npytest sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002Fintegration_tests.py -k mongodb\n```\n\n## Key Concepts\n\n### Providers\nBeam YAML uses \"providers\" to find implementations for transforms requested in the YAML file.\n- **Inline\u002FPython Providers**: Leverage Python functions or PTransforms directly.\n- **Java\u002FExternal Providers**: Use Beam's cross-language capabilities to invoke Java transforms via an expansion service.\n\n### Preprocessing\nBefore execution, a YAML pipeline is preprocessed to resolve schemas, match transforms to providers, and expand shorthand notations (like `chain` or `source`\u002F`sink` composites).\n\n## Common Issues\n\n### Cross-language Failures\nIf a test requires a Java transform, ensure that:\n1. Docker is running (if using testcontainers).\n2. The correct expansion service is available or can be started.\n3. Java environment is correctly configured (sometimes requires specific Java versions like Java 17\u002F21).\n\n### Schema Mismatches\nYAML relies heavily on Beam schemas. Ensure that fields produced by a transform match the fields expected by the next transform. Use explicit mapping if necessary.\n",{"name":5,"description":7},"master","aba9e790548a6870a46d2c085c817c2754f4a301",".agent\u002Fskills\u002Fyaml-development",4812,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam","https:\u002F\u002Fbeam.apache.org\u002F","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-15 01:40:19",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add apache\u002Fbeam --skill yaml-development",{"tool":56,"command":57},"bunx","bunx skills add apache\u002Fbeam --skill yaml-development",{"tool":59,"command":60},"pnpm","pnpm dlx skills add apache\u002Fbeam --skill yaml-development","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fyaml-development\u002FSKILL.md",true,[],[65,82,98,118,132,146],{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":36},"01M6ACDVK2M4QNJ2T0423H23SY","runners","Guides understanding and working with Apache Beam runners (Direct, Dataflow, Flink, Spark, etc.). Use when configuring pipelines for different execution environments or debugging runner-specific issues.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Frunners\u002FSKILL.md",[],[67,9,22,72,73,27,28,74,75,29,76,77,78,79,80,81,35],"for","environments","runner","pipelines","direct","execution","flink","issues","debugging","spark",{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":93,"publishedAt":36},"01M6ACDRVH48A1WBYH9DEETW7F","python-development","Guides Python SDK development in Apache Beam, including environment setup, testing, building, and running pipelines. Use when working with Python code in sdks\u002Fpython\u002F.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fpython-development\u002FSKILL.md",[88],{"categoryNo":89,"categoryName":90,"taxonomyType":91,"taxonomyTypeDesc":92},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",[19,94,34,29,30,95,96,9,31,32,33,35,23,75,97,22,24,26,27,28],"python development","python code","python sdk","running",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":109,"publishedAt":36},"01M6ACDPYBMSHSQNCFXVJ7VJW3","license-compliance","Ensures all new files include proper Apache 2.0 license headers. Use when creating any new file in the Apache Beam repository.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Flicense-compliance\u002FSKILL.md",[104],{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000037","律师与法务","PROFESSION","适用职业",[110,111,112,29,113,114,22,115,35,116,9,117],"compliance","license compliance","license","files","headers","new","file","repository",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":125,"publishedAt":36},"01M6ACDM2W664NC4FH95B3NC14","java-development","Guides Java SDK development in Apache Beam, including building, testing, running examples, and understanding the project structure. Use when working with Java code in sdks\u002Fjava\u002F, runners\u002F, or examples\u002Fjava\u002F.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fjava-development\u002FSKILL.md",[124],{"categoryNo":89,"categoryName":90,"taxonomyType":91,"taxonomyTypeDesc":92},[19,126,127,29,97,22,24,27,28,128,129,130,67,9,31,131,33,35,23],"java","java development","project","structure","examples","project structure",{"skillNo":133,"name":134,"nameZh":6,"description":135,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":136,"categories":137,"keywords":139,"publishedAt":36},"01M6ACDHGVH4F8ADMCFR7T34H8","io-connectors","Guides development and usage of I\u002FO connectors in Apache Beam. Use when working with I\u002FO connectors, creating new connectors, or debugging data source\u002Fsink issues.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fio-connectors\u002FSKILL.md",[138],{"categoryNo":89,"categoryName":90,"taxonomyType":91,"taxonomyTypeDesc":92},[140,141,142,35,143,144,27,28,29,145,9,79,19,80,22,115],"io","io connectors","connectors","source","usage","data",{"skillNo":147,"name":148,"nameZh":6,"description":149,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":150,"categories":151,"keywords":152,"publishedAt":36},"01M6ACDF6YTQG62GC4JKAVQBJS","gradle-build","Guides understanding and using the Gradle build system in Apache Beam. Use when building projects, understanding dependencies, or troubleshooting build issues.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fgradle-build\u002FSKILL.md",[],[153,154,155,22,28,156,29,157,158,159,9,79,160,35],"gradle","build","gradle build","dependencies","troubleshooting","system","projects","building projects"]