[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACDM2W664NC4FH95B3NC14":3,"skill-same:01M6ACDM2W664NC4FH95B3NC14":70},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":42,"linkOnly":43,"markdownContent":44,"markdownBody":45,"frontMatter":46,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":50,"repositoryHtmlUrl":51,"repositoryHomepageUrl":52,"ownerHtmlUrl":53,"ownerBlogUrl":54,"ownerDescription":6,"updatedAt":55,"install":56,"resources":69},"01M6ACDM2W664NC4FH95B3NC14","java-development",null,"Guides Java SDK development in Apache Beam, including building, testing, running examples, and understanding the project structure. Use when working with Java code in sdks\u002Fjava\u002F, runners\u002F, or examples\u002Fjava\u002F.","apache\u002Fbeam","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACCXED4SMY85J25QX8K9SY",8718,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fjava-development\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,9,38,39,40,41],"development","java","java development","runners","running","when","sdks","with","guides","project","structure","use","examples","testing","code","project structure","sdk","beam","2026-11-10 10:12:30",false,"---\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nname: java-development\ndescription: Guides Java SDK development in Apache Beam, including building, testing, running examples, and understanding the project structure. Use when working with Java code in sdks\u002Fjava\u002F, runners\u002F, or examples\u002Fjava\u002F.\n---\n\n# Java Development in Apache Beam\n\n## Project Structure\n\n### Key Directories\n- `sdks\u002Fjava\u002Fcore` - Core Java SDK (PCollection, PTransform, Pipeline)\n- `sdks\u002Fjava\u002Fharness` - SDK harness (container entrypoint)\n- `sdks\u002Fjava\u002Fio\u002F` - I\u002FO connectors (51+ connectors including BigQuery, Kafka, JDBC, etc.)\n- `sdks\u002Fjava\u002Fextensions\u002F` - Extensions (SQL, ML, protobuf, etc.)\n- `runners\u002F` - Runner implementations:\n  - `runners\u002Fdirect-java` - Direct Runner (local execution)\n  - `runners\u002Fflink\u002F` - Flink Runner\n  - `runners\u002Fspark\u002F` - Spark Runner\n  - `runners\u002Fgoogle-cloud-dataflow-java\u002F` - Dataflow Runner\n- `examples\u002Fjava\u002F` - Java examples including WordCount\n\n### Build System\nApache Beam uses Gradle with a custom `BeamModulePlugin`. Every Java project's `build.gradle` starts with:\n```groovy\napply plugin: 'org.apache.beam.module'\napplyJavaNature( ... )\n```\n\n## Common Commands\n\n### Build Commands\n```bash\n# Compile a specific project\n.\u002Fgradlew -p sdks\u002Fjava\u002Fcore compileJava\n\n# Build a project (compile + tests)\n.\u002Fgradlew :sdks:java:harness:build\n\n# Run WordCount example\n.\u002Fgradlew :examples:java:wordCount\n```\n\n### Running Unit Tests\n```bash\n# Run all tests in a project\n.\u002Fgradlew :sdks:java:harness:test\n\n# Run a specific test class\n.\u002Fgradlew :sdks:java:harness:test --tests org.apache.beam.fn.harness.CachesTest\n\n# Run tests matching a pattern\n.\u002Fgradlew :sdks:java:harness:test --tests *CachesTest\n\n# Run a specific test method\n.\u002Fgradlew :sdks:java:harness:test --tests *CachesTest.testClearableCache\n```\n\n### Running Integration Tests\nIntegration tests have filenames ending in `IT.java` and use `TestPipeline`.\n\n```bash\n# Run I\u002FO integration tests on Direct Runner\n.\u002Fgradlew :sdks:java:io:google-cloud-platform:integrationTest\n\n# Run with custom GCP project\n.\u002Fgradlew :sdks:java:io:google-cloud-platform:integrationTest \\\n  -PgcpProject=\u003Cproject> -PgcpTempRoot=gs:\u002F\u002F\u003Cbucket>\u002Fpath\n\n# Run on Dataflow Runner\n.\u002Fgradlew :runners:google-cloud-dataflow-java:examplesJavaRunnerV2IntegrationTest \\\n  -PdisableSpotlessCheck=true -PdisableCheckStyle=true -PskipCheckerFramework \\\n  -PgcpProject=\u003Cproject> -PgcpRegion=us-central1 -PgcsTempRoot=gs:\u002F\u002F\u003Cbucket>\u002Ftmp\n```\n\n### Code Formatting\n```bash\n# Format Java code\n.\u002Fgradlew spotlessApply\n```\n\n## Writing Integration Tests\n\n```java\n@Rule public TestPipeline pipeline = TestPipeline.create();\n\n@Test\npublic void testSomething() {\n  pipeline.apply(...);\n  pipeline.run().waitUntilFinish();\n}\n```\n\nSet pipeline options via `-DbeamTestPipelineOptions='[...]'`:\n```bash\n-DbeamTestPipelineOptions='[\"--runner=TestDataflowRunner\",\"--project=myproject\",\"--region=us-central1\",\"--stagingLocation=gs:\u002F\u002Fbucket\u002Fpath\"]'\n```\n\n## Using Modified Beam Code\n\n### Publish to Maven Local\n```bash\n# Publish a specific module\n.\u002Fgradlew -Ppublishing -p sdks\u002Fjava\u002Fio\u002Fkafka publishToMavenLocal\n\n# Publish all modules\n.\u002Fgradlew -Ppublishing publishToMavenLocal\n```\n\n### Building SDK Container\n```bash\n# Build Java SDK container (for Runner v2)\n.\u002Fgradlew :sdks:java:container:java11:docker\n\n# Tag and push\ndocker tag apache\u002Fbeam_java11_sdk:2.XX.0.dev \\\n  \"us-docker.pkg.dev\u002Fyour-project\u002Fbeam\u002Fbeam_java11_sdk:custom\"\ndocker push \"us-docker.pkg.dev\u002Fyour-project\u002Fbeam\u002Fbeam_java11_sdk:custom\"\n```\n\n### Building Dataflow Worker Jar\n```bash\n.\u002Fgradlew :runners:google-cloud-dataflow-java:worker:shadowJar\n```\n\n## Test Naming Conventions\n- Unit tests: `*Test.java`\n- Integration tests: `*IT.java`\n\n## JUnit Report Location\nAfter running tests, find HTML reports at:\n`\u003Cproject>\u002Fbuild\u002Freports\u002Ftests\u002Ftest\u002Findex.html`\n\n## IDE Setup (IntelliJ)\n1. Open `\u002Fbeam` (the repository root, NOT `sdks\u002Fjava`)\n2. Wait for indexing to complete\n3. Find `examples\u002Fjava\u002Fbuild.gradle` and click Run next to wordCount task to verify setup\n","# Java Development in Apache Beam\n\n## Project Structure\n\n### Key Directories\n- `sdks\u002Fjava\u002Fcore` - Core Java SDK (PCollection, PTransform, Pipeline)\n- `sdks\u002Fjava\u002Fharness` - SDK harness (container entrypoint)\n- `sdks\u002Fjava\u002Fio\u002F` - I\u002FO connectors (51+ connectors including BigQuery, Kafka, JDBC, etc.)\n- `sdks\u002Fjava\u002Fextensions\u002F` - Extensions (SQL, ML, protobuf, etc.)\n- `runners\u002F` - Runner implementations:\n  - `runners\u002Fdirect-java` - Direct Runner (local execution)\n  - `runners\u002Fflink\u002F` - Flink Runner\n  - `runners\u002Fspark\u002F` - Spark Runner\n  - `runners\u002Fgoogle-cloud-dataflow-java\u002F` - Dataflow Runner\n- `examples\u002Fjava\u002F` - Java examples including WordCount\n\n### Build System\nApache Beam uses Gradle with a custom `BeamModulePlugin`. Every Java project's `build.gradle` starts with:\n```groovy\napply plugin: 'org.apache.beam.module'\napplyJavaNature( ... )\n```\n\n## Common Commands\n\n### Build Commands\n```bash\n# Compile a specific project\n.\u002Fgradlew -p sdks\u002Fjava\u002Fcore compileJava\n\n# Build a project (compile + tests)\n.\u002Fgradlew :sdks:java:harness:build\n\n# Run WordCount example\n.\u002Fgradlew :examples:java:wordCount\n```\n\n### Running Unit Tests\n```bash\n# Run all tests in a project\n.\u002Fgradlew :sdks:java:harness:test\n\n# Run a specific test class\n.\u002Fgradlew :sdks:java:harness:test --tests org.apache.beam.fn.harness.CachesTest\n\n# Run tests matching a pattern\n.\u002Fgradlew :sdks:java:harness:test --tests *CachesTest\n\n# Run a specific test method\n.\u002Fgradlew :sdks:java:harness:test --tests *CachesTest.testClearableCache\n```\n\n### Running Integration Tests\nIntegration tests have filenames ending in `IT.java` and use `TestPipeline`.\n\n```bash\n# Run I\u002FO integration tests on Direct Runner\n.\u002Fgradlew :sdks:java:io:google-cloud-platform:integrationTest\n\n# Run with custom GCP project\n.\u002Fgradlew :sdks:java:io:google-cloud-platform:integrationTest \\\n  -PgcpProject=\u003Cproject> -PgcpTempRoot=gs:\u002F\u002F\u003Cbucket>\u002Fpath\n\n# Run on Dataflow Runner\n.\u002Fgradlew :runners:google-cloud-dataflow-java:examplesJavaRunnerV2IntegrationTest \\\n  -PdisableSpotlessCheck=true -PdisableCheckStyle=true -PskipCheckerFramework \\\n  -PgcpProject=\u003Cproject> -PgcpRegion=us-central1 -PgcsTempRoot=gs:\u002F\u002F\u003Cbucket>\u002Ftmp\n```\n\n### Code Formatting\n```bash\n# Format Java code\n.\u002Fgradlew spotlessApply\n```\n\n## Writing Integration Tests\n\n```java\n@Rule public TestPipeline pipeline = TestPipeline.create();\n\n@Test\npublic void testSomething() {\n  pipeline.apply(...);\n  pipeline.run().waitUntilFinish();\n}\n```\n\nSet pipeline options via `-DbeamTestPipelineOptions='[...]'`:\n```bash\n-DbeamTestPipelineOptions='[\"--runner=TestDataflowRunner\",\"--project=myproject\",\"--region=us-central1\",\"--stagingLocation=gs:\u002F\u002Fbucket\u002Fpath\"]'\n```\n\n## Using Modified Beam Code\n\n### Publish to Maven Local\n```bash\n# Publish a specific module\n.\u002Fgradlew -Ppublishing -p sdks\u002Fjava\u002Fio\u002Fkafka publishToMavenLocal\n\n# Publish all modules\n.\u002Fgradlew -Ppublishing publishToMavenLocal\n```\n\n### Building SDK Container\n```bash\n# Build Java SDK container (for Runner v2)\n.\u002Fgradlew :sdks:java:container:java11:docker\n\n# Tag and push\ndocker tag apache\u002Fbeam_java11_sdk:2.XX.0.dev \\\n  \"us-docker.pkg.dev\u002Fyour-project\u002Fbeam\u002Fbeam_java11_sdk:custom\"\ndocker push \"us-docker.pkg.dev\u002Fyour-project\u002Fbeam\u002Fbeam_java11_sdk:custom\"\n```\n\n### Building Dataflow Worker Jar\n```bash\n.\u002Fgradlew :runners:google-cloud-dataflow-java:worker:shadowJar\n```\n\n## Test Naming Conventions\n- Unit tests: `*Test.java`\n- Integration tests: `*IT.java`\n\n## JUnit Report Location\nAfter running tests, find HTML reports at:\n`\u003Cproject>\u002Fbuild\u002Freports\u002Ftests\u002Ftest\u002Findex.html`\n\n## IDE Setup (IntelliJ)\n1. Open `\u002Fbeam` (the repository root, NOT `sdks\u002Fjava`)\n2. Wait for indexing to complete\n3. Find `examples\u002Fjava\u002Fbuild.gradle` and click Run next to wordCount task to verify setup\n",{"name":5,"description":7},"master","aba9e790548a6870a46d2c085c817c2754f4a301",".agent\u002Fskills\u002Fjava-development",4812,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam","https:\u002F\u002Fbeam.apache.org\u002F","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-15 01:40:19",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add apache\u002Fbeam --skill java-development",{"tool":62,"command":63},"bunx","bunx skills add apache\u002Fbeam --skill java-development",{"tool":65,"command":66},"pnpm","pnpm dlx skills add apache\u002Fbeam --skill java-development","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fjava-development\u002FSKILL.md",true,[],[71,85,101,112,132,146],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":42},"01M6ACDYBTKYKDP9Z0BQKF7XMZ","yaml-development","Guides YAML SDK development in Apache Beam, including environment setup, testing, and key concepts. Use when working with Beam YAML code in sdks\u002Fpython\u002Fapache_beam\u002Fyaml\u002F.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fyaml-development\u002FSKILL.md",[],[24,78,79,80,30,81,82,31,32,35,83,9,38,40,84,41,37,29],"yaml development","yaml","setup","concepts","environment setup","environment","python",{"skillNo":86,"name":27,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":42},"01M6ACDVK2M4QNJ2T0423H23SY","Guides understanding and working with Apache Beam runners (Direct, Dataflow, Flink, Spark, etc.). Use when configuring pipelines for different execution environments or debugging runner-specific issues.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Frunners\u002FSKILL.md",[],[27,91,41,92,93,35,9,94,95,96,97,98,29,99,31,32,100],"flink","execution","direct","issues","debugging","spark","environments","pipelines","for","runner",{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":108,"publishedAt":42},"01M6ACDRVH48A1WBYH9DEETW7F","python-development","Guides Python SDK development in Apache Beam, including environment setup, testing, building, and running pipelines. Use when working with Python code in sdks\u002Fpython\u002F.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fpython-development\u002FSKILL.md",[107],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[84,109,24,28,29,30,110,82,31,32,35,83,111,9,38,80,40,41,37,98],"python development","python code","python sdk",{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":123,"publishedAt":42},"01M6ACDPYBMSHSQNCFXVJ7VJW3","license-compliance","Ensures all new files include proper Apache 2.0 license headers. Use when creating any new file in the Apache Beam repository.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Flicense-compliance\u002FSKILL.md",[118],{"categoryNo":119,"categoryName":120,"taxonomyType":121,"taxonomyTypeDesc":122},"01JBSEEDCAT000000000000037","律师与法务","PROFESSION","适用职业",[124,125,126,29,127,128,129,35,130,131,9,41],"compliance","license compliance","license","new","headers","files","repository","file",{"skillNo":133,"name":134,"nameZh":6,"description":135,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":136,"categories":137,"keywords":139,"publishedAt":42},"01M6ACDHGVH4F8ADMCFR7T34H8","io-connectors","Guides development and usage of I\u002FO connectors in Apache Beam. Use when working with I\u002FO connectors, creating new connectors, or debugging data source\u002Fsink issues.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fio-connectors\u002FSKILL.md",[138],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[140,141,142,143,31,32,35,144,9,94,24,95,41,145,127,29],"io","io connectors","connectors","usage","data","source",{"skillNo":147,"name":148,"nameZh":6,"description":149,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":150,"categories":151,"keywords":152,"publishedAt":42},"01M6ACDF6YTQG62GC4JKAVQBJS","gradle-build","Guides understanding and using the Gradle build system in Apache Beam. Use when building projects, understanding dependencies, or troubleshooting build issues.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fbeam\u002Fblob\u002Fmaster\u002F.agent\u002Fskills\u002Fgradle-build\u002FSKILL.md",[],[153,154,155,41,29,32,156,35,157,158,159,9,94,160],"gradle","build","gradle build","dependencies","troubleshooting","system","projects","building projects"]