[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACJR50ZS7F1QX58JEKCGVM":3,"skill-same:01M6ACJR50ZS7F1QX58JEKCGVM":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":58,"ownerDescription":6,"updatedAt":59,"install":60,"resources":73},"01M6ACJR50ZS7F1QX58JEKCGVM","wire-datafusion-function",null,"Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.","apache\u002Fdatafusion-comet","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACHX42TPNCR8TQJEAR8J5J",1322,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"datafusion","wire datafusion function","registration","when","wiring","custom","Existing","for","with","expression","name","tests","upstream","implement","switch","use","api","file","spark","scala","sql","2026-11-10 10:12:30",false,"---\nname: wire-datafusion-function\ndescription: Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.\nargument-hint: \u003Cexpression-name>\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nWire Comet support for the `$ARGUMENTS` Spark expression by reusing an existing DataFusion or `datafusion-spark` function. **No native Rust is written here** — if upstream coverage is missing, stop and run `implement-comet-expression`.\n\n## Wiring patterns\n\nPick the lightest one that satisfies the Spark contract.\n\n| Pattern                        | When                                                                                                                                                                                                                                                                     | What to change                                                                                                                                                                   |\n| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **A** — passthrough            | DF built-in (e.g. `acos`), or `datafusion-spark` UDF already registered in `register_datafusion_spark_function`                                                                                                                                                          | one line in the right map of `QueryPlanSerde.scala`: `classOf[Foo] -> CometScalarFunction(\"foo\")`                                                                                |\n| **B** — register + passthrough | `datafusion-spark` UDF, _not_ yet registered, semantics already match Spark                                                                                                                                                                                              | Pattern A line **plus** `session_ctx.register_udf(ScalarUDF::new_from_impl(SparkFoo::default()));` in `native\u002Fcore\u002Fsrc\u002Fexecution\u002Fjni_api.rs::register_datafusion_spark_function` |\n| **C** — custom serde           | Inputs need preprocessing (cast, `nullIfNegative`, `+0.0` flip), or you need to set return type \u002F `failOnError`, restrict input types via `getSupportLevel`, enforce foldable-only args, or attach `getCompatibleNotes`\u002F`getIncompatibleReasons`\u002F`getUnsupportedReasons` | new `CometXxx` object in the topic file (`math.scala`, `strings.scala`, …); see `CometCeil`, `CometAtan2`, `CometLog`, `CometSha2`, `CometAbs`                                   |\n\nFunction names must match: the string passed to `CometScalarFunction(\"xyz\")` equals `SparkXyz::name()` upstream.\n\n## Workflow\n\n### 1. Study the Spark contract\n\nFind the `case class $ARGUMENTS` (PascalCase). Prefer the user's local Spark clone (agent instructions \u002F project memory); fall back to `git clone --depth 1 https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark.git \u002Ftmp\u002Fspark-master`. If the clone is sandbox-blocked and no local clone is found, **stop and ask** — Spark is the ground truth.\n\nNote `inputTypes`, `dataType`, `eval` \u002F `nullSafeEval`, `require` guards, ANSI mode branches, and foldable-only arguments. These define what Comet must reproduce.\n\n### 2. Find the upstream function (at the pinned version)\n\nThe pinned versions in `native\u002FCargo.toml` are the ground truth — a function on upstream `main` may not exist in the version Comet depends on. Resolve versions first, then search sources matching those exact versions. **Pick one source path; do not mix.**\n\n```bash\nREPO_ROOT=$(git rev-parse --show-toplevel)\n# Portable regex — BSD awk on macOS does not support \\b.\nDF_SPARK_VER=$(awk -F'\"' '\u002F^datafusion-spark[ =]\u002F {print $2; exit}' \"$REPO_ROOT\u002Fnative\u002FCargo.toml\")\nDF_FUNCS_VER=$(awk -F'\"' '\u002F^datafusion[ =]\u002F        {print $2; exit}' \"$REPO_ROOT\u002Fnative\u002FCargo.toml\")\n[ -z \"$DF_SPARK_VER\" ] || [ -z \"$DF_FUNCS_VER\" ] && { echo \"ERROR: version extraction failed\"; exit 1; }\nEXPR='$ARGUMENTS'\n```\n\n**Option A — cargo registry (preferred):**\n\n```bash\nDF_SPARK=$(ls -d ~\u002F.cargo\u002Fregistry\u002Fsrc\u002F*\u002Fdatafusion-spark-${DF_SPARK_VER}\u002F 2>\u002Fdev\u002Fnull | head -1)\nDF_FUNCS=$(ls -d ~\u002F.cargo\u002Fregistry\u002Fsrc\u002F*\u002Fdatafusion-functions-${DF_FUNCS_VER}\u002F 2>\u002Fdev\u002Fnull | head -1)\n# If empty, run `cargo fetch` from native\u002F.\ngrep -rin \"fn name\" \"$DF_SPARK\u002Fsrc\u002Ffunction\u002F\" 2>\u002Fdev\u002Fnull | grep -i \"$EXPR\"\ngrep -rin \"fn name\" \"$DF_FUNCS\u002Fsrc\u002F\"          2>\u002Fdev\u002Fnull | grep -i \"$EXPR\"\n```\n\n**Option B — local DataFusion clone** (only if agent instructions \u002F memory point at one). Grep at the pinned tag — DataFusion uses lightweight tags (`\u003Cversion>`, no `v` prefix), so use `tag -l` not `rev-parse --verify ^{tag}`:\n\n```bash\nDF_CLONE=\u003Cpath from memory>\n[ -z \"$(git -C \"$DF_CLONE\" tag -l \"$DF_SPARK_VER\")\" ] && echo \"WARNING: tag missing — git fetch --tags or use Option A\"\ngit -C \"$DF_CLONE\" grep -in \"fn name\" \"$DF_SPARK_VER\" -- 'datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F' | grep -i \"$EXPR\"\ngit -C \"$DF_CLONE\" grep -in \"fn name\" \"$DF_FUNCS_VER\" -- 'datafusion\u002Ffunctions\u002Fsrc\u002F'      | grep -i \"$EXPR\"\n```\n\nNever grep the clone's working tree directly — it may be on `main`.\n\n### 2a. Decision gate — confirm the source crate\n\n- **Found in `datafusion-spark`** → proceed (Spark-tuned by design). If listed in `register_datafusion_spark_function` → Pattern A; otherwise Pattern B.\n- **Found only in `datafusion-functions`** → **STOP and `AskUserQuestion`**. Pure DataFusion follows standard SQL semantics and frequently diverges from Spark (NULL vs error, negatives, overflow, return type, NaN). Surface what you found in each crate and the divergences you've already identified, then offer:\n  1. Wire from `datafusion-functions` with Pattern C bridging — list the divergences and the masking\u002Fcasting that closes them.\n  2. Stop and switch to `implement-comet-expression` to port to `datafusion-spark` upstream first.\n- **Found in neither** → stop and run `implement-comet-expression`.\n\nIf semantics diverge in a way Pattern C can't bridge, escalate.\n\n### 3. Apply the Scala wiring\n\nAdd the entry to the matching map in `spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002FQueryPlanSerde.scala`: `mathExpressions`, `stringExpressions`, `arrayExpressions`, `mapExpressions`, `structExpressions`, `predicateExpressions`, `conditionalExpressions`, `bitwiseExpressions`, `temporalExpressions`, `hashExpressions`, `conversionExpressions`, `miscExpressions`.\n\nSpark expression classes are in scope via `import org.apache.spark.sql.catalyst.expressions._`. For Pattern C, the topic file uses **explicit** imports — extend the existing `import …expressions.{…}` line and place the new object alongside its peers.\n\nHelpers from `QueryPlanSerde`:\n\n- `scalarFunctionExprToProto(name, args*)` — return type inferred, `failOnError = false`.\n- `scalarFunctionExprToProtoWithReturnType(name, returnType, failOnError, args*)` — explicit type \u002F fail-on-error.\n- `optExprWithInfo(optExpr, expr, children*)` — wrap final result; propagates \"why we couldn't convert\" tags.\n- `withInfo(expr, \"reason\")` — tag a fallback when returning `None`.\n\nIf the Spark behavior you are matching varies by Spark version, resolve the version in Scala and pass the native side a parameter, or pick a function name, that describes the behavior rather than the version (see \"Name the behavior, not the Spark version\" in `adding_a_new_expression.md`).\n\n`getSupportLevel` returning `Incompatible(Some(\"…\"))` gates behind `spark.comet.expr.\u003Cname>.allowIncompatible=true`. `Unsupported(…)` always falls back.\n\n### 4. Register the UDF (Pattern B only)\n\nIn `native\u002Fcore\u002Fsrc\u002Fexecution\u002Fjni_api.rs::register_datafusion_spark_function`, add the `use` and the `register_udf` line, grouped with neighbors. Patterns A and C never touch native code.\n\n### 5. Add SQL file tests\n\nCreate `spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F\u003Ccategory>\u002F$ARGUMENTS.sql`. Mandatory rules:\n\n1. **Test only the directly supported input types** — read `inputTypes` from the Spark source. Do **not** add columns for implicit-cast widening (`ImplicitCastInputTypes`); the cast path has its own tests. E.g. `inputTypes = Seq(IntegerType)` (`Factorial`) → one `int` column, not byte\u002Fshort\u002Flong\u002Ffloat\u002Fdouble\u002Fdecimal. For `TypeCollection(A, B)`, add one column per member.\n2. **All `SELECT`s read from a parquet table** — no literal-only queries (the constant folder skips the read path and column-vector kernel).\n3. **Mix `NULL` into every column** — exercises the validity bitmap.\n4. **Per type, cover the edge-case set:**\n   - Float \u002F Double: `NaN`, `+0.0`, `-0.0`, `±Infinity` via `cast('NaN' as double)` etc.\n   - Integer \u002F decimal: each type's max and min (`127`\u002F`-128`, `2147483647`\u002F`-2147483648`, `9223372036854775807`\u002F`-9223372036854775808`, `Decimal(38, 0)` boundary) for overflow \u002F wrap.\n   - String \u002F binary: empty, multi-byte UTF-8 (`'é'`, `'日本'`), embedded NULs if relevant.\n5. **Cover the Spark-specific edges** found in step 1: in-range boundaries, just-out-of-range values, ANSI error paths, foldable-only args.\n\nUse `query expect_fallback(...)` for inputs where the serde returns `None`. Format: `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`.\n\n### 6. Update docs\n\nHand-curated (`make` does not regenerate these):\n\n- `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md` — add `| \u003CExpressionClass> | \\`\u003Csql_name>\\` |` to the matching category table, alphabetical.\n- `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md` — update the function's Status from ?\u002F? to ✅ (or ⚠️) in the matching category table.\n\nPer-expression compatibility pages under `compatibility\u002Fexpressions\u002F*.md` are auto-generated from `getCompatibleNotes` \u002F `getIncompatibleReasons` \u002F `getUnsupportedReasons` — do not hand-edit.\n\n### 7. Build, audit, finalize\n\n```bash\nmake\ncd native && cargo clippy --all-targets --workspace -- -D warnings\n```\n\nThen run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.4.3 \u002F 3.5.8 \u002F 4.0.1 and surface coverage gaps; iterate on tests.\n\nThe user generally runs tests themselves. If asked for a smoke test:\n\n```bash\n.\u002Fmvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite $ARGUMENTS\" -Dtest=none\n```\n\nPR: fill in `.github\u002Fpull_request_template.md`; under \"What changes are included\", note the skill was used and which upstream function (`datafusion::…` built-in or `datafusion-spark::function::…`) is being wired.\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nWire Comet support for the `$ARGUMENTS` Spark expression by reusing an existing DataFusion or `datafusion-spark` function. **No native Rust is written here** — if upstream coverage is missing, stop and run `implement-comet-expression`.\n\n## Wiring patterns\n\nPick the lightest one that satisfies the Spark contract.\n\n| Pattern                        | When                                                                                                                                                                                                                                                                     | What to change                                                                                                                                                                   |\n| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| **A** — passthrough            | DF built-in (e.g. `acos`), or `datafusion-spark` UDF already registered in `register_datafusion_spark_function`                                                                                                                                                          | one line in the right map of `QueryPlanSerde.scala`: `classOf[Foo] -> CometScalarFunction(\"foo\")`                                                                                |\n| **B** — register + passthrough | `datafusion-spark` UDF, _not_ yet registered, semantics already match Spark                                                                                                                                                                                              | Pattern A line **plus** `session_ctx.register_udf(ScalarUDF::new_from_impl(SparkFoo::default()));` in `native\u002Fcore\u002Fsrc\u002Fexecution\u002Fjni_api.rs::register_datafusion_spark_function` |\n| **C** — custom serde           | Inputs need preprocessing (cast, `nullIfNegative`, `+0.0` flip), or you need to set return type \u002F `failOnError`, restrict input types via `getSupportLevel`, enforce foldable-only args, or attach `getCompatibleNotes`\u002F`getIncompatibleReasons`\u002F`getUnsupportedReasons` | new `CometXxx` object in the topic file (`math.scala`, `strings.scala`, …); see `CometCeil`, `CometAtan2`, `CometLog`, `CometSha2`, `CometAbs`                                   |\n\nFunction names must match: the string passed to `CometScalarFunction(\"xyz\")` equals `SparkXyz::name()` upstream.\n\n## Workflow\n\n### 1. Study the Spark contract\n\nFind the `case class $ARGUMENTS` (PascalCase). Prefer the user's local Spark clone (agent instructions \u002F project memory); fall back to `git clone --depth 1 https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark.git \u002Ftmp\u002Fspark-master`. If the clone is sandbox-blocked and no local clone is found, **stop and ask** — Spark is the ground truth.\n\nNote `inputTypes`, `dataType`, `eval` \u002F `nullSafeEval`, `require` guards, ANSI mode branches, and foldable-only arguments. These define what Comet must reproduce.\n\n### 2. Find the upstream function (at the pinned version)\n\nThe pinned versions in `native\u002FCargo.toml` are the ground truth — a function on upstream `main` may not exist in the version Comet depends on. Resolve versions first, then search sources matching those exact versions. **Pick one source path; do not mix.**\n\n```bash\nREPO_ROOT=$(git rev-parse --show-toplevel)\n# Portable regex — BSD awk on macOS does not support \\b.\nDF_SPARK_VER=$(awk -F'\"' '\u002F^datafusion-spark[ =]\u002F {print $2; exit}' \"$REPO_ROOT\u002Fnative\u002FCargo.toml\")\nDF_FUNCS_VER=$(awk -F'\"' '\u002F^datafusion[ =]\u002F        {print $2; exit}' \"$REPO_ROOT\u002Fnative\u002FCargo.toml\")\n[ -z \"$DF_SPARK_VER\" ] || [ -z \"$DF_FUNCS_VER\" ] && { echo \"ERROR: version extraction failed\"; exit 1; }\nEXPR='$ARGUMENTS'\n```\n\n**Option A — cargo registry (preferred):**\n\n```bash\nDF_SPARK=$(ls -d ~\u002F.cargo\u002Fregistry\u002Fsrc\u002F*\u002Fdatafusion-spark-${DF_SPARK_VER}\u002F 2>\u002Fdev\u002Fnull | head -1)\nDF_FUNCS=$(ls -d ~\u002F.cargo\u002Fregistry\u002Fsrc\u002F*\u002Fdatafusion-functions-${DF_FUNCS_VER}\u002F 2>\u002Fdev\u002Fnull | head -1)\n# If empty, run `cargo fetch` from native\u002F.\ngrep -rin \"fn name\" \"$DF_SPARK\u002Fsrc\u002Ffunction\u002F\" 2>\u002Fdev\u002Fnull | grep -i \"$EXPR\"\ngrep -rin \"fn name\" \"$DF_FUNCS\u002Fsrc\u002F\"          2>\u002Fdev\u002Fnull | grep -i \"$EXPR\"\n```\n\n**Option B — local DataFusion clone** (only if agent instructions \u002F memory point at one). Grep at the pinned tag — DataFusion uses lightweight tags (`\u003Cversion>`, no `v` prefix), so use `tag -l` not `rev-parse --verify ^{tag}`:\n\n```bash\nDF_CLONE=\u003Cpath from memory>\n[ -z \"$(git -C \"$DF_CLONE\" tag -l \"$DF_SPARK_VER\")\" ] && echo \"WARNING: tag missing — git fetch --tags or use Option A\"\ngit -C \"$DF_CLONE\" grep -in \"fn name\" \"$DF_SPARK_VER\" -- 'datafusion\u002Fspark\u002Fsrc\u002Ffunction\u002F' | grep -i \"$EXPR\"\ngit -C \"$DF_CLONE\" grep -in \"fn name\" \"$DF_FUNCS_VER\" -- 'datafusion\u002Ffunctions\u002Fsrc\u002F'      | grep -i \"$EXPR\"\n```\n\nNever grep the clone's working tree directly — it may be on `main`.\n\n### 2a. Decision gate — confirm the source crate\n\n- **Found in `datafusion-spark`** → proceed (Spark-tuned by design). If listed in `register_datafusion_spark_function` → Pattern A; otherwise Pattern B.\n- **Found only in `datafusion-functions`** → **STOP and `AskUserQuestion`**. Pure DataFusion follows standard SQL semantics and frequently diverges from Spark (NULL vs error, negatives, overflow, return type, NaN). Surface what you found in each crate and the divergences you've already identified, then offer:\n  1. Wire from `datafusion-functions` with Pattern C bridging — list the divergences and the masking\u002Fcasting that closes them.\n  2. Stop and switch to `implement-comet-expression` to port to `datafusion-spark` upstream first.\n- **Found in neither** → stop and run `implement-comet-expression`.\n\nIf semantics diverge in a way Pattern C can't bridge, escalate.\n\n### 3. Apply the Scala wiring\n\nAdd the entry to the matching map in `spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002FQueryPlanSerde.scala`: `mathExpressions`, `stringExpressions`, `arrayExpressions`, `mapExpressions`, `structExpressions`, `predicateExpressions`, `conditionalExpressions`, `bitwiseExpressions`, `temporalExpressions`, `hashExpressions`, `conversionExpressions`, `miscExpressions`.\n\nSpark expression classes are in scope via `import org.apache.spark.sql.catalyst.expressions._`. For Pattern C, the topic file uses **explicit** imports — extend the existing `import …expressions.{…}` line and place the new object alongside its peers.\n\nHelpers from `QueryPlanSerde`:\n\n- `scalarFunctionExprToProto(name, args*)` — return type inferred, `failOnError = false`.\n- `scalarFunctionExprToProtoWithReturnType(name, returnType, failOnError, args*)` — explicit type \u002F fail-on-error.\n- `optExprWithInfo(optExpr, expr, children*)` — wrap final result; propagates \"why we couldn't convert\" tags.\n- `withInfo(expr, \"reason\")` — tag a fallback when returning `None`.\n\nIf the Spark behavior you are matching varies by Spark version, resolve the version in Scala and pass the native side a parameter, or pick a function name, that describes the behavior rather than the version (see \"Name the behavior, not the Spark version\" in `adding_a_new_expression.md`).\n\n`getSupportLevel` returning `Incompatible(Some(\"…\"))` gates behind `spark.comet.expr.\u003Cname>.allowIncompatible=true`. `Unsupported(…)` always falls back.\n\n### 4. Register the UDF (Pattern B only)\n\nIn `native\u002Fcore\u002Fsrc\u002Fexecution\u002Fjni_api.rs::register_datafusion_spark_function`, add the `use` and the `register_udf` line, grouped with neighbors. Patterns A and C never touch native code.\n\n### 5. Add SQL file tests\n\nCreate `spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F\u003Ccategory>\u002F$ARGUMENTS.sql`. Mandatory rules:\n\n1. **Test only the directly supported input types** — read `inputTypes` from the Spark source. Do **not** add columns for implicit-cast widening (`ImplicitCastInputTypes`); the cast path has its own tests. E.g. `inputTypes = Seq(IntegerType)` (`Factorial`) → one `int` column, not byte\u002Fshort\u002Flong\u002Ffloat\u002Fdouble\u002Fdecimal. For `TypeCollection(A, B)`, add one column per member.\n2. **All `SELECT`s read from a parquet table** — no literal-only queries (the constant folder skips the read path and column-vector kernel).\n3. **Mix `NULL` into every column** — exercises the validity bitmap.\n4. **Per type, cover the edge-case set:**\n   - Float \u002F Double: `NaN`, `+0.0`, `-0.0`, `±Infinity` via `cast('NaN' as double)` etc.\n   - Integer \u002F decimal: each type's max and min (`127`\u002F`-128`, `2147483647`\u002F`-2147483648`, `9223372036854775807`\u002F`-9223372036854775808`, `Decimal(38, 0)` boundary) for overflow \u002F wrap.\n   - String \u002F binary: empty, multi-byte UTF-8 (`'é'`, `'日本'`), embedded NULs if relevant.\n5. **Cover the Spark-specific edges** found in step 1: in-range boundaries, just-out-of-range values, ANSI error paths, foldable-only args.\n\nUse `query expect_fallback(...)` for inputs where the serde returns `None`. Format: `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`.\n\n### 6. Update docs\n\nHand-curated (`make` does not regenerate these):\n\n- `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md` — add `| \u003CExpressionClass> | \\`\u003Csql_name>\\` |` to the matching category table, alphabetical.\n- `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md` — update the function's Status from ?\u002F? to ✅ (or ⚠️) in the matching category table.\n\nPer-expression compatibility pages under `compatibility\u002Fexpressions\u002F*.md` are auto-generated from `getCompatibleNotes` \u002F `getIncompatibleReasons` \u002F `getUnsupportedReasons` — do not hand-edit.\n\n### 7. Build, audit, finalize\n\n```bash\nmake\ncd native && cargo clippy --all-targets --workspace -- -D warnings\n```\n\nThen run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.4.3 \u002F 3.5.8 \u002F 4.0.1 and surface coverage gaps; iterate on tests.\n\nThe user generally runs tests themselves. If asked for a smoke test:\n\n```bash\n.\u002Fmvnw test -Dsuites=\"org.apache.comet.CometSqlFileTestSuite $ARGUMENTS\" -Dtest=none\n```\n\nPR: fill in `.github\u002Fpull_request_template.md`; under \"What changes are included\", note the skill was used and which upstream function (`datafusion::…` built-in or `datafusion-spark::function::…`) is being wired.\n",{"name":5,"description":7,"argument-hint":50},"\u003Cexpression-name>","main","7deeba21a42e4a9cd34f43e7ed6e960e4e851b26",".ai\u002Fskills\u002Fwire-datafusion-function",418,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet","https:\u002F\u002Fdatafusion.apache.org\u002Fcomet","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-15 03:15:20",{"source":8,"skillName":5,"commands":61,"prompt":71,"zipAvailable":72},[62,65,68],{"tool":63,"command":64},"npx","npx skills add apache\u002Fdatafusion-comet --skill wire-datafusion-function",{"tool":66,"command":67},"bunx","bunx skills add apache\u002Fdatafusion-comet --skill wire-datafusion-function",{"tool":69,"command":70},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-comet --skill wire-datafusion-function","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",true,[74],{"path":75,"resourceType":76,"resourceTypeDesc":77,"sizeBytes":78,"sourceHtmlUrl":79},"OWNERSHIP","OTHER","其他",24,"https:\u002F\u002Fgithub.com\u002FJetBrains\u002Fintellij-community\u002Fblob\u002Fmaster\u002F.agents\u002Fskills\u002Frust-code-style\u002FOWNERSHIP",[81,107,128,145,156,171],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":45},"01M6ACJPDDPND6FHE2PC1CJ4D3","suggest-native-expression","Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fsuggest-native-expression\u002FSKILL.md",[],[88,33,89,90,91,92,37,42,93,94,27,95,96,97,98,99,100,101,102,103,104,105,106],"suggest native expression","native","dispatcher","github","rust","re","files","candidate","log","jvm","next","how","audit","audit log","allocation","issue","throughput","path","save",{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":45},"01M6ACJMHPEN7E76VPKC3P4X6C","review-comet-shuffle-pr","Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",[],[114,115,116,117,118,24,119,39,120,27,121,122,97,89,123,124,125,126,127],"review comet shuffle pr","pr","review","pull","pull request","arrow","load","ipc","number","request","format","integration","block","compression",{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":132,"categories":133,"keywords":134,"publishedAt":45},"01M6ACJHX8BES7ZNNRNVCQ4QWE","review-comet-pr","Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-pr\u002FSKILL.md",[],[115,116,135,117,136,118,137,24,138,139,27,122,140,141,123,142,31,39,143,144],"review comet pr","iceberg","memory","workflow","documentation","reviewer","human","posting","review skills","comments",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":151,"publishedAt":45},"01M6ACJFM0QS61Z1RW516HXN2M","review-comet-memory-pr","Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",[],[115,137,152,116,117,118,24,42,153,154,39,120,27,122,123,155],"review comet memory pr","bridge","reservations","configuration",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":45},"01M6ACJCV64V0FZPMVXDE2K3DX","review-comet-iceberg-write-pr","Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",[],[115,136,163,116,164,117,118,24,92,165,166,167,39,120,168,27,122,169,170,89],"review comet iceberg write pr","write","writer","write plan","partition","pin","split","plan",{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":45},"01M6ACJA72KJRAM6F7WHMX01PN","review-comet-ffi-pr","Use when reviewing a DataFusion Comet pull request that crosses the JVM\u002Fnative boundary, touching Arrow C Data or C Stream interface code, batch export and import, CometExecIterator, ScanExec, NativeUtil, CometVector subclasses, or jni_api. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-ffi-pr\u002FSKILL.md",[],[115,178,116,179,117,118,24,119,180,181,182,183,39,120,27,122,40,97,184,89,123],"review comet ffi pr","ffi","code","batch","import","interface","boundary"]