[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACHZN8CDV3QFB40NX62K3H":3,"skill-same:01M6ACHZN8CDV3QFB40NX62K3H":67},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":51,"ownerDescription":6,"updatedAt":52,"install":53,"resources":66},"01M6ACHZN8CDV3QFB40NX62K3H","audit-comet-expression",null,"Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.","apache\u002Fdatafusion-comet","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACHX42TPNCR8TQJEAR8J5J",1322,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Faudit-comet-expression\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"audit comet expression","audit","expression","implementation","Existing","for","correctness","reviews","missing","name","tests","across","implement","test coverage","coverage","offers","datafusion","spark","test","2026-11-10 10:12:30",false,"---\nname: audit-comet-expression\ndescription: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.\nargument-hint: \u003Cexpression-name>\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nAudit the Comet implementation of the `$ARGUMENTS` expression for correctness and test coverage.\n\n## Overview\n\nThis audit covers:\n\n1. Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1\n2. Comet Scala serde implementation\n3. Comet Rust \u002F DataFusion implementation\n4. Existing test coverage (Comet SQL Tests and Comet Scala Tests)\n5. Gap analysis and test recommendations\n\n---\n\n## Step 1: Locate the Spark Implementations\n\nClone specific Spark version tags (use shallow clones to avoid polluting the workspace). Only clone a version if it is not already present.\n\n```bash\nset -eu -o pipefail\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  if [ ! -d \"$dir\" ]; then\n    git clone --depth 1 --branch \"$tag\" https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark.git \"$dir\"\n  fi\ndone\n```\n\n### Find the expression class in each Spark version\n\nSearch the Catalyst SQL expressions source:\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala\" -name \"*.scala\" | \\\n    xargs grep -l \"case class $ARGUMENTS\\b\\|object $ARGUMENTS\\b\" 2>\u002Fdev\u002Fnull\ndone\n```\n\nIf the expression is not found in catalyst, also check core:\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\" -name \"*.scala\" | \\\n    xargs grep -l \"case class $ARGUMENTS\\b\\|object $ARGUMENTS\\b\" 2>\u002Fdev\u002Fnull\ndone\n```\n\n### Read the Spark source for each version\n\nFor each Spark version, read the expression file and note:\n\n- The `eval`, `nullSafeEval`, and `doGenCode` \u002F `doGenCodeSafe` methods\n- The `inputTypes` and `dataType` fields (accepted input types, return type)\n- Null handling strategy (`nullable`, `nullSafeEval`)\n- ANSI mode behavior (`ansiEnabled`, `failOnError`)\n- Special cases, guards, `require` assertions, and runtime exceptions\n- Any constants or configuration the expression reads\n\n### Compare across Spark versions\n\nProduce a concise diff summary of what changed between:\n\n- 3.4.3 → 3.5.8\n- 3.5.8 → 4.0.1\n- 4.0.1 → 4.1.1\n\nPay attention to:\n\n- New input types added or removed\n- Behavior changes for edge cases (null, overflow, empty, boundary)\n- New ANSI mode branches\n- New parameters or configuration\n- Breaking API changes that Comet must shim\n\n---\n\n## Step 2: Locate the Spark Tests\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\" -name \"*.scala\" -path \"*\u002Ftest\u002F*\" | \\\n    xargs grep -l \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\ndone\n```\n\nRead the relevant Spark test files and produce a list of:\n\n- Input types covered\n- Edge cases exercised (null, empty, overflow, negative, boundary values, special characters, etc.)\n- ANSI mode tests\n- Error cases\n\nThis list will be the reference for the coverage gap analysis in Step 5.\n\n---\n\n## Step 3: Locate the Comet Implementation\n\n### Scala serde\n\n```bash\n# Find the serde object\ngrep -r \"$ARGUMENTS\" spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002F --include=\"*.scala\" -l\ngrep -r \"$ARGUMENTS\" spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002F --include=\"*.scala\" -l\n```\n\nRead the serde implementation and check:\n\n- Which Spark versions the serde handles\n- Whether all input types Spark accepts are handled\n- Whether `convert` validates expression shape (e.g. literal-only arguments) before serializing\n\nThen audit the support-level and reason methods as described below.\n\n#### Audit `getSupportLevel`, `getIncompatibleReasons`, `getUnsupportedReasons`, and `convert`\n\nThese four methods must stay aligned. Each has a distinct purpose, and the\nmost common bugs in Comet serdes are misalignments between them.\n\n**Pick the right support level.**\n\n- `Unsupported(Some(reason))`: Comet cannot run this case at all. The\n  dispatcher in `QueryPlanSerde.exprToProtoInternal` falls back to Spark\n  unconditionally. Use this when an input type, option, or expression shape\n  is not implemented, or when running the native path would crash or error.\n- `Incompatible(Some(reason))`: Comet can run this, but results may differ\n  from Spark. The dispatcher only allows it when\n  `spark.comet.expression.\u003CexprName>.allowIncompatible=true` is set. Use\n  this for known result differences such as locale sensitivity, timezone\n  handling, ordering ambiguity, or floating-point precision.\n- `Compatible(None)`: full Spark compatibility for this combination of\n  inputs and options.\n- `Compatible(Some(note))`: fully compatible but with a docs-only caveat.\n  Note that any `Some(...)` on `Compatible` triggers a runtime\n  `logWarning`, so reserve it for genuinely useful caveats.\n\nDecision rule: if the user would be surprised by a _wrong answer_, it is\n`Incompatible`. If the user would be surprised by a _runtime error or\nunsupported-type message_, it is `Unsupported`.\n\n**Runtime vs docs split.**\n\n- The `notes` field on `Incompatible(Some(...))` and `Unsupported(Some(...))`\n  flows into EXPLAIN output via the dispatcher (see\n  `QueryPlanSerde.exprToProtoInternal`, around the `case Incompatible` \u002F\n  `case Unsupported` branches). This is what users see when they ask why\n  Comet fell back.\n- `getIncompatibleReasons()` and `getUnsupportedReasons()` are read only by\n  `GenerateDocs` when building\n  `docs\u002Fsource\u002Fuser-guide\u002Fcompatibility.md`. They are static (no `expr`\n  argument) and should enumerate every distinct reason the expression\n  could ever return at runtime.\n- `convert` may also call `withInfo(expr, \"reason\")` and return `None` for\n  cases that cannot be detected from the expression alone (for example,\n  non-literal arguments, or child conversion failures). Those reasons\n  belong in `getUnsupportedReasons()` too.\n\n**Consistency rules.**\n\n1. If `getSupportLevel` can return `Incompatible(...)` for any input,\n   override `getIncompatibleReasons()` and include a reason for every\n   incompatible branch.\n2. If `getSupportLevel` can return `Unsupported(...)` for any input,\n   override `getUnsupportedReasons()` and include a reason for every\n   unsupported branch.\n3. If `convert` has its own `withInfo(...); None` fallbacks (e.g. literal\n   checks), enumerate those reasons in `getUnsupportedReasons()` too.\n4. Extract each reason into a `private val` and reference it from both\n   `getSupportLevel` and `get*Reasons()`. Do not duplicate the string\n   inline. Canonical example:\n   `spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002Farrays.scala::CometArrayIntersect`.\n   It declares `private val incompatReason` and\n   `private val unsupportedCollationReason`, and each is referenced from\n   `getSupportLevel` and the matching reasons method.\n5. Prefer `Incompatible(Some(reason))` over `Incompatible(None)`. The\n   `None` form drops the reason from EXPLAIN output, leaving users with a\n   generic \"not fully compatible\" message and forcing them to read the\n   docs to find out why.\n6. Gate compatibility decisions in `getSupportLevel`, not inside `convert`.\n   Putting the check inside `convert` (e.g. reading a config flag and\n   calling `withInfo`) bypasses the dispatcher's `allowIncompatible`\n   handling and the EXPLAIN message becomes inconsistent with what the\n   doc generator produces.\n\n**Wording guidelines for reason strings.**\n\nReasons appear verbatim in the Compatibility Guide (rendered as Markdown)\nand in EXPLAIN output, so they are user-facing.\n\n- Lead with the user-observable effect, then the cause if helpful.\n  ✅ \"Result array element order may differ from Spark when the right array\n  is longer than the left.\"\n  ❌ \"DataFusion probes the longer side.\"\n- Use sentence case and end with a period.\n- Use backticks around config keys, type names, and SQL identifiers.\n- Link to a tracking GitHub issue for known incompatibilities so users can\n  follow progress: `(https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002FNNNN)`.\n  **Verify the issue exists and is open** before citing it\n  (`gh issue view \u003CN> --repo apache\u002Fdatafusion-comet`). Issue numbers\n  invented from context or recalled from memory are a recurring failure\n  mode: a stale link is worse than no link because the reader follows it\n  and finds nothing.\n- Keep it concise. Single sentence is best.\n- Do not write \"Incompatible reason: ...\" or \"Unsupported because ...\".\n  The doc generator adds the framing.\n- Phrase Incompatible reasons as _what differs from Spark_, not _what is\n  missing_. Phrase Unsupported reasons as _what does not run_. If you find\n  yourself writing an \"Incompatible\" reason that says \"Comet only supports\n  X\" or \"Y is not supported\", the support level is probably wrong: it\n  should be `Unsupported`.\n\n**Common antipatterns to flag during the audit.**\n\n- A `private val reason` constant declared near the top of the object, but\n  `getSupportLevel` hardcodes a different string inline. The doc and the\n  EXPLAIN message will drift.\n  Real example: `CometHour` declares `incompatReason` then hardcodes a\n  near-duplicate string in `getSupportLevel`.\n- Reason text duplicated in two places without a shared constant.\n  Real examples: `CometMinute`, `CometSecond`.\n- `Incompatible(None)` paired with a populated `getIncompatibleReasons()`.\n  The reason reaches the docs but not EXPLAIN output.\n  Real example: `CometInitCap`.\n- `getIncompatibleReasons()` overridden but `getSupportLevel` never returns\n  `Incompatible(...)`. Either the reasons method is dead code, or\n  `getSupportLevel` is missing a branch.\n- A reason string phrased as \"X is not supported\" attached to an\n  `Incompatible` branch (or vice versa). Re-read it and decide which\n  support level it really belongs to.\n- `convert` bails out via `withInfo` for a case that is fully knowable\n  from the expression (e.g. an unsupported child data type). Move the\n  check into `getSupportLevel` so the dispatcher handles it uniformly.\n\n### Shims\n\n```bash\nfind spark\u002Fsrc\u002Fmain -name \"CometExprShim.scala\" | xargs grep -l \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\n```\n\nIf shims exist, read them and note any version-specific handling.\n\n### Rust \u002F DataFusion implementation\n\n```bash\n# Search for the function in native\u002Fspark-expr\ngrep -r \"$ARGUMENTS\" native\u002Fspark-expr\u002Fsrc\u002F --include=\"*.rs\" -l\ngrep -r \"$ARGUMENTS\" native\u002Fcore\u002Fsrc\u002F --include=\"*.rs\" -l\n```\n\nIf the expression delegates to DataFusion, find it there too. Set `$DATAFUSION_SRC` to a local DataFusion checkout, or fall back to searching the cargo registry:\n\n```bash\nif [ -n \"${DATAFUSION_SRC:-}\" ]; then\n  grep -r \"$ARGUMENTS\" \"$DATAFUSION_SRC\" --include=\"*.rs\" -l 2>\u002Fdev\u002Fnull | head -10\nelse\n  # Fall back to cargo registry (may include unrelated crates)\n  grep -r \"$ARGUMENTS\" ~\u002F.cargo\u002Fregistry\u002Fsrc\u002F*\u002Fdatafusion* --include=\"*.rs\" -l 2>\u002Fdev\u002Fnull | head -10\nfi\n```\n\nRead the Rust implementation and check:\n\n- Null handling (does it propagate nulls correctly?)\n- Overflow and underflow handling (returns `Err` vs panics)\n- Type dispatch (does it handle all types that Spark supports?)\n- ANSI \u002F fail-on-error mode\n\n---\n\n## Step 4: Locate Existing Comet Tests\n\n### Comet SQL Tests\n\n```bash\n# Find SQL test files for this expression\nfind spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F -name \"*.sql\" | \\\n  xargs grep -l \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\n\n# Also check if there's a dedicated file\nfind spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F -name \"*$(echo $ARGUMENTS | tr '[:upper:]' '[:lower:]')*\"\n```\n\nRead every SQL test file found and list:\n\n- Table schemas and data values used\n- Queries exercised\n- Query modes used (`query`, `spark_answer_only`, `tolerance`, `ignore`, `expect_error`)\n- Any ConfigMatrix directives\n\n### Comet Scala Tests\n\n```bash\ngrep -r \"$ARGUMENTS\" spark\u002Fsrc\u002Ftest\u002Fscala\u002F --include=\"*.scala\" -l\n```\n\nRead the relevant Comet Scala Tests and list:\n\n- Input types covered\n- Edge cases exercised\n- Whether constant folding is disabled for literal tests\n\n---\n\n## Step 5: Gap Analysis\n\nCompare the Spark test coverage (Step 2) against the Comet test coverage (Step 4). Produce a structured gap report:\n\n### Coverage matrix\n\nFor each of the following dimensions, note whether it is covered in Comet tests or missing:\n\n| Dimension                                                                                              | Spark tests it | Comet SQL Test | Comet Scala Test | Gap? |\n| ------------------------------------------------------------------------------------------------------ | -------------- | -------------- | ---------------- | ---- |\n| Column reference argument(s)                                                                           |                |                |                  |      |\n| Literal argument(s)                                                                                    |                |                |                  |      |\n| NULL input                                                                                             |                |                |                  |      |\n| Empty string \u002F empty array \u002F empty map                                                                 |                |                |                  |      |\n| Array\u002Fmap with NULL elements                                                                           |                |                |                  |      |\n| Zero, negative zero, negative values (numeric)                                                         |                |                |                  |      |\n| Underflow, overflow                                                                                    |                |                |                  |      |\n| Boundary values (INT_MIN, INT_MAX, Long.MinValue, minimum positive, etc.)                              |                |                |                  |      |\n| NaN, Infinity, -Infinity, subnormal (float\u002Fdouble)                                                     |                |                |                  |      |\n| Multibyte \u002F special UTF-8 (composed vs decomposed, e.g. `é` U+00E9 vs `e` + U+0301, non-Latin scripts) |                |                |                  |      |\n| ANSI mode (failOnError=true)                                                                           |                |                |                  |      |\n| Non-ANSI mode (failOnError=false)                                                                      |                |                |                  |      |\n| All supported input types                                                                              |                |                |                  |      |\n| Parquet dictionary encoding (ConfigMatrix)                                                             |                |                |                  |      |\n| Cross-version behavior differences                                                                     |                |                |                  |      |\n\n### Implementation gaps\n\nAlso review the Comet implementation (Step 3) against the Spark behavior (Step 1):\n\n- Are there input types that Spark supports but `getSupportLevel` returns `Unsupported` without comment?\n- Are there behavioral differences that are NOT marked `Incompatible` but should be?\n- Are there behavioral differences between Spark versions that the Comet implementation does not account for (missing shim)?\n- Does the Rust implementation match the Spark behavior for all edge cases?\n\n### Support-level consistency audit\n\nWalk through this checklist against the serde. Each failed item is a\nfinding for Step 6.\n\n1. **Support level matches behavior.** For each branch of\n   `getSupportLevel`, decide whether the user-observable effect is a wrong\n   answer (`Incompatible`) or a fallback \u002F error (`Unsupported`). Flag any\n   branch where the label does not match the behavior.\n2. **Reasons cover every branch.** Every distinct reason that\n   `getSupportLevel` can return as `Incompatible(Some(r))` must appear in\n   `getIncompatibleReasons()`. Same for `Unsupported(Some(r))` and\n   `getUnsupportedReasons()`. Missing reasons silently drop from the\n   Compatibility Guide.\n3. **Reasons are not dead code.** If `getIncompatibleReasons()` is\n   overridden but `getSupportLevel` never returns `Incompatible(...)`,\n   either the reason is stale or `getSupportLevel` is missing a branch.\n   Same for `getUnsupportedReasons()`.\n4. **Reason strings are shared via `private val`.** If the same reason\n   appears as a string literal in two places, flag it: changes to one\n   will not propagate to the other.\n5. **Inline reason matches the constant.** If a `private val reason` is\n   declared but `getSupportLevel` uses a different string literal, flag\n   it as a drift bug.\n6. **`Incompatible(None)` has no docs-only reason.** If\n   `getSupportLevel` returns `Incompatible(None)` but\n   `getIncompatibleReasons()` is non-empty, flag it: the EXPLAIN message\n   will be generic while the docs show a specific reason. Switch to\n   `Incompatible(Some(reason))`.\n7. **`convert` fallbacks are documented.** If `convert` calls\n   `withInfo(expr, \"...\")` and returns `None` for cases not covered by\n   `getSupportLevel` (e.g. non-literal arguments, unsupported expression\n   shapes), confirm the reason is also listed in\n   `getUnsupportedReasons()`.\n8. **Compatibility decisions live in `getSupportLevel`.** If `convert`\n   reads a config flag and bails out, prefer moving the check into\n   `getSupportLevel` so the dispatcher handles `allowIncompatible`\n   uniformly. `CometCaseConversionBase` is an example of the in-`convert`\n   pattern that this skill recommends against.\n9. **Reason wording.** Each reason should describe the user-observable\n   effect, use sentence case with a period, backtick config keys and\n   types, and link a tracking issue when one exists. Flag reasons that\n   read like internal implementation notes (\"DataFusion probes the longer\n   side\") or that mismatch their support level (an \"Incompatible\" reason\n   that says \"X is not supported\").\n10. **Expression-shape restrictions live in `getSupportLevel`.** Any\n    restriction that is knowable from the expression alone (literal-only\n    arguments, unsupported child data type, foldable-only options, a\n    specific operator shape) must be declared as an\n    `Unsupported(Some(reason))` branch in `getSupportLevel`, not gated\n    inside `convert` with a `withInfo` + `return None`. Putting the\n    check in `convert` means EXPLAIN surfaces the reason only at\n    conversion time, the doc generator never sees it, and the\n    dispatcher cannot route around it. The literal-only `len`\n    restrictions on `CometLeft`, `CometRight`, and `CometSubstring`\n    are the canonical example of the in-`convert` pattern that this\n    skill forbids: lift them into `getSupportLevel`.\n11. **Spark 4.0 collation divergences are flagged, not glossed over.**\n    If the Spark 4.0\u002F4.1 implementation routes through\n    `CollationSupport.X.exec(..., collationId)` (or uses\n    `StringTypeWithCollation` \u002F `StringTypeNonCSAICollation` for input\n    types) and the Comet path does not propagate collation, the\n    expression is `Incompatible` for non-default collations. Mark the\n    branch `Incompatible(Some(reason))` linking to the collation\n    umbrella issue\n    (https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F4496) so the\n    follow-up sweep can find every site. \"Behaviour unchanged for\n    `UTF8_BINARY`\" alone is not a justification for leaving the\n    support level at `Compatible`: users running with non-default\n    collations get silently wrong answers.\n12. **Known divergences flip the support level.** If you find yourself\n    writing the words \"Known divergence\" or \"Known limitation\" in the\n    support-doc sub-bullet while leaving `getSupportLevel` returning\n    `Compatible`, the audit has skipped its job. A documented\n    divergence is by definition not `Compatible`. Promote the branch\n    to `Incompatible(Some(reason))` (or `Unsupported` if the native\n    path errors rather than producing a wrong answer) and link the\n    tracking issue. The `replace` empty-search-string divergence with\n    DataFusion is the canonical example of this anti-pattern.\n13. **Unreachable serde mappings are removed.** Expressions registered\n    as `RuntimeReplaceable` (or otherwise rewritten by an analyzer\n    rule before serde) never reach `QueryPlanSerde.exprToProtoInternal`\n    with their original class. If the audit finds that a registered\n    `CometScalarFunction(\"name\")` or `CometExpressionSerde` entry can\n    never be hit (e.g. the `btrim` mapping for `StringTrimBoth`, which\n    is rewritten to `StringTrim` before serde runs), delete the\n    registration in the same audit PR. Documenting the dead code in\n    the support doc is not enough.\n\n---\n\n## Step 6: Recommendations\n\nSummarize findings as a prioritized list.\n\n### High priority: correctness divergences and high-risk coverage gaps\n\nCases where Comet produces a different observable result from Spark\n(wrong value, missing exception, accepted-instead-of-rejected input,\netc.), **and** untested cases on a known-fragile axis where a\ndivergence is plausible: overflow \u002F out-of-range inputs, NULL handling,\ntimezones and DST transitions, ANSI mode, leap years, version-specific\nSpark semantics, and any edge case explicitly called out in Step 1.\n\nTreat an untested high-risk case as a likely divergence until proven\notherwise. Each item in this bucket becomes a captured test in Step 7.\nFor an untested case, run it manually first to determine the current\nbehaviour, then commit either a regression test (passes) or a\n`query ignore(\u003Cissue-url>)` test (fails).\n\nEvery item in this bucket either becomes an inline fix + test, or a\nfiled GitHub issue + ignored regression test, in the audit PR. Step 7\nspells out the workflow: never leave a high-priority finding as PR-body\nprose only.\n\n### Medium priority: missing test coverage\n\nLow-risk coverage gaps: additional input permutations on already-tested\ncode paths, redundant happy-path values, or cases where Comet and Spark\nshare a well-exercised implementation. The behaviour appears to match\nand the axis is not on the high-risk list above.\n\n### Low priority: cosmetic and consistency issues\n\nReason-string drift, missing `get*Reasons()` overrides, dead branches,\netc. These come from the Step 5 consistency audit.\n\n---\n\n## Step 7: Apply Findings (Tests and Fixes), Then Offer the Rest\n\nHigh-priority findings (correctness divergences and high-risk coverage\ngaps) and consistency issues from Step 5 \u002F Step 6 must not be left as\nprose. Apply them in the same PR as the audit. Anything you cannot fix\ninline (because it needs a semantics decision, native code change, or\nlarger design work) must still be captured as a GitHub issue per the\n\"Findings that need follow-up\" section below: prose recommendations in\nthe PR body alone are insufficient. Only low-risk missing coverage\nrequires the user's go-ahead, because adding tests for cases that\nalready work on well-exercised paths is incremental polish.\n\n### High-priority findings: capture as tests\n\nEvery high-priority finding becomes a regression test in the same PR as\nthe audit, so future readers can run it. This covers both confirmed\ndivergences and high-risk untested cases. For the latter, run the case\nmanually first to determine whether it currently passes or fails, then\nfollow the same workflow below: a passing case is committed as a plain\n`query` regression test; a failing case is committed as\n`query ignore(\u003Cissue-url>)` after filing the bug.\n\nFor each high-priority finding, do the following in this order:\n\n1. **Search for an existing tracking issue.**\n\n   ```bash\n   gh issue list --search \"\u003Cexpression> \u003Csymptom> in:title,body\" --state all --limit 5\n   ```\n\n   Match on both the expression name and a distinguishing keyword\n   (ANSI, timezone, NTZ, overflow, etc.).\n\n2. **If no issue exists, file one.** Use the `correctness` label plus\n   the relevant area label (e.g. `temporal expressions`). Keep the\n   title in the form \"[Bug] \u003Cexpression> \u003Cone-line symptom>\". Include\n   the Spark version, a minimal repro, and the divergent result.\n\n3. **If the fix is trivial and you are confident, fix it inline.**\n   Then add the test in the default `query` mode so it locks in the\n   fix. \"Trivial\" means a few lines in one file with no native code\n   changes, no support-level reshuffling, and no semantics decisions\n   that the user should weigh in on.\n\n4. **Otherwise, add the test in `query ignore(\u003Cissue-url>)` mode** with\n   a one-line SQL comment above the query explaining the symptom. The\n   test file lives next to the existing tests for the expression. Do\n   not skip writing the test because the bug is unfixed: the captured\n   reproduction is the whole point of this step.\n\n   ```sql\n   -- Comet returns NULL where Spark throws under spark.sql.ansi.enabled=true\n   query ignore(https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002FNNNN)\n   SELECT next_day(date('2024-01-01'), 'NOT_A_DAY')\n   ```\n\n   When the underlying bug is fixed, the `ignore(...)` is removed and\n   the test starts running. This is also the contract documented in\n   `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`.\n\n### Support-level consistency: apply fixes automatically\n\nApply every finding from the Step 5 consistency audit in the same PR.\nThese are mechanical edits with no behavioural impact, so they do not\nneed user approval. The classes of fix are:\n\n- Extract a duplicated or drifting reason string into a `private val`\n  and reference it from both `getSupportLevel` and `get*Reasons()`.\n- Switch `Incompatible(None)` to `Incompatible(Some(reason))` so the\n  reason reaches EXPLAIN output, not only the docs.\n- Add a missing `get*Reasons()` override that enumerates every\n  reason the support level can return.\n- Move a compatibility check out of `convert` and into\n  `getSupportLevel` so the dispatcher handles `allowIncompatible`\n  uniformly (the `withInfo` call in `convert` should disappear).\n- Hoist a reason shared by multiple serdes (e.g. a recurring\n  TimestampNTZ caveat) into a small `private object` companion in the\n  same file, mirroring `UTCTimestampSerde`.\n- Lift expression-shape restrictions (literal-only argument, foldable\n  child, unsupported child data type) out of `convert`'s `withInfo` +\n  `return None` and into an `Unsupported(Some(reason))` branch in\n  `getSupportLevel`. The `convert` body should then assume the\n  precondition holds and stop calling `withInfo` for that case.\n- Promote a documented \"Known divergence\" or \"Known limitation\" sub-\n  bullet from a `Compatible` branch to `Incompatible(Some(reason))`\n  (or `Unsupported` if the native path errors) and link the tracking\n  issue. The sub-bullet stays as user-facing documentation. The\n  support level catches up to match.\n- Mark expressions whose Spark 4.0+ path routes through\n  `CollationSupport.X.exec` (or accepts `StringTypeWithCollation` \u002F\n  `StringTypeNonCSAICollation`) as `Incompatible(Some(reason))` for\n  non-default collations, linking\n  https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F4496.\n- Delete unreachable serde registrations (`RuntimeReplaceable` rewrites\n  the expression before serde runs, an analyzer rule strips the case,\n  etc.) rather than documenting them as a curiosity.\n\nEach fix is one of these patterns. If a finding requires a semantics\ndecision (e.g. is a specific branch really `Unsupported`, or is it\n`Incompatible`?), do not guess: **file a GitHub issue per the\n\"Findings that need follow-up\" section below** and link it from the PR\ndescription. Do not leave the recommendation as prose only: prose in a\nPR description gets buried as soon as the PR merges.\n\nAfter every fix, build the affected module to make sure the edit\ncompiles. Do not run the full suite; targeted tests suffice if the\nfix could plausibly affect behaviour.\n\n### Findings that need follow-up: always file a tracking issue\n\nAny high-priority finding (correctness divergence, robustness gap,\nbehavioural difference from Spark, missing-collation guard, etc.) that\nthis PR does not fix inline **must** be filed as a GitHub issue before\nthe PR is opened. This includes:\n\n- Semantics decisions the audit surfaces but should not unilaterally\n  resolve (e.g. promote `Compatible` to `Incompatible`, change a\n  default).\n- Architectural concerns that span multiple expressions (e.g. Spark 4.0\n  collation propagation across an entire family).\n- Bugs that are fixable in principle but need more design or native\n  changes than fit in the audit PR.\n- Documentation gaps surfaced by the audit (e.g. an expression that\n  doesn't appear in the auto-generated compatibility doc).\n\nFor each follow-up:\n\n1. Search for an existing issue first\n   (`gh issue list --search \"\u003Cexpression> \u003Csymptom> in:title,body\" --state all --limit 5`).\n   If a candidate match comes back, **open it\n   (`gh issue view \u003CN> --repo apache\u002Fdatafusion-comet`) and confirm the\n   title and body actually describe the divergence you found, and that\n   the issue is still `OPEN`**. A closed-but-fixed issue cited as\n   \"known divergence\" is worse than no citation, because the reader\n   follows the link and finds a fix that was already shipped. If it\n   matches, link it from the PR description and the support-doc\n   sub-bullet, and stop.\n2. If no issue exists, file one with `gh issue create` using the\n   `correctness` label (or `documentation` for doc-only gaps) plus any\n   relevant area labels (e.g. `spark 4.0`). Title format:\n   `[Bug] \u003Cexpression> \u003Cone-line symptom>`. Body includes: Spark version\n   range affected, a minimal repro, the divergent result, the relevant\n   Comet file\u002Fline, and a one-line note that the issue was surfaced by\n   this audit PR.\n3. Reference the issue number from both the support-doc sub-bullet and\n   the PR description \"What changes are included\" section so reviewers\n   can see what work the audit intentionally deferred.\n\nProse-only \"future work\" notes in the PR description are not enough.\nThe whole point of the audit is to leave behind durable artefacts; an\nunfiled finding evaporates after the PR merges.\n\n### Low-risk missing test coverage: ask the user\n\nThis is the only Step 7 category that pauses for user input. It only\ncovers the Medium-priority bucket from Step 6: low-risk gaps on\nwell-exercised code paths. High-risk untested cases belong above, in\n\"High-priority findings: capture as tests\". Adding tests for cases\nthat already work on well-exercised paths is incremental polish, not\npart of the audit's required output.\n\n> Spark exercises the following cases that have no Comet test. Would\n> you like me to add them?\n>\n> - [list each missing test case]\n>\n> I can add them as Comet SQL Tests in `spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F\u003Ccategory>\u002F$ARGUMENTS.sql`\n> (or as Comet Scala Tests in `CometExpressionSuite` for cases that require programmatic setup).\n\nIf the user says yes, implement the tests following the format described\nin `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`. Prefer Comet SQL\nTests over Comet Scala Tests.\n\n### Comet SQL Tests template\n\n```sql\n-- Licensed to the Apache Software Foundation (ASF) under one\n-- or more contributor license agreements.  See the NOTICE file\n-- distributed with this work for additional information\n-- regarding copyright ownership.  The ASF licenses this file\n-- to you under the Apache License, Version 2.0 (the\n-- \"License\"); you may not use this file except in compliance\n-- with the License.  You may obtain a copy of the License at\n--\n--   http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n--\n-- Unless required by applicable law or agreed to in writing,\n-- software distributed under the License is distributed on an\n-- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n-- KIND, either express or implied.  See the License for the\n-- specific language governing permissions and limitations\n-- under the License.\n\n-- ConfigMatrix: parquet.enable.dictionary=false,true\n\nstatement\nCREATE TABLE test_$ARGUMENTS(...) USING parquet\n\nstatement\nINSERT INTO test_$ARGUMENTS VALUES\n  (...),\n  (NULL)\n\n-- column argument\nquery\nSELECT $ARGUMENTS(col) FROM test_$ARGUMENTS\n\n-- literal arguments\nquery\nSELECT $ARGUMENTS('value'), $ARGUMENTS(''), $ARGUMENTS(NULL)\n```\n\n### Verify the tests pass\n\nAfter implementing tests, tell the user how to run them:\n\n```bash\n.\u002Fmvnw test -DwildcardSuites=\"CometSqlFileTestSuite\" -Dsuites=\"org.apache.comet.CometSqlFileTestSuite $ARGUMENTS\" -Dtest=none\n```\n\n---\n\n## Step 8: Record the Audit\n\nAfter completing the audit (whether or not tests were added), record the findings on the\nexpression's category page under `docs\u002Fsource\u002Fcontributor-guide\u002Fexpression-audits\u002F`.\nThe page is named after the Spark function-registry category (e.g. `agg_funcs.md`,\n`string_funcs.md`, `datetime_funcs.md`). If you are unsure of the category, match the one\nused for the expression in `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md`.\n\n- If the category page does not exist yet, create it with the ASF license header, a\n  `# \u003Ccategory> Expression Audits` title, and the standard intro blockquote used by the\n  other pages.\n- Add (or update) a `## \u003Cfunction_name>` section, keeping sections alphabetically ordered.\n- Under that heading, add one bullet per Spark version checked, each including:\n  - Spark version (e.g. 3.4.3, 3.5.8, 4.0.1, 4.1.1)\n  - Today's date\n  - A brief note for any version-specific finding (behavioral difference, known\n    incompatibility); omit the note if nothing notable.\n\n---\n\n## Output Format\n\nPresent the audit as:\n\n1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input\u002Foutput types, and null behavior\n2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1\n3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns\n4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps\n5. **Recommendations** - Prioritized list from Step 6\n6. **Offer to add tests** - The prompt from Step 7\n\n## Tone and Style\n\n- Write in clear, concise prose\n- Use backticks around code references (function names, file paths, class names, types, config keys)\n- Avoid robotic or formulaic language\n- Be constructive and acknowledge what is already well-covered before raising gaps\n- Avoid em dashes and semicolons; use separate sentences instead\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nAudit the Comet implementation of the `$ARGUMENTS` expression for correctness and test coverage.\n\n## Overview\n\nThis audit covers:\n\n1. Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1\n2. Comet Scala serde implementation\n3. Comet Rust \u002F DataFusion implementation\n4. Existing test coverage (Comet SQL Tests and Comet Scala Tests)\n5. Gap analysis and test recommendations\n\n---\n\n## Step 1: Locate the Spark Implementations\n\nClone specific Spark version tags (use shallow clones to avoid polluting the workspace). Only clone a version if it is not already present.\n\n```bash\nset -eu -o pipefail\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  if [ ! -d \"$dir\" ]; then\n    git clone --depth 1 --branch \"$tag\" https:\u002F\u002Fgithub.com\u002Fapache\u002Fspark.git \"$dir\"\n  fi\ndone\n```\n\n### Find the expression class in each Spark version\n\nSearch the Catalyst SQL expressions source:\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\u002Fcatalyst\u002Fsrc\u002Fmain\u002Fscala\" -name \"*.scala\" | \\\n    xargs grep -l \"case class $ARGUMENTS\\b\\|object $ARGUMENTS\\b\" 2>\u002Fdev\u002Fnull\ndone\n```\n\nIf the expression is not found in catalyst, also check core:\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\" -name \"*.scala\" | \\\n    xargs grep -l \"case class $ARGUMENTS\\b\\|object $ARGUMENTS\\b\" 2>\u002Fdev\u002Fnull\ndone\n```\n\n### Read the Spark source for each version\n\nFor each Spark version, read the expression file and note:\n\n- The `eval`, `nullSafeEval`, and `doGenCode` \u002F `doGenCodeSafe` methods\n- The `inputTypes` and `dataType` fields (accepted input types, return type)\n- Null handling strategy (`nullable`, `nullSafeEval`)\n- ANSI mode behavior (`ansiEnabled`, `failOnError`)\n- Special cases, guards, `require` assertions, and runtime exceptions\n- Any constants or configuration the expression reads\n\n### Compare across Spark versions\n\nProduce a concise diff summary of what changed between:\n\n- 3.4.3 → 3.5.8\n- 3.5.8 → 4.0.1\n- 4.0.1 → 4.1.1\n\nPay attention to:\n\n- New input types added or removed\n- Behavior changes for edge cases (null, overflow, empty, boundary)\n- New ANSI mode branches\n- New parameters or configuration\n- Breaking API changes that Comet must shim\n\n---\n\n## Step 2: Locate the Spark Tests\n\n```bash\nfor tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do\n  dir=\"\u002Ftmp\u002Fspark-${tag}\"\n  echo \"=== $tag ===\"\n  find \"$dir\u002Fsql\" -name \"*.scala\" -path \"*\u002Ftest\u002F*\" | \\\n    xargs grep -l \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\ndone\n```\n\nRead the relevant Spark test files and produce a list of:\n\n- Input types covered\n- Edge cases exercised (null, empty, overflow, negative, boundary values, special characters, etc.)\n- ANSI mode tests\n- Error cases\n\nThis list will be the reference for the coverage gap analysis in Step 5.\n\n---\n\n## Step 3: Locate the Comet Implementation\n\n### Scala serde\n\n```bash\n# Find the serde object\ngrep -r \"$ARGUMENTS\" spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002F --include=\"*.scala\" -l\ngrep -r \"$ARGUMENTS\" spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002F --include=\"*.scala\" -l\n```\n\nRead the serde implementation and check:\n\n- Which Spark versions the serde handles\n- Whether all input types Spark accepts are handled\n- Whether `convert` validates expression shape (e.g. literal-only arguments) before serializing\n\nThen audit the support-level and reason methods as described below.\n\n#### Audit `getSupportLevel`, `getIncompatibleReasons`, `getUnsupportedReasons`, and `convert`\n\nThese four methods must stay aligned. Each has a distinct purpose, and the\nmost common bugs in Comet serdes are misalignments between them.\n\n**Pick the right support level.**\n\n- `Unsupported(Some(reason))`: Comet cannot run this case at all. The\n  dispatcher in `QueryPlanSerde.exprToProtoInternal` falls back to Spark\n  unconditionally. Use this when an input type, option, or expression shape\n  is not implemented, or when running the native path would crash or error.\n- `Incompatible(Some(reason))`: Comet can run this, but results may differ\n  from Spark. The dispatcher only allows it when\n  `spark.comet.expression.\u003CexprName>.allowIncompatible=true` is set. Use\n  this for known result differences such as locale sensitivity, timezone\n  handling, ordering ambiguity, or floating-point precision.\n- `Compatible(None)`: full Spark compatibility for this combination of\n  inputs and options.\n- `Compatible(Some(note))`: fully compatible but with a docs-only caveat.\n  Note that any `Some(...)` on `Compatible` triggers a runtime\n  `logWarning`, so reserve it for genuinely useful caveats.\n\nDecision rule: if the user would be surprised by a _wrong answer_, it is\n`Incompatible`. If the user would be surprised by a _runtime error or\nunsupported-type message_, it is `Unsupported`.\n\n**Runtime vs docs split.**\n\n- The `notes` field on `Incompatible(Some(...))` and `Unsupported(Some(...))`\n  flows into EXPLAIN output via the dispatcher (see\n  `QueryPlanSerde.exprToProtoInternal`, around the `case Incompatible` \u002F\n  `case Unsupported` branches). This is what users see when they ask why\n  Comet fell back.\n- `getIncompatibleReasons()` and `getUnsupportedReasons()` are read only by\n  `GenerateDocs` when building\n  `docs\u002Fsource\u002Fuser-guide\u002Fcompatibility.md`. They are static (no `expr`\n  argument) and should enumerate every distinct reason the expression\n  could ever return at runtime.\n- `convert` may also call `withInfo(expr, \"reason\")` and return `None` for\n  cases that cannot be detected from the expression alone (for example,\n  non-literal arguments, or child conversion failures). Those reasons\n  belong in `getUnsupportedReasons()` too.\n\n**Consistency rules.**\n\n1. If `getSupportLevel` can return `Incompatible(...)` for any input,\n   override `getIncompatibleReasons()` and include a reason for every\n   incompatible branch.\n2. If `getSupportLevel` can return `Unsupported(...)` for any input,\n   override `getUnsupportedReasons()` and include a reason for every\n   unsupported branch.\n3. If `convert` has its own `withInfo(...); None` fallbacks (e.g. literal\n   checks), enumerate those reasons in `getUnsupportedReasons()` too.\n4. Extract each reason into a `private val` and reference it from both\n   `getSupportLevel` and `get*Reasons()`. Do not duplicate the string\n   inline. Canonical example:\n   `spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fserde\u002Farrays.scala::CometArrayIntersect`.\n   It declares `private val incompatReason` and\n   `private val unsupportedCollationReason`, and each is referenced from\n   `getSupportLevel` and the matching reasons method.\n5. Prefer `Incompatible(Some(reason))` over `Incompatible(None)`. The\n   `None` form drops the reason from EXPLAIN output, leaving users with a\n   generic \"not fully compatible\" message and forcing them to read the\n   docs to find out why.\n6. Gate compatibility decisions in `getSupportLevel`, not inside `convert`.\n   Putting the check inside `convert` (e.g. reading a config flag and\n   calling `withInfo`) bypasses the dispatcher's `allowIncompatible`\n   handling and the EXPLAIN message becomes inconsistent with what the\n   doc generator produces.\n\n**Wording guidelines for reason strings.**\n\nReasons appear verbatim in the Compatibility Guide (rendered as Markdown)\nand in EXPLAIN output, so they are user-facing.\n\n- Lead with the user-observable effect, then the cause if helpful.\n  ✅ \"Result array element order may differ from Spark when the right array\n  is longer than the left.\"\n  ❌ \"DataFusion probes the longer side.\"\n- Use sentence case and end with a period.\n- Use backticks around config keys, type names, and SQL identifiers.\n- Link to a tracking GitHub issue for known incompatibilities so users can\n  follow progress: `(https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002FNNNN)`.\n  **Verify the issue exists and is open** before citing it\n  (`gh issue view \u003CN> --repo apache\u002Fdatafusion-comet`). Issue numbers\n  invented from context or recalled from memory are a recurring failure\n  mode: a stale link is worse than no link because the reader follows it\n  and finds nothing.\n- Keep it concise. Single sentence is best.\n- Do not write \"Incompatible reason: ...\" or \"Unsupported because ...\".\n  The doc generator adds the framing.\n- Phrase Incompatible reasons as _what differs from Spark_, not _what is\n  missing_. Phrase Unsupported reasons as _what does not run_. If you find\n  yourself writing an \"Incompatible\" reason that says \"Comet only supports\n  X\" or \"Y is not supported\", the support level is probably wrong: it\n  should be `Unsupported`.\n\n**Common antipatterns to flag during the audit.**\n\n- A `private val reason` constant declared near the top of the object, but\n  `getSupportLevel` hardcodes a different string inline. The doc and the\n  EXPLAIN message will drift.\n  Real example: `CometHour` declares `incompatReason` then hardcodes a\n  near-duplicate string in `getSupportLevel`.\n- Reason text duplicated in two places without a shared constant.\n  Real examples: `CometMinute`, `CometSecond`.\n- `Incompatible(None)` paired with a populated `getIncompatibleReasons()`.\n  The reason reaches the docs but not EXPLAIN output.\n  Real example: `CometInitCap`.\n- `getIncompatibleReasons()` overridden but `getSupportLevel` never returns\n  `Incompatible(...)`. Either the reasons method is dead code, or\n  `getSupportLevel` is missing a branch.\n- A reason string phrased as \"X is not supported\" attached to an\n  `Incompatible` branch (or vice versa). Re-read it and decide which\n  support level it really belongs to.\n- `convert` bails out via `withInfo` for a case that is fully knowable\n  from the expression (e.g. an unsupported child data type). Move the\n  check into `getSupportLevel` so the dispatcher handles it uniformly.\n\n### Shims\n\n```bash\nfind spark\u002Fsrc\u002Fmain -name \"CometExprShim.scala\" | xargs grep -l \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\n```\n\nIf shims exist, read them and note any version-specific handling.\n\n### Rust \u002F DataFusion implementation\n\n```bash\n# Search for the function in native\u002Fspark-expr\ngrep -r \"$ARGUMENTS\" native\u002Fspark-expr\u002Fsrc\u002F --include=\"*.rs\" -l\ngrep -r \"$ARGUMENTS\" native\u002Fcore\u002Fsrc\u002F --include=\"*.rs\" -l\n```\n\nIf the expression delegates to DataFusion, find it there too. Set `$DATAFUSION_SRC` to a local DataFusion checkout, or fall back to searching the cargo registry:\n\n```bash\nif [ -n \"${DATAFUSION_SRC:-}\" ]; then\n  grep -r \"$ARGUMENTS\" \"$DATAFUSION_SRC\" --include=\"*.rs\" -l 2>\u002Fdev\u002Fnull | head -10\nelse\n  # Fall back to cargo registry (may include unrelated crates)\n  grep -r \"$ARGUMENTS\" ~\u002F.cargo\u002Fregistry\u002Fsrc\u002F*\u002Fdatafusion* --include=\"*.rs\" -l 2>\u002Fdev\u002Fnull | head -10\nfi\n```\n\nRead the Rust implementation and check:\n\n- Null handling (does it propagate nulls correctly?)\n- Overflow and underflow handling (returns `Err` vs panics)\n- Type dispatch (does it handle all types that Spark supports?)\n- ANSI \u002F fail-on-error mode\n\n---\n\n## Step 4: Locate Existing Comet Tests\n\n### Comet SQL Tests\n\n```bash\n# Find SQL test files for this expression\nfind spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F -name \"*.sql\" | \\\n  xargs grep -l \"$ARGUMENTS\" 2>\u002Fdev\u002Fnull\n\n# Also check if there's a dedicated file\nfind spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F -name \"*$(echo $ARGUMENTS | tr '[:upper:]' '[:lower:]')*\"\n```\n\nRead every SQL test file found and list:\n\n- Table schemas and data values used\n- Queries exercised\n- Query modes used (`query`, `spark_answer_only`, `tolerance`, `ignore`, `expect_error`)\n- Any ConfigMatrix directives\n\n### Comet Scala Tests\n\n```bash\ngrep -r \"$ARGUMENTS\" spark\u002Fsrc\u002Ftest\u002Fscala\u002F --include=\"*.scala\" -l\n```\n\nRead the relevant Comet Scala Tests and list:\n\n- Input types covered\n- Edge cases exercised\n- Whether constant folding is disabled for literal tests\n\n---\n\n## Step 5: Gap Analysis\n\nCompare the Spark test coverage (Step 2) against the Comet test coverage (Step 4). Produce a structured gap report:\n\n### Coverage matrix\n\nFor each of the following dimensions, note whether it is covered in Comet tests or missing:\n\n| Dimension                                                                                              | Spark tests it | Comet SQL Test | Comet Scala Test | Gap? |\n| ------------------------------------------------------------------------------------------------------ | -------------- | -------------- | ---------------- | ---- |\n| Column reference argument(s)                                                                           |                |                |                  |      |\n| Literal argument(s)                                                                                    |                |                |                  |      |\n| NULL input                                                                                             |                |                |                  |      |\n| Empty string \u002F empty array \u002F empty map                                                                 |                |                |                  |      |\n| Array\u002Fmap with NULL elements                                                                           |                |                |                  |      |\n| Zero, negative zero, negative values (numeric)                                                         |                |                |                  |      |\n| Underflow, overflow                                                                                    |                |                |                  |      |\n| Boundary values (INT_MIN, INT_MAX, Long.MinValue, minimum positive, etc.)                              |                |                |                  |      |\n| NaN, Infinity, -Infinity, subnormal (float\u002Fdouble)                                                     |                |                |                  |      |\n| Multibyte \u002F special UTF-8 (composed vs decomposed, e.g. `é` U+00E9 vs `e` + U+0301, non-Latin scripts) |                |                |                  |      |\n| ANSI mode (failOnError=true)                                                                           |                |                |                  |      |\n| Non-ANSI mode (failOnError=false)                                                                      |                |                |                  |      |\n| All supported input types                                                                              |                |                |                  |      |\n| Parquet dictionary encoding (ConfigMatrix)                                                             |                |                |                  |      |\n| Cross-version behavior differences                                                                     |                |                |                  |      |\n\n### Implementation gaps\n\nAlso review the Comet implementation (Step 3) against the Spark behavior (Step 1):\n\n- Are there input types that Spark supports but `getSupportLevel` returns `Unsupported` without comment?\n- Are there behavioral differences that are NOT marked `Incompatible` but should be?\n- Are there behavioral differences between Spark versions that the Comet implementation does not account for (missing shim)?\n- Does the Rust implementation match the Spark behavior for all edge cases?\n\n### Support-level consistency audit\n\nWalk through this checklist against the serde. Each failed item is a\nfinding for Step 6.\n\n1. **Support level matches behavior.** For each branch of\n   `getSupportLevel`, decide whether the user-observable effect is a wrong\n   answer (`Incompatible`) or a fallback \u002F error (`Unsupported`). Flag any\n   branch where the label does not match the behavior.\n2. **Reasons cover every branch.** Every distinct reason that\n   `getSupportLevel` can return as `Incompatible(Some(r))` must appear in\n   `getIncompatibleReasons()`. Same for `Unsupported(Some(r))` and\n   `getUnsupportedReasons()`. Missing reasons silently drop from the\n   Compatibility Guide.\n3. **Reasons are not dead code.** If `getIncompatibleReasons()` is\n   overridden but `getSupportLevel` never returns `Incompatible(...)`,\n   either the reason is stale or `getSupportLevel` is missing a branch.\n   Same for `getUnsupportedReasons()`.\n4. **Reason strings are shared via `private val`.** If the same reason\n   appears as a string literal in two places, flag it: changes to one\n   will not propagate to the other.\n5. **Inline reason matches the constant.** If a `private val reason` is\n   declared but `getSupportLevel` uses a different string literal, flag\n   it as a drift bug.\n6. **`Incompatible(None)` has no docs-only reason.** If\n   `getSupportLevel` returns `Incompatible(None)` but\n   `getIncompatibleReasons()` is non-empty, flag it: the EXPLAIN message\n   will be generic while the docs show a specific reason. Switch to\n   `Incompatible(Some(reason))`.\n7. **`convert` fallbacks are documented.** If `convert` calls\n   `withInfo(expr, \"...\")` and returns `None` for cases not covered by\n   `getSupportLevel` (e.g. non-literal arguments, unsupported expression\n   shapes), confirm the reason is also listed in\n   `getUnsupportedReasons()`.\n8. **Compatibility decisions live in `getSupportLevel`.** If `convert`\n   reads a config flag and bails out, prefer moving the check into\n   `getSupportLevel` so the dispatcher handles `allowIncompatible`\n   uniformly. `CometCaseConversionBase` is an example of the in-`convert`\n   pattern that this skill recommends against.\n9. **Reason wording.** Each reason should describe the user-observable\n   effect, use sentence case with a period, backtick config keys and\n   types, and link a tracking issue when one exists. Flag reasons that\n   read like internal implementation notes (\"DataFusion probes the longer\n   side\") or that mismatch their support level (an \"Incompatible\" reason\n   that says \"X is not supported\").\n10. **Expression-shape restrictions live in `getSupportLevel`.** Any\n    restriction that is knowable from the expression alone (literal-only\n    arguments, unsupported child data type, foldable-only options, a\n    specific operator shape) must be declared as an\n    `Unsupported(Some(reason))` branch in `getSupportLevel`, not gated\n    inside `convert` with a `withInfo` + `return None`. Putting the\n    check in `convert` means EXPLAIN surfaces the reason only at\n    conversion time, the doc generator never sees it, and the\n    dispatcher cannot route around it. The literal-only `len`\n    restrictions on `CometLeft`, `CometRight`, and `CometSubstring`\n    are the canonical example of the in-`convert` pattern that this\n    skill forbids: lift them into `getSupportLevel`.\n11. **Spark 4.0 collation divergences are flagged, not glossed over.**\n    If the Spark 4.0\u002F4.1 implementation routes through\n    `CollationSupport.X.exec(..., collationId)` (or uses\n    `StringTypeWithCollation` \u002F `StringTypeNonCSAICollation` for input\n    types) and the Comet path does not propagate collation, the\n    expression is `Incompatible` for non-default collations. Mark the\n    branch `Incompatible(Some(reason))` linking to the collation\n    umbrella issue\n    (https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F4496) so the\n    follow-up sweep can find every site. \"Behaviour unchanged for\n    `UTF8_BINARY`\" alone is not a justification for leaving the\n    support level at `Compatible`: users running with non-default\n    collations get silently wrong answers.\n12. **Known divergences flip the support level.** If you find yourself\n    writing the words \"Known divergence\" or \"Known limitation\" in the\n    support-doc sub-bullet while leaving `getSupportLevel` returning\n    `Compatible`, the audit has skipped its job. A documented\n    divergence is by definition not `Compatible`. Promote the branch\n    to `Incompatible(Some(reason))` (or `Unsupported` if the native\n    path errors rather than producing a wrong answer) and link the\n    tracking issue. The `replace` empty-search-string divergence with\n    DataFusion is the canonical example of this anti-pattern.\n13. **Unreachable serde mappings are removed.** Expressions registered\n    as `RuntimeReplaceable` (or otherwise rewritten by an analyzer\n    rule before serde) never reach `QueryPlanSerde.exprToProtoInternal`\n    with their original class. If the audit finds that a registered\n    `CometScalarFunction(\"name\")` or `CometExpressionSerde` entry can\n    never be hit (e.g. the `btrim` mapping for `StringTrimBoth`, which\n    is rewritten to `StringTrim` before serde runs), delete the\n    registration in the same audit PR. Documenting the dead code in\n    the support doc is not enough.\n\n---\n\n## Step 6: Recommendations\n\nSummarize findings as a prioritized list.\n\n### High priority: correctness divergences and high-risk coverage gaps\n\nCases where Comet produces a different observable result from Spark\n(wrong value, missing exception, accepted-instead-of-rejected input,\netc.), **and** untested cases on a known-fragile axis where a\ndivergence is plausible: overflow \u002F out-of-range inputs, NULL handling,\ntimezones and DST transitions, ANSI mode, leap years, version-specific\nSpark semantics, and any edge case explicitly called out in Step 1.\n\nTreat an untested high-risk case as a likely divergence until proven\notherwise. Each item in this bucket becomes a captured test in Step 7.\nFor an untested case, run it manually first to determine the current\nbehaviour, then commit either a regression test (passes) or a\n`query ignore(\u003Cissue-url>)` test (fails).\n\nEvery item in this bucket either becomes an inline fix + test, or a\nfiled GitHub issue + ignored regression test, in the audit PR. Step 7\nspells out the workflow: never leave a high-priority finding as PR-body\nprose only.\n\n### Medium priority: missing test coverage\n\nLow-risk coverage gaps: additional input permutations on already-tested\ncode paths, redundant happy-path values, or cases where Comet and Spark\nshare a well-exercised implementation. The behaviour appears to match\nand the axis is not on the high-risk list above.\n\n### Low priority: cosmetic and consistency issues\n\nReason-string drift, missing `get*Reasons()` overrides, dead branches,\netc. These come from the Step 5 consistency audit.\n\n---\n\n## Step 7: Apply Findings (Tests and Fixes), Then Offer the Rest\n\nHigh-priority findings (correctness divergences and high-risk coverage\ngaps) and consistency issues from Step 5 \u002F Step 6 must not be left as\nprose. Apply them in the same PR as the audit. Anything you cannot fix\ninline (because it needs a semantics decision, native code change, or\nlarger design work) must still be captured as a GitHub issue per the\n\"Findings that need follow-up\" section below: prose recommendations in\nthe PR body alone are insufficient. Only low-risk missing coverage\nrequires the user's go-ahead, because adding tests for cases that\nalready work on well-exercised paths is incremental polish.\n\n### High-priority findings: capture as tests\n\nEvery high-priority finding becomes a regression test in the same PR as\nthe audit, so future readers can run it. This covers both confirmed\ndivergences and high-risk untested cases. For the latter, run the case\nmanually first to determine whether it currently passes or fails, then\nfollow the same workflow below: a passing case is committed as a plain\n`query` regression test; a failing case is committed as\n`query ignore(\u003Cissue-url>)` after filing the bug.\n\nFor each high-priority finding, do the following in this order:\n\n1. **Search for an existing tracking issue.**\n\n   ```bash\n   gh issue list --search \"\u003Cexpression> \u003Csymptom> in:title,body\" --state all --limit 5\n   ```\n\n   Match on both the expression name and a distinguishing keyword\n   (ANSI, timezone, NTZ, overflow, etc.).\n\n2. **If no issue exists, file one.** Use the `correctness` label plus\n   the relevant area label (e.g. `temporal expressions`). Keep the\n   title in the form \"[Bug] \u003Cexpression> \u003Cone-line symptom>\". Include\n   the Spark version, a minimal repro, and the divergent result.\n\n3. **If the fix is trivial and you are confident, fix it inline.**\n   Then add the test in the default `query` mode so it locks in the\n   fix. \"Trivial\" means a few lines in one file with no native code\n   changes, no support-level reshuffling, and no semantics decisions\n   that the user should weigh in on.\n\n4. **Otherwise, add the test in `query ignore(\u003Cissue-url>)` mode** with\n   a one-line SQL comment above the query explaining the symptom. The\n   test file lives next to the existing tests for the expression. Do\n   not skip writing the test because the bug is unfixed: the captured\n   reproduction is the whole point of this step.\n\n   ```sql\n   -- Comet returns NULL where Spark throws under spark.sql.ansi.enabled=true\n   query ignore(https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002FNNNN)\n   SELECT next_day(date('2024-01-01'), 'NOT_A_DAY')\n   ```\n\n   When the underlying bug is fixed, the `ignore(...)` is removed and\n   the test starts running. This is also the contract documented in\n   `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`.\n\n### Support-level consistency: apply fixes automatically\n\nApply every finding from the Step 5 consistency audit in the same PR.\nThese are mechanical edits with no behavioural impact, so they do not\nneed user approval. The classes of fix are:\n\n- Extract a duplicated or drifting reason string into a `private val`\n  and reference it from both `getSupportLevel` and `get*Reasons()`.\n- Switch `Incompatible(None)` to `Incompatible(Some(reason))` so the\n  reason reaches EXPLAIN output, not only the docs.\n- Add a missing `get*Reasons()` override that enumerates every\n  reason the support level can return.\n- Move a compatibility check out of `convert` and into\n  `getSupportLevel` so the dispatcher handles `allowIncompatible`\n  uniformly (the `withInfo` call in `convert` should disappear).\n- Hoist a reason shared by multiple serdes (e.g. a recurring\n  TimestampNTZ caveat) into a small `private object` companion in the\n  same file, mirroring `UTCTimestampSerde`.\n- Lift expression-shape restrictions (literal-only argument, foldable\n  child, unsupported child data type) out of `convert`'s `withInfo` +\n  `return None` and into an `Unsupported(Some(reason))` branch in\n  `getSupportLevel`. The `convert` body should then assume the\n  precondition holds and stop calling `withInfo` for that case.\n- Promote a documented \"Known divergence\" or \"Known limitation\" sub-\n  bullet from a `Compatible` branch to `Incompatible(Some(reason))`\n  (or `Unsupported` if the native path errors) and link the tracking\n  issue. The sub-bullet stays as user-facing documentation. The\n  support level catches up to match.\n- Mark expressions whose Spark 4.0+ path routes through\n  `CollationSupport.X.exec` (or accepts `StringTypeWithCollation` \u002F\n  `StringTypeNonCSAICollation`) as `Incompatible(Some(reason))` for\n  non-default collations, linking\n  https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fissues\u002F4496.\n- Delete unreachable serde registrations (`RuntimeReplaceable` rewrites\n  the expression before serde runs, an analyzer rule strips the case,\n  etc.) rather than documenting them as a curiosity.\n\nEach fix is one of these patterns. If a finding requires a semantics\ndecision (e.g. is a specific branch really `Unsupported`, or is it\n`Incompatible`?), do not guess: **file a GitHub issue per the\n\"Findings that need follow-up\" section below** and link it from the PR\ndescription. Do not leave the recommendation as prose only: prose in a\nPR description gets buried as soon as the PR merges.\n\nAfter every fix, build the affected module to make sure the edit\ncompiles. Do not run the full suite; targeted tests suffice if the\nfix could plausibly affect behaviour.\n\n### Findings that need follow-up: always file a tracking issue\n\nAny high-priority finding (correctness divergence, robustness gap,\nbehavioural difference from Spark, missing-collation guard, etc.) that\nthis PR does not fix inline **must** be filed as a GitHub issue before\nthe PR is opened. This includes:\n\n- Semantics decisions the audit surfaces but should not unilaterally\n  resolve (e.g. promote `Compatible` to `Incompatible`, change a\n  default).\n- Architectural concerns that span multiple expressions (e.g. Spark 4.0\n  collation propagation across an entire family).\n- Bugs that are fixable in principle but need more design or native\n  changes than fit in the audit PR.\n- Documentation gaps surfaced by the audit (e.g. an expression that\n  doesn't appear in the auto-generated compatibility doc).\n\nFor each follow-up:\n\n1. Search for an existing issue first\n   (`gh issue list --search \"\u003Cexpression> \u003Csymptom> in:title,body\" --state all --limit 5`).\n   If a candidate match comes back, **open it\n   (`gh issue view \u003CN> --repo apache\u002Fdatafusion-comet`) and confirm the\n   title and body actually describe the divergence you found, and that\n   the issue is still `OPEN`**. A closed-but-fixed issue cited as\n   \"known divergence\" is worse than no citation, because the reader\n   follows the link and finds a fix that was already shipped. If it\n   matches, link it from the PR description and the support-doc\n   sub-bullet, and stop.\n2. If no issue exists, file one with `gh issue create` using the\n   `correctness` label (or `documentation` for doc-only gaps) plus any\n   relevant area labels (e.g. `spark 4.0`). Title format:\n   `[Bug] \u003Cexpression> \u003Cone-line symptom>`. Body includes: Spark version\n   range affected, a minimal repro, the divergent result, the relevant\n   Comet file\u002Fline, and a one-line note that the issue was surfaced by\n   this audit PR.\n3. Reference the issue number from both the support-doc sub-bullet and\n   the PR description \"What changes are included\" section so reviewers\n   can see what work the audit intentionally deferred.\n\nProse-only \"future work\" notes in the PR description are not enough.\nThe whole point of the audit is to leave behind durable artefacts; an\nunfiled finding evaporates after the PR merges.\n\n### Low-risk missing test coverage: ask the user\n\nThis is the only Step 7 category that pauses for user input. It only\ncovers the Medium-priority bucket from Step 6: low-risk gaps on\nwell-exercised code paths. High-risk untested cases belong above, in\n\"High-priority findings: capture as tests\". Adding tests for cases\nthat already work on well-exercised paths is incremental polish, not\npart of the audit's required output.\n\n> Spark exercises the following cases that have no Comet test. Would\n> you like me to add them?\n>\n> - [list each missing test case]\n>\n> I can add them as Comet SQL Tests in `spark\u002Fsrc\u002Ftest\u002Fresources\u002Fsql-tests\u002Fexpressions\u002F\u003Ccategory>\u002F$ARGUMENTS.sql`\n> (or as Comet Scala Tests in `CometExpressionSuite` for cases that require programmatic setup).\n\nIf the user says yes, implement the tests following the format described\nin `docs\u002Fsource\u002Fcontributor-guide\u002Fsql-file-tests.md`. Prefer Comet SQL\nTests over Comet Scala Tests.\n\n### Comet SQL Tests template\n\n```sql\n-- Licensed to the Apache Software Foundation (ASF) under one\n-- or more contributor license agreements.  See the NOTICE file\n-- distributed with this work for additional information\n-- regarding copyright ownership.  The ASF licenses this file\n-- to you under the Apache License, Version 2.0 (the\n-- \"License\"); you may not use this file except in compliance\n-- with the License.  You may obtain a copy of the License at\n--\n--   http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n--\n-- Unless required by applicable law or agreed to in writing,\n-- software distributed under the License is distributed on an\n-- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n-- KIND, either express or implied.  See the License for the\n-- specific language governing permissions and limitations\n-- under the License.\n\n-- ConfigMatrix: parquet.enable.dictionary=false,true\n\nstatement\nCREATE TABLE test_$ARGUMENTS(...) USING parquet\n\nstatement\nINSERT INTO test_$ARGUMENTS VALUES\n  (...),\n  (NULL)\n\n-- column argument\nquery\nSELECT $ARGUMENTS(col) FROM test_$ARGUMENTS\n\n-- literal arguments\nquery\nSELECT $ARGUMENTS('value'), $ARGUMENTS(''), $ARGUMENTS(NULL)\n```\n\n### Verify the tests pass\n\nAfter implementing tests, tell the user how to run them:\n\n```bash\n.\u002Fmvnw test -DwildcardSuites=\"CometSqlFileTestSuite\" -Dsuites=\"org.apache.comet.CometSqlFileTestSuite $ARGUMENTS\" -Dtest=none\n```\n\n---\n\n## Step 8: Record the Audit\n\nAfter completing the audit (whether or not tests were added), record the findings on the\nexpression's category page under `docs\u002Fsource\u002Fcontributor-guide\u002Fexpression-audits\u002F`.\nThe page is named after the Spark function-registry category (e.g. `agg_funcs.md`,\n`string_funcs.md`, `datetime_funcs.md`). If you are unsure of the category, match the one\nused for the expression in `docs\u002Fsource\u002Fuser-guide\u002Flatest\u002Fexpressions.md`.\n\n- If the category page does not exist yet, create it with the ASF license header, a\n  `# \u003Ccategory> Expression Audits` title, and the standard intro blockquote used by the\n  other pages.\n- Add (or update) a `## \u003Cfunction_name>` section, keeping sections alphabetically ordered.\n- Under that heading, add one bullet per Spark version checked, each including:\n  - Spark version (e.g. 3.4.3, 3.5.8, 4.0.1, 4.1.1)\n  - Today's date\n  - A brief note for any version-specific finding (behavioral difference, known\n    incompatibility); omit the note if nothing notable.\n\n---\n\n## Output Format\n\nPresent the audit as:\n\n1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input\u002Foutput types, and null behavior\n2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1\n3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns\n4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps\n5. **Recommendations** - Prioritized list from Step 6\n6. **Offer to add tests** - The prompt from Step 7\n\n## Tone and Style\n\n- Write in clear, concise prose\n- Use backticks around code references (function names, file paths, class names, types, config keys)\n- Avoid robotic or formulaic language\n- Be constructive and acknowledge what is already well-covered before raising gaps\n- Avoid em dashes and semicolons; use separate sentences instead\n",{"name":5,"description":7,"argument-hint":43},"\u003Cexpression-name>","main","7deeba21a42e4a9cd34f43e7ed6e960e4e851b26",".ai\u002Fskills\u002Faudit-comet-expression",418,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet","https:\u002F\u002Fdatafusion.apache.org\u002Fcomet","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-15 03:15:20",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add apache\u002Fdatafusion-comet --skill audit-comet-expression",{"tool":59,"command":60},"bunx","bunx skills add apache\u002Fdatafusion-comet --skill audit-comet-expression",{"tool":62,"command":63},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-comet --skill audit-comet-expression","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Faudit-comet-expression\u002FSKILL.md",true,[],[68,93,118,139,156,167],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":79,"publishedAt":38},"01M6ACJR50ZS7F1QX58JEKCGVM","wire-datafusion-function","Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",[74],{"categoryNo":75,"categoryName":76,"taxonomyType":77,"taxonomyTypeDesc":78},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[35,80,81,82,31,36,83,84,24,85,86,21,29,87,23,88,89,90,91,92,28],"wire datafusion function","sql","scala","registration","switch","upstream","custom","use","when","file","api","wiring","with",{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":38},"01M6ACJPDDPND6FHE2PC1CJ4D3","suggest-native-expression","Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fsuggest-native-expression\u002FSKILL.md",[],[100,21,101,102,103,104,31,36,105,106,88,107,108,109,110,111,20,112,113,114,115,116,117],"suggest native expression","native","dispatcher","github","rust","re","files","candidate","log","jvm","next","how","audit log","allocation","issue","throughput","path","save",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":38},"01M6ACJMHPEN7E76VPKC3P4X6C","review-comet-shuffle-pr","Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",[],[125,126,127,128,129,35,130,87,131,88,132,133,109,101,134,135,136,137,138],"review comet shuffle pr","pr","review","pull","pull request","arrow","load","ipc","number","request","format","integration","block","compression",{"skillNo":140,"name":141,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":143,"categories":144,"keywords":145,"publishedAt":38},"01M6ACJHX8BES7ZNNRNVCQ4QWE","review-comet-pr","Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-pr\u002FSKILL.md",[],[126,127,146,128,147,129,148,35,149,150,88,133,151,152,134,153,24,87,154,155],"review comet pr","iceberg","memory","workflow","documentation","reviewer","human","posting","review skills","comments",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":38},"01M6ACJFM0QS61Z1RW516HXN2M","review-comet-memory-pr","Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",[],[126,148,163,127,128,129,35,36,164,165,87,131,88,133,134,166],"review comet memory pr","bridge","reservations","configuration",{"skillNo":168,"name":169,"nameZh":6,"description":170,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":171,"categories":172,"keywords":173,"publishedAt":38},"01M6ACJCV64V0FZPMVXDE2K3DX","review-comet-iceberg-write-pr","Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",[],[126,147,174,127,175,128,129,35,104,176,177,178,87,131,179,88,133,180,181,101],"review comet iceberg write pr","write","writer","write plan","partition","pin","split","plan"]