implement-comet-expression
Use when implementing a new Spark expression in DataFusion Comet. Walks through cloning latest Spark master to study the canonical implementation, checking the upstream datafusion-spark crate before writing native code, building the Comet serde and Rust wire-up from the contributor guide, then running audit-comet-expression to drive a test-coverage iteration loop.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 1.3k
- Forks
- 418
01 前言 Front Matter
- name
- implement-comet-expression
- description
- Use when implementing a new Spark expression in DataFusion Comet. Walks through cloning latest Spark master to study the canonical implementation, checking the upstream datafusion-spark crate before writing native code, building the Comet serde and Rust wire-up from the contributor guide, then running audit-comet-expression to drive a test-coverage iteration loop.
- argument-hint
- <expression-name>
02 资源文件 1
- 其他OWNERSHIP13 B
03 SKILL.md 正文
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
Implement Comet support for the $ARGUMENTS Spark expression.
Background reading
The contributor guide is the canonical reference. Read these before writing code:
docs/source/contributor-guide/adding_a_new_expression.mdcovers the Scala serde, protobuf, Rust scalar function flow, support levels, shims, and tests.docs/source/contributor-guide/sql-file-tests.mddescribes the Comet SQL Tests format.docs/source/user-guide/latest/expressions.mdlists the support status for every expression.
Workflow
1. Study the Spark master implementation first
Always start from the latest Spark master. Shallow clone if not already present:
if [ ! -d /tmp/spark-master ]; then
git clone --depth 1 https://github.com/apache/spark.git /tmp/spark-master
fi
Find the expression class and tests:
find /tmp/spark-master/sql -name "*.scala" | \
xargs grep -l "case class $ARGUMENTS\b\|object $ARGUMENTS\b" 2>/dev/null
find /tmp/spark-master/sql -name "*.scala" -path "*/test/*" | \
xargs grep -l "$ARGUMENTS" 2>/dev/null
Read the source. Note inputTypes, dataType, eval / nullSafeEval, ANSI mode branches, and any require guards. These define the contract Comet must match.
2. Check for an upstream datafusion-spark implementation
Before writing a Comet-specific native function, check whether the expression is already available in the upstream datafusion-spark crate. It is a Spark-compatible function library maintained alongside DataFusion, so its semantics are usually a closer match to Spark than a generic datafusion-functions built-in.
grep -rn "fn name\|SparkFunctionName" ~/.cargo/registry/src/*/datafusion-spark-*/src/function/ 2>/dev/null | grep -i "$ARGUMENTS"
Functions are organized as datafusion_spark::function::<category>::<name>::Spark<Name>. Existing wire-ups can be found in native/core/src/execution/planner.rs (e.g. SparkDateAdd, SparkDateSub, SparkCollectSet).
When the upstream implementation matches Spark's semantics, prefer it: register the ScalarUDF from datafusion-spark rather than re-implementing. This keeps the maintenance burden upstream. If the upstream version is missing, incomplete, or diverges from Spark, fall through to step 3 and write the function locally.
3. Implement the initial version
Follow adding_a_new_expression.md:
- Add a
CometExpressionSerde[T]in the appropriate file underspark/src/main/scala/org/apache/comet/serde/. - Register it in the matching map in
QueryPlanSerde.scala. - If the function name collides with a DataFusion built-in that has a different signature, use
scalarFunctionExprToProtoWithReturnType(see "When to set the return type explicitly"). - For a new scalar function, add a match case in
native/spark-expr/src/comet_scalar_funcs.rs::create_comet_physical_fun. If step 2 found an upstream implementation, wire that in. Otherwise implement the function undernative/spark-expr/src/. - If Spark's behavior differs between versions, resolve the version in the serde or a shim and pass native code a parameter named for the behavior (
wrap_second_millisecond_overflow), not the version (spark_420_plus). See "Name the behavior, not the Spark version" inadding_a_new_expression.md. - Add at least one Comet SQL Test at
spark/src/test/resources/sql-tests/expressions/<category>/$ARGUMENTS.sqlexercising column references, literals, andNULL.
Build and smoke-test:
make
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite $ARGUMENTS" -Dtest=none
4. Run the audit skill
Once the initial implementation passes its smoke test, run the audit-comet-expression skill on $ARGUMENTS. It compares the implementation and tests against Spark 3.4.3, 3.5.8, and 4.0.1 and produces a prioritized list of gaps.
5. Implement audit-recommended tests and iterate
Add the missing test cases the audit recommends, then re-run the targeted suite:
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite $ARGUMENTS" -Dtest=none
Surface findings to the user and ask whether the coverage is sufficient. Continue iterating (adding tests, fixing bugs, refining getSupportLevel / getIncompatibleReasons / getUnsupportedReasons) until the user confirms they are happy.
6. Final checks
Before opening a PR:
make format
cd native && cargo clippy --all-targets --workspace -- -D warnings
7. Open the PR
Use the repo's PR template at .github/pull_request_template.md and fill in every section: "Which issue does this PR close?", "Rationale for this change", "What changes are included in this PR?", and "How are these changes tested?". Do not add a separate test plan section.
In the "What changes are included in this PR?" section, add a brief note that the implement-comet-expression skill was used to scaffold the implementation, so reviewers know which workflow produced the change.
04 同仓库更多 Skills
wire-datafusion-function
Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging / restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.
suggest-native-expression
Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.
review-comet-shuffle-pr
Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.
review-comet-pr
Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.
review-comet-memory-pr
Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.
review-comet-iceberg-write-pr
Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.