review-comet-memory-pr
Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 1.3k
- Forks
- 418
01 前言 Front Matter
- name
- review-comet-memory-pr
- description
- Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.
- argument-hint
- <pr-number>
02 资源文件 1
- 其他OWNERSHIP37 B
03 SKILL.md 正文
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
Memory-specific review for Comet PR #$ARGUMENTS.
REQUIRED BACKGROUND: Use review-comet-pr for PR metadata, existing comments, CI, the review
bar, and the output format. This skill only covers memory.
Read the Contributor Guide First
| Doc | What you need from it |
|---|---|
docs/source/contributor-guide/memory_management.md |
The three budgets, the allocator inventory, the pool stack, the accounting gap |
docs/source/contributor-guide/ffi.md |
Where buffers physically live when a batch crosses the boundary |
docs/source/contributor-guide/debugging.md |
"Debugging Memory Reservations", for what evidence to ask the author for |
docs/source/contributor-guide/tracing.md |
jemalloc_allocated against summed reservations, the only measurement of the gap |
Read memory_management.md in full before the diff. Most bad memory PRs are not wrong about code,
they are wrong about which of the three budgets they are affecting.
1. Name the Budget
There are three budgets, enforced by three different parties, and a PR that improves one can make another worse:
| Budget | Enforced by | Failure mode |
|---|---|---|
| JVM heap | The JVM | OutOfMemoryError, executor exits with code 52 |
| Spark memory pools | Spark bookkeeping | A consumer spills, or the task fails with SparkOutOfMemoryError |
| Container RSS | OS or cgroup | SIGKILL of the whole executor, exit 137, OOMKilled |
The first two are accounting, a running total of voluntarily declared bytes. The third is physical.
Ask which one the PR claims to fix and check that the mechanism actually reaches it. Lowering a
declared reservation does not reduce resident pages. Reducing resident pages does not by itself
prevent a SparkOutOfMemoryError.
Comet's declared reservations are a lower bound on real usage, for structural reasons the guide
enumerates: undeclared allocations in builders and kernels, 64-byte padding and doubling growth,
allocator fragmentation and retained pages, non-Rust allocations, and batches in flight across FFI.
A PR that treats reserved() as the real footprint is built on a false premise.
2. Reservations
Native operators reserve through DataFusion's MemoryConsumer and MemoryReservation API.
- [ ]
try_growfailure turns into spilling, not an error. For a spillable operator,ResourcesExhaustedis the signal to spill and retry. A new operator that propagates it as a query failure has converted a recoverable condition into a lost task. - [ ] Treat a new
growcall site as potential unbacked memory.growcannot fail. When Spark grants less than asked, the Comet pools record the full amount anyway and carry the shortfall as overcommit, which Spark does not know about and can hand to another consumer or task. Nothing caps how far successivegrowcalls overcommit short of the container limit. That is acceptable only for memory that already exists and cannot be spilled, such as a spilled batch read back from disk, so a new call site needs that justification. Memory the caller is about to allocate belongs behindtry_grow. - [ ] Every
try_growhas a matchingshrink, including on the error path. A reservation leaked on an early return is charged for the life of the task. - [ ] In
try_grow, a partial grant is released, not kept.acquired < additionalmust release and reportResourcesExhausted. Onlygrowkeeps a partial grant, as overcommit. - [ ] An operator that buffers without reserving is invisible to the pool. If the PR adds buffering, ask where the reservation is. "It is only a few batches" is how the accounting gap grows.
- [ ] Reservation size matches what is actually held. Check the PR uses buffer sizes read off
the
ArrayData, asget_reserved_bytes_for_record_batchandget_record_batch_memory_sizedo, rather than a row-count estimate.
3. The Pool Stack
create_memory_pool builds a base pool and wraps it in decorators. In the default off-heap
configuration, from the inside out:
[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
[TaskSharedMemoryPool] <- RAII handle for the per-task registry
[TrackConsumersPool] <- DataFusion, names the top 10 consumers in error messages
[CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
- [ ] A new decorator forwards every
MemoryPoolmethod to its inner pool. A partial implementation makesreserved()disagree between levels. - [ ]
fair_unifiedholds the consumers that can spill to a share of the task's share. The task's share isspark.memory.offHeap.sizedivided by the tasks on the executor running native plans, whichRunningTaskscounts while each task holds its task-shared pool, orpool_sizeif that is smaller. A consumer that can spill is refused if what it holds plus the request exceeds what the consumers that cannot spill leave of the task's share, divided by the registered consumers that can spill, or if the task's total plus the request exceeds the task's share. A consumer that cannot spill is refused only if the task's total plus the request exceedspool_size, which binds before Spark's own limit only while the deprecated fraction is below1 / N. With eight running tasks, a 16 GiB pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds 512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations thatnew_empty(),split()andtake()create count against one share. A PR that checksreservation.size()instead lets an operator with several reservations, such as a sort's streaming merge, take other consumers' shares. It also depends on when DataFusion updates the size, which is after it callstry_growbut before it callsshrink. A PR that limits a consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back #7088: one operator takes the task's share from Spark and a merge or hash join build fails the task. If the PR changes any of these comparisons it changes when every query spills, so it needs benchmark evidence, not reasoning. - [ ] The consumers that divide the task's share are every consumer that can spill in the task, across every native plan, because the pool is task-shared.
- [ ] Task-shared pool lifetime.
acquire_task_shared_poolkeeps a process-wideHashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>. The returnedArcis the only lifetime handle, so there is no explicit release to forget and acreatePlanthat fails partway cleans up on unwind. A PR that adds an explicit release call, or stores theArcsomewhere longer-lived, breaks that property.TaskSharedMemoryPool::dropcompares pointers so it only removes an entry that is still its own, which handles the race where anacquireobserves an expiredWeakand inserts a replacement first. Do not let that check be simplified away. - [ ]
fair_unifiedandgreedy_unifiedare the only pool types. On-heap mode ignores the pool-type string and always getsUnboundedMemoryPool: it exists so the Spark SQL test suite can run against Comet, it accounts for nothing, and it must not be used in production. A PR re-adding a sized on-heap pool is reintroducing a budget that bounds nothing real (see issue #6063).
4. The Spark Bridge
CometUnifiedMemoryPool and CometFairMemoryPool call CometTaskMemoryManager.acquireMemory over
JNI, which goes through Spark's ordinary TaskMemoryManager. Two asymmetries matter:
- Spark can force its own consumers in the same task to spill to satisfy a Comet request.
- Comet cannot be forced to spill by Spark.
NativeMemoryConsumer.spill()returns0, so a JVM consumer blocked behind native reservations has no way to reclaim them. Native operators spill only when their owntry_growfails.
If the PR claims Spark-side reclamation works, check spill(). If the PR implements native
reclamation, that is a significant change and needs to say what it does to in-flight operators.
5. Budget and Configuration
CometExecIterator.getMemoryConfig computes the budget once per executor and passes it across JNI
as memory_limit, along with spark.memory.offHeap.size itself as off_heap_size, which
fair_unified divides among the tasks running native plans:
memory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction
off_heap_size = spark.memory.offHeap.size
- [ ] A new memory config goes through
getMemoryConfigrather than being read separately in native code, and a new value crossing JNI is added tocreatePlanon both sides. - [ ] Changing
memoryPool.fractionsemantics affects every deployment that tuned it as a haircut for the accounting gap. - [ ] On Kubernetes,
spark.memory.offHeap.sizeis part of the pod limit, not headroom on top of it. A PR whose fix is "raise the off-heap size" is asking for fewer executors per node.spark.executor.memoryOverheadis the only real slack in the container, and JVM non-heap usage already consumes much of it.
6. Unaccounted Allocators
Two allocators sit outside Comet's pool entirely, and they are easy to confuse:
CometArrowAllocator(spark/src/main/scala/org/apache/comet/package.scala) is a process-wideRootAllocator(Long.MaxValue). It is unbounded and no budget sees it. Child allocators are cut from it for FFI stream export, broadcast coalescing, andCometSparkToColumnarExec. A PR that adds a child allocator, or makes an existing one hold more, is adding uncounted container RSS. Say so even if the volume is small.- JVM shuffle pages go through
CometShuffleMemoryAllocator.getInstance, which returnsCometUnifiedShuffleMemoryAllocator, an ordinary SparkMemoryConsumerdrawing fromspark.memory.offHeap.size. These are arbitrated by Spark against its other consumers. Do not review them as if they were native allocations.
7. Evidence
Memory changes are hard to unit test, so the review question is what evidence exists rather than what test was added. Ask for at least one of:
spark.comet.debug.memory=trueoutput fromLoggingMemoryPool, showing register, grow, and shrink with consumer names- A trace comparing
jemalloc_allocatedagainst the summedthread_NNN_comet_memory_reservedvalues, which is the only way to see the accounting gap - Spill counts from
spark.comet.explain.native.enabled=true, before and after - For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the ability to spill again during that replay, and only a run at a small off-heap size showed it (#6254).
- For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or
OOMKilledis the cgroup. 52 withjava.lang.OutOfMemoryErroris JVM heap. A failed task withSparkOutOfMemoryErrorand a surviving executor is Spark's pool, the only one of the three that is recoverable at task level.
spark/src/test/scala/org/apache/spark/CometTaskMemoryManagerSuite.scala and
CometUnboundedShuffleMemoryAllocatorSuite.scala are the existing JVM-side tests. A change to the
bridge or an allocator should extend one of them.
8. Does the PR Make memory_management.md Stale?
This doc states invariants and enumerates known gaps, so it goes stale in ways a file-path check will not catch. Check:
- The Who allocates what table, if the PR adds an allocator or changes what bounds one
- The pool stack diagram, if a decorator is added, removed, or reordered
- The pool type table, if a pool type is added, removed, or changes how it is sized
- The
memory_limitformula, if the budget calculation changes - The fair pool description, which spells out the task share, the share and task-total checks, and their consequences
- The Crossing the FFI boundary section, which names the operators that reserve for imported batches
- The Open problems list. If the PR closes one of these gaps, the entry must be removed or rewritten in the same PR. Leaving a fixed problem listed as open is worse than no doc, because the next contributor will re-fix it.
- The Debugging memory issues table and checklist, if a tool or signal is added or removed
04 同仓库更多 Skills
wire-datafusion-function
Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging / restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.
suggest-native-expression
Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.
review-comet-shuffle-pr
Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.
review-comet-pr
Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.
review-comet-iceberg-write-pr
Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.
review-comet-ffi-pr
Use when reviewing a DataFusion Comet pull request that crosses the JVM/native boundary, touching Arrow C Data or C Stream interface code, batch export and import, CometExecIterator, ScanExec, NativeUtil, CometVector subclasses, or jni_api. Load alongside review-comet-pr.