[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACJFM0QS61Z1RW516HXN2M":3,"skill-same:01M6ACJFM0QS61Z1RW516HXN2M":70},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":35,"linkOnly":36,"markdownContent":37,"markdownBody":38,"frontMatter":39,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":44,"repositoryHtmlUrl":45,"repositoryHomepageUrl":46,"ownerHtmlUrl":47,"ownerBlogUrl":48,"ownerDescription":6,"updatedAt":49,"install":50,"resources":63},"01M6ACJFM0QS61Z1RW516HXN2M","review-comet-memory-pr",null,"Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.","apache\u002Fdatafusion-comet","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACHX42TPNCR8TQJEAR8J5J",1322,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34],"review","memory","pr","review comet memory pr","spark","pull request","pull","datafusion","bridge","use","configuration","reservations","number","load","request","when","2026-11-10 10:12:30",false,"---\nname: review-comet-memory-pr\ndescription: Use when reviewing a DataFusion Comet pull request that touches memory pools, memory reservations, spilling, the Spark TaskMemoryManager bridge, allocators, or memory-related configuration. Load alongside review-comet-pr.\nargument-hint: \u003Cpr-number>\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nMemory-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers memory.\n\n## Read the Contributor Guide First\n\n| Doc                                                  | What you need from it                                                             |\n| ---------------------------------------------------- | --------------------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fmemory_management.md` | The three budgets, the allocator inventory, the pool stack, the accounting gap    |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fffi.md`               | Where buffers physically live when a batch crosses the boundary                   |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fdebugging.md`         | \"Debugging Memory Reservations\", for what evidence to ask the author for          |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ftracing.md`           | `jemalloc_allocated` against summed reservations, the only measurement of the gap |\n\nRead `memory_management.md` in full before the diff. Most bad memory PRs are not wrong about code,\nthey are wrong about which of the three budgets they are affecting.\n\n## 1. Name the Budget\n\nThere are three budgets, enforced by three different parties, and a PR that improves one can make\nanother worse:\n\n| Budget             | Enforced by       | Failure mode                                                      |\n| ------------------ | ----------------- | ----------------------------------------------------------------- |\n| JVM heap           | The JVM           | `OutOfMemoryError`, executor exits with code 52                   |\n| Spark memory pools | Spark bookkeeping | A consumer spills, or the task fails with `SparkOutOfMemoryError` |\n| Container RSS      | OS or cgroup      | `SIGKILL` of the whole executor, exit 137, `OOMKilled`            |\n\nThe first two are accounting, a running total of voluntarily declared bytes. The third is physical.\nAsk which one the PR claims to fix and check that the mechanism actually reaches it. Lowering a\ndeclared reservation does not reduce resident pages. Reducing resident pages does not by itself\nprevent a `SparkOutOfMemoryError`.\n\nComet's declared reservations are a **lower bound** on real usage, for structural reasons the guide\nenumerates: undeclared allocations in builders and kernels, 64-byte padding and doubling growth,\nallocator fragmentation and retained pages, non-Rust allocations, and batches in flight across FFI.\nA PR that treats `reserved()` as the real footprint is built on a false premise.\n\n## 2. Reservations\n\nNative operators reserve through DataFusion's `MemoryConsumer` and `MemoryReservation` API.\n\n- [ ] **`try_grow` failure turns into spilling, not an error.** For a spillable operator,\n      `ResourcesExhausted` is the signal to spill and retry. A new operator that propagates it as a\n      query failure has converted a recoverable condition into a lost task.\n- [ ] **Treat a new `grow` call site as potential unbacked memory.** `grow` cannot fail. When\n      Spark grants less than asked, the Comet pools record the full amount anyway and carry the\n      shortfall as overcommit, which Spark does not know about and can hand to another consumer\n      or task. Nothing caps how far successive `grow` calls overcommit short of the container\n      limit. That is acceptable only for memory that already exists and cannot be spilled, such\n      as a spilled batch read back from disk, so a new call site needs that justification.\n      Memory the caller is about to allocate belongs behind `try_grow`.\n- [ ] **Every `try_grow` has a matching `shrink`, including on the error path.** A reservation\n      leaked on an early return is charged for the life of the task.\n- [ ] **In `try_grow`, a partial grant is released, not kept.** `acquired \u003C additional` must\n      release and report `ResourcesExhausted`. Only `grow` keeps a partial grant, as overcommit.\n- [ ] **An operator that buffers without reserving is invisible to the pool.** If the PR adds\n      buffering, ask where the reservation is. \"It is only a few batches\" is how the accounting gap\n      grows.\n- [ ] **Reservation size matches what is actually held.** Check the PR uses buffer sizes read off\n      the `ArrayData`, as `get_reserved_bytes_for_record_batch` and `get_record_batch_memory_size`\n      do, rather than a row-count estimate.\n\n## 3. The Pool Stack\n\n`create_memory_pool` builds a base pool and wraps it in decorators. In the default off-heap\nconfiguration, from the inside out:\n\n```text\n[LoggingMemoryPool]        \u003C- only when spark.comet.debug.memory=true\n  [TaskSharedMemoryPool]   \u003C- RAII handle for the per-task registry\n    [TrackConsumersPool]   \u003C- DataFusion, names the top 10 consumers in error messages\n      [CometFairMemoryPool]  \u003C- delegates acquire\u002Frelease to Spark over JNI\n```\n\n- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial\n      implementation makes `reserved()` disagree between levels.\n- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The\n      task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running\n      native plans, which `RunningTasks` counts while each task holds its task-shared pool, or\n      `pool_size` if that is smaller. A consumer that can spill is\n      refused if what it holds plus the request exceeds what the consumers that cannot spill leave\n      of the task's share, divided by the registered consumers that can spill, or if the task's\n      total plus the request exceeds the task's share. A consumer that cannot spill is refused only\n      if the task's total plus the request exceeds `pool_size`, which binds before Spark's own\n      limit only while the deprecated fraction is below `1 \u002F N`. With eight running tasks, a 16 GiB\n      pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds\n      512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations\n      that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks\n      `reservation.size()` instead lets an operator with several reservations, such as a sort's\n      streaming merge, take other consumers' shares. It also depends on when DataFusion updates the\n      size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a\n      consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back\n      #7088: one operator takes the task's share from Spark and a merge or hash join build fails\n      the task. If the PR changes any of these comparisons it changes when every query spills, so\n      it needs benchmark evidence, not reasoning.\n- [ ] **The consumers that divide the task's share are every consumer that can spill in the\n      task**, across every native plan, because the pool is task-shared.\n- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide\n      `HashMap\u003Ctask_attempt_id, Weak\u003CTaskSharedMemoryPool>>`. The returned `Arc` is the only\n      lifetime handle, so there is no explicit release to forget and a `createPlan` that fails\n      partway cleans up on unwind. A PR that adds an explicit release call, or stores the `Arc`\n      somewhere longer-lived, breaks that property. `TaskSharedMemoryPool::drop` compares pointers\n      so it only removes an entry that is still its own, which handles the race where an `acquire`\n      observes an expired `Weak` and inserts a replacement first. Do not let that check be\n      simplified away.\n- [ ] **`fair_unified` and `greedy_unified` are the only pool types.** On-heap mode ignores the\n      pool-type string and always gets `UnboundedMemoryPool`: it exists so the Spark SQL test suite\n      can run against Comet, it accounts for nothing, and it must not be used in production. A PR\n      re-adding a sized on-heap pool is reintroducing a budget that bounds nothing real (see\n      issue #6063).\n\n## 4. The Spark Bridge\n\n`CometUnifiedMemoryPool` and `CometFairMemoryPool` call `CometTaskMemoryManager.acquireMemory` over\nJNI, which goes through Spark's ordinary `TaskMemoryManager`. Two asymmetries matter:\n\n- Spark **can** force its own consumers in the same task to spill to satisfy a Comet request.\n- Comet **cannot** be forced to spill by Spark. `NativeMemoryConsumer.spill()` returns `0`, so a\n  JVM consumer blocked behind native reservations has no way to reclaim them. Native operators\n  spill only when their own `try_grow` fails.\n\nIf the PR claims Spark-side reclamation works, check `spill()`. If the PR implements native\nreclamation, that is a significant change and needs to say what it does to in-flight operators.\n\n## 5. Budget and Configuration\n\n`CometExecIterator.getMemoryConfig` computes the budget once per executor and passes it across JNI\nas `memory_limit`, along with `spark.memory.offHeap.size` itself as `off_heap_size`, which\n`fair_unified` divides among the tasks running native plans:\n\n```text\nmemory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction\noff_heap_size = spark.memory.offHeap.size\n```\n\n- [ ] A new memory config goes through `getMemoryConfig` rather than being read separately in\n      native code, and a new value crossing JNI is added to `createPlan` on both sides.\n- [ ] Changing `memoryPool.fraction` semantics affects every deployment that tuned it as a haircut\n      for the accounting gap.\n- [ ] On Kubernetes, `spark.memory.offHeap.size` is **part of** the pod limit, not headroom on top\n      of it. A PR whose fix is \"raise the off-heap size\" is asking for fewer executors per node.\n      `spark.executor.memoryOverhead` is the only real slack in the container, and JVM non-heap\n      usage already consumes much of it.\n\n## 6. Unaccounted Allocators\n\nTwo allocators sit outside Comet's pool entirely, and they are easy to confuse:\n\n- **`CometArrowAllocator`** (`spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fpackage.scala`) is a\n  process-wide `RootAllocator(Long.MaxValue)`. It is unbounded and no budget sees it. Child\n  allocators are cut from it for FFI stream export, broadcast coalescing, and\n  `CometSparkToColumnarExec`. A PR that adds a child allocator, or makes an existing one hold more,\n  is adding uncounted container RSS. Say so even if the volume is small.\n- **JVM shuffle pages** go through `CometShuffleMemoryAllocator.getInstance`, which returns\n  `CometUnifiedShuffleMemoryAllocator`, an ordinary Spark `MemoryConsumer` drawing from\n  `spark.memory.offHeap.size`. These **are** arbitrated by Spark against its other consumers. Do\n  not review them as if they were native allocations.\n\n## 7. Evidence\n\nMemory changes are hard to unit test, so the review question is what evidence exists rather than\nwhat test was added. Ask for at least one of:\n\n- `spark.comet.debug.memory=true` output from `LoggingMemoryPool`, showing register, grow, and\n  shrink with consumer names\n- A trace comparing `jemalloc_allocated` against the summed\n  `thread_NNN_comet_memory_reserved` values, which is the only way to see the accounting gap\n- Spill counts from `spark.comet.explain.native.enabled=true`, before and after\n- For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills\n  under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the\n  ability to spill again during that replay, and only a run at a small off-heap size showed it\n  (#6254).\n- For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or\n  `OOMKilled` is the cgroup. 52 with `java.lang.OutOfMemoryError` is JVM heap. A failed task with\n  `SparkOutOfMemoryError` and a surviving executor is Spark's pool, the only one of the three that\n  is recoverable at task level.\n\n`spark\u002Fsrc\u002Ftest\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002FCometTaskMemoryManagerSuite.scala` and\n`CometUnboundedShuffleMemoryAllocatorSuite.scala` are the existing JVM-side tests. A change to the\nbridge or an allocator should extend one of them.\n\n## 8. Does the PR Make `memory_management.md` Stale?\n\nThis doc states invariants and enumerates known gaps, so it goes stale in ways a file-path check\nwill not catch. Check:\n\n- The **Who allocates what** table, if the PR adds an allocator or changes what bounds one\n- The **pool stack** diagram, if a decorator is added, removed, or reordered\n- The **pool type** table, if a pool type is added, removed, or changes how it is sized\n- The `memory_limit` formula, if the budget calculation changes\n- The **fair pool** description, which spells out the task share, the share and task-total\n  checks, and their consequences\n- The **Crossing the FFI boundary** section, which names the operators that reserve for imported\n  batches\n- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or\n  rewritten in the same PR. Leaving a fixed problem listed as open is worse than no doc, because\n  the next contributor will re-fix it.\n- The **Debugging memory issues** table and checklist, if a tool or signal is added or removed\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n-->\n\nMemory-specific review for Comet PR #$ARGUMENTS.\n\n**REQUIRED BACKGROUND:** Use `review-comet-pr` for PR metadata, existing comments, CI, the review\nbar, and the output format. This skill only covers memory.\n\n## Read the Contributor Guide First\n\n| Doc                                                  | What you need from it                                                             |\n| ---------------------------------------------------- | --------------------------------------------------------------------------------- |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fmemory_management.md` | The three budgets, the allocator inventory, the pool stack, the accounting gap    |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fffi.md`               | Where buffers physically live when a batch crosses the boundary                   |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Fdebugging.md`         | \"Debugging Memory Reservations\", for what evidence to ask the author for          |\n| `docs\u002Fsource\u002Fcontributor-guide\u002Ftracing.md`           | `jemalloc_allocated` against summed reservations, the only measurement of the gap |\n\nRead `memory_management.md` in full before the diff. Most bad memory PRs are not wrong about code,\nthey are wrong about which of the three budgets they are affecting.\n\n## 1. Name the Budget\n\nThere are three budgets, enforced by three different parties, and a PR that improves one can make\nanother worse:\n\n| Budget             | Enforced by       | Failure mode                                                      |\n| ------------------ | ----------------- | ----------------------------------------------------------------- |\n| JVM heap           | The JVM           | `OutOfMemoryError`, executor exits with code 52                   |\n| Spark memory pools | Spark bookkeeping | A consumer spills, or the task fails with `SparkOutOfMemoryError` |\n| Container RSS      | OS or cgroup      | `SIGKILL` of the whole executor, exit 137, `OOMKilled`            |\n\nThe first two are accounting, a running total of voluntarily declared bytes. The third is physical.\nAsk which one the PR claims to fix and check that the mechanism actually reaches it. Lowering a\ndeclared reservation does not reduce resident pages. Reducing resident pages does not by itself\nprevent a `SparkOutOfMemoryError`.\n\nComet's declared reservations are a **lower bound** on real usage, for structural reasons the guide\nenumerates: undeclared allocations in builders and kernels, 64-byte padding and doubling growth,\nallocator fragmentation and retained pages, non-Rust allocations, and batches in flight across FFI.\nA PR that treats `reserved()` as the real footprint is built on a false premise.\n\n## 2. Reservations\n\nNative operators reserve through DataFusion's `MemoryConsumer` and `MemoryReservation` API.\n\n- [ ] **`try_grow` failure turns into spilling, not an error.** For a spillable operator,\n      `ResourcesExhausted` is the signal to spill and retry. A new operator that propagates it as a\n      query failure has converted a recoverable condition into a lost task.\n- [ ] **Treat a new `grow` call site as potential unbacked memory.** `grow` cannot fail. When\n      Spark grants less than asked, the Comet pools record the full amount anyway and carry the\n      shortfall as overcommit, which Spark does not know about and can hand to another consumer\n      or task. Nothing caps how far successive `grow` calls overcommit short of the container\n      limit. That is acceptable only for memory that already exists and cannot be spilled, such\n      as a spilled batch read back from disk, so a new call site needs that justification.\n      Memory the caller is about to allocate belongs behind `try_grow`.\n- [ ] **Every `try_grow` has a matching `shrink`, including on the error path.** A reservation\n      leaked on an early return is charged for the life of the task.\n- [ ] **In `try_grow`, a partial grant is released, not kept.** `acquired \u003C additional` must\n      release and report `ResourcesExhausted`. Only `grow` keeps a partial grant, as overcommit.\n- [ ] **An operator that buffers without reserving is invisible to the pool.** If the PR adds\n      buffering, ask where the reservation is. \"It is only a few batches\" is how the accounting gap\n      grows.\n- [ ] **Reservation size matches what is actually held.** Check the PR uses buffer sizes read off\n      the `ArrayData`, as `get_reserved_bytes_for_record_batch` and `get_record_batch_memory_size`\n      do, rather than a row-count estimate.\n\n## 3. The Pool Stack\n\n`create_memory_pool` builds a base pool and wraps it in decorators. In the default off-heap\nconfiguration, from the inside out:\n\n```text\n[LoggingMemoryPool]        \u003C- only when spark.comet.debug.memory=true\n  [TaskSharedMemoryPool]   \u003C- RAII handle for the per-task registry\n    [TrackConsumersPool]   \u003C- DataFusion, names the top 10 consumers in error messages\n      [CometFairMemoryPool]  \u003C- delegates acquire\u002Frelease to Spark over JNI\n```\n\n- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial\n      implementation makes `reserved()` disagree between levels.\n- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The\n      task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running\n      native plans, which `RunningTasks` counts while each task holds its task-shared pool, or\n      `pool_size` if that is smaller. A consumer that can spill is\n      refused if what it holds plus the request exceeds what the consumers that cannot spill leave\n      of the task's share, divided by the registered consumers that can spill, or if the task's\n      total plus the request exceeds the task's share. A consumer that cannot spill is refused only\n      if the task's total plus the request exceeds `pool_size`, which binds before Spark's own\n      limit only while the deprecated fraction is below `1 \u002F N`. With eight running tasks, a 16 GiB\n      pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds\n      512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations\n      that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks\n      `reservation.size()` instead lets an operator with several reservations, such as a sort's\n      streaming merge, take other consumers' shares. It also depends on when DataFusion updates the\n      size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a\n      consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back\n      #7088: one operator takes the task's share from Spark and a merge or hash join build fails\n      the task. If the PR changes any of these comparisons it changes when every query spills, so\n      it needs benchmark evidence, not reasoning.\n- [ ] **The consumers that divide the task's share are every consumer that can spill in the\n      task**, across every native plan, because the pool is task-shared.\n- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide\n      `HashMap\u003Ctask_attempt_id, Weak\u003CTaskSharedMemoryPool>>`. The returned `Arc` is the only\n      lifetime handle, so there is no explicit release to forget and a `createPlan` that fails\n      partway cleans up on unwind. A PR that adds an explicit release call, or stores the `Arc`\n      somewhere longer-lived, breaks that property. `TaskSharedMemoryPool::drop` compares pointers\n      so it only removes an entry that is still its own, which handles the race where an `acquire`\n      observes an expired `Weak` and inserts a replacement first. Do not let that check be\n      simplified away.\n- [ ] **`fair_unified` and `greedy_unified` are the only pool types.** On-heap mode ignores the\n      pool-type string and always gets `UnboundedMemoryPool`: it exists so the Spark SQL test suite\n      can run against Comet, it accounts for nothing, and it must not be used in production. A PR\n      re-adding a sized on-heap pool is reintroducing a budget that bounds nothing real (see\n      issue #6063).\n\n## 4. The Spark Bridge\n\n`CometUnifiedMemoryPool` and `CometFairMemoryPool` call `CometTaskMemoryManager.acquireMemory` over\nJNI, which goes through Spark's ordinary `TaskMemoryManager`. Two asymmetries matter:\n\n- Spark **can** force its own consumers in the same task to spill to satisfy a Comet request.\n- Comet **cannot** be forced to spill by Spark. `NativeMemoryConsumer.spill()` returns `0`, so a\n  JVM consumer blocked behind native reservations has no way to reclaim them. Native operators\n  spill only when their own `try_grow` fails.\n\nIf the PR claims Spark-side reclamation works, check `spill()`. If the PR implements native\nreclamation, that is a significant change and needs to say what it does to in-flight operators.\n\n## 5. Budget and Configuration\n\n`CometExecIterator.getMemoryConfig` computes the budget once per executor and passes it across JNI\nas `memory_limit`, along with `spark.memory.offHeap.size` itself as `off_heap_size`, which\n`fair_unified` divides among the tasks running native plans:\n\n```text\nmemory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction\noff_heap_size = spark.memory.offHeap.size\n```\n\n- [ ] A new memory config goes through `getMemoryConfig` rather than being read separately in\n      native code, and a new value crossing JNI is added to `createPlan` on both sides.\n- [ ] Changing `memoryPool.fraction` semantics affects every deployment that tuned it as a haircut\n      for the accounting gap.\n- [ ] On Kubernetes, `spark.memory.offHeap.size` is **part of** the pod limit, not headroom on top\n      of it. A PR whose fix is \"raise the off-heap size\" is asking for fewer executors per node.\n      `spark.executor.memoryOverhead` is the only real slack in the container, and JVM non-heap\n      usage already consumes much of it.\n\n## 6. Unaccounted Allocators\n\nTwo allocators sit outside Comet's pool entirely, and they are easy to confuse:\n\n- **`CometArrowAllocator`** (`spark\u002Fsrc\u002Fmain\u002Fscala\u002Forg\u002Fapache\u002Fcomet\u002Fpackage.scala`) is a\n  process-wide `RootAllocator(Long.MaxValue)`. It is unbounded and no budget sees it. Child\n  allocators are cut from it for FFI stream export, broadcast coalescing, and\n  `CometSparkToColumnarExec`. A PR that adds a child allocator, or makes an existing one hold more,\n  is adding uncounted container RSS. Say so even if the volume is small.\n- **JVM shuffle pages** go through `CometShuffleMemoryAllocator.getInstance`, which returns\n  `CometUnifiedShuffleMemoryAllocator`, an ordinary Spark `MemoryConsumer` drawing from\n  `spark.memory.offHeap.size`. These **are** arbitrated by Spark against its other consumers. Do\n  not review them as if they were native allocations.\n\n## 7. Evidence\n\nMemory changes are hard to unit test, so the review question is what evidence exists rather than\nwhat test was added. Ask for at least one of:\n\n- `spark.comet.debug.memory=true` output from `LoggingMemoryPool`, showing register, grow, and\n  shrink with consumer names\n- A trace comparing `jemalloc_allocated` against the summed\n  `thread_NNN_comet_memory_reserved` values, which is the only way to see the accounting gap\n- Spill counts from `spark.comet.explain.native.enabled=true`, before and after\n- For a DataFusion upgrade, or a change to a spilling operator, a run where the operator spills\n  under a tight pool and then reads its spill back. The DataFusion 55.1 final aggregate lost the\n  ability to spill again during that replay, and only a run at a small off-heap size showed it\n  (#6254).\n- For a fix to an OOM report, the exit code that identifies which budget was exceeded. 137 or\n  `OOMKilled` is the cgroup. 52 with `java.lang.OutOfMemoryError` is JVM heap. A failed task with\n  `SparkOutOfMemoryError` and a surviving executor is Spark's pool, the only one of the three that\n  is recoverable at task level.\n\n`spark\u002Fsrc\u002Ftest\u002Fscala\u002Forg\u002Fapache\u002Fspark\u002FCometTaskMemoryManagerSuite.scala` and\n`CometUnboundedShuffleMemoryAllocatorSuite.scala` are the existing JVM-side tests. A change to the\nbridge or an allocator should extend one of them.\n\n## 8. Does the PR Make `memory_management.md` Stale?\n\nThis doc states invariants and enumerates known gaps, so it goes stale in ways a file-path check\nwill not catch. Check:\n\n- The **Who allocates what** table, if the PR adds an allocator or changes what bounds one\n- The **pool stack** diagram, if a decorator is added, removed, or reordered\n- The **pool type** table, if a pool type is added, removed, or changes how it is sized\n- The `memory_limit` formula, if the budget calculation changes\n- The **fair pool** description, which spells out the task share, the share and task-total\n  checks, and their consequences\n- The **Crossing the FFI boundary** section, which names the operators that reserve for imported\n  batches\n- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or\n  rewritten in the same PR. Leaving a fixed problem listed as open is worse than no doc, because\n  the next contributor will re-fix it.\n- The **Debugging memory issues** table and checklist, if a tool or signal is added or removed\n",{"name":5,"description":7,"argument-hint":40},"\u003Cpr-number>","main","7deeba21a42e4a9cd34f43e7ed6e960e4e851b26",".ai\u002Fskills\u002Freview-comet-memory-pr",418,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet","https:\u002F\u002Fdatafusion.apache.org\u002Fcomet","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-15 03:15:20",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add apache\u002Fdatafusion-comet --skill review-comet-memory-pr",{"tool":56,"command":57},"bunx","bunx skills add apache\u002Fdatafusion-comet --skill review-comet-memory-pr",{"tool":59,"command":60},"pnpm","pnpm dlx skills add apache\u002Fdatafusion-comet --skill review-comet-memory-pr","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-memory-pr\u002FSKILL.md",true,[64],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"OWNERSHIP","OTHER","其他",37,"https:\u002F\u002Fgithub.com\u002FJetBrains\u002Fintellij-community\u002Fblob\u002Fmaster\u002F.agents\u002Fskills\u002Fplugin-model-analyzer\u002FOWNERSHIP",[71,100,126,140,156,171],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":82,"publishedAt":35},"01M6ACJR50ZS7F1QX58JEKCGVM","wire-datafusion-function","Use when wiring an existing DataFusion or datafusion-spark function into Comet for a Spark expression. Identifies the right wiring pattern (one-line passthrough, datafusion-spark UDF registration, or custom serde with input massaging \u002F restrictions), applies the Scala serde, registers the UDF in jni_api when needed, and adds SQL file tests. Assumes the function already exists upstream — if not, switch to `implement-comet-expression`.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fwire-datafusion-function\u002FSKILL.md",[77],{"categoryNo":78,"categoryName":79,"taxonomyType":80,"taxonomyTypeDesc":81},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[26,83,84,85,86,23,87,88,89,90,91,92,93,28,94,34,95,96,97,98,99],"wire datafusion function","sql","scala","implement","registration","switch","for","upstream","custom","expression","tests","Existing","file","api","wiring","with","name",{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":35},"01M6ACJPDDPND6FHE2PC1CJ4D3","suggest-native-expression","Use when picking the next Spark expression to implement natively in Comet, or when asked whether a specific codegen-dispatched expression is worth a native Rust implementation. Scores the candidate on how likely a native path can be 100% Spark-compatible and how much throughput or allocation it would save versus the JVM codegen dispatcher, records the verdict in the per-expression audit log so disqualified candidates are not re-litigated, and files a GitHub issue for the recommendation.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fsuggest-native-expression\u002FSKILL.md",[],[107,92,108,109,110,111,86,23,112,113,34,114,115,116,117,118,119,120,121,122,123,124,125],"suggest native expression","native","dispatcher","github","rust","re","files","candidate","log","jvm","next","how","audit","audit log","allocation","issue","throughput","path","save",{"skillNo":127,"name":128,"nameZh":6,"description":129,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":130,"categories":131,"keywords":132,"publishedAt":35},"01M6ACJMHPEN7E76VPKC3P4X6C","review-comet-shuffle-pr","Use when reviewing a DataFusion Comet pull request that touches native or JVM columnar shuffle, the shuffle writers and readers, partitioning, the Arrow IPC block format, shuffle compression, or the Celeborn integration. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-shuffle-pr\u002FSKILL.md",[],[133,21,19,25,24,26,134,28,32,34,135,31,116,108,33,136,137,138,139],"review comet shuffle pr","arrow","ipc","format","integration","block","compression",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":146,"publishedAt":35},"01M6ACJHX8BES7ZNNRNVCQ4QWE","review-comet-pr","Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-pr\u002FSKILL.md",[],[21,19,147,25,148,24,20,26,149,150,34,31,151,152,33,153,89,28,154,155],"review comet pr","iceberg","workflow","documentation","reviewer","human","posting","review skills","comments",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":35},"01M6ACJCV64V0FZPMVXDE2K3DX","review-comet-iceberg-write-pr","Use when reviewing a DataFusion Comet pull request that touches Iceberg writes, the split-operator write plan (IcebergWriteStrategy, IcebergWriteExec, IcebergCommitExec), the native Iceberg writer (CometIcebergWriteExec, iceberg_write.rs, iceberg_partition_path.rs), its eligibility gate (CometIcebergNativeWrite), or the iceberg-rust pin. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-iceberg-write-pr\u002FSKILL.md",[],[21,148,163,19,164,25,24,26,111,165,166,167,28,32,168,34,31,169,170,108],"review comet iceberg write pr","write","writer","write plan","partition","pin","split","plan",{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":35},"01M6ACJA72KJRAM6F7WHMX01PN","review-comet-ffi-pr","Use when reviewing a DataFusion Comet pull request that crosses the JVM\u002Fnative boundary, touching Arrow C Data or C Stream interface code, batch export and import, CometExecIterator, ScanExec, NativeUtil, CometVector subclasses, or jni_api. Load alongside review-comet-pr.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-comet\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Freview-comet-ffi-pr\u002FSKILL.md",[],[21,178,19,179,25,24,26,134,180,181,182,183,28,32,34,31,96,116,184,108,33],"review comet ffi pr","ffi","code","batch","import","interface","boundary"]