[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACDG4GVV9PQ2ZACWXJNR2B":3,"skill-same:01M6ACDG4GVV9PQ2ZACWXJNR2B":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":51,"ownerDescription":6,"updatedAt":52,"install":53,"resources":66},"01M6ACDG4GVV9PQ2ZACWXJNR2B","datafusion-ffi",null,"Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion\u002Fffi\u002F` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.","apache\u002Fdatafusion","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACD7H7PW9G5CPPV291Y97C",9595,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fdatafusion-ffi\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"datafusion","datafusion ffi","ffi","touch","new","when","prs","Existing","changes","for","cover","more","reviews","review","patterns","checklist","Wrapper","use","review checklist","code","2026-11-10 10:12:30",false,"---\nname: datafusion-ffi\ndescription: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion\u002Fffi\u002F` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.\n---\n\n# DataFusion FFI Skill\n\nThis crate exposes a stable C ABI for DataFusion traits so that independently-compiled libraries (different Rust versions, plugins, `datafusion-python`, etc.) can interoperate at runtime. Stability and correctness here are load-bearing: a missed pattern can cause segfaults, leaks, or silently dropped trait behavior on the consumer side.\n\nRead the crate's `README.md` first if you have not — it establishes the vocabulary (`FFI_X` \u002F `ForeignX`, `library_marker_id`, `release`, `TaskContextProvider`, stabby vs `#[repr(C)]`).\n\n## When to use\n\nTrigger this skill any time the work touches `datafusion\u002Fffi\u002F`:\n\n- Adding a new `FFI_\u003CTrait>` + `Foreign\u003CTrait>` pair\n- Adding a method to an existing `FFI_X` struct\n- Reviewing a PR that touches this crate\n- Changing the codec \u002F proto serialization layer\n- Bumping the wrapped DataFusion trait surface (e.g. a new default method appeared upstream)\n\n## Hard rules\n\n1. **No `datafusion` dependency.** `datafusion-ffi` must not depend on the umbrella `datafusion` crate. Use the leaf crates (`datafusion-common`, `datafusion-expr`, `datafusion-catalog`, `datafusion-physical-plan`, etc.). `datafusion` is fine in `[dev-dependencies]`.\n2. **`#[repr(C)]` on every `FFI_X` struct**, not `#[stabby::stabby]`. Stabby is used for `SString`\u002F`SVec` only. Reasons documented in the README (build time, Arrow types lack `IStable`).\n3. **`unsafe extern \"C\"` on every function-pointer field — including `version`.** The one exception is the `library_marker_id` field, which is plain `extern \"C\" fn() -> usize`. Plain (safe) `extern \"C\"` also applies to the standalone function defs in `src\u002Flib.rs` — `pub extern \"C\" fn version()` and `pub extern \"C\" fn get_library_marker_id()` — which coerce into the `unsafe extern \"C\"` `version` field slot at construction.\n4. **Match `Send`\u002F`Sync` to the wrapped trait.** Raw `*mut c_void` makes every `FFI_X` `!Send + !Sync` by default — `unsafe impl` whichever bounds the consumer-facing trait requires. Most DataFusion traits (`TableProvider`, `ExecutionPlan`, all UDFs, codecs) need both. `Send`-only: `RecordBatchStream`, `Accumulator`, `GroupsAccumulator`, `PartitionEvaluator` (mutable \u002F stream APIs). The matching `ForeignX` always carries the same bounds — pick consistently.\n5. **`#![deny(clippy::clone_on_ref_ptr)]`** is on at the crate root. Use `Arc::clone(&x)`, never `x.clone()` on `Arc`.\n6. **Run before pushing:** `cargo fmt --all`, `cargo clippy -p datafusion-ffi --all-targets --all-features -- -D warnings`, `cargo test -p datafusion-ffi`.\n7. **`api change` label required.** Any PR that modifies an `FFI_X` struct layout (adds\u002Fremoves\u002Freorders fields, changes a function-pointer signature, adds a variant to an FFI enum, or changes the `version` extern) must carry the `api change` GitHub label. Layout changes break ABI for already-compiled consumer libraries. The label is the project-wide convention for highlighting breaking public-API changes in release notes — see `docs\u002Fsource\u002Fcontributor-guide\u002Fapi-health.md` §\"What to do when making breaking API changes?\" (step 1 names the label explicitly). Downstream users (e.g. `datafusion-python`, plugin authors) read the labelled notes to know they must recompile against the new DataFusion major. The `version()` extern in `src\u002Flib.rs` returns the major of workspace `CARGO_PKG_VERSION`; consumers compare it at load time and can refuse mismatched producers. Apply via `gh pr edit \u003Cnum> --add-label \"api change\"` (label name contains a space — must be quoted). When reviewing such a PR, block merge until label present.\n8. **No FFI struct changes in patch releases.** Patch releases ship from branches matching `^branch-\\d+$` (e.g. `branch-53`, `branch-52`). FFI struct layout changes (anything that would earn rule 7's `api change` label) **must not** target a release branch and must not be back-ported. Patch releases are ABI-stable by contract — a consumer compiled against `53.1.0` must keep working against `53.1.1`. Before reviewing\u002Fapproving an FFI PR, check the PR's base branch: if it matches the regex above, or the PR description \u002F labels indicate patch \u002F back-port, reject the FFI struct change and ask the author to retarget `main`. Bugfixes that do not alter struct layout (e.g. fixing a function-pointer body) are fine to back-port. Quick check: `gh pr view \u003Cnum> --json baseRefName,labels --jq '.baseRefName'` then match against `^branch-\\d+$`. Do **not** glob-match `branch-*` — back-port \u002F cherry-pick working branches (e.g. `branch-53-cherry-pick-1`) also share that prefix but are not release branches; only the strict `branch-\u003Cdigits>` form is the freeze target.\n\n## The standard wrapper shape\n\nA new `FFI_X` for trait `X` must follow this template. Use `FFI_CatalogProvider` (`src\u002Fcatalog_provider.rs`) as the canonical reference — it shows the full shape (codec field, nested FFI types, `FFI_Option`\u002F`FFI_Result` returns, Arc-backed `PrivateData`) without async or capability-flag noise. `FFI_TableProvider` (`src\u002Ftable_provider.rs`) covers async (`scan`, `FFI_SessionRef`, `FfiFuture`) and the one `Option\u003Cfn>` capability flag (`supports_filters_pushdown`).\n\n### 1. The `FFI_X` struct\n\n```rust\n#[repr(C)]\n#[derive(Debug)]\npub struct FFI_X {\n    some_method: unsafe extern \"C\" fn(this: &Self, ...) -> FFI_Result\u003C...>,\n    optional_method: Option\u003Cunsafe extern \"C\" fn(&Self, ...) -> FFI_Result\u003C...>>,\n    pub logical_codec: FFI_LogicalExtensionCodec,\n\n    clone: unsafe extern \"C\" fn(&Self) -> Self,\n    release: unsafe extern \"C\" fn(&mut Self),\n    pub version: unsafe extern \"C\" fn() -> u64,\n\n    private_data: *mut c_void,\n    pub library_marker_id: extern \"C\" fn() -> usize,\n}\n\nunsafe impl Send for FFI_X {}\nunsafe impl Sync for FFI_X {}\n```\n\nField rules:\n\n- **One `unsafe extern \"C\" fn` per trait method.** Always populate — `Arc\u003Cdyn Trait>` dispatch picks override-or-default at call time, so the producer side gets the right answer without the consumer needing to know. See § \"Method coverage\".\n- **`Option\u003Cfn>` is the capability-flag exception**, not a template. Crate uses it exactly once: `FFI_TableProvider::supports_filters_pushdown`. See § \"Method coverage\".\n- **Codec field** (`FFI_LogicalExtensionCodec` \u002F `FFI_PhysicalExtensionCodec`) only if the trait moves `Expr`s \u002F `LogicalPlan`s \u002F `ExecutionPlan`s across the boundary.\n- **Method function pointers are private by default.** Mark `pub` only if a downstream library needs to invoke them directly (rare — typically only `version`, `library_marker_id`, embedded codecs are `pub`).\n- **`version: super::version` is mandatory.** Consumers gate compatibility on it.\n- **`library_marker_id: crate::get_library_marker_id` is mandatory *when the wrapper uses the standard `ForeignX` adapter pattern*.** Two flavors exist:\n  - **Arc-backed (immutable \u002F shareable traits — `TableProvider`, `ExecutionPlan`, all UDFs, codecs):** consumer-side `From\u003C&FFI_X> for Arc\u003Cdyn X>` consults the marker to choose `Arc::clone(inner)` vs `Arc::new(ForeignX(...))`.\n  - **Box-backed (mutable \u002F move-only traits — `Accumulator`, `GroupsAccumulator`, `PartitionEvaluator`):** consumer-side `From\u003CFFI_X> for Box\u003Cdyn X>` consults the marker to take the inner `Box` directly vs `Box::new(ForeignX(...))`. Producer-side `From\u003CBox\u003Cdyn X>> for FFI_X` *also* uses an `is::\u003CForeignX>()` downcast as an additional re-wrap bypass; the marker check covers the reverse direction.\n\n  The field is dead ABI surface — and must be omitted with a one-line module-doc rationale — only when **neither** flavor applies: no `ForeignX` adapter, no reverse `From\u003CFFI_X> for {Arc,Box}\u003Cdyn X>`, and the trait is impl'd directly on `FFI_X`. Canonical example: `FFI_RecordBatchStream` (`impl RecordBatchStream for FFI_RecordBatchStream` at `record_batch_stream.rs:149`, no `ForeignRecordBatchStream`, no reverse `From`).\n\n  Before flagging a missing `library_marker_id` as a gap, run all three greps on the wrapper file: `Foreign\u003Cwrapper-name>`, `From\u003C&?FFI_X> for Arc\u003C`, `From\u003CFFI_X> for Box\u003C`. Hit on any → marker is required and its absence is a real gap. Zero hits on all three → marker is intentionally not needed, not a gap.\n- **`Send`\u002F`Sync` bounds match the wrapped trait** (rule 4). Most wrappers want both; `Send`-only for streams \u002F mutable traits.\n\n### 2. `PrivateData` shape\n\nDefault — for read-only, shareable traits — use `Arc`:\n\n```rust\nstruct XPrivateData {\n    inner: Arc\u003Cdyn X>,\n    runtime: Option\u003CHandle>, \u002F\u002F include when any async method exists\n}\n```\n\nFor traits that require `&mut self` (e.g. `Accumulator`, `GroupsAccumulator`, `PartitionEvaluator`), use `Box\u003Cdyn X>`:\n\n```rust\nstruct XPrivateData {\n    inner: Box\u003Cdyn X>,\n    runtime: Option\u003CHandle>, \u002F\u002F include when any async method exists\n}\n```\n\nA `Box`-backed `FFI_X` **cannot implement `Clone`**; document this and skip the `clone` function pointer, or hand-write a release path that distinguishes producer vs consumer side. Canonical example: `FFI_Accumulator` in `src\u002Fudaf\u002Faccumulator.rs`. See also `FFI_GroupsAccumulator` (`src\u002Fudaf\u002Fgroups_accumulator.rs`) and `FFI_PartitionEvaluator` (`src\u002Fudwf\u002Fpartition_evaluator.rs`).\n\n### 3. Function-pointer wrappers\n\nNaming convention: `\u003Cmethod>_fn_wrapper`.\n\n```rust\nunsafe extern \"C\" fn some_method_fn_wrapper(this: &FFI_X, ...) -> FFI_Result\u003CT> {\n    \u002F\u002F 1. Recover inner via this.inner()\n    \u002F\u002F 2. Translate FFI types → native types\n    \u002F\u002F 3. Call native method\n    \u002F\u002F 4. Translate native Result → FFI_Result via sresult_return! or .into()\n}\n```\n\n### 4. `clone` \u002F `release` \u002F `Drop`\n\n```rust\nunsafe extern \"C\" fn clone_fn_wrapper(this: &FFI_X) -> FFI_X { \u002F* re-Box new private_data, copy fn ptrs *\u002F }\n\nunsafe extern \"C\" fn release_fn_wrapper(this: &mut FFI_X) {\n    unsafe {\n        debug_assert!(!this.private_data.is_null());\n        drop(Box::from_raw(this.private_data as *mut XPrivateData));\n        this.private_data = std::ptr::null_mut();\n    }\n}\n\nimpl Drop for FFI_X { fn drop(&mut self) { unsafe { (self.release)(self) } } }\nimpl Clone for FFI_X { fn clone(&self) -> Self { unsafe { (self.clone)(self) } } }\n```\n\n`release` must null `private_data` so a double-free debug-asserts loudly.\n\n### 5. Constructor split\n\n```rust\nimpl FFI_X {\n    pub fn new(inner: Arc\u003Cdyn X>, runtime: Option\u003CHandle>,\n               task_ctx_provider: impl Into\u003CFFI_TaskContextProvider>,\n               logical_codec: Option\u003CArc\u003Cdyn LogicalExtensionCodec>>) -> Self {\n        \u002F\u002F build FFI_LogicalExtensionCodec from defaults, then forward\n        Self::new_with_ffi_codec(inner, runtime, ffi_codec)\n    }\n\n    pub fn new_with_ffi_codec(inner: Arc\u003Cdyn X>, runtime: Option\u003CHandle>,\n                              logical_codec: FFI_LogicalExtensionCodec) -> Self {\n        \u002F\u002F Round-trip downcast: if inner is already a ForeignX, return its FFI directly.\n        if let Some(foreign) = inner.downcast_ref::\u003CForeignX>() {\n            return foreign.0.clone();\n        }\n        \u002F\u002F …allocate XPrivateData and populate fn ptrs…\n    }\n}\n```\n\nThe round-trip downcast is **mandatory** — without it, repeated FFI hops nest `ForeignX(FFI_X(ForeignX(...)))` and you pay the boundary cost every layer.\n\n### 6. The `Foreign\u003CX>` consumer\n\n```rust\n#[derive(Debug)]\npub struct ForeignX(pub FFI_X);\nunsafe impl Send for ForeignX {}\nunsafe impl Sync for ForeignX {}\n\nimpl From\u003C&FFI_X> for Arc\u003Cdyn X> {\n    fn from(p: &FFI_X) -> Self {\n        if (p.library_marker_id)() == crate::get_library_marker_id() {\n            Arc::clone(unsafe { p.inner() })\n        } else {\n            Arc::new(ForeignX(p.clone()))\n        }\n    }\n}\n\nimpl X for ForeignX { \u002F* call each fn pointer, translate types back *\u002F }\n```\n\nThe marker-id check is **mandatory** for every `From\u003C&FFI_X> for Arc\u003Cdyn X>`. Skipping it breaks the local-bypass optimization and forces the producer's data through serialization.\n\n### 7. Tests\n\nThe crate has **two distinct test surfaces** and a new wrapper usually needs entries in both. They are not interchangeable; they catch different classes of bug.\n\n#### a. In-process unit tests (`#[cfg(test)] mod tests` inside `src\u002F\u003Cmod>.rs`)\n\nRun on every `cargo test -p datafusion-ffi`. Producer and consumer live in the same compilation unit, so `library_marker_id` returns the same value on both sides. To force the foreign path you must override the marker:\n\n```rust\nlet mut ffi_x = FFI_X::new(provider, …);\nffi_x.library_marker_id = crate::mock_foreign_marker_id; \u002F\u002F forces the ForeignX branch\nlet arc: Arc\u003Cdyn X> = (&ffi_x).into();\nassert!(arc.downcast_ref::\u003CForeignX>().is_some());\n```\n\nEvery wrapper must include at minimum:\n\n- A **local-bypass test** — build `FFI_X` from a concrete native type, convert to `Arc\u003Cdyn X>`, `downcast_ref::\u003CConcreteType>()` must succeed.\n- A **forced-foreign test** — set `library_marker_id = crate::mock_foreign_marker_id`, convert, `downcast_ref::\u003CForeignX>()` must succeed, then exercise every method end-to-end.\n\nTemplates: `test_ffi_table_provider_local_bypass` and `test_round_trip_ffi_table_provider_scan` in `src\u002Ftable_provider.rs`.\n\nWhat unit tests catch: Rust-level correctness (translation logic, lifetime bugs, leaks under valgrind\u002Fmiri, Send\u002FSync, error propagation, codec round-trips). What they **cannot** catch: real ABI bugs. Both producer and consumer share `#[repr(C)]` layout because they are the exact same struct definition in memory.\n\n#### b. Cross-library integration tests (`integration-tests\u002Ftests\u002F`, gated by the `integration-tests` feature)\n\nThe integration tests live in the unpublished `datafusion-ffi-integration-tests` crate in `datafusion\u002Fffi\u002Fintegration-tests\u002F`, so that `datafusion-ffi` itself is a plain rlib. That crate is built as a `cdylib` that exports `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern \"C\"` entry point defined in `src\u002Ftests\u002Fmod.rs` and gated by `#[cfg(feature = \"integration-tests\")]`. Its tests use `libloading` to `dlopen` that `cdylib`. The test executable links against the `datafusion-ffi` rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.\n\nRun with:\n\n```bash\ncargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests\n```\n\nTo add coverage for a new wrapper:\n\n1. **Add a constructor** in `src\u002Ftests\u002F\u003Carea>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.\n2. **Wire it into `ForeignLibraryModule`** in `src\u002Ftests\u002Fmod.rs`: add a field of type `extern \"C\" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.\n3. **Add the test** in `integration-tests\u002Ftests\u002Fcases\u002F\u003Carea>.rs`, registered in `cases\u002Fmod.rs`, under `mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc\u003Cdyn X>`, and exercise every method.\n\nWhen adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.\n\nWhat integration tests catch that unit tests cannot:\n\n- **Real ABI layout bugs.** Two builds means the consumer's view of `FFI_X` is reconstructed from declaration, not aliased to the producer's memory. Mismatched alignment, padding, niche optimization, or accidentally non-`#[repr(C)]` types surface here.\n- **Symbol visibility \u002F `no_mangle`** issues.\n- **`library_marker_id` correctness without mocking** — the two libraries genuinely have different statics, so the foreign branch is taken for real.\n- **Drop \u002F leak ordering** when the producer side is in a `dlopen`'d image.\n\n#### Which tests does my change need?\n\n| Change                                                                | Unit | Integration |\n| --------------------------------------------------------------------- | ---- | ----------- |\n| New `FFI_X` wrapper                                                   | Yes  | Yes         |\n| New method on existing `FFI_X`                                        | Yes  | Yes if the method takes\u002Freturns a non-trivial FFI type. See note below. |\n| Bugfix to a wrapper body, no signature change                         | Yes  | Only if reproducing the bug requires cross-library symbol lookup or `dlopen` semantics |\n| Layout change (`#[repr(C)]` field add\u002Fremove\u002Freorder, fn-ptr sig)     | Yes  | **Mandatory** — this is exactly the bug class integration tests exist for |\n| New `From\u003CX> for FFI_X` or codec change                               | Yes  | Yes if the codec is exercised by the cross-library round-trip |\n\n**\"Non-trivial FFI type\" for the table above** — anything other than:\n\n- Primitives (`u8`\u002F`u64`\u002F`bool`\u002F`usize`, etc.) and `#[repr(u8)]` FFI enums (`FFI_TableType`, `Volatility`, `InsertOp`, `TableProviderFilterPushDown`).\n- A `stabby::string::String` (`SString`) returned by value, with no other args or returns.\n\nConcrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec\u003CT>`, `FFI_Option\u003CT>`, `FFI_Result\u003CT>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`\u002F`*const` pointer. These exercise alignment \u002F padding \u002F niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.\n\nIf you skip the integration test for a layout change, you have effectively shipped untested ABI.\n\n## Method coverage — the silent-default gap\n\n**This is the area where the crate currently has real holes.** When the wrapped trait has methods with *default implementations*, those defaults are typically the trait's \"no-op \u002F unsupported\" answer (`None`, `false`, `Unsupported`, `not_impl_err!()`). If the producer overrides a default but the FFI struct does not carry a function pointer for it, the consumer's `Foreign\u003CX>` falls back to the trait default — **silently losing the override**. The producer thinks it implemented `delete_from`; the consumer behaves as if it never did.\n\n### Rule\n\nWhen adding or auditing an `FFI_X`, **enumerate every method on the wrapped trait, including defaulted ones**, and for each one decide:\n\n| Category                                                                              | Action                                                                                                                                |\n| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |\n| Required method (no default)                                                          | Mandatory `unsafe extern \"C\" fn` field.                                                                                               |\n| Defaulted, plausible override (statistics, distribution, ordering, simplify, DML, …)  | Mandatory `unsafe extern \"C\" fn` field — same as a required method. The wrapper body calls `inner.method(...)` and `Arc\u003Cdyn Trait>` dispatch picks override-or-default for free. |\n| Defaulted, deprecated or vestigial                                                    | Document the skip in a `\u002F\u002F FFI omitted: …` comment.                                                                                   |\n| Defaulted, derived purely from other methods already plumbed                          | Skip — but call out the derivation in a comment.                                                                                      |\n\n**Do not use `Option\u003Cfn>` just because the underlying trait has a default.** `Arc\u003Cdyn Trait>` erases override-vs-default info, so the producer side cannot know whether to populate the slot. Always plumb the fn pointer; the wrapper body invokes the trait method and dynamic dispatch does the right thing.\n\n`Option\u003Cfn>` is used exactly once in the crate today: `FFI_TableProvider::supports_filters_pushdown`, gated by the `can_support_pushdown_filters: bool` argument to `FFI_TableProvider::new`. It is an exception, not a template. Reach for it only when (a) the producer's constructor takes an explicit capability flag and (b) skipping the FFI call is meaningfully cheaper than letting the trait default run on the producer side. Otherwise plumb the fn pointer unconditionally.\n\n### Known gaps to close\n\nTracked gaps live on GitHub under the [`ffi`](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues?q=is%3Aissue+is%3Aopen+label%3Affi) label — that query is the source of truth and stays current as issues are filed or closed. Treat new PRs in those areas as opportunities to fix the listed methods; treat new wrappers as required to avoid creating more. Each open issue names the specific wrapper, the missing methods, and the severity.\n\nQuick CLI list:\n\n```bash\ngh issue list --repo apache\u002Fdatafusion --label ffi --state open --limit 50\n```\n\nCommon severity classes seen on the label today:\n\n- **DML \u002F optimizer-relevant defaults silently lost** — e.g. `delete_from`\u002F`update`\u002F`truncate` on table providers, distribution \u002F ordering \u002F pushdown on execution plans, `value_from_stats` on aggregates. These demote producer capability on the foreign side.\n- **SQL surface area silently absent** — naming hooks (`display_name`, `schema_name`, `documentation`), null-handling and within-group clauses on UDAFs, etc.\n- **Performance regressions, not correctness** — e.g. `memoize` on partition evaluators.\n- **Open design questions** — none currently tracked. (Historical entry: whether `FFI_RecordBatchStream` needs `library_marker_id` — resolved no, it impls the trait directly on `FFI_X` with no `Foreign` adapter and no reverse `From`, so the marker has no consultation site. See `library_marker_id` rule in §\"Method coverage\".)\n\nWhen in doubt, open the label query; do not assume the list above is exhaustive. Wrappers without an open issue are **not** certified complete — re-enumerate the trait surface (see \"How to audit a wrapper's coverage\" below) whenever you audit one; upstream trait drift can introduce new defaulted methods at any time and silently re-open the silent-override-loss bug class.\n\nConversely, an open issue under the `ffi` label is a **claim of a gap, not proof of one.** Past audits have filed false positives by enumerating gaps from memory rather than from current source (e.g. #22335 claimed `size` missing on `FFI_GroupsAccumulator` when it had been plumbed since PR #14775). Before acting on a listed gap — opening a fix PR, re-citing it in a new audit, or extending the list — run the dual-grep audit below and confirm the field is actually absent. If the issue is a false positive, close it as `not planned`, link the `file:line` of the existing plumbing, and remove the corresponding bullet from this skill.\n\nWhen *closing* a gap, add the fn pointer unconditionally — the wrapper body calls the trait method on the inner `Arc\u003Cdyn Trait>` and Rust's dynamic dispatch picks the producer's override or falls back to the default. Use `Option\u003Cfn>` only when the new method also gains a corresponding capability flag on the producer's `new()`. Either way the layout changes, so the PR is an ABI break: mark `api change`, do not back-port to `branch-\u003Cmajor>`, and the workspace major bump in the next release makes the `version()` extern surface the change to consumers at load time.\n\n### How to audit a wrapper's coverage\n\nEvery audit — opening an issue, filing a fix PR, or re-confirming a listed gap — must compare two sides drawn from current source. Never enumerate either side from memory or from a prior audit; trait surface and FFI struct both drift.\n\n**Side A — trait defaults (what could go missing):**\n\n```bash\n# Find the trait definition\ngrep -rn \"pub trait X\" datafusion\u002F --include='*.rs'\n\n# Inspect for `fn method(...) { default_body }` — the body marks it as a default\n```\n\n**Side B — FFI wrapper coverage (what is already plumbed):**\n\n```bash\n# List every fn-pointer field on the FFI struct\ngrep -nE 'pub [a-z_]+: (unsafe )?extern \"C\" fn' datafusion\u002Fffi\u002Fsrc\u002F...\u002FX.rs\n```\n\nDiff Side A against Side B. Any claim of a gap — in an issue body, audit summary, or PR description — must cite `file:line` for **both** the trait default and the FFI struct line where the field is (or is not). An issue body with only one side cited is incomplete and likely a false positive; reject it pending a re-grep.\n\nIf a method's body is non-trivial, the consumer-side default is non-trivial too. Decide explicitly whether the FFI should let the consumer recompute the same default, or whether the producer's override is what should travel.\n\n## Type-bridging conventions\n\n- **Strings \u002F vecs** crossing the boundary: `stabby::string::String as SString`, `stabby::vec::Vec as SVec`. Native conversion is `Vec::into_iter().collect::\u003CSVec\u003C_>>()` and back.\n- **Optional \u002F fallible** values: use this crate's `FFI_Option\u003CT>` and `FFI_Result\u003CT>` (`src\u002Fffi_option.rs`), *not* stabby's, because ours do not require `T: IStable`.\n- **Schema \u002F arrays**: `WrappedSchema` (`src\u002Farrow_wrappers.rs`) wraps `FFI_ArrowSchema`. Never expose `FFI_ArrowSchema` directly.\n- **Logical `Expr` \u002F `LogicalPlan`**: serialize via `datafusion-proto` using the embedded `FFI_LogicalExtensionCodec`. Same for physical plans → `FFI_PhysicalExtensionCodec`.\n- **Enums** (`Volatility`, `TableType`, `InsertOp`, `TableProviderFilterPushDown`): `#[repr(u8)]`, with `From\u003CNative> for FFI_X` and `From\u003C&FFI_X> for Native`. Always write a round-trip unit test that exercises every variant.\n- **Errors**: every `FFI_X` method that can fail returns `FFI_Result\u003CT>`. Use the `sresult!`, `sresult_return!`, `df_result!` macros from `src\u002Futil.rs` — do not roll your own.\n- **Infallible trait methods**: if an FFI call can fail but the native trait cannot return the error, log the transport error before returning the trait's fallback (`None`, `false`, or a default). Never discard it with `.ok()`, `.unwrap_or_default()`, or equivalent.\n- **Owned FFI returns**: consume an owned `FFI_X` with `From\u003CFFI_X>` instead of converting through `&FFI_X` and cloning across the boundary. Keep the borrowed conversion's local marker fast path.\n\n## Async, sessions, and task context\n\n- Any async method becomes `unsafe extern \"C\" fn(...) -> FfiFuture\u003CFFI_Result\u003CT>>`. The wrapper body uses `async move { ... }.into_ffi()`. Store `Option\u003Ctokio::runtime::Handle>` in `PrivateData` so the producer side can re-enter its own runtime if needed.\n- Methods taking `&dyn Session` cross the boundary as `FFI_SessionRef`. On the consumer side, try `session.as_local()` first; only construct a `ForeignSession::try_from(&session)` if that returns `None`. See `scan_fn_wrapper` in `src\u002Ftable_provider.rs`.\n- Anything that needs to deserialize an `Expr` \u002F `LogicalPlan` needs a `TaskContext`. Threading a fresh `TaskContext` per call is wrong because new UDFs may have been registered since construction. Use `FFI_TaskContextProvider` (`src\u002Fexecution\u002Ftask_ctx_provider.rs`), which holds a `Weak` ref to a `TaskContextProvider`. If the weak ref is dead at call time, return a clear error — do not panic.\n\n## Memory model checklist for every new `FFI_X`\n\n- [ ] `private_data` is `Box::into_raw`-ed exactly once at construction.\n- [ ] Every constructor path (including `clone_fn_wrapper`) allocates a fresh `Box` for its own `private_data`.\n- [ ] `release_fn_wrapper` `Box::from_raw`s it and nulls the pointer.\n- [ ] `Drop` calls `release`.\n- [ ] No method touches `private_data` directly outside the producer side. Consumer-side methods on `ForeignX` use only the function pointers.\n- [ ] `library_marker_id` and `version` are populated in **every** constructor (including `clone`).\n- [ ] No method dereferences a pointer it did not check for nullness (debug-assert at minimum).\n\n## Quick PR-review checklist\n\nWhen reviewing a PR that touches `datafusion\u002Fffi\u002F`:\n\n1. **Trait coverage.** Pull up the underlying trait. List its methods. Confirm every non-defaulted method has a function pointer. For each *defaulted* method, ask whether a real-world producer would override it — if yes, the PR must either plumb it through as a plain `unsafe extern \"C\" fn` (let dynamic dispatch on `Arc\u003Cdyn Trait>` pick override-or-default) or explicitly justify the omission in a comment. Reserve `Option\u003Cfn>` for the capability-flag pattern described in §\"Method coverage\" — do not use it just because the trait has a default.\n2. **Layout fields.** `clone`, `release`, `version`, `private_data` all present? `library_marker_id` present **iff** wrapper has a `ForeignX` adapter and a reverse `From\u003C&?FFI_X> for {Arc,Box}\u003Cdyn X>` consultation site (Arc-backed for shareable traits, Box-backed for `&mut self` traits); if the trait is impl'd directly on `FFI_X` (no `ForeignX`, no reverse `From`), `library_marker_id` is dead surface and must be omitted with a one-line rationale.\n3. **Marker-id bypass** in `From\u003C&FFI_X>` (where applicable per rule 2)?\n4. **Round-trip downcast** in the constructor?\n5. **`Drop` calling `release`**, and `release` nulling the pointer?\n6. **`Send`\u002F`Sync` unsafe impls** match the wrapped trait's bounds (rule 4), and `FFI_X` + `ForeignX` agree?\n7. **Stabby types** for strings\u002Fvecs; crate-local `FFI_Option`\u002F`FFI_Result` for optional\u002Ffallible payloads?\n8. **Async** uses `FfiFuture` + `.into_ffi()`, never blocking?\n9. **Codec** present on any method that ships an `Expr` \u002F plan?\n10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `integration-tests\u002Ftests\u002F` exist for any wrapper that takes\u002Freturns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests` must pass before merging an ABI-affecting PR.\n11. **No `datafusion` runtime dep** crept into `Cargo.toml`?\n12. **`Arc::clone(&x)`** everywhere — no implicit `x.clone()` on `Arc` (the lint will reject it but worth pre-flagging).\n13. **`cargo clippy --all-targets --all-features -- -D warnings`** clean on the crate?\n14. **`api change` label** on the PR if any `FFI_X` struct layout \u002F fn-ptr signature \u002F FFI enum \u002F `version` extern changed? Block merge until applied.\n15. **Base branch check.** If FFI struct layout changed, base branch must be `main`, never a release branch matching `^branch-\\d+$` (e.g. `branch-53`). Reject back-ports of ABI-breaking changes to patch-release branches. Verify with `gh pr view \u003Cnum> --json baseRefName,labels --jq '.baseRefName'`; match strictly against `^branch-\\d+$` — cherry-pick working branches like `branch-53-cherry-pick-1` also start with `branch-` and must not false-positive.\n\n## References\n\n- Crate README: `datafusion\u002Fffi\u002FREADME.md` — vocabulary + memory-model rationale.\n- Canonical wrapper to model after: `src\u002Fcatalog_provider.rs`. Async + capability-flag variants: `src\u002Ftable_provider.rs`.\n- Mutable-trait variant: `src\u002Fudaf\u002Faccumulator.rs` (`Box\u003Cdyn Accumulator>`).\n- Optional-method pattern: `FFI_TableProvider::supports_filters_pushdown`.\n- Codec wiring: `src\u002Fproto\u002Flogical_extension_codec.rs`, `src\u002Fproto\u002Fphysical_extension_codec.rs`.\n- Examples crate: `datafusion-examples\u002Fexamples\u002Fffi` (end-to-end producer + consumer).\n","# DataFusion FFI Skill\n\nThis crate exposes a stable C ABI for DataFusion traits so that independently-compiled libraries (different Rust versions, plugins, `datafusion-python`, etc.) can interoperate at runtime. Stability and correctness here are load-bearing: a missed pattern can cause segfaults, leaks, or silently dropped trait behavior on the consumer side.\n\nRead the crate's `README.md` first if you have not — it establishes the vocabulary (`FFI_X` \u002F `ForeignX`, `library_marker_id`, `release`, `TaskContextProvider`, stabby vs `#[repr(C)]`).\n\n## When to use\n\nTrigger this skill any time the work touches `datafusion\u002Fffi\u002F`:\n\n- Adding a new `FFI_\u003CTrait>` + `Foreign\u003CTrait>` pair\n- Adding a method to an existing `FFI_X` struct\n- Reviewing a PR that touches this crate\n- Changing the codec \u002F proto serialization layer\n- Bumping the wrapped DataFusion trait surface (e.g. a new default method appeared upstream)\n\n## Hard rules\n\n1. **No `datafusion` dependency.** `datafusion-ffi` must not depend on the umbrella `datafusion` crate. Use the leaf crates (`datafusion-common`, `datafusion-expr`, `datafusion-catalog`, `datafusion-physical-plan`, etc.). `datafusion` is fine in `[dev-dependencies]`.\n2. **`#[repr(C)]` on every `FFI_X` struct**, not `#[stabby::stabby]`. Stabby is used for `SString`\u002F`SVec` only. Reasons documented in the README (build time, Arrow types lack `IStable`).\n3. **`unsafe extern \"C\"` on every function-pointer field — including `version`.** The one exception is the `library_marker_id` field, which is plain `extern \"C\" fn() -> usize`. Plain (safe) `extern \"C\"` also applies to the standalone function defs in `src\u002Flib.rs` — `pub extern \"C\" fn version()` and `pub extern \"C\" fn get_library_marker_id()` — which coerce into the `unsafe extern \"C\"` `version` field slot at construction.\n4. **Match `Send`\u002F`Sync` to the wrapped trait.** Raw `*mut c_void` makes every `FFI_X` `!Send + !Sync` by default — `unsafe impl` whichever bounds the consumer-facing trait requires. Most DataFusion traits (`TableProvider`, `ExecutionPlan`, all UDFs, codecs) need both. `Send`-only: `RecordBatchStream`, `Accumulator`, `GroupsAccumulator`, `PartitionEvaluator` (mutable \u002F stream APIs). The matching `ForeignX` always carries the same bounds — pick consistently.\n5. **`#![deny(clippy::clone_on_ref_ptr)]`** is on at the crate root. Use `Arc::clone(&x)`, never `x.clone()` on `Arc`.\n6. **Run before pushing:** `cargo fmt --all`, `cargo clippy -p datafusion-ffi --all-targets --all-features -- -D warnings`, `cargo test -p datafusion-ffi`.\n7. **`api change` label required.** Any PR that modifies an `FFI_X` struct layout (adds\u002Fremoves\u002Freorders fields, changes a function-pointer signature, adds a variant to an FFI enum, or changes the `version` extern) must carry the `api change` GitHub label. Layout changes break ABI for already-compiled consumer libraries. The label is the project-wide convention for highlighting breaking public-API changes in release notes — see `docs\u002Fsource\u002Fcontributor-guide\u002Fapi-health.md` §\"What to do when making breaking API changes?\" (step 1 names the label explicitly). Downstream users (e.g. `datafusion-python`, plugin authors) read the labelled notes to know they must recompile against the new DataFusion major. The `version()` extern in `src\u002Flib.rs` returns the major of workspace `CARGO_PKG_VERSION`; consumers compare it at load time and can refuse mismatched producers. Apply via `gh pr edit \u003Cnum> --add-label \"api change\"` (label name contains a space — must be quoted). When reviewing such a PR, block merge until label present.\n8. **No FFI struct changes in patch releases.** Patch releases ship from branches matching `^branch-\\d+$` (e.g. `branch-53`, `branch-52`). FFI struct layout changes (anything that would earn rule 7's `api change` label) **must not** target a release branch and must not be back-ported. Patch releases are ABI-stable by contract — a consumer compiled against `53.1.0` must keep working against `53.1.1`. Before reviewing\u002Fapproving an FFI PR, check the PR's base branch: if it matches the regex above, or the PR description \u002F labels indicate patch \u002F back-port, reject the FFI struct change and ask the author to retarget `main`. Bugfixes that do not alter struct layout (e.g. fixing a function-pointer body) are fine to back-port. Quick check: `gh pr view \u003Cnum> --json baseRefName,labels --jq '.baseRefName'` then match against `^branch-\\d+$`. Do **not** glob-match `branch-*` — back-port \u002F cherry-pick working branches (e.g. `branch-53-cherry-pick-1`) also share that prefix but are not release branches; only the strict `branch-\u003Cdigits>` form is the freeze target.\n\n## The standard wrapper shape\n\nA new `FFI_X` for trait `X` must follow this template. Use `FFI_CatalogProvider` (`src\u002Fcatalog_provider.rs`) as the canonical reference — it shows the full shape (codec field, nested FFI types, `FFI_Option`\u002F`FFI_Result` returns, Arc-backed `PrivateData`) without async or capability-flag noise. `FFI_TableProvider` (`src\u002Ftable_provider.rs`) covers async (`scan`, `FFI_SessionRef`, `FfiFuture`) and the one `Option\u003Cfn>` capability flag (`supports_filters_pushdown`).\n\n### 1. The `FFI_X` struct\n\n```rust\n#[repr(C)]\n#[derive(Debug)]\npub struct FFI_X {\n    some_method: unsafe extern \"C\" fn(this: &Self, ...) -> FFI_Result\u003C...>,\n    optional_method: Option\u003Cunsafe extern \"C\" fn(&Self, ...) -> FFI_Result\u003C...>>,\n    pub logical_codec: FFI_LogicalExtensionCodec,\n\n    clone: unsafe extern \"C\" fn(&Self) -> Self,\n    release: unsafe extern \"C\" fn(&mut Self),\n    pub version: unsafe extern \"C\" fn() -> u64,\n\n    private_data: *mut c_void,\n    pub library_marker_id: extern \"C\" fn() -> usize,\n}\n\nunsafe impl Send for FFI_X {}\nunsafe impl Sync for FFI_X {}\n```\n\nField rules:\n\n- **One `unsafe extern \"C\" fn` per trait method.** Always populate — `Arc\u003Cdyn Trait>` dispatch picks override-or-default at call time, so the producer side gets the right answer without the consumer needing to know. See § \"Method coverage\".\n- **`Option\u003Cfn>` is the capability-flag exception**, not a template. Crate uses it exactly once: `FFI_TableProvider::supports_filters_pushdown`. See § \"Method coverage\".\n- **Codec field** (`FFI_LogicalExtensionCodec` \u002F `FFI_PhysicalExtensionCodec`) only if the trait moves `Expr`s \u002F `LogicalPlan`s \u002F `ExecutionPlan`s across the boundary.\n- **Method function pointers are private by default.** Mark `pub` only if a downstream library needs to invoke them directly (rare — typically only `version`, `library_marker_id`, embedded codecs are `pub`).\n- **`version: super::version` is mandatory.** Consumers gate compatibility on it.\n- **`library_marker_id: crate::get_library_marker_id` is mandatory *when the wrapper uses the standard `ForeignX` adapter pattern*.** Two flavors exist:\n  - **Arc-backed (immutable \u002F shareable traits — `TableProvider`, `ExecutionPlan`, all UDFs, codecs):** consumer-side `From\u003C&FFI_X> for Arc\u003Cdyn X>` consults the marker to choose `Arc::clone(inner)` vs `Arc::new(ForeignX(...))`.\n  - **Box-backed (mutable \u002F move-only traits — `Accumulator`, `GroupsAccumulator`, `PartitionEvaluator`):** consumer-side `From\u003CFFI_X> for Box\u003Cdyn X>` consults the marker to take the inner `Box` directly vs `Box::new(ForeignX(...))`. Producer-side `From\u003CBox\u003Cdyn X>> for FFI_X` *also* uses an `is::\u003CForeignX>()` downcast as an additional re-wrap bypass; the marker check covers the reverse direction.\n\n  The field is dead ABI surface — and must be omitted with a one-line module-doc rationale — only when **neither** flavor applies: no `ForeignX` adapter, no reverse `From\u003CFFI_X> for {Arc,Box}\u003Cdyn X>`, and the trait is impl'd directly on `FFI_X`. Canonical example: `FFI_RecordBatchStream` (`impl RecordBatchStream for FFI_RecordBatchStream` at `record_batch_stream.rs:149`, no `ForeignRecordBatchStream`, no reverse `From`).\n\n  Before flagging a missing `library_marker_id` as a gap, run all three greps on the wrapper file: `Foreign\u003Cwrapper-name>`, `From\u003C&?FFI_X> for Arc\u003C`, `From\u003CFFI_X> for Box\u003C`. Hit on any → marker is required and its absence is a real gap. Zero hits on all three → marker is intentionally not needed, not a gap.\n- **`Send`\u002F`Sync` bounds match the wrapped trait** (rule 4). Most wrappers want both; `Send`-only for streams \u002F mutable traits.\n\n### 2. `PrivateData` shape\n\nDefault — for read-only, shareable traits — use `Arc`:\n\n```rust\nstruct XPrivateData {\n    inner: Arc\u003Cdyn X>,\n    runtime: Option\u003CHandle>, \u002F\u002F include when any async method exists\n}\n```\n\nFor traits that require `&mut self` (e.g. `Accumulator`, `GroupsAccumulator`, `PartitionEvaluator`), use `Box\u003Cdyn X>`:\n\n```rust\nstruct XPrivateData {\n    inner: Box\u003Cdyn X>,\n    runtime: Option\u003CHandle>, \u002F\u002F include when any async method exists\n}\n```\n\nA `Box`-backed `FFI_X` **cannot implement `Clone`**; document this and skip the `clone` function pointer, or hand-write a release path that distinguishes producer vs consumer side. Canonical example: `FFI_Accumulator` in `src\u002Fudaf\u002Faccumulator.rs`. See also `FFI_GroupsAccumulator` (`src\u002Fudaf\u002Fgroups_accumulator.rs`) and `FFI_PartitionEvaluator` (`src\u002Fudwf\u002Fpartition_evaluator.rs`).\n\n### 3. Function-pointer wrappers\n\nNaming convention: `\u003Cmethod>_fn_wrapper`.\n\n```rust\nunsafe extern \"C\" fn some_method_fn_wrapper(this: &FFI_X, ...) -> FFI_Result\u003CT> {\n    \u002F\u002F 1. Recover inner via this.inner()\n    \u002F\u002F 2. Translate FFI types → native types\n    \u002F\u002F 3. Call native method\n    \u002F\u002F 4. Translate native Result → FFI_Result via sresult_return! or .into()\n}\n```\n\n### 4. `clone` \u002F `release` \u002F `Drop`\n\n```rust\nunsafe extern \"C\" fn clone_fn_wrapper(this: &FFI_X) -> FFI_X { \u002F* re-Box new private_data, copy fn ptrs *\u002F }\n\nunsafe extern \"C\" fn release_fn_wrapper(this: &mut FFI_X) {\n    unsafe {\n        debug_assert!(!this.private_data.is_null());\n        drop(Box::from_raw(this.private_data as *mut XPrivateData));\n        this.private_data = std::ptr::null_mut();\n    }\n}\n\nimpl Drop for FFI_X { fn drop(&mut self) { unsafe { (self.release)(self) } } }\nimpl Clone for FFI_X { fn clone(&self) -> Self { unsafe { (self.clone)(self) } } }\n```\n\n`release` must null `private_data` so a double-free debug-asserts loudly.\n\n### 5. Constructor split\n\n```rust\nimpl FFI_X {\n    pub fn new(inner: Arc\u003Cdyn X>, runtime: Option\u003CHandle>,\n               task_ctx_provider: impl Into\u003CFFI_TaskContextProvider>,\n               logical_codec: Option\u003CArc\u003Cdyn LogicalExtensionCodec>>) -> Self {\n        \u002F\u002F build FFI_LogicalExtensionCodec from defaults, then forward\n        Self::new_with_ffi_codec(inner, runtime, ffi_codec)\n    }\n\n    pub fn new_with_ffi_codec(inner: Arc\u003Cdyn X>, runtime: Option\u003CHandle>,\n                              logical_codec: FFI_LogicalExtensionCodec) -> Self {\n        \u002F\u002F Round-trip downcast: if inner is already a ForeignX, return its FFI directly.\n        if let Some(foreign) = inner.downcast_ref::\u003CForeignX>() {\n            return foreign.0.clone();\n        }\n        \u002F\u002F …allocate XPrivateData and populate fn ptrs…\n    }\n}\n```\n\nThe round-trip downcast is **mandatory** — without it, repeated FFI hops nest `ForeignX(FFI_X(ForeignX(...)))` and you pay the boundary cost every layer.\n\n### 6. The `Foreign\u003CX>` consumer\n\n```rust\n#[derive(Debug)]\npub struct ForeignX(pub FFI_X);\nunsafe impl Send for ForeignX {}\nunsafe impl Sync for ForeignX {}\n\nimpl From\u003C&FFI_X> for Arc\u003Cdyn X> {\n    fn from(p: &FFI_X) -> Self {\n        if (p.library_marker_id)() == crate::get_library_marker_id() {\n            Arc::clone(unsafe { p.inner() })\n        } else {\n            Arc::new(ForeignX(p.clone()))\n        }\n    }\n}\n\nimpl X for ForeignX { \u002F* call each fn pointer, translate types back *\u002F }\n```\n\nThe marker-id check is **mandatory** for every `From\u003C&FFI_X> for Arc\u003Cdyn X>`. Skipping it breaks the local-bypass optimization and forces the producer's data through serialization.\n\n### 7. Tests\n\nThe crate has **two distinct test surfaces** and a new wrapper usually needs entries in both. They are not interchangeable; they catch different classes of bug.\n\n#### a. In-process unit tests (`#[cfg(test)] mod tests` inside `src\u002F\u003Cmod>.rs`)\n\nRun on every `cargo test -p datafusion-ffi`. Producer and consumer live in the same compilation unit, so `library_marker_id` returns the same value on both sides. To force the foreign path you must override the marker:\n\n```rust\nlet mut ffi_x = FFI_X::new(provider, …);\nffi_x.library_marker_id = crate::mock_foreign_marker_id; \u002F\u002F forces the ForeignX branch\nlet arc: Arc\u003Cdyn X> = (&ffi_x).into();\nassert!(arc.downcast_ref::\u003CForeignX>().is_some());\n```\n\nEvery wrapper must include at minimum:\n\n- A **local-bypass test** — build `FFI_X` from a concrete native type, convert to `Arc\u003Cdyn X>`, `downcast_ref::\u003CConcreteType>()` must succeed.\n- A **forced-foreign test** — set `library_marker_id = crate::mock_foreign_marker_id`, convert, `downcast_ref::\u003CForeignX>()` must succeed, then exercise every method end-to-end.\n\nTemplates: `test_ffi_table_provider_local_bypass` and `test_round_trip_ffi_table_provider_scan` in `src\u002Ftable_provider.rs`.\n\nWhat unit tests catch: Rust-level correctness (translation logic, lifetime bugs, leaks under valgrind\u002Fmiri, Send\u002FSync, error propagation, codec round-trips). What they **cannot** catch: real ABI bugs. Both producer and consumer share `#[repr(C)]` layout because they are the exact same struct definition in memory.\n\n#### b. Cross-library integration tests (`integration-tests\u002Ftests\u002F`, gated by the `integration-tests` feature)\n\nThe integration tests live in the unpublished `datafusion-ffi-integration-tests` crate in `datafusion\u002Fffi\u002Fintegration-tests\u002F`, so that `datafusion-ffi` itself is a plain rlib. That crate is built as a `cdylib` that exports `datafusion_ffi_get_module` — a `#[unsafe(no_mangle)] extern \"C\"` entry point defined in `src\u002Ftests\u002Fmod.rs` and gated by `#[cfg(feature = \"integration-tests\")]`. Its tests use `libloading` to `dlopen` that `cdylib`. The test executable links against the `datafusion-ffi` rlib (consumer side); the dlopen'd cdylib is the producer side. Even though both are built from the same source, they are independent compilation outputs going through the actual FFI symbol path.\n\nRun with:\n\n```bash\ncargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests\n```\n\nTo add coverage for a new wrapper:\n\n1. **Add a constructor** in `src\u002Ftests\u002F\u003Carea>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.\n2. **Wire it into `ForeignLibraryModule`** in `src\u002Ftests\u002Fmod.rs`: add a field of type `extern \"C\" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.\n3. **Add the test** in `integration-tests\u002Ftests\u002Fcases\u002F\u003Carea>.rs`, registered in `cases\u002Fmod.rs`, under `mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc\u003Cdyn X>`, and exercise every method.\n\nWhen adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.\n\nWhat integration tests catch that unit tests cannot:\n\n- **Real ABI layout bugs.** Two builds means the consumer's view of `FFI_X` is reconstructed from declaration, not aliased to the producer's memory. Mismatched alignment, padding, niche optimization, or accidentally non-`#[repr(C)]` types surface here.\n- **Symbol visibility \u002F `no_mangle`** issues.\n- **`library_marker_id` correctness without mocking** — the two libraries genuinely have different statics, so the foreign branch is taken for real.\n- **Drop \u002F leak ordering** when the producer side is in a `dlopen`'d image.\n\n#### Which tests does my change need?\n\n| Change                                                                | Unit | Integration |\n| --------------------------------------------------------------------- | ---- | ----------- |\n| New `FFI_X` wrapper                                                   | Yes  | Yes         |\n| New method on existing `FFI_X`                                        | Yes  | Yes if the method takes\u002Freturns a non-trivial FFI type. See note below. |\n| Bugfix to a wrapper body, no signature change                         | Yes  | Only if reproducing the bug requires cross-library symbol lookup or `dlopen` semantics |\n| Layout change (`#[repr(C)]` field add\u002Fremove\u002Freorder, fn-ptr sig)     | Yes  | **Mandatory** — this is exactly the bug class integration tests exist for |\n| New `From\u003CX> for FFI_X` or codec change                               | Yes  | Yes if the codec is exercised by the cross-library round-trip |\n\n**\"Non-trivial FFI type\" for the table above** — anything other than:\n\n- Primitives (`u8`\u002F`u64`\u002F`bool`\u002F`usize`, etc.) and `#[repr(u8)]` FFI enums (`FFI_TableType`, `Volatility`, `InsertOp`, `TableProviderFilterPushDown`).\n- A `stabby::string::String` (`SString`) returned by value, with no other args or returns.\n\nConcrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec\u003CT>`, `FFI_Option\u003CT>`, `FFI_Result\u003CT>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`\u002F`*const` pointer. These exercise alignment \u002F padding \u002F niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.\n\nIf you skip the integration test for a layout change, you have effectively shipped untested ABI.\n\n## Method coverage — the silent-default gap\n\n**This is the area where the crate currently has real holes.** When the wrapped trait has methods with *default implementations*, those defaults are typically the trait's \"no-op \u002F unsupported\" answer (`None`, `false`, `Unsupported`, `not_impl_err!()`). If the producer overrides a default but the FFI struct does not carry a function pointer for it, the consumer's `Foreign\u003CX>` falls back to the trait default — **silently losing the override**. The producer thinks it implemented `delete_from`; the consumer behaves as if it never did.\n\n### Rule\n\nWhen adding or auditing an `FFI_X`, **enumerate every method on the wrapped trait, including defaulted ones**, and for each one decide:\n\n| Category                                                                              | Action                                                                                                                                |\n| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |\n| Required method (no default)                                                          | Mandatory `unsafe extern \"C\" fn` field.                                                                                               |\n| Defaulted, plausible override (statistics, distribution, ordering, simplify, DML, …)  | Mandatory `unsafe extern \"C\" fn` field — same as a required method. The wrapper body calls `inner.method(...)` and `Arc\u003Cdyn Trait>` dispatch picks override-or-default for free. |\n| Defaulted, deprecated or vestigial                                                    | Document the skip in a `\u002F\u002F FFI omitted: …` comment.                                                                                   |\n| Defaulted, derived purely from other methods already plumbed                          | Skip — but call out the derivation in a comment.                                                                                      |\n\n**Do not use `Option\u003Cfn>` just because the underlying trait has a default.** `Arc\u003Cdyn Trait>` erases override-vs-default info, so the producer side cannot know whether to populate the slot. Always plumb the fn pointer; the wrapper body invokes the trait method and dynamic dispatch does the right thing.\n\n`Option\u003Cfn>` is used exactly once in the crate today: `FFI_TableProvider::supports_filters_pushdown`, gated by the `can_support_pushdown_filters: bool` argument to `FFI_TableProvider::new`. It is an exception, not a template. Reach for it only when (a) the producer's constructor takes an explicit capability flag and (b) skipping the FFI call is meaningfully cheaper than letting the trait default run on the producer side. Otherwise plumb the fn pointer unconditionally.\n\n### Known gaps to close\n\nTracked gaps live on GitHub under the [`ffi`](https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fissues?q=is%3Aissue+is%3Aopen+label%3Affi) label — that query is the source of truth and stays current as issues are filed or closed. Treat new PRs in those areas as opportunities to fix the listed methods; treat new wrappers as required to avoid creating more. Each open issue names the specific wrapper, the missing methods, and the severity.\n\nQuick CLI list:\n\n```bash\ngh issue list --repo apache\u002Fdatafusion --label ffi --state open --limit 50\n```\n\nCommon severity classes seen on the label today:\n\n- **DML \u002F optimizer-relevant defaults silently lost** — e.g. `delete_from`\u002F`update`\u002F`truncate` on table providers, distribution \u002F ordering \u002F pushdown on execution plans, `value_from_stats` on aggregates. These demote producer capability on the foreign side.\n- **SQL surface area silently absent** — naming hooks (`display_name`, `schema_name`, `documentation`), null-handling and within-group clauses on UDAFs, etc.\n- **Performance regressions, not correctness** — e.g. `memoize` on partition evaluators.\n- **Open design questions** — none currently tracked. (Historical entry: whether `FFI_RecordBatchStream` needs `library_marker_id` — resolved no, it impls the trait directly on `FFI_X` with no `Foreign` adapter and no reverse `From`, so the marker has no consultation site. See `library_marker_id` rule in §\"Method coverage\".)\n\nWhen in doubt, open the label query; do not assume the list above is exhaustive. Wrappers without an open issue are **not** certified complete — re-enumerate the trait surface (see \"How to audit a wrapper's coverage\" below) whenever you audit one; upstream trait drift can introduce new defaulted methods at any time and silently re-open the silent-override-loss bug class.\n\nConversely, an open issue under the `ffi` label is a **claim of a gap, not proof of one.** Past audits have filed false positives by enumerating gaps from memory rather than from current source (e.g. #22335 claimed `size` missing on `FFI_GroupsAccumulator` when it had been plumbed since PR #14775). Before acting on a listed gap — opening a fix PR, re-citing it in a new audit, or extending the list — run the dual-grep audit below and confirm the field is actually absent. If the issue is a false positive, close it as `not planned`, link the `file:line` of the existing plumbing, and remove the corresponding bullet from this skill.\n\nWhen *closing* a gap, add the fn pointer unconditionally — the wrapper body calls the trait method on the inner `Arc\u003Cdyn Trait>` and Rust's dynamic dispatch picks the producer's override or falls back to the default. Use `Option\u003Cfn>` only when the new method also gains a corresponding capability flag on the producer's `new()`. Either way the layout changes, so the PR is an ABI break: mark `api change`, do not back-port to `branch-\u003Cmajor>`, and the workspace major bump in the next release makes the `version()` extern surface the change to consumers at load time.\n\n### How to audit a wrapper's coverage\n\nEvery audit — opening an issue, filing a fix PR, or re-confirming a listed gap — must compare two sides drawn from current source. Never enumerate either side from memory or from a prior audit; trait surface and FFI struct both drift.\n\n**Side A — trait defaults (what could go missing):**\n\n```bash\n# Find the trait definition\ngrep -rn \"pub trait X\" datafusion\u002F --include='*.rs'\n\n# Inspect for `fn method(...) { default_body }` — the body marks it as a default\n```\n\n**Side B — FFI wrapper coverage (what is already plumbed):**\n\n```bash\n# List every fn-pointer field on the FFI struct\ngrep -nE 'pub [a-z_]+: (unsafe )?extern \"C\" fn' datafusion\u002Fffi\u002Fsrc\u002F...\u002FX.rs\n```\n\nDiff Side A against Side B. Any claim of a gap — in an issue body, audit summary, or PR description — must cite `file:line` for **both** the trait default and the FFI struct line where the field is (or is not). An issue body with only one side cited is incomplete and likely a false positive; reject it pending a re-grep.\n\nIf a method's body is non-trivial, the consumer-side default is non-trivial too. Decide explicitly whether the FFI should let the consumer recompute the same default, or whether the producer's override is what should travel.\n\n## Type-bridging conventions\n\n- **Strings \u002F vecs** crossing the boundary: `stabby::string::String as SString`, `stabby::vec::Vec as SVec`. Native conversion is `Vec::into_iter().collect::\u003CSVec\u003C_>>()` and back.\n- **Optional \u002F fallible** values: use this crate's `FFI_Option\u003CT>` and `FFI_Result\u003CT>` (`src\u002Fffi_option.rs`), *not* stabby's, because ours do not require `T: IStable`.\n- **Schema \u002F arrays**: `WrappedSchema` (`src\u002Farrow_wrappers.rs`) wraps `FFI_ArrowSchema`. Never expose `FFI_ArrowSchema` directly.\n- **Logical `Expr` \u002F `LogicalPlan`**: serialize via `datafusion-proto` using the embedded `FFI_LogicalExtensionCodec`. Same for physical plans → `FFI_PhysicalExtensionCodec`.\n- **Enums** (`Volatility`, `TableType`, `InsertOp`, `TableProviderFilterPushDown`): `#[repr(u8)]`, with `From\u003CNative> for FFI_X` and `From\u003C&FFI_X> for Native`. Always write a round-trip unit test that exercises every variant.\n- **Errors**: every `FFI_X` method that can fail returns `FFI_Result\u003CT>`. Use the `sresult!`, `sresult_return!`, `df_result!` macros from `src\u002Futil.rs` — do not roll your own.\n- **Infallible trait methods**: if an FFI call can fail but the native trait cannot return the error, log the transport error before returning the trait's fallback (`None`, `false`, or a default). Never discard it with `.ok()`, `.unwrap_or_default()`, or equivalent.\n- **Owned FFI returns**: consume an owned `FFI_X` with `From\u003CFFI_X>` instead of converting through `&FFI_X` and cloning across the boundary. Keep the borrowed conversion's local marker fast path.\n\n## Async, sessions, and task context\n\n- Any async method becomes `unsafe extern \"C\" fn(...) -> FfiFuture\u003CFFI_Result\u003CT>>`. The wrapper body uses `async move { ... }.into_ffi()`. Store `Option\u003Ctokio::runtime::Handle>` in `PrivateData` so the producer side can re-enter its own runtime if needed.\n- Methods taking `&dyn Session` cross the boundary as `FFI_SessionRef`. On the consumer side, try `session.as_local()` first; only construct a `ForeignSession::try_from(&session)` if that returns `None`. See `scan_fn_wrapper` in `src\u002Ftable_provider.rs`.\n- Anything that needs to deserialize an `Expr` \u002F `LogicalPlan` needs a `TaskContext`. Threading a fresh `TaskContext` per call is wrong because new UDFs may have been registered since construction. Use `FFI_TaskContextProvider` (`src\u002Fexecution\u002Ftask_ctx_provider.rs`), which holds a `Weak` ref to a `TaskContextProvider`. If the weak ref is dead at call time, return a clear error — do not panic.\n\n## Memory model checklist for every new `FFI_X`\n\n- [ ] `private_data` is `Box::into_raw`-ed exactly once at construction.\n- [ ] Every constructor path (including `clone_fn_wrapper`) allocates a fresh `Box` for its own `private_data`.\n- [ ] `release_fn_wrapper` `Box::from_raw`s it and nulls the pointer.\n- [ ] `Drop` calls `release`.\n- [ ] No method touches `private_data` directly outside the producer side. Consumer-side methods on `ForeignX` use only the function pointers.\n- [ ] `library_marker_id` and `version` are populated in **every** constructor (including `clone`).\n- [ ] No method dereferences a pointer it did not check for nullness (debug-assert at minimum).\n\n## Quick PR-review checklist\n\nWhen reviewing a PR that touches `datafusion\u002Fffi\u002F`:\n\n1. **Trait coverage.** Pull up the underlying trait. List its methods. Confirm every non-defaulted method has a function pointer. For each *defaulted* method, ask whether a real-world producer would override it — if yes, the PR must either plumb it through as a plain `unsafe extern \"C\" fn` (let dynamic dispatch on `Arc\u003Cdyn Trait>` pick override-or-default) or explicitly justify the omission in a comment. Reserve `Option\u003Cfn>` for the capability-flag pattern described in §\"Method coverage\" — do not use it just because the trait has a default.\n2. **Layout fields.** `clone`, `release`, `version`, `private_data` all present? `library_marker_id` present **iff** wrapper has a `ForeignX` adapter and a reverse `From\u003C&?FFI_X> for {Arc,Box}\u003Cdyn X>` consultation site (Arc-backed for shareable traits, Box-backed for `&mut self` traits); if the trait is impl'd directly on `FFI_X` (no `ForeignX`, no reverse `From`), `library_marker_id` is dead surface and must be omitted with a one-line rationale.\n3. **Marker-id bypass** in `From\u003C&FFI_X>` (where applicable per rule 2)?\n4. **Round-trip downcast** in the constructor?\n5. **`Drop` calling `release`**, and `release` nulling the pointer?\n6. **`Send`\u002F`Sync` unsafe impls** match the wrapped trait's bounds (rule 4), and `FFI_X` + `ForeignX` agree?\n7. **Stabby types** for strings\u002Fvecs; crate-local `FFI_Option`\u002F`FFI_Result` for optional\u002Ffallible payloads?\n8. **Async** uses `FfiFuture` + `.into_ffi()`, never blocking?\n9. **Codec** present on any method that ships an `Expr` \u002F plan?\n10. **Unit tests** include both local-bypass and `mock_foreign_marker_id` forced-foreign cases? **Integration tests** in `integration-tests\u002Ftests\u002F` exist for any wrapper that takes\u002Freturns non-trivial FFI types, and for *every* layout change? `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests` must pass before merging an ABI-affecting PR.\n11. **No `datafusion` runtime dep** crept into `Cargo.toml`?\n12. **`Arc::clone(&x)`** everywhere — no implicit `x.clone()` on `Arc` (the lint will reject it but worth pre-flagging).\n13. **`cargo clippy --all-targets --all-features -- -D warnings`** clean on the crate?\n14. **`api change` label** on the PR if any `FFI_X` struct layout \u002F fn-ptr signature \u002F FFI enum \u002F `version` extern changed? Block merge until applied.\n15. **Base branch check.** If FFI struct layout changed, base branch must be `main`, never a release branch matching `^branch-\\d+$` (e.g. `branch-53`). Reject back-ports of ABI-breaking changes to patch-release branches. Verify with `gh pr view \u003Cnum> --json baseRefName,labels --jq '.baseRefName'`; match strictly against `^branch-\\d+$` — cherry-pick working branches like `branch-53-cherry-pick-1` also start with `branch-` and must not false-positive.\n\n## References\n\n- Crate README: `datafusion\u002Fffi\u002FREADME.md` — vocabulary + memory-model rationale.\n- Canonical wrapper to model after: `src\u002Fcatalog_provider.rs`. Async + capability-flag variants: `src\u002Ftable_provider.rs`.\n- Mutable-trait variant: `src\u002Fudaf\u002Faccumulator.rs` (`Box\u003Cdyn Accumulator>`).\n- Optional-method pattern: `FFI_TableProvider::supports_filters_pushdown`.\n- Codec wiring: `src\u002Fproto\u002Flogical_extension_codec.rs`, `src\u002Fproto\u002Fphysical_extension_codec.rs`.\n- Examples crate: `datafusion-examples\u002Fexamples\u002Fffi` (end-to-end producer + consumer).\n",{"name":5,"description":7},"main","5db4b233efde58f6b34299fb15b756e7bdef2698",".ai\u002Fskills\u002Fdatafusion-ffi",2649,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion","https:\u002F\u002Fdatafusion.apache.org\u002F","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-15 03:08:42",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add apache\u002Fdatafusion --skill datafusion-ffi",{"tool":59,"command":60},"bunx","bunx skills add apache\u002Fdatafusion --skill datafusion-ffi",{"tool":62,"command":63},"pnpm","pnpm dlx skills add apache\u002Fdatafusion --skill datafusion-ffi","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fdatafusion-ffi\u002FSKILL.md",true,[67,73],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"scripts\u002Fget_os.py","SCRIPT","脚本",271,"https:\u002F\u002Fgithub.com\u002Fbylatt\u002Fdotfiles\u002Fblob\u002Fmaster\u002Fdot-config\u002Fopencode\u002Fskills\u002Fget-os-info\u002Fscripts\u002Fget_os.py",{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"TEMPLATE.md","REFERENCE","参考文档",588,"https:\u002F\u002Fgithub.com\u002Fbylatt\u002Fdotfiles\u002Fblob\u002Fmaster\u002Fdot-config\u002Fopencode\u002Fskills\u002Fget-os-info\u002FTEMPLATE.md",[80,97],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":39},"01M6ACDJXN4NB50439K7E7KKMP","pr_review","Review Apache DataFusion pull requests following the project's PR review guide. Use whenever asked to review a DataFusion PR or PR URL, and whenever creating a PR, to check the changes against the same criteria before submitting.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fpr_review\u002FSKILL.md",[],[32,87,88,89,27,90,91,92,93,36,94,95,19,9,96],"pr review","pr","pull","requests","URL","project","review guide","guide","check","pull requests",{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":39},"01M6ACDA8M3Z9W6BXCXVHZSB4M","add-benchmarks","Guidelines for designing or extending DataFusion benchmarks.","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion\u002Fblob\u002Fmain\u002F.ai\u002Fskills\u002Fadd-benchmarks\u002FSKILL.md",[],[104,105,106,107,108,28,19],"add benchmarks","benchmarks","add","guidelines","extending"]