[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KKX58GGXA81WHTCPJJG902":3,"skill-same:01M6KKX58GGXA81WHTCPJJG902":58},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":30,"linkOnly":31,"markdownContent":32,"markdownBody":33,"frontMatter":34,"refName":36,"commitSha":37,"skillDirectory":38,"forksCount":39,"repositoryHtmlUrl":40,"repositoryHomepageUrl":6,"ownerHtmlUrl":41,"ownerBlogUrl":42,"ownerDescription":6,"updatedAt":43,"install":44,"resources":57},"01M6KKX58GGXA81WHTCPJJG902","profile-hardware-ceiling",null,"Measure raw hardware bandwidth ceilings for all data paths (NVMe, GPU, memory, P2P, GDS, GDRCopy) on the current platform","AI-native-Systems-Research\u002Fai-native-storage-certus","AI-native-Systems-Research","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F270755458?v=4","01M6KKQWSJX0D4X33E0KKSR4WM","AI-native Systems Research","01M6KKQWXTKR6MYVQMPKVJ5WD5",3,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002F.claude\u002Fskills\u002Fprofile-hardware-ceiling\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29],"profile","hardware","profile hardware ceiling","memory","gpu","data","p2p","platform","paths","output","for","2026-11-10 10:12:30",false,"---\nname: profile-hardware-ceiling\ndescription: Measure raw hardware bandwidth ceilings for all data paths (NVMe, GPU, memory, P2P, GDS, GDRCopy) on the current platform\nargument-hint: \"[output-dir]\"\n---\n\nMeasure the raw hardware bandwidth ceilings for every data path the Certus storage system\nuses. This skill is portable — it auto-detects hardware, adapts to whatever GPUs\u002Fdrives\u002Ftools\nare present, and produces a comprehensive matrix report.\n\nOutput:\n- `\u003Coutput-dir>\u002Fhardware-ceiling-\u003CYYYY-MM-DD>.md` (default output-dir: `profiling\u002F`)\n- `\u003Coutput-dir>\u002Fhardware_profile_\u003CYYYY-MM-DD>.yaml`\n\n## Data Paths to Measure\n\nCertus has multiple data paths depending on cold\u002Fhot and P2P vs memory-tier modes:\n\n**Measured paths (active benchmarks):**\n1. **NVMe → Host RAM read** (raw drive read bandwidth — also gives DMA IO size sweep)\n2. **NVMe → Host RAM write** (raw drive sequential write bandwidth)\n3. **Host RAM → GPU \u002F GPU → Host RAM** (cudaMemcpy H2D + D2H, NUMA-local and cross-NUMA)\n4. **GPU D2D** (staging ring → final destination, part of P2P pipeline)\n5. **GDRCopy CPU ↔ GPU BAR1** (CPU-initiated BAR1 access, diagnostic)\n6. **NVMe → GPU via GDS** (GPUDirect Storage — nvidia_fs\u002FcuFile, if available)\n7. **Host RAM memcpy** (baseline — same-NUMA and cross-NUMA)\n8. **NVMe DMA → BAR1** (P2P microbenchmark — closest isolated cold-path hardware ceiling)\n\n**Derived analysis (no additional benchmarks):**\n9. **Topology-Aware P2P Matrix** (routes each drive-group\u002FGPU pairing through the topology)\n\n## Platform Detection (Step 1)\n\nAuto-detect and record:\n- CPU: model, cores, NUMA nodes, cache sizes (`lscpu`)\n- Hostname: use `hostname -s` (short name only — strip domain suffix)\n- OS: distro, kernel (`uname -r`, `\u002Fetc\u002Fos-release`)\n- GPU(s): model, PCIe gen\u002Fwidth, BAR1 size, CUDA device index, PCI address\n  - `nvidia-smi -q` for BAR1\u002Fmemory\n  - `lspci -vvs \u003Caddr>` for LnkSta (speed\u002Fwidth)\n- NVMe drive(s): model, PCIe gen\u002Fwidth, PCI address\n  - `lspci -vvs \u003Caddr>` for LnkSta\n- PCIe topology: which root complex each device is on (`lspci -tv`)\n  - Map which drives are \"same-root\" with which GPUs\n- Available tools:\n  - `nvidia_fs` module: `lsmod | grep nvidia_fs`\n  - `gdsio`: `which gdsio` or `find \u002Fusr\u002Flocal\u002Fcuda*\u002Fgds\u002Ftools\u002Fgdsio`\n  - `gdrcopy_copybw`: `which gdrcopy_copybw` or find in `\u002Fusr\u002Flocal\u002Fgdrcopy*\u002F`\n  - `nvcc`: find under `\u002Fusr\u002Flocal\u002Fcuda*`\n  - `spdk_nvme_perf`: find under `deps\u002Fspdk-build\u002Fbin\u002F`\n  - `nvme-bar1-bench`: find under `target\u002Frelease\u002F`\n- Compute theoretical per-link bandwidth:\n  - Gen3: 8 GT\u002Fs × lanes × 128\u002F130 encoding\n  - Gen4: 16 GT\u002Fs × lanes × 128\u002F130 encoding\n  - Gen5: 32 GT\u002Fs × lanes × 128\u002F130 encoding\n\n## Command Style (Important for Permissions)\n\nTo avoid permission prompts on compound commands, use ONE simple command per\nBash tool call. Do NOT chain with `&&`, `|`, or `2>&1` redirects. The project's\npermission allowlist matches against the full command string — compound commands\nwon't match patterns like `Bash(ls \u002Fdev\u002Fvfio\u002F)`.\n\nGood: separate tool calls for `ls \u002Fdev\u002Fvfio\u002F` and `ls \u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F`\nBad: `ls \u002Fdev\u002Fvfio\u002F 2>&1 && ls \u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F 2>&1 | head -20`\n\n## CPU Feature Check (Step 2)\n\nRun `tools\u002Fcheck-amd-proc-features.py` to characterize the CPU's SIMD capabilities\nrelevant to hashing and KV cache integrity:\n\n```\npython3 tools\u002Fcheck-amd-proc-features.py\n```\n\nCapture and include the full output in the report. Key fields to extract into the YAML profile:\n- CPU vendor\u002Fbrand\u002FZen generation (if AMD)\n- XXH3 throughput tier (AVX-512 \u002F AVX2 \u002F SSE2 \u002F Scalar)\n- SIMD flags present: avx512f, avx512bw, avx2, sse4_2, pclmulqdq, aes\n- Certus algorithm recommendation\n\nIf `py-cpuinfo` is missing, note it and install: `pip install py-cpuinfo`.\n\n## NVMe Device Inspection (Step 3)\n\nFor each NVMe drive detected (drives must be bound to the **kernel NVMe driver** for this step,\nnot vfio-pci), run `tools\u002Finspect-nvme.py`. This characterizes drive behavior critical to\nbenchmark tuning:\n\n```\nsudo python3 tools\u002Finspect-nvme.py \u002Fdev\u002Fnvme\u003CN>n1 --skip-gc --skip-power\n```\n\nUse `--skip-gc` and `--skip-power` to skip the long-running settle and power-state tests\n(those take 10–30 minutes per drive). Run the full inspection only when you have time:\n\n```\nsudo python3 tools\u002Finspect-nvme.py \u002Fdev\u002Fnvme\u003CN>n1\n```\n\nKey fields to extract into the YAML profile per drive:\n- `model`, `firmware_rev`, `serial` (from `identify`)\n- `capacity_bytes` → convert to TB: `capacity_tb = capacity_bytes \u002F 1e12`\n- `mdts_bytes`, `block_size` (from `identify`)\n- `vwc_enabled`: `bool(identify['vwc'] & 0x1)` (volatile write cache)\n- `has_read_cache`, `cache_speedup` (from `read_cache`, if run)\n- `sustained_write_mbps` (from `gc_settle.sustained_write_mbps`, if run)\n- `recommended_gc_settle_s` (from `gc_settle.recommended_gc_settle_s`, if run)\n- `power_state_transitions_at_s` (first element of `power_states.power_state_transitions_at_s`, if run; null if empty list)\n- Read latency: `read_profile` entries — extract avg_us for qd=1, qd=16, qd=32\n\nAfter running inspect-nvme.py, rebind drives to vfio-pci before SPDK benchmarks.\n\n## Setup (Step 4)\n\n- Kill any running `certus-server-yaml` (it holds VFIO groups)\n- Wait for VFIO groups to release (`fuser \u002Fdev\u002Fvfio\u002F*`)\n- Bind all NVMe drives to vfio-pci: run the SPDK setup script\n  - Find it: `deps\u002Fspdk\u002Fscripts\u002Fsetup.sh` or `deps\u002Fspdk-build\u002F..\u002Fscripts\u002Fsetup.sh`\n  - If not found, try `HUGEMEM=4096 scripts\u002Fsetup.sh` from SPDK source dir\n- Verify all detected NVMe drives appear in `\u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F`\n\n## Path 1: NVMe Sequential Read (Step 5)\n\nTool: `spdk_nvme_perf`\n\nEvery NVMe command in the SPDK path is a DMA transfer — the NVMe controller DMAs data\ndirectly into the destination buffer (host RAM or BAR1) over PCIe using PRPs. The block\nsize passed to `spdk_nvme_perf` controls how many NVMe commands are grouped into one\nlogical IO, but each individual NVMe command is still bounded by MDTS (typically 128 KiB).\nThe block size matrix below therefore spans the full range from native MDTS size up through\nlarge logical groupings, giving both the per-command efficiency curve and the link\nsaturation behavior in one pass.\n\nMatrix dimensions:\n- **Drive counts**: 1, 2 (if ≥2 drives), 4 (if ≥4), 6 (if ≥6), all-detected\n- **Block sizes**: 4 KiB, 16 KiB, 64 KiB, 128 KiB, 256 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB\n  - 4–16 KiB: well below MDTS — CPU submission overhead and PCIe TLP inefficiency dominate;\n    throughput will be low but shows the floor and confirms MDTS is not the only factor\n  - 64–128 KiB: sub-MDTS and at-MDTS — isolates per-command PCIe efficiency; at QD=32\n    only 2–4 MiB in flight so the link may not yet be saturated at these sizes\n  - 256 KiB–1 MiB: MDTS-straddling zone where logical IOs split into 2–8 NVMe commands;\n    throughput typically plateaus here as the link saturates\n  - 2–8 MiB: large logical groupings; throughput should match 1 MiB if link-saturated.\n    Confirms saturation and exercises the logical IO grouping path.\n  Note: at small block sizes (4–128 KiB), QD=32 may not fully saturate the link — this\n  is expected and informative. The goal is to see where the throughput knee occurs, not\n  to reach absolute peak at every size. QD=32 is sufficient to observe the knee clearly.\n- **Queue depths**: 16, 32\n  QD=16 typically saturates the PCIe Gen4 x4 link at block sizes ≥256 KiB. Include\n  QD=32 to confirm saturation (identical throughput = link-bound) and to catch drives\n  where QD=16 is insufficient (smaller MDTS, higher per-IO latency, or BAR1 targets\n  with posted-write ordering delays).\n- **Duration**: 5 seconds per run\n- **Workload**: sequential read (`-w read`)\n\nFor multi-drive tests, pass multiple `-r 'trtype:pcie traddr:0000:XX:00.0'` flags.\nGroup drives by root complex when possible (test same-root groups and cross-root groups separately).\n\nParse output: the \"Total\" line gives IOPS, MiB\u002Fs, and average\u002Fmin\u002Fmax latency.\nConvert MiB\u002Fs to decimal GB\u002Fs using: `GB\u002Fs = MiB\u002Fs × 1.048576 \u002F 1000`.\nUse this formula consistently across all paths — the evaluator compares percentages\nagainst these numbers, so approximate conversions cause drift.\n\nOutput: Two matrices (throughput GB\u002Fs, latency us) with rows = (drives, block_size) and\ncolumns = QD. Note the **optimal read chunk size**: the smallest block size that achieves\n≥95% of the peak throughput for that drive count. This is the recommended `--chunk-size`\nfor `nvme-bar1-bench` and the NVMe command size in the dispatcher cold path.\n\nIf QD=16 and QD=32 produce identical throughput at a given size, note that the drive is\nlink-saturated at that size — deeper queuing adds only latency.\n\n## Path 2: NVMe Sequential Write (Step 6)\n\nTool: `spdk_nvme_perf`\n\nSame block size matrix as Path 1 — identical sizes allow direct read\u002Fwrite comparison\nand the same efficiency-curve analysis. Write throughput is the ceiling for the Certus\nwrite-through path (background flusher). Write efficiency vs size may differ from reads\ndue to FTL write amplification, especially at sub-MDTS sizes (64 KiB) where partial NAND\npage writes increase write amplification.\n\nMatrix dimensions:\n- **Drive counts**: 1, 2 (if ≥2 drives), 4 (if ≥4), all-detected\n  (6-drive write is less critical — the write path is single-drive-per-extent)\n- **Block sizes**: 4 KiB, 16 KiB, 64 KiB, 128 KiB, 256 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB\n- **Queue depths**: 16, 32\n- **Duration**: 5 seconds per run\n- **Workload**: sequential write (`-w write`)\n\n```\nsudo deps\u002Fspdk-build\u002Fbin\u002Fspdk_nvme_perf \\\n    -r 'trtype:pcie traddr:0000:XX:00.0' \\\n    -q 16 -o 131072 -w write -t 5\n```\n\nParse output identically to the read path. Convert MiB\u002Fs to decimal GB\u002Fs.\n\nOutput: Two matrices (throughput GB\u002Fs, latency us) with rows = (drives, block_size) and\ncolumns = QD. Note the **optimal write chunk size**: smallest block size achieving ≥95%\nof peak write throughput. Compare peak write GB\u002Fs against the `inspect-nvme.py` sustained\nwrite figure to verify SPDK and kernel-mode numbers are consistent.\n\n## Path 3: GPU PCIe Bandwidth — cudaMemcpy (Step 7)\n\nSource: `benchmarks\u002Fhw-ceiling\u002Fcuda_bw_test.cu`. Binary output: `\u002Ftmp\u002Fcuda_bw_test`.\n\nBuild step:\n```\n# Preferred — source already in repo:\nmake -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp cuda_bw_test\n\n# If benchmarks\u002Fhw-ceiling\u002F is absent: write the source to benchmarks\u002Fhw-ceiling\u002Fcuda_bw_test.cu\n# (creating the directory), then run make as above. Do NOT write to \u002Ftmp\u002F only —\n# saving to benchmarks\u002Fhw-ceiling\u002F means future runs skip the write step entirely.\n```\n\n**NUMA locality matters here.** H2D bandwidth depends on where the pinned host buffer\nis allocated relative to the GPU's root complex. If the GPU's PCIe root is on NUMA node 1\nbut the calling thread (and therefore `cudaMallocHost`) is on NUMA node 0, the DMA engine\nmust read the source buffer across the Infinity Fabric inter-socket link — this can reduce\nH2D throughput by 2–3+ GB\u002Fs compared to a NUMA-local allocation. Run the benchmark twice:\n\n```\n# NUMA-local: pin to the NUMA node that owns the GPU's root complex\n# First determine which NUMA node each GPU's PCIe root is on (from lspci\u002Fnumactl --hardware)\nnumactl --cpunodebind=\u003Cgpu_numa_node> --membind=\u003Cgpu_numa_node> \u002Ftmp\u002Fcuda_bw_test\n\n# Cross-NUMA: force host buffer onto the opposite NUMA node\nnumactl --cpunodebind=\u003Cother_numa_node> --membind=\u003Cother_numa_node> \u002Ftmp\u002Fcuda_bw_test\n```\n\nThe difference between these two runs is the Infinity Fabric H2D penalty. The NUMA-local\nresult is the true hardware ceiling; the cross-NUMA result is what Certus sees if CPU\naffinity is not set correctly.\n\nProgram behaviour:\n- `cudaMallocHost` + `cudaMemcpy` (pinned host memory)\n- Measure H2D and D2H\n- Transfer sizes: 128 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB, 16 MiB, 32 MiB, 64 MiB, 256 MiB\n- 50 iterations, 5 warmup\n- `cudaEvent` timing\n- Runs on ALL GPUs in a single invocation (loops over `cudaGetDeviceCount`)\n- Also measures H2D with 2, 4, and 8 concurrent `cudaMemcpyAsync` on separate streams\n  (each stream copies a different pinned buffer → different GPU destination).\n  This matters because single-stream H2D at 4 MiB may only reach ~10 GB\u002Fs on a x16 link\n  that can do ~25+ GB\u002Fs with concurrency.\n\nOutput: Per-GPU table for each NUMA binding: Size | H2D 1-stream (GB\u002Fs) | H2D 4-stream (GB\u002Fs) | D2H (GB\u002Fs).\nNote the per-GPU NUMA-local vs cross-NUMA delta — this is the cost of incorrect CPU affinity.\n\n## Path 4: GPU D2D Bandwidth — cudaMemcpyDeviceToDevice (Step 8)\n\nSource: `benchmarks\u002Fhw-ceiling\u002Fcuda_d2d_test.cu`. Binary output: `\u002Ftmp\u002Fcuda_d2d_test`.\n\nBuild step:\n```\n# Preferred — source already in repo:\nmake -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp cuda_d2d_test\n\n# If benchmarks\u002Fhw-ceiling\u002F is absent: write the source to benchmarks\u002Fhw-ceiling\u002Fcuda_d2d_test.cu\n# (creating the directory if needed), then run make as above.\n```\n\nThe P2P cold path uses a staging ring (GDRCopy BAR1-mapped GPU memory) as the NVMe DMA\ntarget, then does a D2D copy from staging → client's final GPU destination. This D2D\ncopy is part of the critical path and its bandwidth is a ceiling on throughput.\n\nProgram behaviour:\n- Source: `cudaMalloc` region A; Destination: `cudaMalloc` region B\n- Transfer sizes: 128 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB, 16 MiB\n- 100 iterations, 10 warmup\n- `cudaEvent` timing\n- Runs on ALL GPUs in a single invocation\n- Also measures with 2 and 4 concurrent async D2D copies on separate streams\n\nThis tells us whether D2D is fast enough to not bottleneck the pipeline,\nand whether multiple streams can overlap.\n\nOutput: Table: Size | 1-stream D2D (GB\u002Fs) | 2-stream (GB\u002Fs) | 4-stream (GB\u002Fs).\n\n## Path 5: GDRCopy BAR1 Mapping Bandwidth (Step 9)\n\nGDRCopy provides the BAR1 mapping that makes NVMe P2P work. It also exposes\nCPU-accessible bandwidth to GPU memory via BAR1, relevant for:\n- Metadata writes to GPU memory\n- Verifying BAR1 mapping is functional\n\n**Important context**: In the Certus P2P path, GDRCopy is used for MAPPING SETUP\n(gdr_pin_buffer + gdr_map + spdk_mem_register), NOT for data transfer. The actual\ndata transfer is NVMe DMA through the BAR1 aperture. But `gdrcopy_copybw` still\ntells us the BAR1 aperture bandwidth from the CPU side, which is a useful diagnostic.\n\nIf `gdrcopy_copybw` or `gdrcopy_copylat` is found:\n```\ngdrcopy_copybw\ngdrcopy_copylat\n```\n\nIf not installed, try to find and compile from source:\n```\nfind \u002F -path \"*\u002Fgdrcopy*\u002Ftests\" -type d 2>\u002Fdev\u002Fnull\n# or from the kernel\u002Fmodules\u002Fgdrcopy tree in this repo\ncd kernel\u002Fmodules\u002Fgdrcopy && make PREFIX=\u002Ftmp\u002Fgdrcopy lib lib_install\ncd tests && make\n```\n\nIf GDRCopy is not available at all, note in report and provide install instructions.\n\nOutput: Table of sizes vs H2D\u002FD2H bandwidth (GB\u002Fs).\nNote: CPU→BAR1 writes are often several GB\u002Fs to ~20+ GB\u002Fs depending on platform,\nNUMA placement, PCIe generation, and write-combining behavior.\nCPU←BAR1 reads are typically 1-5 GB\u002Fs (uncacheable reads, severe amplification).\n\n## Path 6: GPUDirect Storage — gdsio (Step 10)\n\nIf `lsmod | grep nvidia_fs` succeeds AND `gdsio` binary is found:\n\nUnbind drives from vfio-pci and rebind to kernel NVMe driver first:\n```\ndeps\u002Fspdk\u002Fscripts\u002Fsetup.sh reset\n```\n\nThen run gdsio for each topology pairing:\n- Same-root drives → same-root GPU\n- Cross-root drives → cross-root GPU (to measure penalty)\n\nMatrix:\n- Drive counts: 1, 2, 4\n- Block sizes: 4M, 8M, 16M\n- GPU: each available GPU\n- IO depth: 16, 32\n\n```\ngdsio -f \u002Fdev\u002FnvmeXn1 -d \u003Cgpu-index> -w 4 -s \u003Cblock-size> -x 0 -I 1 -T 10\n```\n\nAfter GDS tests, rebind drives back to vfio-pci for other tests.\n\nIf nvidia_fs is NOT loaded, note in report:\n```\nGDS not available.\nnvidia_fs module status: [not installed | installed but fails to load | loaded]\nIf not working, check driver version compatibility:\n  modinfo nvidia | grep version\n  modinfo nvidia-fs | grep version\nInstall: sudo dnf install nvidia-gds nvidia-fs-dkms\n```\n\nOutput: Table with (drives, block_size, GPU, topology) → bandwidth GB\u002Fs.\n\n## Path 7: Host RAM Bandwidth — memcpy baseline (Step 11)\n\nSource: `benchmarks\u002Fhw-ceiling\u002Fmemcpy_bench.c`. Binary output: `\u002Ftmp\u002Fmemcpy_bench`.\n\nBuild step:\n```\n# Preferred — source already in repo:\nmake -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp memcpy_bench\n\n# If benchmarks\u002Fhw-ceiling\u002F is absent: write the source to benchmarks\u002Fhw-ceiling\u002Fmemcpy_bench.c\n# (creating the directory if needed), then run make as above.\n```\n\nProgram behaviour:\n- Same-NUMA memcpy (both allocs on node 0, via `numa_alloc_onnode`)\n- Cross-NUMA memcpy (src on node 0, dst on node 1) — if ≥2 NUMA nodes\n- Sizes: 4 MiB, 16 MiB, 64 MiB, 256 MiB\n- 200 iters for ≤16 MiB, 50 iters for ≥64 MiB\n\nThis establishes the memory-tier ceiling when data is in host RAM (before GPU transfer).\n\nOutput: Table: Size | Same-NUMA (GB\u002Fs) | Cross-NUMA (GB\u002Fs).\n\n## Path 8: Topology-Aware P2P Matrix (Step 12)\n\nFor each (drive-group, GPU) pairing, summarize the expected path:\n- Same root complex: direct PCIe path\n- Cross root complex: traverses CPU\u002FInfinity Fabric\n- Cross NUMA: traverses inter-socket link\n\nCreate a topology matrix showing the routing for every possible pairing.\n\n## Path 9: NVMe DMA → BAR1 Microbenchmark (Step 13)\n\nTool: `nvme-bar1-bench` (`apps\u002Fnvme-bar1-bench\u002F`)\n\nThis is the most important measurement — it gives the closest isolated P2P cold-path\nhardware ceiling. Note: it still includes SPDK FFI, qpair behavior, poller\nimplementation, GDRCopy mapping, and VFIO\u002FIOMMU effects — it is the practical\nceiling for Certus, not a pure PCIe-theoretical ceiling. It bypasses the actor\u002Fchannel model and calls SPDK FFI directly. Each drive\ngets a dedicated CPU-pinned poller thread with its own qpair. Runs two modes\nback-to-back: host-ram (baseline) and bar1 (GDRCopy-mapped GPU memory).\n\nRequires: `gdrdrv` kernel module loaded. Depending on the driver stack,\n`nvidia_peermem` may also be required; detect and report whether it is loaded.\nIf `gdrdrv` is absent, skip this path and note in the report.\n\n```\ncargo build --release -p nvme-bar1-bench\n```\n\n**Critical: use a large total-bytes (≥2 GiB) to read past the drive's DRAM cache.**\nSmall streams (16–64 MiB) will report inflated numbers from cache hits.\n\n**Critical: report exactly which drive BDFs were selected for each run.**\nDo not interpret topology results unless the selected drives are shown. The\n`--drive-count N` flag uses the first N discovered drives, which may not correspond\nto the drives same-root with the target GPU.\n\n**NUMA and topology pinning are critical for correct ceiling measurements.**\n`nvme-bar1-bench` spawns one poller thread per drive. Those threads issue NVMe commands\nand wait for completions — their NUMA locality determines where the host-RAM DMA target\nbuffer is allocated and which memory controller handles the PCIe read\u002Fwrite traffic.\n\n- **NUMA-local run**: pin poller threads to the NUMA node that owns the target GPU's PCIe\n  root complex, AND select drives from that same root complex. This gives the true hardware\n  ceiling with no cross-fabric penalty.\n- **Cross-NUMA run**: pin to the opposite NUMA node. Quantifies the Infinity Fabric penalty\n  that Certus incurs when CPU affinity is not set. Required to understand observed production\n  performance, not just the theoretical ceiling.\n\nUse `sudo numactl --cpunodebind=N --membind=N` to wrap the benchmark. Determine the correct\nNUMA node for each GPU from `numactl --hardware` and the GPU's PCI BDF (cross-reference\nwith `lspci -tv`). Verify that `--drive-count N` picks drives from the right root complex\nby checking the BDFs printed in the benchmark output — the tool sorts drives by PCI address\nand takes the first N, which may not be the same-root drives for the target GPU.\n\nRun matrix:\n```\n# --- NUMA-local runs (true ceiling) ---\n# Substitute \u003Cgpu0_numa> with the NUMA node of GPU 0's PCIe root complex.\n# Verify BDFs in output confirm drives are same-root with GPU 0.\n\n# 1 drive, NUMA-local, GPU 0\nsudo numactl --cpunodebind=\u003Cgpu0_numa> --membind=\u003Cgpu0_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 1 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 16 --iterations 3\n\n# 4 drives, NUMA-local, GPU 0\nsudo numactl --cpunodebind=\u003Cgpu0_numa> --membind=\u003Cgpu0_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 4 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n\n# 6 drives, NUMA-local, GPU 0\nsudo numactl --cpunodebind=\u003Cgpu0_numa> --membind=\u003Cgpu0_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 6 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n\n# 4 drives, NUMA-local, GPU 1 (topology-best pairing for GPU 1)\nsudo numactl --cpunodebind=\u003Cgpu1_numa> --membind=\u003Cgpu1_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 4 --gpu 1 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n\n# --- Cross-NUMA run (quantify Infinity Fabric penalty) ---\n# Pin to the NUMA node opposite to GPU 0's root complex.\n# Same drive count and GPU as the NUMA-local 4-drive run for a direct apples-to-apples comparison.\n\n# 4 drives, cross-NUMA, GPU 0\nsudo numactl --cpunodebind=\u003Cother_numa> --membind=\u003Cother_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 4 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n```\n\nCLI reference:\n- `--drive-count N`: use first N discovered drives (sorted by PCI address)\n- `--gpu INDEX`: CUDA device index for BAR1 target [default: 0]\n- `--chunk-size BYTES`: NVMe IO size, must be ≤ drive MDTS (typically 128 KiB) [default: 131072]\n- `--total-bytes BYTES`: total bytes per drive per iteration [default: 16 MiB — TOO SMALL, use 2G]\n- `--queue-depth N`: in-flight commands per drive [default: 16]\n- `--warmup N`: warmup iterations [default: 3]\n- `--iterations N`: measured iterations [default: 20]\n\nSanity check: if 1-drive host-RAM result is >20% below the SPDK perf single-drive\nnumber, the microbenchmark itself may be adding overhead (e.g. missing doorbell\nbatching). Note this in the report — the host-ram vs bar1 comparison remains valid\nbut the absolute numbers may understate the hardware ceiling.\n\nObserved on current node (Intel Sentinel Rock, Gen4 x4, NVIDIA A30):\n- 1 drive: host-ram ≈ bar1 ≈ 4.3 GB\u002Fs (0% BAR1 overhead)\n- 4 drives: host-ram ≈ 24 GB\u002Fs, bar1 ≈ 20 GB\u002Fs (~17% BAR1 overhead)\n- 6 drives: host-ram ≈ 35 GB\u002Fs, bar1 ≈ 21.5 GB\u002Fs (~39% overhead, GPU BAR1 saturates)\n\nThese are node-specific and should not be treated as universal expected values.\n\n## Report Generation (Step 14)\n\nProduce TWO outputs:\n\n### 1. Markdown report: `\u003Coutput-dir>\u002Fhardware-ceiling-\u003CYYYY-MM-DD>.md`\n\nWrite the report with:\n1. Platform section (all detected hardware, topology diagram)\n2. CPU feature report (full output of `check-amd-proc-features.py`, formatted as a code block)\n3. NVMe device inspection summary (from `inspect-nvme.py` per drive — key fields, not full output)\n4. PCIe theoretical bandwidth table\n5. Per-path matrices (Paths 1–8, each gets its own section; read and write matrices include the IO size knee annotation)\n6. Topology routing matrix (Path 9 — derived, no additional benchmarks)\n7. Summary table (see below)\n8. Observations and recommendations\n\nThe summary must contain **four sub-tables**:\n\n**Sub-table 1 — Raw hardware paths** (measured primitives, no dispatcher logic):\n```\n| Data Path                              | Peak BW (GB\u002Fs) | Limiting Factor              | Notes              |\n|----------------------------------------|----------------|------------------------------|--------------------|\n| NVMe → Host read (1 drive)             |                | Drive PCIe x4                |                    |\n| NVMe → Host read (4 drives)            |                | 4× PCIe x4 aggregate         |                    |\n| NVMe → Host read (6\u002F7 drives)          |                | N× PCIe x4 aggregate         |                    |\n| NVMe → Host write (1 drive)            |                | Drive PCIe x4 \u002F FTL          |                    |\n| NVMe → Host write (4 drives)           |                | 4× PCIe x4 aggregate \u002F FTL   |                    |\n| Optimal read chunk size                |                | (bytes — knee of curve)      |                    |\n| Optimal write chunk size               |                | (bytes — knee of curve)      |                    |\n| Host → GPU 0 H2D (NUMA-local, peak)    |                | GPU PCIe x16                 |                    |\n| Host → GPU 0 H2D (cross-NUMA, peak)    |                | Infinity Fabric              | penalty vs local   |\n| Host → GPU 1 H2D (NUMA-local, peak)    |                | GPU PCIe x16                 |                    |\n| Host → GPU 1 H2D (cross-NUMA, peak)    |                | Infinity Fabric              | penalty vs local   |\n| GPU N → Host (D2H)                     |                | PCIe read amplification      |                    |\n| GPU D2D (1-stream sustained)           |                | HBM internal bus             |                    |\n| GDRCopy CPU→BAR1 write                 |                | BAR1 write-combining         |                    |\n| GDRCopy CPU←BAR1 read                  |                | BAR1 uncacheable reads       |                    |\n| Host RAM same-NUMA (sustained)         |                | Memory controller            |                    |\n| Host RAM cross-NUMA (sustained)        |                | Inter-socket link            |                    |\n| GDS NVMe → GPU                         | N\u002FA or XX.XX   | nvidia_fs \u002F true P2P DMA     |                    |\n```\n\n**Sub-table 2 — `dispatcher` warm path** (both `dispatcher` and `dispatcher-p2p` share this):\n```\nRoute: Memory-Tier (CUDA-pinned DRAM) ──H2D DMA──▶ GPU\nBoth components use memcpy_h2d_async from a DRAM memory-tier slot for warm hits.\nCeiling = H2D PCIe bandwidth (NVMe not involved).\n\n| GPU | Ceiling (GB\u002Fs) | Bound by        |\n|-----|----------------|-----------------|\n| 0   |                | H2D PCIe x16    |\n| 1   |                | H2D PCIe x16    |\n```\n\n**Sub-table 3 — `dispatcher` cold path** (NVMe → DRAM bounce → H2D):\n```\nRoute: NVMe ──DMA──▶ Memory-Tier (DRAM) ──H2D DMA──▶ GPU\nTwo-stage pipeline. Ceiling = min(NVMe_aggregate, H2D). Crossover point where H2D\nbecomes the cap = H2D_ceiling \u002F per_drive_bandwidth drives.\n\n| Drives | GPU | Ceiling (GB\u002Fs) | Bound by                          |\n|--------|-----|----------------|-----------------------------------|\n| 1      | N   |                | NVMe x4                           |\n| 4      | N   |                | NVMe aggregate (if \u003C H2D ceiling) |\n| 6      | N   |                | H2D PCIe x16 or NVMe aggregate    |\n```\n\n**Sub-table 4 — `dispatcher-p2p` cold path** (NVMe → BAR1 direct):\n```\nRoute: NVMe ──DMA──▶ BAR1 (GDRCopy-mapped GPU memory)\nSingle-stage. Ceiling = nvme-bar1-bench BAR1 result (includes PCIe + BAR1 overhead).\nBAR1 saturates around 21–22 GB\u002Fs on A30; adding drives beyond that point yields no gain.\n\n| Drives | GPU | BAR1 ceiling (GB\u002Fs) | BAR1 overhead vs host-ram | Notes                    |\n|--------|-----|---------------------|--------------------------|--------------------------|\n| 1      | 0   |                     | ~0%                      | NUMA-local               |\n| 4      | 0   |                     | measured %               | NUMA-local               |\n| 4      | 0   |                     | measured %               | cross-NUMA (penalty run) |\n| 4      | 1   |                     | measured %               | NUMA-local, topology-best|\n| 6      | 0   |                     | measured %               | NUMA-local, saturated?   |\n```\n\nAfter the four sub-tables, add a **crossover comparison**:\n```\n| Drive count | dispatcher cold ceiling | dispatcher-p2p cold ceiling | Winner        |\n|-------------|------------------------|-----------------------------|---------------|\n| 1           |                        |                             |               |\n| 4           |                        |                             | ~tie or one   |\n| 6           |                        |                             | dispatcher or p2p |\n```\nThis exposes the key insight: at low drive counts dispatcher-p2p may win on latency\n(single-stage, no DRAM bounce) but at high drive counts dispatcher wins on throughput\n(H2D cap > BAR1 saturation ceiling).\n\n### 2. Machine-readable profile: `\u003Coutput-dir>\u002Fhardware_profile_\u003CYYYY-MM-DD>.yaml`\n\nThe filename includes the date so multiple profiles from different runs are preserved\nside-by-side for comparison. Use only measured values — omit any path that was\nskipped due to missing tools (do not fill with estimates or zeros).\n\n```yaml\n# Auto-generated by profile-hardware-ceiling skill\n# Date: YYYY-MM-DD\n# Node: \u003Chostname>\n\nplatform:\n  hostname: \u003Chostname>\n  cpu: \u003Cmodel>\n  numa_nodes: \u003CN>\n  kernel: \u003Cversion>\n  os: \u003Cdistro version>\n\ncpu_features:\n  vendor: \u003CAuthenticAMD | GenuineIntel | ...>\n  zen_generation: \u003C1-5 or null>\n  zen_label: \u003C\"Zen 4\" or \"N\u002FA (non-AMD)\">\n  has_avx512: \u003Ctrue\u002Ffalse>\n  has_avx2: \u003Ctrue\u002Ffalse>\n  has_sse4_2: \u003Ctrue\u002Ffalse>\n  has_pclmulqdq: \u003Ctrue\u002Ffalse>\n  has_aes: \u003Ctrue\u002Ffalse>\n  xxh3_tier: \u003C\"AVX-512\" | \"AVX2\" | \"SSE2\" | \"Scalar\">\n  xxh3_throughput_estimate: \u003C\"~40-50 GB\u002Fs\" or similar string>\n  certus_recommended_hash: \u003C\"XXH3_128bits (AVX-512 path)\" or similar>\n\ngpus:\n  gpu0:\n    model: \u003Cname>\n    pcie_gen: 4\n    pcie_width: 16\n    bar1_gib: 32\n    pci_bdf: \"a1:00.0\"\n    numa_node: \u003Cint>                        # NUMA node of GPU's PCIe root complex\n    # NUMA-local: cudaMallocHost on same NUMA node as GPU's root complex\n    h2d_4m_1stream_numa_local_gbps: \u003Cmeasured>\n    h2d_peak_numa_local_gbps: \u003Cmeasured at best transfer size>\n    d2h_peak_numa_local_gbps: \u003Cmeasured>\n    # Cross-NUMA: host buffer on opposite NUMA node\n    h2d_4m_1stream_cross_numa_gbps: \u003Cmeasured>\n    h2d_peak_cross_numa_gbps: \u003Cmeasured>\n    cross_numa_h2d_penalty_gbps: \u003Ccomputed: local - cross>\n    d2d_1stream_gbps: \u003Cmeasured>\n    d2d_4stream_gbps: \u003Cmeasured>\n  gpu1:\n    # same structure\n\nnvme:\n  drives:\n    - bdf: \"61:00.0\"\n      model: \u003Cname>\n      firmware_rev: \u003Cstring>\n      serial: \u003Cstring>\n      capacity_tb: \u003Cfloat>          # capacity_bytes \u002F 1e12\n      mdts_bytes: 131072\n      pcie_gen: 4\n      pcie_width: 4\n      numa_node: \u003Cint>              # from sysfs numa_node attribute\n      vwc_enabled: \u003Ctrue\u002Ffalse>     # bool(vwc & 0x1)\n      has_read_cache: \u003Ctrue\u002Ffalse>\n      # from inspect-nvme.py gc_settle (if run):\n      sustained_write_mbps: \u003Cmeasured or null>\n      recommended_gc_settle_s: \u003Cint or null>\n      # from inspect-nvme.py power_states (if run):\n      power_state_transition_at_s: \u003Cint or null>\n      # from inspect-nvme.py read_profile (if run):\n      read_latency_qd1_us: \u003Cfloat or null>\n      read_latency_qd16_us: \u003Cfloat or null>\n      read_latency_qd32_us: \u003Cfloat or null>\n    - bdf: \"62:00.0\"\n      # ...\n  mdts_bytes: 131072         # representative MDTS (use smallest across all drives)\n  raw_read_gbps:\n    1_drive: \u003Cmeasured>\n    4_drive: \u003Cmeasured>\n    6_drive: \u003Cmeasured>\n  raw_write_gbps:\n    1_drive: \u003Cmeasured>\n    4_drive: \u003Cmeasured>\n  # IO size sweep: 1 drive, QD=32. Read and write use the same sizes.\n  # Derived from Path 1 (read) and Path 2 (write) 1-drive QD=32 rows.\n  io_size_sweep_read_gbps:\n    4kib: \u003Cmeasured>\n    16kib: \u003Cmeasured>\n    64kib: \u003Cmeasured>\n    128kib: \u003Cmeasured>\n    256kib: \u003Cmeasured>\n    512kib: \u003Cmeasured>\n    1mib: \u003Cmeasured>\n    2mib: \u003Cmeasured>\n    4mib: \u003Cmeasured>\n    8mib: \u003Cmeasured>\n    optimal_bytes: \u003Cint — smallest size achieving ≥95% of peak>\n  io_size_sweep_write_gbps:      # 1 drive, QD=32\n    4kib: \u003Cmeasured>\n    16kib: \u003Cmeasured>\n    64kib: \u003Cmeasured>\n    128kib: \u003Cmeasured>\n    256kib: \u003Cmeasured>\n    512kib: \u003Cmeasured>\n    1mib: \u003Cmeasured>\n    2mib: \u003Cmeasured>\n    4mib: \u003Cmeasured>\n    8mib: \u003Cmeasured>\n    optimal_bytes: \u003Cint>\n\nbar1:\n  # nvme-bar1-bench results (per GPU target)\n  # Each entry has numa_local (cpunodebind = GPU's NUMA node) and cross_numa variants.\n  # The numa_local result is the true hardware ceiling.\n  # The cross_numa result quantifies the Infinity Fabric penalty.\n  gpu0:\n    numa_local:\n      host_ram_1drive_gbps: \u003Cmeasured>\n      host_ram_4drive_gbps: \u003Cmeasured>\n      host_ram_6drive_gbps: \u003Cmeasured>\n      bar1_1drive_gbps: \u003Cmeasured>\n      bar1_4drive_gbps: \u003Cmeasured>\n      bar1_6drive_gbps: \u003Cmeasured>\n      overhead_4drive_pct: \u003Ccomputed>\n      overhead_6drive_pct: \u003Ccomputed>\n    cross_numa:\n      host_ram_4drive_gbps: \u003Cmeasured>\n      bar1_4drive_gbps: \u003Cmeasured>\n      penalty_vs_local_pct: \u003Ccomputed>\n  gpu1:\n    numa_local:\n      host_ram_1drive_gbps: \u003Cmeasured>\n      host_ram_4drive_gbps: \u003Cmeasured>\n      bar1_1drive_gbps: \u003Cmeasured>\n      bar1_4drive_gbps: \u003Cmeasured>\n      overhead_4drive_pct: \u003Ccomputed>\n\ngdrcopy:\n  cpu_to_bar1_gbps: \u003Cmeasured or null>\n  bar1_to_cpu_gbps: \u003Cmeasured or null>\n\ngds:\n  available: \u003Ctrue\u002Ffalse>\n  # if available:\n  1drive_gbps: \u003Cmeasured>\n  4drive_gbps: \u003Cmeasured>\n\nhost_ram:\n  same_numa_gbps: \u003Cmeasured>\n  cross_numa_gbps: \u003Cmeasured or null>\n\ntopology:\n  # Which drives share a root complex with which GPU\n  gpu0_same_root_drives: [\"c1:00.0\", \"c2:00.0\"]\n  gpu1_same_root_drives: [\"61:00.0\", \"62:00.0\", \"63:00.0\", \"64:00.0\"]\n\n# Derived ceilings — keyed by Certus dispatcher path, for investigator\u002Fevaluator\n#\n# Certus has two dispatchers, each with a warm and cold path:\n#\n#   dispatcher (memory-tier DRAM bounce):\n#     warm:  Memory-Tier (DRAM) ──H2D──▶ GPU           ceiling = H2D bandwidth\n#     cold:  NVMe ──DMA──▶ Memory-Tier ──H2D──▶ GPU    ceiling = min(NVMe_agg, H2D)\n#\n#   dispatcher-p2p (direct BAR1, no DRAM):\n#     warm:  Memory-Tier (DRAM) ──H2D──▶ GPU           ceiling = H2D bandwidth (same as above)\n#     cold:  NVMe ──DMA──▶ BAR1 (GPU)                  ceiling = bar1 result from nvme-bar1-bench\n#\nceilings:\n  # All ceiling values use NUMA-local measurements (true hardware ceiling).\n  # Cross-NUMA penalty fields show the cost of incorrect CPU affinity in production.\n\n  # --- warm path (shared by both dispatchers) ---\n  # Use NUMA-local H2D — this is the ceiling Certus achieves with correct affinity.\n  warm_gpu0_gbps: \u003Cgpus.gpu0.h2d_4m_1stream_numa_local_gbps>\n  warm_gpu0_peak_gbps: \u003Cgpus.gpu0.h2d_peak_numa_local_gbps>\n  warm_gpu1_gbps: \u003Cgpus.gpu1.h2d_4m_1stream_numa_local_gbps>\n  warm_gpu1_peak_gbps: \u003Cgpus.gpu1.h2d_peak_numa_local_gbps>\n  # Penalty if affinity is wrong:\n  warm_gpu0_cross_numa_penalty_gbps: \u003Cgpus.gpu0.cross_numa_h2d_penalty_gbps>\n  warm_gpu1_cross_numa_penalty_gbps: \u003Cgpus.gpu1.cross_numa_h2d_penalty_gbps>\n\n  # --- dispatcher cold path: NVMe → DRAM → H2D ---\n  dispatcher_cold_1drive_gbps: \u003Cmin(nvme.raw_read_gbps.1_drive, warm_gpuX_peak_gbps)>\n  dispatcher_cold_4drive_gbps: \u003Cmin(nvme.raw_read_gbps.4_drive, warm_gpuX_peak_gbps)>\n  dispatcher_cold_6drive_gbps: \u003Cmin(nvme.raw_read_gbps.6_drive, warm_gpuX_peak_gbps)>\n  dispatcher_cold_bottleneck_at_4drive: \u003C\"nvme\" or \"h2d\">\n  dispatcher_cold_bottleneck_at_6drive: \u003C\"nvme\" or \"h2d\">\n  dispatcher_cold_crossover_drives: \u003Ccomputed: h2d_peak_gbps \u002F per_drive_gbps>\n\n  # --- dispatcher-p2p cold path: NVMe → BAR1 (NUMA-local) ---\n  p2p_cold_1drive_gbps: \u003Cbar1.gpuX.numa_local.bar1_1drive_gbps>\n  p2p_cold_4drive_gpu0_gbps: \u003Cbar1.gpu0.numa_local.bar1_4drive_gbps>\n  p2p_cold_4drive_gpu1_gbps: \u003Cbar1.gpu1.numa_local.bar1_4drive_gbps>\n  p2p_cold_6drive_gbps: \u003Cbar1.gpuX.numa_local.bar1_6drive_gbps>\n  p2p_cold_bar1_saturated: \u003Ctrue if adding drives past N yields \u003C5% gain>\n  # Cross-NUMA penalty:\n  p2p_cold_4drive_gpu0_cross_numa_gbps: \u003Cbar1.gpu0.cross_numa.bar1_4drive_gbps>\n  p2p_cold_cross_numa_penalty_pct: \u003Cbar1.gpu0.cross_numa.penalty_vs_local_pct>\n\n  # --- NVMe raw (no GPU overhead, reference) ---\n  nvme_raw_1drive_read_gbps: \u003Cnvme.raw_read_gbps.1_drive>\n  nvme_raw_4drive_read_gbps: \u003Cnvme.raw_read_gbps.4_drive>\n  nvme_raw_6drive_read_gbps: \u003Cnvme.raw_read_gbps.6_drive>\n  nvme_raw_1drive_write_gbps: \u003Cnvme.raw_write_gbps.1_drive>\n  nvme_raw_4drive_write_gbps: \u003Cnvme.raw_write_gbps.4_drive>\n\n  # --- DMA chunk size recommendation ---\n  nvme_optimal_read_chunk_bytes: \u003Cnvme.io_size_sweep_read_gbps.optimal_bytes>\n  nvme_optimal_write_chunk_bytes: \u003Cnvme.io_size_sweep_write_gbps.optimal_bytes>\n\n  # --- D2D (GPU internal; not a bottleneck for either dispatcher) ---\n  gpu_d2d_sustained_gbps: \u003Cgpus.gpuX.d2d_1stream_gbps at 8 MiB>\n\n  # --- Host RAM ---\n  host_ram_same_numa_gbps: \u003Chost_ram.same_numa_gbps>\n  host_ram_cross_numa_gbps: \u003Chost_ram.cross_numa_gbps>\n```\n\nThe `ceilings` section is consumed by:\n- **Rule-based investigator**: `efficiency = certus_measured \u002F ceiling[relevant_path]`\n- **Evaluation function**: pick the ceiling matching the active dispatch mode and drive count\n\nOnly populate entries where measurements succeeded. Use topology-correct GPU\u002Fdrive pairings\n(same-NUMA where available). Omit rather than estimate for skipped paths.\n\n## Notes for Portability\n\n- Do NOT hardcode PCI addresses — detect them from `lspci`\n- Do NOT hardcode drive counts — use what's available\n- Do NOT assume specific GPU models — query capabilities\n- If a tool is missing (nvcc, gdsio, gdrcopy_copybw, spdk_nvme_perf, nvme-bar1-bench),\n  skip that path and note in the report with install\u002Fbuild instructions\n- Handle single-GPU and multi-GPU systems\n- Handle single-NUMA and multi-NUMA systems\n- **CUDA\u002FC benchmark sources live in `benchmarks\u002Fhw-ceiling\u002F`** (`cuda_bw_test.cu`,\n  `cuda_d2d_test.cu`, `memcpy_bench.c`, `Makefile`). Always use these.\n  Build with `make -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp \u003Ctarget>` — binaries go to `\u002Ftmp\u002F`.\n  Only write sources to `\u002Ftmp\u002F` if the repo checkout is somehow read-only (rare).\n- The YAML profile must be valid YAML parseable by any standard library\n- `inspect-nvme.py` requires drives bound to the **kernel NVMe driver** (not vfio-pci).\n  Run it before the SPDK setup step, or after GDS tests when drives are already rebound.\n- `check-amd-proc-features.py` requires `py-cpuinfo` (`pip install py-cpuinfo`).\n  It is read-only and can run at any point without affecting drive binding state.\n\n### Permissions setup for new users\n\nThe repo's `.claude\u002Fsettings.json` pre-allows read-only detection commands (lscpu,\nlspci, nvidia-smi, cargo, etc.). The sudo\u002Fhardware commands needed for actual\nbenchmarking require a per-user `.claude\u002Fsettings.local.json` (gitignored).\n\nTo run this skill without permission prompts, create `.claude\u002Fsettings.local.json`:\n```json\n{\n  \"permissions\": {\n    \"allow\": [\n      \"Bash(sudo lspci*)\",\n      \"Bash(sudo modprobe *)\",\n      \"Bash(sudo pkill*certus*)\",\n      \"Bash(sudo kill*)\",\n      \"Bash(sudo chmod * \u002Fdev\u002Fvfio\u002F*)\",\n      \"Bash(sudo fuser \u002Fdev\u002Fvfio\u002F*)\",\n      \"Bash(sudo *\u002Fdeps\u002Fspdk*\u002Fscripts\u002Fsetup.sh*)\",\n      \"Bash(sudo *\u002Fdeps\u002Fspdk-build\u002Fbin\u002Fspdk_nvme_perf*)\",\n      \"Bash(sudo *\u002Ftarget\u002Frelease\u002Fnvme-bar1-bench*)\",\n      \"Bash(sudo numactl*)\",\n      \"Bash(sudo *\u002Ftarget\u002Frelease\u002Fcertus-server-yaml*)\",\n      \"Bash(sudo python3 tools\u002Finspect-nvme.py*)\",\n      \"Bash(python3 tools\u002Fcheck-amd-proc-features.py*)\",\n      \"Bash(make -C benchmarks\u002Fhw-ceiling*)\",\n      \"Bash(\u002Fusr\u002Flocal\u002Fcuda*\u002Fbin\u002Fnvcc*)\",\n      \"Bash(CUDA_VISIBLE_DEVICES=* \u002Ftmp\u002F*)\",\n      \"Bash(\u002Ftmp\u002F*)\",\n      \"Bash(find \u002F -name *gdrcopy*)\",\n      \"Bash(find \u002F -name *gdsio*)\",\n      \"Bash(ls \u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F)\",\n      \"Bash(ls \u002Fdev\u002Fvfio\u002F)\",\n      \"Write(*\u002Fprofiling\u002F*)\",\n      \"Write(\u002Ftmp\u002F*)\"\n    ]\n  }\n}\n```\n\nOr let Claude prompt you on first run and approve each category once.\n\n## Cleanup (Step 15)\n\nLeave drives in whatever state the last test needed (vfio-pci if GDS was not run,\nkernel driver if GDS was run last). Do NOT restart certus-server — leave the system\nready for the user to run further tests. Report which driver the drives are currently bound to.\n","Measure the raw hardware bandwidth ceilings for every data path the Certus storage system\nuses. This skill is portable — it auto-detects hardware, adapts to whatever GPUs\u002Fdrives\u002Ftools\nare present, and produces a comprehensive matrix report.\n\nOutput:\n- `\u003Coutput-dir>\u002Fhardware-ceiling-\u003CYYYY-MM-DD>.md` (default output-dir: `profiling\u002F`)\n- `\u003Coutput-dir>\u002Fhardware_profile_\u003CYYYY-MM-DD>.yaml`\n\n## Data Paths to Measure\n\nCertus has multiple data paths depending on cold\u002Fhot and P2P vs memory-tier modes:\n\n**Measured paths (active benchmarks):**\n1. **NVMe → Host RAM read** (raw drive read bandwidth — also gives DMA IO size sweep)\n2. **NVMe → Host RAM write** (raw drive sequential write bandwidth)\n3. **Host RAM → GPU \u002F GPU → Host RAM** (cudaMemcpy H2D + D2H, NUMA-local and cross-NUMA)\n4. **GPU D2D** (staging ring → final destination, part of P2P pipeline)\n5. **GDRCopy CPU ↔ GPU BAR1** (CPU-initiated BAR1 access, diagnostic)\n6. **NVMe → GPU via GDS** (GPUDirect Storage — nvidia_fs\u002FcuFile, if available)\n7. **Host RAM memcpy** (baseline — same-NUMA and cross-NUMA)\n8. **NVMe DMA → BAR1** (P2P microbenchmark — closest isolated cold-path hardware ceiling)\n\n**Derived analysis (no additional benchmarks):**\n9. **Topology-Aware P2P Matrix** (routes each drive-group\u002FGPU pairing through the topology)\n\n## Platform Detection (Step 1)\n\nAuto-detect and record:\n- CPU: model, cores, NUMA nodes, cache sizes (`lscpu`)\n- Hostname: use `hostname -s` (short name only — strip domain suffix)\n- OS: distro, kernel (`uname -r`, `\u002Fetc\u002Fos-release`)\n- GPU(s): model, PCIe gen\u002Fwidth, BAR1 size, CUDA device index, PCI address\n  - `nvidia-smi -q` for BAR1\u002Fmemory\n  - `lspci -vvs \u003Caddr>` for LnkSta (speed\u002Fwidth)\n- NVMe drive(s): model, PCIe gen\u002Fwidth, PCI address\n  - `lspci -vvs \u003Caddr>` for LnkSta\n- PCIe topology: which root complex each device is on (`lspci -tv`)\n  - Map which drives are \"same-root\" with which GPUs\n- Available tools:\n  - `nvidia_fs` module: `lsmod | grep nvidia_fs`\n  - `gdsio`: `which gdsio` or `find \u002Fusr\u002Flocal\u002Fcuda*\u002Fgds\u002Ftools\u002Fgdsio`\n  - `gdrcopy_copybw`: `which gdrcopy_copybw` or find in `\u002Fusr\u002Flocal\u002Fgdrcopy*\u002F`\n  - `nvcc`: find under `\u002Fusr\u002Flocal\u002Fcuda*`\n  - `spdk_nvme_perf`: find under `deps\u002Fspdk-build\u002Fbin\u002F`\n  - `nvme-bar1-bench`: find under `target\u002Frelease\u002F`\n- Compute theoretical per-link bandwidth:\n  - Gen3: 8 GT\u002Fs × lanes × 128\u002F130 encoding\n  - Gen4: 16 GT\u002Fs × lanes × 128\u002F130 encoding\n  - Gen5: 32 GT\u002Fs × lanes × 128\u002F130 encoding\n\n## Command Style (Important for Permissions)\n\nTo avoid permission prompts on compound commands, use ONE simple command per\nBash tool call. Do NOT chain with `&&`, `|`, or `2>&1` redirects. The project's\npermission allowlist matches against the full command string — compound commands\nwon't match patterns like `Bash(ls \u002Fdev\u002Fvfio\u002F)`.\n\nGood: separate tool calls for `ls \u002Fdev\u002Fvfio\u002F` and `ls \u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F`\nBad: `ls \u002Fdev\u002Fvfio\u002F 2>&1 && ls \u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F 2>&1 | head -20`\n\n## CPU Feature Check (Step 2)\n\nRun `tools\u002Fcheck-amd-proc-features.py` to characterize the CPU's SIMD capabilities\nrelevant to hashing and KV cache integrity:\n\n```\npython3 tools\u002Fcheck-amd-proc-features.py\n```\n\nCapture and include the full output in the report. Key fields to extract into the YAML profile:\n- CPU vendor\u002Fbrand\u002FZen generation (if AMD)\n- XXH3 throughput tier (AVX-512 \u002F AVX2 \u002F SSE2 \u002F Scalar)\n- SIMD flags present: avx512f, avx512bw, avx2, sse4_2, pclmulqdq, aes\n- Certus algorithm recommendation\n\nIf `py-cpuinfo` is missing, note it and install: `pip install py-cpuinfo`.\n\n## NVMe Device Inspection (Step 3)\n\nFor each NVMe drive detected (drives must be bound to the **kernel NVMe driver** for this step,\nnot vfio-pci), run `tools\u002Finspect-nvme.py`. This characterizes drive behavior critical to\nbenchmark tuning:\n\n```\nsudo python3 tools\u002Finspect-nvme.py \u002Fdev\u002Fnvme\u003CN>n1 --skip-gc --skip-power\n```\n\nUse `--skip-gc` and `--skip-power` to skip the long-running settle and power-state tests\n(those take 10–30 minutes per drive). Run the full inspection only when you have time:\n\n```\nsudo python3 tools\u002Finspect-nvme.py \u002Fdev\u002Fnvme\u003CN>n1\n```\n\nKey fields to extract into the YAML profile per drive:\n- `model`, `firmware_rev`, `serial` (from `identify`)\n- `capacity_bytes` → convert to TB: `capacity_tb = capacity_bytes \u002F 1e12`\n- `mdts_bytes`, `block_size` (from `identify`)\n- `vwc_enabled`: `bool(identify['vwc'] & 0x1)` (volatile write cache)\n- `has_read_cache`, `cache_speedup` (from `read_cache`, if run)\n- `sustained_write_mbps` (from `gc_settle.sustained_write_mbps`, if run)\n- `recommended_gc_settle_s` (from `gc_settle.recommended_gc_settle_s`, if run)\n- `power_state_transitions_at_s` (first element of `power_states.power_state_transitions_at_s`, if run; null if empty list)\n- Read latency: `read_profile` entries — extract avg_us for qd=1, qd=16, qd=32\n\nAfter running inspect-nvme.py, rebind drives to vfio-pci before SPDK benchmarks.\n\n## Setup (Step 4)\n\n- Kill any running `certus-server-yaml` (it holds VFIO groups)\n- Wait for VFIO groups to release (`fuser \u002Fdev\u002Fvfio\u002F*`)\n- Bind all NVMe drives to vfio-pci: run the SPDK setup script\n  - Find it: `deps\u002Fspdk\u002Fscripts\u002Fsetup.sh` or `deps\u002Fspdk-build\u002F..\u002Fscripts\u002Fsetup.sh`\n  - If not found, try `HUGEMEM=4096 scripts\u002Fsetup.sh` from SPDK source dir\n- Verify all detected NVMe drives appear in `\u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F`\n\n## Path 1: NVMe Sequential Read (Step 5)\n\nTool: `spdk_nvme_perf`\n\nEvery NVMe command in the SPDK path is a DMA transfer — the NVMe controller DMAs data\ndirectly into the destination buffer (host RAM or BAR1) over PCIe using PRPs. The block\nsize passed to `spdk_nvme_perf` controls how many NVMe commands are grouped into one\nlogical IO, but each individual NVMe command is still bounded by MDTS (typically 128 KiB).\nThe block size matrix below therefore spans the full range from native MDTS size up through\nlarge logical groupings, giving both the per-command efficiency curve and the link\nsaturation behavior in one pass.\n\nMatrix dimensions:\n- **Drive counts**: 1, 2 (if ≥2 drives), 4 (if ≥4), 6 (if ≥6), all-detected\n- **Block sizes**: 4 KiB, 16 KiB, 64 KiB, 128 KiB, 256 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB\n  - 4–16 KiB: well below MDTS — CPU submission overhead and PCIe TLP inefficiency dominate;\n    throughput will be low but shows the floor and confirms MDTS is not the only factor\n  - 64–128 KiB: sub-MDTS and at-MDTS — isolates per-command PCIe efficiency; at QD=32\n    only 2–4 MiB in flight so the link may not yet be saturated at these sizes\n  - 256 KiB–1 MiB: MDTS-straddling zone where logical IOs split into 2–8 NVMe commands;\n    throughput typically plateaus here as the link saturates\n  - 2–8 MiB: large logical groupings; throughput should match 1 MiB if link-saturated.\n    Confirms saturation and exercises the logical IO grouping path.\n  Note: at small block sizes (4–128 KiB), QD=32 may not fully saturate the link — this\n  is expected and informative. The goal is to see where the throughput knee occurs, not\n  to reach absolute peak at every size. QD=32 is sufficient to observe the knee clearly.\n- **Queue depths**: 16, 32\n  QD=16 typically saturates the PCIe Gen4 x4 link at block sizes ≥256 KiB. Include\n  QD=32 to confirm saturation (identical throughput = link-bound) and to catch drives\n  where QD=16 is insufficient (smaller MDTS, higher per-IO latency, or BAR1 targets\n  with posted-write ordering delays).\n- **Duration**: 5 seconds per run\n- **Workload**: sequential read (`-w read`)\n\nFor multi-drive tests, pass multiple `-r 'trtype:pcie traddr:0000:XX:00.0'` flags.\nGroup drives by root complex when possible (test same-root groups and cross-root groups separately).\n\nParse output: the \"Total\" line gives IOPS, MiB\u002Fs, and average\u002Fmin\u002Fmax latency.\nConvert MiB\u002Fs to decimal GB\u002Fs using: `GB\u002Fs = MiB\u002Fs × 1.048576 \u002F 1000`.\nUse this formula consistently across all paths — the evaluator compares percentages\nagainst these numbers, so approximate conversions cause drift.\n\nOutput: Two matrices (throughput GB\u002Fs, latency us) with rows = (drives, block_size) and\ncolumns = QD. Note the **optimal read chunk size**: the smallest block size that achieves\n≥95% of the peak throughput for that drive count. This is the recommended `--chunk-size`\nfor `nvme-bar1-bench` and the NVMe command size in the dispatcher cold path.\n\nIf QD=16 and QD=32 produce identical throughput at a given size, note that the drive is\nlink-saturated at that size — deeper queuing adds only latency.\n\n## Path 2: NVMe Sequential Write (Step 6)\n\nTool: `spdk_nvme_perf`\n\nSame block size matrix as Path 1 — identical sizes allow direct read\u002Fwrite comparison\nand the same efficiency-curve analysis. Write throughput is the ceiling for the Certus\nwrite-through path (background flusher). Write efficiency vs size may differ from reads\ndue to FTL write amplification, especially at sub-MDTS sizes (64 KiB) where partial NAND\npage writes increase write amplification.\n\nMatrix dimensions:\n- **Drive counts**: 1, 2 (if ≥2 drives), 4 (if ≥4), all-detected\n  (6-drive write is less critical — the write path is single-drive-per-extent)\n- **Block sizes**: 4 KiB, 16 KiB, 64 KiB, 128 KiB, 256 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB\n- **Queue depths**: 16, 32\n- **Duration**: 5 seconds per run\n- **Workload**: sequential write (`-w write`)\n\n```\nsudo deps\u002Fspdk-build\u002Fbin\u002Fspdk_nvme_perf \\\n    -r 'trtype:pcie traddr:0000:XX:00.0' \\\n    -q 16 -o 131072 -w write -t 5\n```\n\nParse output identically to the read path. Convert MiB\u002Fs to decimal GB\u002Fs.\n\nOutput: Two matrices (throughput GB\u002Fs, latency us) with rows = (drives, block_size) and\ncolumns = QD. Note the **optimal write chunk size**: smallest block size achieving ≥95%\nof peak write throughput. Compare peak write GB\u002Fs against the `inspect-nvme.py` sustained\nwrite figure to verify SPDK and kernel-mode numbers are consistent.\n\n## Path 3: GPU PCIe Bandwidth — cudaMemcpy (Step 7)\n\nSource: `benchmarks\u002Fhw-ceiling\u002Fcuda_bw_test.cu`. Binary output: `\u002Ftmp\u002Fcuda_bw_test`.\n\nBuild step:\n```\n# Preferred — source already in repo:\nmake -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp cuda_bw_test\n\n# If benchmarks\u002Fhw-ceiling\u002F is absent: write the source to benchmarks\u002Fhw-ceiling\u002Fcuda_bw_test.cu\n# (creating the directory), then run make as above. Do NOT write to \u002Ftmp\u002F only —\n# saving to benchmarks\u002Fhw-ceiling\u002F means future runs skip the write step entirely.\n```\n\n**NUMA locality matters here.** H2D bandwidth depends on where the pinned host buffer\nis allocated relative to the GPU's root complex. If the GPU's PCIe root is on NUMA node 1\nbut the calling thread (and therefore `cudaMallocHost`) is on NUMA node 0, the DMA engine\nmust read the source buffer across the Infinity Fabric inter-socket link — this can reduce\nH2D throughput by 2–3+ GB\u002Fs compared to a NUMA-local allocation. Run the benchmark twice:\n\n```\n# NUMA-local: pin to the NUMA node that owns the GPU's root complex\n# First determine which NUMA node each GPU's PCIe root is on (from lspci\u002Fnumactl --hardware)\nnumactl --cpunodebind=\u003Cgpu_numa_node> --membind=\u003Cgpu_numa_node> \u002Ftmp\u002Fcuda_bw_test\n\n# Cross-NUMA: force host buffer onto the opposite NUMA node\nnumactl --cpunodebind=\u003Cother_numa_node> --membind=\u003Cother_numa_node> \u002Ftmp\u002Fcuda_bw_test\n```\n\nThe difference between these two runs is the Infinity Fabric H2D penalty. The NUMA-local\nresult is the true hardware ceiling; the cross-NUMA result is what Certus sees if CPU\naffinity is not set correctly.\n\nProgram behaviour:\n- `cudaMallocHost` + `cudaMemcpy` (pinned host memory)\n- Measure H2D and D2H\n- Transfer sizes: 128 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB, 16 MiB, 32 MiB, 64 MiB, 256 MiB\n- 50 iterations, 5 warmup\n- `cudaEvent` timing\n- Runs on ALL GPUs in a single invocation (loops over `cudaGetDeviceCount`)\n- Also measures H2D with 2, 4, and 8 concurrent `cudaMemcpyAsync` on separate streams\n  (each stream copies a different pinned buffer → different GPU destination).\n  This matters because single-stream H2D at 4 MiB may only reach ~10 GB\u002Fs on a x16 link\n  that can do ~25+ GB\u002Fs with concurrency.\n\nOutput: Per-GPU table for each NUMA binding: Size | H2D 1-stream (GB\u002Fs) | H2D 4-stream (GB\u002Fs) | D2H (GB\u002Fs).\nNote the per-GPU NUMA-local vs cross-NUMA delta — this is the cost of incorrect CPU affinity.\n\n## Path 4: GPU D2D Bandwidth — cudaMemcpyDeviceToDevice (Step 8)\n\nSource: `benchmarks\u002Fhw-ceiling\u002Fcuda_d2d_test.cu`. Binary output: `\u002Ftmp\u002Fcuda_d2d_test`.\n\nBuild step:\n```\n# Preferred — source already in repo:\nmake -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp cuda_d2d_test\n\n# If benchmarks\u002Fhw-ceiling\u002F is absent: write the source to benchmarks\u002Fhw-ceiling\u002Fcuda_d2d_test.cu\n# (creating the directory if needed), then run make as above.\n```\n\nThe P2P cold path uses a staging ring (GDRCopy BAR1-mapped GPU memory) as the NVMe DMA\ntarget, then does a D2D copy from staging → client's final GPU destination. This D2D\ncopy is part of the critical path and its bandwidth is a ceiling on throughput.\n\nProgram behaviour:\n- Source: `cudaMalloc` region A; Destination: `cudaMalloc` region B\n- Transfer sizes: 128 KiB, 512 KiB, 1 MiB, 2 MiB, 4 MiB, 8 MiB, 16 MiB\n- 100 iterations, 10 warmup\n- `cudaEvent` timing\n- Runs on ALL GPUs in a single invocation\n- Also measures with 2 and 4 concurrent async D2D copies on separate streams\n\nThis tells us whether D2D is fast enough to not bottleneck the pipeline,\nand whether multiple streams can overlap.\n\nOutput: Table: Size | 1-stream D2D (GB\u002Fs) | 2-stream (GB\u002Fs) | 4-stream (GB\u002Fs).\n\n## Path 5: GDRCopy BAR1 Mapping Bandwidth (Step 9)\n\nGDRCopy provides the BAR1 mapping that makes NVMe P2P work. It also exposes\nCPU-accessible bandwidth to GPU memory via BAR1, relevant for:\n- Metadata writes to GPU memory\n- Verifying BAR1 mapping is functional\n\n**Important context**: In the Certus P2P path, GDRCopy is used for MAPPING SETUP\n(gdr_pin_buffer + gdr_map + spdk_mem_register), NOT for data transfer. The actual\ndata transfer is NVMe DMA through the BAR1 aperture. But `gdrcopy_copybw` still\ntells us the BAR1 aperture bandwidth from the CPU side, which is a useful diagnostic.\n\nIf `gdrcopy_copybw` or `gdrcopy_copylat` is found:\n```\ngdrcopy_copybw\ngdrcopy_copylat\n```\n\nIf not installed, try to find and compile from source:\n```\nfind \u002F -path \"*\u002Fgdrcopy*\u002Ftests\" -type d 2>\u002Fdev\u002Fnull\n# or from the kernel\u002Fmodules\u002Fgdrcopy tree in this repo\ncd kernel\u002Fmodules\u002Fgdrcopy && make PREFIX=\u002Ftmp\u002Fgdrcopy lib lib_install\ncd tests && make\n```\n\nIf GDRCopy is not available at all, note in report and provide install instructions.\n\nOutput: Table of sizes vs H2D\u002FD2H bandwidth (GB\u002Fs).\nNote: CPU→BAR1 writes are often several GB\u002Fs to ~20+ GB\u002Fs depending on platform,\nNUMA placement, PCIe generation, and write-combining behavior.\nCPU←BAR1 reads are typically 1-5 GB\u002Fs (uncacheable reads, severe amplification).\n\n## Path 6: GPUDirect Storage — gdsio (Step 10)\n\nIf `lsmod | grep nvidia_fs` succeeds AND `gdsio` binary is found:\n\nUnbind drives from vfio-pci and rebind to kernel NVMe driver first:\n```\ndeps\u002Fspdk\u002Fscripts\u002Fsetup.sh reset\n```\n\nThen run gdsio for each topology pairing:\n- Same-root drives → same-root GPU\n- Cross-root drives → cross-root GPU (to measure penalty)\n\nMatrix:\n- Drive counts: 1, 2, 4\n- Block sizes: 4M, 8M, 16M\n- GPU: each available GPU\n- IO depth: 16, 32\n\n```\ngdsio -f \u002Fdev\u002FnvmeXn1 -d \u003Cgpu-index> -w 4 -s \u003Cblock-size> -x 0 -I 1 -T 10\n```\n\nAfter GDS tests, rebind drives back to vfio-pci for other tests.\n\nIf nvidia_fs is NOT loaded, note in report:\n```\nGDS not available.\nnvidia_fs module status: [not installed | installed but fails to load | loaded]\nIf not working, check driver version compatibility:\n  modinfo nvidia | grep version\n  modinfo nvidia-fs | grep version\nInstall: sudo dnf install nvidia-gds nvidia-fs-dkms\n```\n\nOutput: Table with (drives, block_size, GPU, topology) → bandwidth GB\u002Fs.\n\n## Path 7: Host RAM Bandwidth — memcpy baseline (Step 11)\n\nSource: `benchmarks\u002Fhw-ceiling\u002Fmemcpy_bench.c`. Binary output: `\u002Ftmp\u002Fmemcpy_bench`.\n\nBuild step:\n```\n# Preferred — source already in repo:\nmake -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp memcpy_bench\n\n# If benchmarks\u002Fhw-ceiling\u002F is absent: write the source to benchmarks\u002Fhw-ceiling\u002Fmemcpy_bench.c\n# (creating the directory if needed), then run make as above.\n```\n\nProgram behaviour:\n- Same-NUMA memcpy (both allocs on node 0, via `numa_alloc_onnode`)\n- Cross-NUMA memcpy (src on node 0, dst on node 1) — if ≥2 NUMA nodes\n- Sizes: 4 MiB, 16 MiB, 64 MiB, 256 MiB\n- 200 iters for ≤16 MiB, 50 iters for ≥64 MiB\n\nThis establishes the memory-tier ceiling when data is in host RAM (before GPU transfer).\n\nOutput: Table: Size | Same-NUMA (GB\u002Fs) | Cross-NUMA (GB\u002Fs).\n\n## Path 8: Topology-Aware P2P Matrix (Step 12)\n\nFor each (drive-group, GPU) pairing, summarize the expected path:\n- Same root complex: direct PCIe path\n- Cross root complex: traverses CPU\u002FInfinity Fabric\n- Cross NUMA: traverses inter-socket link\n\nCreate a topology matrix showing the routing for every possible pairing.\n\n## Path 9: NVMe DMA → BAR1 Microbenchmark (Step 13)\n\nTool: `nvme-bar1-bench` (`apps\u002Fnvme-bar1-bench\u002F`)\n\nThis is the most important measurement — it gives the closest isolated P2P cold-path\nhardware ceiling. Note: it still includes SPDK FFI, qpair behavior, poller\nimplementation, GDRCopy mapping, and VFIO\u002FIOMMU effects — it is the practical\nceiling for Certus, not a pure PCIe-theoretical ceiling. It bypasses the actor\u002Fchannel model and calls SPDK FFI directly. Each drive\ngets a dedicated CPU-pinned poller thread with its own qpair. Runs two modes\nback-to-back: host-ram (baseline) and bar1 (GDRCopy-mapped GPU memory).\n\nRequires: `gdrdrv` kernel module loaded. Depending on the driver stack,\n`nvidia_peermem` may also be required; detect and report whether it is loaded.\nIf `gdrdrv` is absent, skip this path and note in the report.\n\n```\ncargo build --release -p nvme-bar1-bench\n```\n\n**Critical: use a large total-bytes (≥2 GiB) to read past the drive's DRAM cache.**\nSmall streams (16–64 MiB) will report inflated numbers from cache hits.\n\n**Critical: report exactly which drive BDFs were selected for each run.**\nDo not interpret topology results unless the selected drives are shown. The\n`--drive-count N` flag uses the first N discovered drives, which may not correspond\nto the drives same-root with the target GPU.\n\n**NUMA and topology pinning are critical for correct ceiling measurements.**\n`nvme-bar1-bench` spawns one poller thread per drive. Those threads issue NVMe commands\nand wait for completions — their NUMA locality determines where the host-RAM DMA target\nbuffer is allocated and which memory controller handles the PCIe read\u002Fwrite traffic.\n\n- **NUMA-local run**: pin poller threads to the NUMA node that owns the target GPU's PCIe\n  root complex, AND select drives from that same root complex. This gives the true hardware\n  ceiling with no cross-fabric penalty.\n- **Cross-NUMA run**: pin to the opposite NUMA node. Quantifies the Infinity Fabric penalty\n  that Certus incurs when CPU affinity is not set. Required to understand observed production\n  performance, not just the theoretical ceiling.\n\nUse `sudo numactl --cpunodebind=N --membind=N` to wrap the benchmark. Determine the correct\nNUMA node for each GPU from `numactl --hardware` and the GPU's PCI BDF (cross-reference\nwith `lspci -tv`). Verify that `--drive-count N` picks drives from the right root complex\nby checking the BDFs printed in the benchmark output — the tool sorts drives by PCI address\nand takes the first N, which may not be the same-root drives for the target GPU.\n\nRun matrix:\n```\n# --- NUMA-local runs (true ceiling) ---\n# Substitute \u003Cgpu0_numa> with the NUMA node of GPU 0's PCIe root complex.\n# Verify BDFs in output confirm drives are same-root with GPU 0.\n\n# 1 drive, NUMA-local, GPU 0\nsudo numactl --cpunodebind=\u003Cgpu0_numa> --membind=\u003Cgpu0_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 1 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 16 --iterations 3\n\n# 4 drives, NUMA-local, GPU 0\nsudo numactl --cpunodebind=\u003Cgpu0_numa> --membind=\u003Cgpu0_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 4 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n\n# 6 drives, NUMA-local, GPU 0\nsudo numactl --cpunodebind=\u003Cgpu0_numa> --membind=\u003Cgpu0_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 6 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n\n# 4 drives, NUMA-local, GPU 1 (topology-best pairing for GPU 1)\nsudo numactl --cpunodebind=\u003Cgpu1_numa> --membind=\u003Cgpu1_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 4 --gpu 1 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n\n# --- Cross-NUMA run (quantify Infinity Fabric penalty) ---\n# Pin to the NUMA node opposite to GPU 0's root complex.\n# Same drive count and GPU as the NUMA-local 4-drive run for a direct apples-to-apples comparison.\n\n# 4 drives, cross-NUMA, GPU 0\nsudo numactl --cpunodebind=\u003Cother_numa> --membind=\u003Cother_numa> \\\n    target\u002Frelease\u002Fnvme-bar1-bench --drive-count 4 --gpu 0 \\\n    --chunk-size 131072 --total-bytes 2147483648 --queue-depth 32 --iterations 3\n```\n\nCLI reference:\n- `--drive-count N`: use first N discovered drives (sorted by PCI address)\n- `--gpu INDEX`: CUDA device index for BAR1 target [default: 0]\n- `--chunk-size BYTES`: NVMe IO size, must be ≤ drive MDTS (typically 128 KiB) [default: 131072]\n- `--total-bytes BYTES`: total bytes per drive per iteration [default: 16 MiB — TOO SMALL, use 2G]\n- `--queue-depth N`: in-flight commands per drive [default: 16]\n- `--warmup N`: warmup iterations [default: 3]\n- `--iterations N`: measured iterations [default: 20]\n\nSanity check: if 1-drive host-RAM result is >20% below the SPDK perf single-drive\nnumber, the microbenchmark itself may be adding overhead (e.g. missing doorbell\nbatching). Note this in the report — the host-ram vs bar1 comparison remains valid\nbut the absolute numbers may understate the hardware ceiling.\n\nObserved on current node (Intel Sentinel Rock, Gen4 x4, NVIDIA A30):\n- 1 drive: host-ram ≈ bar1 ≈ 4.3 GB\u002Fs (0% BAR1 overhead)\n- 4 drives: host-ram ≈ 24 GB\u002Fs, bar1 ≈ 20 GB\u002Fs (~17% BAR1 overhead)\n- 6 drives: host-ram ≈ 35 GB\u002Fs, bar1 ≈ 21.5 GB\u002Fs (~39% overhead, GPU BAR1 saturates)\n\nThese are node-specific and should not be treated as universal expected values.\n\n## Report Generation (Step 14)\n\nProduce TWO outputs:\n\n### 1. Markdown report: `\u003Coutput-dir>\u002Fhardware-ceiling-\u003CYYYY-MM-DD>.md`\n\nWrite the report with:\n1. Platform section (all detected hardware, topology diagram)\n2. CPU feature report (full output of `check-amd-proc-features.py`, formatted as a code block)\n3. NVMe device inspection summary (from `inspect-nvme.py` per drive — key fields, not full output)\n4. PCIe theoretical bandwidth table\n5. Per-path matrices (Paths 1–8, each gets its own section; read and write matrices include the IO size knee annotation)\n6. Topology routing matrix (Path 9 — derived, no additional benchmarks)\n7. Summary table (see below)\n8. Observations and recommendations\n\nThe summary must contain **four sub-tables**:\n\n**Sub-table 1 — Raw hardware paths** (measured primitives, no dispatcher logic):\n```\n| Data Path                              | Peak BW (GB\u002Fs) | Limiting Factor              | Notes              |\n|----------------------------------------|----------------|------------------------------|--------------------|\n| NVMe → Host read (1 drive)             |                | Drive PCIe x4                |                    |\n| NVMe → Host read (4 drives)            |                | 4× PCIe x4 aggregate         |                    |\n| NVMe → Host read (6\u002F7 drives)          |                | N× PCIe x4 aggregate         |                    |\n| NVMe → Host write (1 drive)            |                | Drive PCIe x4 \u002F FTL          |                    |\n| NVMe → Host write (4 drives)           |                | 4× PCIe x4 aggregate \u002F FTL   |                    |\n| Optimal read chunk size                |                | (bytes — knee of curve)      |                    |\n| Optimal write chunk size               |                | (bytes — knee of curve)      |                    |\n| Host → GPU 0 H2D (NUMA-local, peak)    |                | GPU PCIe x16                 |                    |\n| Host → GPU 0 H2D (cross-NUMA, peak)    |                | Infinity Fabric              | penalty vs local   |\n| Host → GPU 1 H2D (NUMA-local, peak)    |                | GPU PCIe x16                 |                    |\n| Host → GPU 1 H2D (cross-NUMA, peak)    |                | Infinity Fabric              | penalty vs local   |\n| GPU N → Host (D2H)                     |                | PCIe read amplification      |                    |\n| GPU D2D (1-stream sustained)           |                | HBM internal bus             |                    |\n| GDRCopy CPU→BAR1 write                 |                | BAR1 write-combining         |                    |\n| GDRCopy CPU←BAR1 read                  |                | BAR1 uncacheable reads       |                    |\n| Host RAM same-NUMA (sustained)         |                | Memory controller            |                    |\n| Host RAM cross-NUMA (sustained)        |                | Inter-socket link            |                    |\n| GDS NVMe → GPU                         | N\u002FA or XX.XX   | nvidia_fs \u002F true P2P DMA     |                    |\n```\n\n**Sub-table 2 — `dispatcher` warm path** (both `dispatcher` and `dispatcher-p2p` share this):\n```\nRoute: Memory-Tier (CUDA-pinned DRAM) ──H2D DMA──▶ GPU\nBoth components use memcpy_h2d_async from a DRAM memory-tier slot for warm hits.\nCeiling = H2D PCIe bandwidth (NVMe not involved).\n\n| GPU | Ceiling (GB\u002Fs) | Bound by        |\n|-----|----------------|-----------------|\n| 0   |                | H2D PCIe x16    |\n| 1   |                | H2D PCIe x16    |\n```\n\n**Sub-table 3 — `dispatcher` cold path** (NVMe → DRAM bounce → H2D):\n```\nRoute: NVMe ──DMA──▶ Memory-Tier (DRAM) ──H2D DMA──▶ GPU\nTwo-stage pipeline. Ceiling = min(NVMe_aggregate, H2D). Crossover point where H2D\nbecomes the cap = H2D_ceiling \u002F per_drive_bandwidth drives.\n\n| Drives | GPU | Ceiling (GB\u002Fs) | Bound by                          |\n|--------|-----|----------------|-----------------------------------|\n| 1      | N   |                | NVMe x4                           |\n| 4      | N   |                | NVMe aggregate (if \u003C H2D ceiling) |\n| 6      | N   |                | H2D PCIe x16 or NVMe aggregate    |\n```\n\n**Sub-table 4 — `dispatcher-p2p` cold path** (NVMe → BAR1 direct):\n```\nRoute: NVMe ──DMA──▶ BAR1 (GDRCopy-mapped GPU memory)\nSingle-stage. Ceiling = nvme-bar1-bench BAR1 result (includes PCIe + BAR1 overhead).\nBAR1 saturates around 21–22 GB\u002Fs on A30; adding drives beyond that point yields no gain.\n\n| Drives | GPU | BAR1 ceiling (GB\u002Fs) | BAR1 overhead vs host-ram | Notes                    |\n|--------|-----|---------------------|--------------------------|--------------------------|\n| 1      | 0   |                     | ~0%                      | NUMA-local               |\n| 4      | 0   |                     | measured %               | NUMA-local               |\n| 4      | 0   |                     | measured %               | cross-NUMA (penalty run) |\n| 4      | 1   |                     | measured %               | NUMA-local, topology-best|\n| 6      | 0   |                     | measured %               | NUMA-local, saturated?   |\n```\n\nAfter the four sub-tables, add a **crossover comparison**:\n```\n| Drive count | dispatcher cold ceiling | dispatcher-p2p cold ceiling | Winner        |\n|-------------|------------------------|-----------------------------|---------------|\n| 1           |                        |                             |               |\n| 4           |                        |                             | ~tie or one   |\n| 6           |                        |                             | dispatcher or p2p |\n```\nThis exposes the key insight: at low drive counts dispatcher-p2p may win on latency\n(single-stage, no DRAM bounce) but at high drive counts dispatcher wins on throughput\n(H2D cap > BAR1 saturation ceiling).\n\n### 2. Machine-readable profile: `\u003Coutput-dir>\u002Fhardware_profile_\u003CYYYY-MM-DD>.yaml`\n\nThe filename includes the date so multiple profiles from different runs are preserved\nside-by-side for comparison. Use only measured values — omit any path that was\nskipped due to missing tools (do not fill with estimates or zeros).\n\n```yaml\n# Auto-generated by profile-hardware-ceiling skill\n# Date: YYYY-MM-DD\n# Node: \u003Chostname>\n\nplatform:\n  hostname: \u003Chostname>\n  cpu: \u003Cmodel>\n  numa_nodes: \u003CN>\n  kernel: \u003Cversion>\n  os: \u003Cdistro version>\n\ncpu_features:\n  vendor: \u003CAuthenticAMD | GenuineIntel | ...>\n  zen_generation: \u003C1-5 or null>\n  zen_label: \u003C\"Zen 4\" or \"N\u002FA (non-AMD)\">\n  has_avx512: \u003Ctrue\u002Ffalse>\n  has_avx2: \u003Ctrue\u002Ffalse>\n  has_sse4_2: \u003Ctrue\u002Ffalse>\n  has_pclmulqdq: \u003Ctrue\u002Ffalse>\n  has_aes: \u003Ctrue\u002Ffalse>\n  xxh3_tier: \u003C\"AVX-512\" | \"AVX2\" | \"SSE2\" | \"Scalar\">\n  xxh3_throughput_estimate: \u003C\"~40-50 GB\u002Fs\" or similar string>\n  certus_recommended_hash: \u003C\"XXH3_128bits (AVX-512 path)\" or similar>\n\ngpus:\n  gpu0:\n    model: \u003Cname>\n    pcie_gen: 4\n    pcie_width: 16\n    bar1_gib: 32\n    pci_bdf: \"a1:00.0\"\n    numa_node: \u003Cint>                        # NUMA node of GPU's PCIe root complex\n    # NUMA-local: cudaMallocHost on same NUMA node as GPU's root complex\n    h2d_4m_1stream_numa_local_gbps: \u003Cmeasured>\n    h2d_peak_numa_local_gbps: \u003Cmeasured at best transfer size>\n    d2h_peak_numa_local_gbps: \u003Cmeasured>\n    # Cross-NUMA: host buffer on opposite NUMA node\n    h2d_4m_1stream_cross_numa_gbps: \u003Cmeasured>\n    h2d_peak_cross_numa_gbps: \u003Cmeasured>\n    cross_numa_h2d_penalty_gbps: \u003Ccomputed: local - cross>\n    d2d_1stream_gbps: \u003Cmeasured>\n    d2d_4stream_gbps: \u003Cmeasured>\n  gpu1:\n    # same structure\n\nnvme:\n  drives:\n    - bdf: \"61:00.0\"\n      model: \u003Cname>\n      firmware_rev: \u003Cstring>\n      serial: \u003Cstring>\n      capacity_tb: \u003Cfloat>          # capacity_bytes \u002F 1e12\n      mdts_bytes: 131072\n      pcie_gen: 4\n      pcie_width: 4\n      numa_node: \u003Cint>              # from sysfs numa_node attribute\n      vwc_enabled: \u003Ctrue\u002Ffalse>     # bool(vwc & 0x1)\n      has_read_cache: \u003Ctrue\u002Ffalse>\n      # from inspect-nvme.py gc_settle (if run):\n      sustained_write_mbps: \u003Cmeasured or null>\n      recommended_gc_settle_s: \u003Cint or null>\n      # from inspect-nvme.py power_states (if run):\n      power_state_transition_at_s: \u003Cint or null>\n      # from inspect-nvme.py read_profile (if run):\n      read_latency_qd1_us: \u003Cfloat or null>\n      read_latency_qd16_us: \u003Cfloat or null>\n      read_latency_qd32_us: \u003Cfloat or null>\n    - bdf: \"62:00.0\"\n      # ...\n  mdts_bytes: 131072         # representative MDTS (use smallest across all drives)\n  raw_read_gbps:\n    1_drive: \u003Cmeasured>\n    4_drive: \u003Cmeasured>\n    6_drive: \u003Cmeasured>\n  raw_write_gbps:\n    1_drive: \u003Cmeasured>\n    4_drive: \u003Cmeasured>\n  # IO size sweep: 1 drive, QD=32. Read and write use the same sizes.\n  # Derived from Path 1 (read) and Path 2 (write) 1-drive QD=32 rows.\n  io_size_sweep_read_gbps:\n    4kib: \u003Cmeasured>\n    16kib: \u003Cmeasured>\n    64kib: \u003Cmeasured>\n    128kib: \u003Cmeasured>\n    256kib: \u003Cmeasured>\n    512kib: \u003Cmeasured>\n    1mib: \u003Cmeasured>\n    2mib: \u003Cmeasured>\n    4mib: \u003Cmeasured>\n    8mib: \u003Cmeasured>\n    optimal_bytes: \u003Cint — smallest size achieving ≥95% of peak>\n  io_size_sweep_write_gbps:      # 1 drive, QD=32\n    4kib: \u003Cmeasured>\n    16kib: \u003Cmeasured>\n    64kib: \u003Cmeasured>\n    128kib: \u003Cmeasured>\n    256kib: \u003Cmeasured>\n    512kib: \u003Cmeasured>\n    1mib: \u003Cmeasured>\n    2mib: \u003Cmeasured>\n    4mib: \u003Cmeasured>\n    8mib: \u003Cmeasured>\n    optimal_bytes: \u003Cint>\n\nbar1:\n  # nvme-bar1-bench results (per GPU target)\n  # Each entry has numa_local (cpunodebind = GPU's NUMA node) and cross_numa variants.\n  # The numa_local result is the true hardware ceiling.\n  # The cross_numa result quantifies the Infinity Fabric penalty.\n  gpu0:\n    numa_local:\n      host_ram_1drive_gbps: \u003Cmeasured>\n      host_ram_4drive_gbps: \u003Cmeasured>\n      host_ram_6drive_gbps: \u003Cmeasured>\n      bar1_1drive_gbps: \u003Cmeasured>\n      bar1_4drive_gbps: \u003Cmeasured>\n      bar1_6drive_gbps: \u003Cmeasured>\n      overhead_4drive_pct: \u003Ccomputed>\n      overhead_6drive_pct: \u003Ccomputed>\n    cross_numa:\n      host_ram_4drive_gbps: \u003Cmeasured>\n      bar1_4drive_gbps: \u003Cmeasured>\n      penalty_vs_local_pct: \u003Ccomputed>\n  gpu1:\n    numa_local:\n      host_ram_1drive_gbps: \u003Cmeasured>\n      host_ram_4drive_gbps: \u003Cmeasured>\n      bar1_1drive_gbps: \u003Cmeasured>\n      bar1_4drive_gbps: \u003Cmeasured>\n      overhead_4drive_pct: \u003Ccomputed>\n\ngdrcopy:\n  cpu_to_bar1_gbps: \u003Cmeasured or null>\n  bar1_to_cpu_gbps: \u003Cmeasured or null>\n\ngds:\n  available: \u003Ctrue\u002Ffalse>\n  # if available:\n  1drive_gbps: \u003Cmeasured>\n  4drive_gbps: \u003Cmeasured>\n\nhost_ram:\n  same_numa_gbps: \u003Cmeasured>\n  cross_numa_gbps: \u003Cmeasured or null>\n\ntopology:\n  # Which drives share a root complex with which GPU\n  gpu0_same_root_drives: [\"c1:00.0\", \"c2:00.0\"]\n  gpu1_same_root_drives: [\"61:00.0\", \"62:00.0\", \"63:00.0\", \"64:00.0\"]\n\n# Derived ceilings — keyed by Certus dispatcher path, for investigator\u002Fevaluator\n#\n# Certus has two dispatchers, each with a warm and cold path:\n#\n#   dispatcher (memory-tier DRAM bounce):\n#     warm:  Memory-Tier (DRAM) ──H2D──▶ GPU           ceiling = H2D bandwidth\n#     cold:  NVMe ──DMA──▶ Memory-Tier ──H2D──▶ GPU    ceiling = min(NVMe_agg, H2D)\n#\n#   dispatcher-p2p (direct BAR1, no DRAM):\n#     warm:  Memory-Tier (DRAM) ──H2D──▶ GPU           ceiling = H2D bandwidth (same as above)\n#     cold:  NVMe ──DMA──▶ BAR1 (GPU)                  ceiling = bar1 result from nvme-bar1-bench\n#\nceilings:\n  # All ceiling values use NUMA-local measurements (true hardware ceiling).\n  # Cross-NUMA penalty fields show the cost of incorrect CPU affinity in production.\n\n  # --- warm path (shared by both dispatchers) ---\n  # Use NUMA-local H2D — this is the ceiling Certus achieves with correct affinity.\n  warm_gpu0_gbps: \u003Cgpus.gpu0.h2d_4m_1stream_numa_local_gbps>\n  warm_gpu0_peak_gbps: \u003Cgpus.gpu0.h2d_peak_numa_local_gbps>\n  warm_gpu1_gbps: \u003Cgpus.gpu1.h2d_4m_1stream_numa_local_gbps>\n  warm_gpu1_peak_gbps: \u003Cgpus.gpu1.h2d_peak_numa_local_gbps>\n  # Penalty if affinity is wrong:\n  warm_gpu0_cross_numa_penalty_gbps: \u003Cgpus.gpu0.cross_numa_h2d_penalty_gbps>\n  warm_gpu1_cross_numa_penalty_gbps: \u003Cgpus.gpu1.cross_numa_h2d_penalty_gbps>\n\n  # --- dispatcher cold path: NVMe → DRAM → H2D ---\n  dispatcher_cold_1drive_gbps: \u003Cmin(nvme.raw_read_gbps.1_drive, warm_gpuX_peak_gbps)>\n  dispatcher_cold_4drive_gbps: \u003Cmin(nvme.raw_read_gbps.4_drive, warm_gpuX_peak_gbps)>\n  dispatcher_cold_6drive_gbps: \u003Cmin(nvme.raw_read_gbps.6_drive, warm_gpuX_peak_gbps)>\n  dispatcher_cold_bottleneck_at_4drive: \u003C\"nvme\" or \"h2d\">\n  dispatcher_cold_bottleneck_at_6drive: \u003C\"nvme\" or \"h2d\">\n  dispatcher_cold_crossover_drives: \u003Ccomputed: h2d_peak_gbps \u002F per_drive_gbps>\n\n  # --- dispatcher-p2p cold path: NVMe → BAR1 (NUMA-local) ---\n  p2p_cold_1drive_gbps: \u003Cbar1.gpuX.numa_local.bar1_1drive_gbps>\n  p2p_cold_4drive_gpu0_gbps: \u003Cbar1.gpu0.numa_local.bar1_4drive_gbps>\n  p2p_cold_4drive_gpu1_gbps: \u003Cbar1.gpu1.numa_local.bar1_4drive_gbps>\n  p2p_cold_6drive_gbps: \u003Cbar1.gpuX.numa_local.bar1_6drive_gbps>\n  p2p_cold_bar1_saturated: \u003Ctrue if adding drives past N yields \u003C5% gain>\n  # Cross-NUMA penalty:\n  p2p_cold_4drive_gpu0_cross_numa_gbps: \u003Cbar1.gpu0.cross_numa.bar1_4drive_gbps>\n  p2p_cold_cross_numa_penalty_pct: \u003Cbar1.gpu0.cross_numa.penalty_vs_local_pct>\n\n  # --- NVMe raw (no GPU overhead, reference) ---\n  nvme_raw_1drive_read_gbps: \u003Cnvme.raw_read_gbps.1_drive>\n  nvme_raw_4drive_read_gbps: \u003Cnvme.raw_read_gbps.4_drive>\n  nvme_raw_6drive_read_gbps: \u003Cnvme.raw_read_gbps.6_drive>\n  nvme_raw_1drive_write_gbps: \u003Cnvme.raw_write_gbps.1_drive>\n  nvme_raw_4drive_write_gbps: \u003Cnvme.raw_write_gbps.4_drive>\n\n  # --- DMA chunk size recommendation ---\n  nvme_optimal_read_chunk_bytes: \u003Cnvme.io_size_sweep_read_gbps.optimal_bytes>\n  nvme_optimal_write_chunk_bytes: \u003Cnvme.io_size_sweep_write_gbps.optimal_bytes>\n\n  # --- D2D (GPU internal; not a bottleneck for either dispatcher) ---\n  gpu_d2d_sustained_gbps: \u003Cgpus.gpuX.d2d_1stream_gbps at 8 MiB>\n\n  # --- Host RAM ---\n  host_ram_same_numa_gbps: \u003Chost_ram.same_numa_gbps>\n  host_ram_cross_numa_gbps: \u003Chost_ram.cross_numa_gbps>\n```\n\nThe `ceilings` section is consumed by:\n- **Rule-based investigator**: `efficiency = certus_measured \u002F ceiling[relevant_path]`\n- **Evaluation function**: pick the ceiling matching the active dispatch mode and drive count\n\nOnly populate entries where measurements succeeded. Use topology-correct GPU\u002Fdrive pairings\n(same-NUMA where available). Omit rather than estimate for skipped paths.\n\n## Notes for Portability\n\n- Do NOT hardcode PCI addresses — detect them from `lspci`\n- Do NOT hardcode drive counts — use what's available\n- Do NOT assume specific GPU models — query capabilities\n- If a tool is missing (nvcc, gdsio, gdrcopy_copybw, spdk_nvme_perf, nvme-bar1-bench),\n  skip that path and note in the report with install\u002Fbuild instructions\n- Handle single-GPU and multi-GPU systems\n- Handle single-NUMA and multi-NUMA systems\n- **CUDA\u002FC benchmark sources live in `benchmarks\u002Fhw-ceiling\u002F`** (`cuda_bw_test.cu`,\n  `cuda_d2d_test.cu`, `memcpy_bench.c`, `Makefile`). Always use these.\n  Build with `make -C benchmarks\u002Fhw-ceiling OUTDIR=\u002Ftmp \u003Ctarget>` — binaries go to `\u002Ftmp\u002F`.\n  Only write sources to `\u002Ftmp\u002F` if the repo checkout is somehow read-only (rare).\n- The YAML profile must be valid YAML parseable by any standard library\n- `inspect-nvme.py` requires drives bound to the **kernel NVMe driver** (not vfio-pci).\n  Run it before the SPDK setup step, or after GDS tests when drives are already rebound.\n- `check-amd-proc-features.py` requires `py-cpuinfo` (`pip install py-cpuinfo`).\n  It is read-only and can run at any point without affecting drive binding state.\n\n### Permissions setup for new users\n\nThe repo's `.claude\u002Fsettings.json` pre-allows read-only detection commands (lscpu,\nlspci, nvidia-smi, cargo, etc.). The sudo\u002Fhardware commands needed for actual\nbenchmarking require a per-user `.claude\u002Fsettings.local.json` (gitignored).\n\nTo run this skill without permission prompts, create `.claude\u002Fsettings.local.json`:\n```json\n{\n  \"permissions\": {\n    \"allow\": [\n      \"Bash(sudo lspci*)\",\n      \"Bash(sudo modprobe *)\",\n      \"Bash(sudo pkill*certus*)\",\n      \"Bash(sudo kill*)\",\n      \"Bash(sudo chmod * \u002Fdev\u002Fvfio\u002F*)\",\n      \"Bash(sudo fuser \u002Fdev\u002Fvfio\u002F*)\",\n      \"Bash(sudo *\u002Fdeps\u002Fspdk*\u002Fscripts\u002Fsetup.sh*)\",\n      \"Bash(sudo *\u002Fdeps\u002Fspdk-build\u002Fbin\u002Fspdk_nvme_perf*)\",\n      \"Bash(sudo *\u002Ftarget\u002Frelease\u002Fnvme-bar1-bench*)\",\n      \"Bash(sudo numactl*)\",\n      \"Bash(sudo *\u002Ftarget\u002Frelease\u002Fcertus-server-yaml*)\",\n      \"Bash(sudo python3 tools\u002Finspect-nvme.py*)\",\n      \"Bash(python3 tools\u002Fcheck-amd-proc-features.py*)\",\n      \"Bash(make -C benchmarks\u002Fhw-ceiling*)\",\n      \"Bash(\u002Fusr\u002Flocal\u002Fcuda*\u002Fbin\u002Fnvcc*)\",\n      \"Bash(CUDA_VISIBLE_DEVICES=* \u002Ftmp\u002F*)\",\n      \"Bash(\u002Ftmp\u002F*)\",\n      \"Bash(find \u002F -name *gdrcopy*)\",\n      \"Bash(find \u002F -name *gdsio*)\",\n      \"Bash(ls \u002Fsys\u002Fbus\u002Fpci\u002Fdrivers\u002Fvfio-pci\u002F)\",\n      \"Bash(ls \u002Fdev\u002Fvfio\u002F)\",\n      \"Write(*\u002Fprofiling\u002F*)\",\n      \"Write(\u002Ftmp\u002F*)\"\n    ]\n  }\n}\n```\n\nOr let Claude prompt you on first run and approve each category once.\n\n## Cleanup (Step 15)\n\nLeave drives in whatever state the last test needed (vfio-pci if GDS was not run,\nkernel driver if GDS was run last). Do NOT restart certus-server — leave the system\nready for the user to run further tests. Report which driver the drives are currently bound to.\n",{"name":5,"description":7,"argument-hint":35},"[output-dir]","unstable","dc564aa356769b7754391c9fa4fdaa0f179f476c",".claude\u002Fskills\u002Fprofile-hardware-ceiling",4,"https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research","https:\u002F\u002Fai-native-systems-research.github.io\u002Fai-native-systems-research\u002F","2026-11-12 06:08:32",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add AI-native-Systems-Research\u002Fai-native-storage-certus --skill profile-hardware-ceiling",{"tool":50,"command":51},"bunx","bunx skills add AI-native-Systems-Research\u002Fai-native-storage-certus --skill profile-hardware-ceiling",{"tool":53,"command":54},"pnpm","pnpm dlx skills add AI-native-Systems-Research\u002Fai-native-storage-certus --skill profile-hardware-ceiling","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002F.claude\u002Fskills\u002Fprofile-hardware-ceiling\u002FSKILL.md",true,[],[59,78,93,104,121,133],{"skillNo":60,"name":61,"nameZh":6,"description":62,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":63,"categories":64,"keywords":65,"publishedAt":30},"01M6KP29TWSNGADF3AJ47BR7TY","tools-spin-sync","Synchronize Spin\u002FPromela models with current source code and re-verify","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002Fcomponents\u002Fzyre\u002F.claude\u002Fskills\u002Ftools-spin-sync\u002FSKILL.md",[],[66,67,68,69,70,71,72,73,74,75,76,77],"tools spin sync","Tools","sync","code","verify","model","models","name","with","re","source","source code",{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":30},"01M6KP277TDY3ZAJEBP5P6MDT8","tools-spin-model","Create a new Spin\u002FPromela formal verification model for a system property","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002Fcomponents\u002Fzyre\u002F.claude\u002Fskills\u002Ftools-spin-model\u002FSKILL.md",[],[85,71,67,86,87,88,29,89,90,91,92],"tools spin model","system","formal verification","number","prove","new","create","verification",{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":30},"01M6KP250W406CEDSF02BRXT9D","tools-creusot-install","Install the Creusot Rust verification tool","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002Fcomponents\u002Fzyre\u002F.claude\u002Fskills\u002Ftools-creusot-install\u002FSKILL.md",[],[100,101,67,102,92,103],"tools creusot install","install","rust","tool",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":30},"01M6KP22H28FQ1X0V7VJHEFWXD","tools-creusot-init","Prepare an existing Cargo project to use Creusot formal verification","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002Fcomponents\u002Fzyre\u002F.claude\u002Fskills\u002Ftools-creusot-init\u002FSKILL.md",[],[111,67,112,113,114,115,116,92,117,87,118,119,120],"tools creusot init","init","project","directory","components","prepare","use","cargo","manager","Existing",{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":30},"01M6KP1V506J9YM8DKKVDG1NRE","tools-creusot-coq-install","Add Coq 8.20 as an additional prover to an existing Creusot installation, for verification conditions that SMT solvers cannot discharge","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002Fcomponents\u002Fzyre\u002F.claude\u002Fskills\u002Ftools-creusot-coq-install\u002FSKILL.md",[],[128,129,101,67,29,120,130,131,92,132],"tools creusot coq install","Coq","as","installation","add",{"skillNo":134,"name":135,"nameZh":6,"description":136,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":137,"categories":138,"keywords":139,"publishedAt":30},"01M6KP1PQJF3H765TYZAZ7P3EF","tools-creusot-annotate-interfaces","Formally verify a component's properties with Creusot and annotate its interface definitions","https:\u002F\u002Fgithub.com\u002FAI-native-Systems-Research\u002Fai-native-storage-certus\u002Fblob\u002Funstable\u002Fcomponents\u002Fzyre\u002F.claude\u002Fskills\u002Ftools-creusot-annotate-interfaces\u002FSKILL.md",[],[67,140,141,74,73,142,70,143,144],"tools creusot annotate interfaces","annotate","properties","component","interface"]