[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CP41N22W0PPT3F3T032JQZ":3,"skill-same:01M6CP41N22W0PPT3F3T032JQZ":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":51,"linkOnly":52,"markdownContent":53,"markdownBody":54,"frontMatter":55,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":59,"repositoryHtmlUrl":60,"repositoryHomepageUrl":61,"ownerHtmlUrl":62,"ownerBlogUrl":63,"ownerDescription":6,"updatedAt":64,"install":65,"resources":78},"01M6CP41N22W0PPT3F3T032JQZ","nsys-profiling",null,"Profile GPU workloads with NVIDIA Nsight Systems (nsys). Use this skill when the user says 'profile', 'nsys', 'capture a trace', 'GPU profiling', 'kernel breakdown', 'where is time spent', 'decode is slow', 'prefill is slow', 'TPOT regression', 'TTFT regression', or wants to understand where GPU time is going. Also trigger when the user pastes nsys output and asks for help interpreting it. This skill covers the full lifecycle: capturing traces with the right flags, analyzing kernel\u002FAPI summaries, detecting tail latency problems, and diagnosing performance regressions.","pegainfer-project\u002Fpegainfer","pegainfer-project","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F292134277?v=4","01M6CP3N2567A4DACEYMN5W8S0","01M6CP3N66CWB0M39SGYE63S7D",739,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50],"profiling","nsys profiling","use","prefill","breakdown","when","gpu profiling","summaries","for","detecting","systems","latency","full","time","flags","kernel","profile","user","capture","trace","regression","api","lifecycle","analyzing","NVIDIA","gpu","help","performance","2026-11-10 10:12:30",false,"---\nname: nsys-profiling\ndescription: \"Profile GPU workloads with NVIDIA Nsight Systems (nsys). Use this skill when the user says 'profile', 'nsys', 'capture a trace', 'GPU profiling', 'kernel breakdown', 'where is time spent', 'decode is slow', 'prefill is slow', 'TPOT regression', 'TTFT regression', or wants to understand where GPU time is going. Also trigger when the user pastes nsys output and asks for help interpreting it. This skill covers the full lifecycle: capturing traces with the right flags, analyzing kernel\u002FAPI summaries, detecting tail latency problems, and diagnosing performance regressions.\"\n---\n\n# nsys Profiling\n\nYou are helping a developer profile GPU workloads in a Rust + CUDA inference engine using NVIDIA Nsight Systems (`nsys`). The goal is always to answer \"where is time going and what should I optimize next?\"\n\n## Before you start\n\n1. **Confirm the workload.** Ask which model, which path (prefill vs decode vs both), and what prompt\u002Foutput shapes matter. If the user is vague, default to the two standard profiles:\n   - Prefill-heavy: `--prompt-len 2048 --output-len 1`\n   - Decode-heavy: `--prompt-len 1 --output-len 128`\n\n2. **Check `--release`.** Debug builds slow GPU kernels by 10x+ and produce misleading traces. If you see `cargo run` without `-r` or `--release`, stop and fix this first.\n\n3. **Check nsys is available.** Run `nsys --version`. If it fails, the user likely needs to add `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin` to PATH — a common gotcha in SSH or tmux sessions where `~\u002F.bashrc` doesn't source. Fix: `export PATH=\"\u002Fusr\u002Flocal\u002Fcuda\u002Fbin:$PATH\"`, or use full path `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fnsys`.\n\n4. **Check profiling environment.** Run `nsys status -e` to verify the system can profile. It checks kernel paranoid level, perf_event support, and LBR availability. If paranoid level is too high (>1), CPU sampling won't work without root.\n\n## Capturing a trace\n\n### Standard capture command\n\n```bash\nnsys profile --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002F\u003Ctrace_name> \\\n  cargo run -r --bin bench_serving -- request \\\n    --prompt-len \u003CPROMPT> --output-len \u003COUTPUT> --warmup 1 --iters 1\n```\n\nBuild this command for the user based on their workload. Always include all four flags:\n\n| Flag | Why |\n|------|-----|\n| `--force-overwrite=true` | nsys errors on existing files without this |\n| `--cuda-graph-trace=node` | Without it, CUDA Graph replay is opaque — you can't see individual kernels. This is the single most common nsys pitfall. Default is `graph` on driver ≥11.7, which hides everything inside the graph. |\n| `--export=sqlite` | Produces a `.sqlite` file that `nsys stats` reads directly |\n| `-o target\u002Fprofiling\u002F\u003Cname>` | Keep traces organized in one place |\n\n### Precise capture with cudaProfilerApi\n\nIf the `bench_serving` binary supports `--cuda-profiler-capture`, use it to capture only the measured iterations — this excludes model loading and warmup from the trace, producing a much cleaner profile:\n\n```bash\nnsys profile --force-overwrite=true --cuda-graph-trace=node \\\n  -c cudaProfilerApi --export=sqlite -o target\u002Fprofiling\u002F\u003Ctrace_name> \\\n  cargo run -r --bin bench_serving -- --cuda-profiler-capture request \\\n    --prompt-len \u003CPROMPT> --output-len \u003COUTPUT> --warmup 1 --iters 1\n```\n\nUse `--capture-range-end=repeat[:N]` to capture multiple separate ranges (e.g., N iterations) into the same trace. Each range produces its own segment.\n\n### Delay\u002Fduration capture (when cudaProfilerApi isn't available)\n\nFor workloads without profiler API integration, use `--delay` to skip startup and `--duration` to limit capture:\n\n```bash\nnsys profile --delay=10 --duration=30 --cuda-graph-trace=node \\\n  --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002F\u003Ctrace_name> \\\n  \u003Ccommand>\n```\n\n### Disabling CUDA Graph for cleaner kernel analysis\n\nCUDA Graph replay groups many kernels into a single opaque launch. Even with `--cuda-graph-trace=node`, profiler overhead inflates times differently than ungraphed launches. For precise per-kernel timing (e.g., comparing short vs long context), disable CUDA Graph:\n\n```bash\ncargo run -r --bin bench_serving -- --cuda-graph false request --prompt-len 2048 --output-len 32 ...\n```\n\nTradeoff: ungraphed traces show true individual kernel times but include ~5μs CPU launch overhead per kernel that the production (graphed) path avoids. Use graphed traces for real TPOT, ungraphed for kernel-level comparison.\n\n### Interactive profiling (long-running servers)\n\nFor profiling a server that's already running, use the launch\u002Fstart\u002Fstop workflow:\n\n```bash\n# Terminal 1: launch the server under nsys control, but don't start collecting yet\nnsys launch --trace=cuda,nvtx --cuda-graph-trace=node \\\n  --session-new=my_session -- cargo run -r --bin pegainfer-server -- ...\n\n# Terminal 2: start\u002Fstop collection on demand\nnsys start --session=my_session\n# ... send requests ...\nnsys stop --session=my_session\n```\n\nOr use `--start-later=true` with `nsys profile` to get the same deferred-start behavior in a single command.\n\n### Lighter traces\n\nIf the trace is too large or slow to process, limit tracing scope:\n\n```bash\n# Only CUDA + NVTX, skip OS scheduling noise — smaller trace, faster to open\nnsys profile --trace=cuda,nvtx --cuda-graph-trace=node ...\n```\n\nOther `--trace` options: `cublas` (trace cuBLAS API calls with parameters), `cudnn`, `osrt` (OS runtime like pthread), `mpi`, `python-gil`. Multiple values comma-separated.\n\n### Output naming tricks\n\nThe `-o` flag supports pattern substitution:\n- `%h` — hostname (useful for multi-node)\n- `%p` — PID of target process\n- `%n` — auto-incrementing number (avoids overwrite without `--force-overwrite`)\n- `%q{ENV_VAR}` — environment variable expansion\n- `%%` — literal `%`\n\nExample: `-o traces\u002F%h_rank%q{RANK}_%n` → `traces\u002Fgpu-node-01_rank0_1.nsys-rep`\n\n### Reproducible configs with `--command-file`\n\nStore nsys flags in a file to avoid long command lines:\n\n```bash\n# nsys_decode.cfg\n--force-overwrite=true\n--cuda-graph-trace=node\n--export=sqlite\n--trace=cuda,nvtx\n```\n\nThen: `nsys profile --command-file=nsys_decode.cfg -o target\u002Fprofiling\u002Ftrace \u003Ccommand>`\n\nCommand-line flags override file flags.\n\n### One-shot stats with `--stats=true`\n\nSkip the separate `nsys stats` step entirely:\n\n```bash\nnsys profile --stats=true --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nThis automatically generates all default summary reports (kern_sum, api_sum, mem_sum, etc.) to the console after collection finishes. Convenient for quick checks, but the output is long — better for scripted workflows.\n\n## Analyzing a trace\n\nRun these analyses in sequence. Each answers a different question.\n\n### Step 1: Kernel time breakdown\n\n```bash\nnsys stats --report cuda_gpu_kern_sum target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis tells you **which GPU kernels consume the most time**. Normal patterns:\n- MLP + GEMV typically >90% of decode time\n- Attention grows with sequence length (O(seq_len)) while MLP\u002FGEMV stay flat\n- RMSNorm and activation kernels are lightweight (\u003C3%)\n\n**Use `:base` to tame mangled names.** By default, kernel names are fully mangled C++ templates — unreadable. Append `:base` to strip template parameters:\n\n```bash\nnsys stats --report cuda_gpu_kern_sum:base target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis turns `std::enable_if\u003C!T7, void>::type internal::gemvx::kernel\u003Cint, int, __nv_bfloat16, ...>(T13)` into just `kernel`. Much easier to scan, but note that it merges kernels with different template instantiations. Use the full name when you need to distinguish shapes.\n\nThere's also `:mangled` if you need the raw linker symbol.\n\n### Step 2: CUDA API overhead\n\n```bash\nnsys stats --report cuda_api_sum target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis tells you **where the CPU\u002Fhost is spending time**. Watch for:\n- `cuMemcpyHtoDAsync` dominated by model loading (normal if one-time)\n- `cuStreamSynchronize` — if this is large during inference (not loading), the host is unnecessarily waiting for the GPU\n- `cuMemAllocAsync` \u002F `cuMemFreeAsync` — high counts or time means hot-path allocation churn\n- `cudaLaunchKernel` — large count × ~5μs each = significant CPU launch overhead\n\n### Step 3: Tail-aware stats\n\n```bash\npython3 tools\u002Fnsys_tail_stats.py target\u002Fprofiling\u002F\u003Ctrace>.sqlite --limit 15\n```\n\nThis is the team's custom tool that goes beyond nsys's built-in p50\u002Favg reporting. It shows `p50, p95, p99, max` and a tail score for each kernel. The p50 can hide serious problems:\n\n- **max\u002Fp50 > 2×**: Kernel has outlier invocations. Common causes: route imbalance in MoE, rank arrival skew in NCCL, allocator stalls.\n- **p99\u002Fp50 > 1.5×**: Systematic tail — not a one-off spike. Look for stream contention or variable work per invocation.\n- **High tail score but low total time**: The kernel is cheap on average but occasionally blocks everything. These are the sneakiest bottlenecks.\n\n### Step 4: Bandwidth efficiency (for GEMV-bound workloads)\n\nWhen GEMV dominates (>90% of GPU time, typical for bs=1 decode), compute memory bandwidth utilization to answer \"is there room to optimize, or are we already at the hardware limit?\"\n\n```\nBW_achieved = weight_bytes \u002F kernel_time\nBW_efficiency = BW_achieved \u002F GPU_peak_BW\n```\n\nFor each major GEMV shape, calculate weight size (M × K × sizeof(bf16)) and divide by the nsys avg kernel time. Compare against GPU peak memory bandwidth (e.g., RTX 5070 Ti = 896 GB\u002Fs, H100 = 3.35 TB\u002Fs). Large GEMVs (>20MB weights) typically reach 85-90% efficiency; small GEMVs (\u003C5MB) drop to 60-70% due to launch overhead and tail effects.\n\nIf overall efficiency is >80%, GEMV optimization has limited headroom — quantization or batching is the path forward. If \u003C70%, there's room for custom kernels.\n\n### Step 5 (optional): Auto-analysis\n\n```bash\nnsys analyze target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nRuns NVIDIA's built-in rules: `cuda_memcpy_async`, `cuda_memcpy_sync`, `cuda_memset_sync`, `cuda_api_sync`, `gpu_gaps`, `gpu_time_util`. It flags synchronous memcpy, unnecessary device syncs, and periods where the GPU is idle >500ms. Quick sanity check, but won't catch domain-specific problems.\n\nRun a specific rule: `nsys analyze --rule cuda_api_sync \u003Ctrace>.sqlite`\n\n## Advanced analysis techniques\n\n### Time-window filtering\n\nIsolate just the inference phase by filtering out model loading:\n\n```bash\n# Skip first 5 seconds (model loading), analyze everything after\nnsys stats --report cuda_gpu_kern_sum --filter-time \"5s\u002F\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n\n# Analyze only a specific 2-second window\nnsys stats --report cuda_gpu_kern_sum --filter-time \"5s\u002F7s\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nTime units are composable: `1s2ms3us4ns` = 1,002,003,004ns. This is much more precise than `--delay`\u002F`--duration` — it's post-hoc filtering on an already-captured trace.\n\nYou can also filter by NVTX range if the code has NVTX annotations:\n```bash\nnsys stats --report cuda_gpu_kern_sum --filter-nvtx \"decode_step@inference\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### Change time units\n\nDefault output is nanoseconds. Switch to milliseconds for readability:\n\n```bash\nnsys stats --report cuda_gpu_kern_sum --timeunit msec target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### Machine-readable output\n\nFor scripted analysis or piping into other tools:\n\n```bash\n# JSON — directly parseable by Python\u002Fjq\nnsys stats --report cuda_gpu_kern_sum --format json target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n\n# CSV — for spreadsheets or pandas\nnsys stats --report cuda_gpu_kern_sum --format csv --output . target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n\n# Pipe to grep for quick filtering\nnsys stats --report cuda_api_sum --format table --output @\"grep -E '(-|Name|cudaFree)'\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### Launch-to-execution correlation\n\n```bash\nnsys stats --report cuda_kern_exec_sum target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis report shows three time columns per kernel:\n- **AAvg** (API time) — how long the host-side launch API took\n- **QAvg** (queue time) — how long the kernel sat in the stream queue before the GPU picked it up\n- **KAvg** (kernel time) — actual GPU execution time\n\nIf QAvg is large compared to KAvg, kernels are queuing because the GPU is still busy with previous work, or the launch stream is congested.\n\n### Per-launch GPU trace\n\n```bash\nnsys stats --report cuda_gpu_trace target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nShows every individual kernel launch with its timestamp, grid dimensions (`GrdX\u002FY\u002FZ`), block dimensions (`BlkX\u002FY\u002FZ`), register count (`Reg\u002FTrd`), and shared memory usage. Useful for:\n- Verifying launch configurations match what you expect\n- Finding specific outlier launches (sort by Duration)\n- Checking if grid dimensions vary across iterations (sign of dynamic shapes)\n\n### Multi-report in one command\n\nGenerate multiple reports in a single pass:\n\n```bash\nnsys stats --report cuda_gpu_kern_sum --report cuda_api_sum --report cuda_gpu_mem_time_sum \\\n  target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### GPU memory tracking\n\nCapture GPU memory allocation\u002Fdeallocation patterns over time:\n\n```bash\nnsys profile --cuda-memory-usage=true --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nAdds overhead but shows memory watermarks. Useful when debugging OOM or fragmentation.\n\n### CUDA backtrace for slow API calls\n\nFind which host code is issuing slow CUDA API calls:\n\n```bash\nnsys profile --cudabacktrace=kernel:500 --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nCollects CPU backtraces for CUDA kernel launches that take >500ns (the threshold). The backtrace tells you exactly which Rust\u002FC++ function triggered the launch. Options: `all`, `kernel`, `memory`, `sync`, `other`, or comma-combined. Significant overhead — use selectively.\n\n### GPU hardware metrics\n\n```bash\nnsys profile --gpu-metrics-devices=all --gpu-metrics-frequency=10000 \\\n  --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nSamples SM utilization, memory throughput, and other hardware counters at 10kHz. Check available metric sets: `nsys profile --gpu-metrics-set=help`.\n\n**Caveat**: on consumer GPUs (GeForce), this requires setting GPU performance counter access. If you see `ERR_NVGPUCTRPERM`, run `nsys status -e` and check NVIDIA's permission guide.\n\n### Experimental plugins\n\n```bash\n# Power and temperature monitoring during profiling\nnsys profile --enable=nvml_metrics --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n\n# Network adapter metrics\nnsys profile --enable=network_interface --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nList all plugins: `nsys profile --enable=help`\n\n### Export to other formats\n\n```bash\n# Parquet — for pandas\u002FSpark analysis on large traces\nnsys export --type=parquetdir -o target\u002Fprofiling\u002Ftrace_parquet target\u002Fprofiling\u002Ftrace.nsys-rep\n\n# Arrow — for Apache Arrow consumers\nnsys export --type=arrowdir -o target\u002Fprofiling\u002Ftrace_arrow target\u002Fprofiling\u002Ftrace.nsys-rep\n\n# Export only specific tables (faster for large traces)\nnsys export --type=sqlite --tables=CUPTI_ACTIVITY_KIND_KERNEL,StringIds \\\n  -o target\u002Fprofiling\u002Fkernels_only target\u002Fprofiling\u002Ftrace.nsys-rep\n\n# Time-filtered export — extract just the interesting window\nnsys export --type=sqlite --times=5s\u002F10s \\\n  -o target\u002Fprofiling\u002Finference_only target\u002Fprofiling\u002Ftrace.nsys-rep\n```\n\n### Recipes (multi-file analysis)\n\n`nsys recipe` runs higher-level analyses across one or more trace files. Requires `pandas` — install with:\n```bash\npython3 \u002Fopt\u002Fnvidia\u002Fnsight-systems\u002F*\u002Ftarget-linux-x64\u002Fpython\u002Fpackages\u002Fnsys_recipe\u002Finstall.py\n```\n\nUseful recipes for this project:\n\n| Recipe | What it does |\n|--------|-------------|\n| `diff` | Side-by-side comparison of two traces — kernel time, API time, memory ops |\n| `nccl_sum` | NCCL collective summary (multi-GPU\u002Fmulti-node) |\n| `nccl_gpu_overlap_trace` | NCCL\u002Fcompute overlap analysis — are collectives hiding behind compute? |\n| `cuda_gpu_kern_pace` | Kernel pacing — are kernels launching at a steady rate or bursty? |\n| `cuda_gpu_kern_hist` | Duration histogram per kernel — visualize the distribution, not just percentiles |\n| `cuda_gpu_time_util_map` | GPU utilization heatmap over time |\n\nExample:\n```bash\n# Diff two traces to see what changed\nnsys recipe diff -i trace_before.sqlite trace_after.sqlite\n\n# NCCL analysis for distributed profiling\nnsys recipe nccl_sum -i rank0.sqlite rank1.sqlite rank2.sqlite rank3.sqlite\n```\n\n## Available report types (quick reference)\n\n| Report | Purpose |\n|--------|---------|\n| `cuda_gpu_kern_sum[:base\\|:mangled]` | Kernel time totals — first thing to look at |\n| `cuda_api_sum` | Host-side CUDA API time |\n| `cuda_gpu_trace` | Per-launch kernel trace with grid\u002Fblock dims and registers |\n| `cuda_kern_exec_sum` | Launch → queue → execute timing per kernel |\n| `cuda_gpu_mem_size_sum` | Memory ops grouped by transfer size |\n| `cuda_gpu_mem_time_sum` | Memory ops grouped by duration |\n| `cuda_api_gpu_sum` | Combined API + kernel + memops view |\n| `nvtx_sum` \u002F `nvtx_pushpop_sum` | NVTX range summaries (if code has annotations) |\n| `nvtx_kern_sum` | Kernels grouped by NVTX range |\n| `osrt_sum` | OS runtime API summary (pthread, file I\u002FO) |\n\nFull list: `nsys stats --help-reports`\n\n## Interpreting results\n\n### What to report to the user\n\nAfter analysis, present:\n\n1. **Top 3-5 kernels by total time** — with percentage of total GPU time\n2. **Any tail anomalies** — kernels where max\u002Fp50 > 2×\n3. **Host overhead** — is `cuStreamSynchronize` or `cudaLaunchKernel` significant?\n4. **Diagnosis** — which kernel family is the optimization target?\n5. **Next action** — what to microbench or optimize\n\n### Mapping kernel names to model operations\n\nnsys kernel names are mangled C++ symbols. Map them to logical operations:\n- `gemvx::kernel` → cuBLAS GEMV (matrix-vector multiply for all linear projections)\n- `FusedAddRMSNormKernel` → residual connection + RMS normalization\n- `BatchDecodeWithPagedKVCacheKernel` → FlashInfer paged attention decode\n- `BatchPrefillWithPagedKVCacheKernel` → FlashInfer paged attention prefill\n- `AppendPagedKVCacheKernel` → KV cache page append\n- `RadixTopKKernel_Unified` → FlashInfer sampling (token selection)\n- `silu_mul_fused_kernel` → SiLU activation × gate (MLP)\n- `prefill_qk_norm_rope_kernel` → QK normalization + rotary position embedding\n- `embedding_batched_kernel` → token embedding lookup\n\nUse `:base` to get these short names directly: `nsys stats --report cuda_gpu_kern_sum:base`.\n\n### Diagnosing sequence-length degradation\n\nIf TPOT grows with context length, run two traces with different prompt lengths:\n\n```bash\n# Short context\nnsys profile ... -o target\u002Fprofiling\u002Fctx_short \\\n  cargo run -r --bin bench_serving -- request --prompt-len 1 --output-len 128 --warmup 1 --iters 1\n\n# Long context\nnsys profile ... -o target\u002Fprofiling\u002Fctx_long \\\n  cargo run -r --bin bench_serving -- request --prompt-len 2048 --output-len 128 --warmup 1 --iters 1\n```\n\nCompare `cuda_gpu_kern_sum` between the two. Only attention decode should scale with context — if other kernels also grow, investigate.\n\nIf `nsys recipe diff` is available (pandas installed), use it for an automated side-by-side:\n```bash\nnsys recipe diff -i target\u002Fprofiling\u002Fctx_short.sqlite target\u002Fprofiling\u002Fctx_long.sqlite\n```\n\n## Critical pitfalls (team knowledge)\n\nThese are real lessons from production profiling on this codebase:\n\n1. **`--cuda-graph-trace=node` inflates absolute times by 30-60%.** Always measure actual TPOT with `bench_serving` *without* nsys, and use nsys only for kernel time *proportions* and *composition*. Never quote a TPOT number from an nsys trace as ground truth.\n\n2. **Default `--cuda-graph-trace` is `graph`, not `node`.** On driver ≥11.7, nsys defaults to `graph` granularity — the entire CUDA Graph replay appears as a single opaque block. You MUST explicitly pass `--cuda-graph-trace=node` to see individual kernels. This is the #1 \"my trace is useless\" moment.\n\n3. **p50 is not enough.** A kernel can look cheap at p50 while dominating p99. In MoE workloads, NCCL all-reduce and routed expert kernels are especially prone to this — the p50 reflects pure compute but the p99 includes rank arrival skew. Always run `nsys_tail_stats.py` when tail behavior matters.\n\n4. **CPU launch overhead is multiplicative.** Each `cudaLaunchKernel` costs ~5μs on the host. A per-token loop launching 20,000 tiny kernels adds ~100ms of pure CPU overhead per decode step. If `cudaLaunchKernel` count is surprisingly high, look for kernel fusion or CUDA Graph opportunities. Use `cuda_kern_exec_sum` to see queue time vs execution time.\n\n5. **`cuStreamSynchronize` during inference is a red flag.** During model loading it's normal. During decode it means the host is blocking — look for unnecessary D2H copies or sync points. Use `--filter-time` to isolate inference-only API stats.\n\n6. **NCCL wait-inclusive time ≠ pure transfer time.** In distributed traces, if NCCL collectives look expensive, separate rank arrival skew from actual data movement. Two ranks may do the same collective but one \"takes 1.2ms\" only because the other arrived 1.18ms late. Use `nsys recipe nccl_gpu_overlap_trace` to analyze this.\n\n7. **CUDA toolkit version affects kernel performance.** Same code, same GPU: CUDA 13.1 generates ~3% faster code than 12.8 for custom CUDA kernels on sm_120. Always note the toolkit version in profiling reports. Check with `nvcc --version`.\n\n8. **Allocation churn on the hot path.** High `cuMemAllocAsync` \u002F `cuMemFreeAsync` counts during inference (not loading) indicate per-step allocations that should be pre-allocated or pooled. Check the CUDA API tail candidates table — allocation stalls show up as high max\u002Fp50 ratios.\n\n9. **PATH in non-interactive shells.** When profiling via SSH or detached tmux, `nsys` may not be in PATH because `~\u002F.bashrc` is only sourced for interactive shells. Either export PATH explicitly or use the full path `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fnsys`.\n\n10. **GPU metrics require privilege on consumer GPUs.** `--gpu-metrics-devices=all` fails on GeForce cards with `ERR_NVGPUCTRPERM`. On datacenter GPUs (A100\u002FH100\u002FB200), it works out of the box. Run `nsys status -e` to check.\n","# nsys Profiling\n\nYou are helping a developer profile GPU workloads in a Rust + CUDA inference engine using NVIDIA Nsight Systems (`nsys`). The goal is always to answer \"where is time going and what should I optimize next?\"\n\n## Before you start\n\n1. **Confirm the workload.** Ask which model, which path (prefill vs decode vs both), and what prompt\u002Foutput shapes matter. If the user is vague, default to the two standard profiles:\n   - Prefill-heavy: `--prompt-len 2048 --output-len 1`\n   - Decode-heavy: `--prompt-len 1 --output-len 128`\n\n2. **Check `--release`.** Debug builds slow GPU kernels by 10x+ and produce misleading traces. If you see `cargo run` without `-r` or `--release`, stop and fix this first.\n\n3. **Check nsys is available.** Run `nsys --version`. If it fails, the user likely needs to add `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin` to PATH — a common gotcha in SSH or tmux sessions where `~\u002F.bashrc` doesn't source. Fix: `export PATH=\"\u002Fusr\u002Flocal\u002Fcuda\u002Fbin:$PATH\"`, or use full path `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fnsys`.\n\n4. **Check profiling environment.** Run `nsys status -e` to verify the system can profile. It checks kernel paranoid level, perf_event support, and LBR availability. If paranoid level is too high (>1), CPU sampling won't work without root.\n\n## Capturing a trace\n\n### Standard capture command\n\n```bash\nnsys profile --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002F\u003Ctrace_name> \\\n  cargo run -r --bin bench_serving -- request \\\n    --prompt-len \u003CPROMPT> --output-len \u003COUTPUT> --warmup 1 --iters 1\n```\n\nBuild this command for the user based on their workload. Always include all four flags:\n\n| Flag | Why |\n|------|-----|\n| `--force-overwrite=true` | nsys errors on existing files without this |\n| `--cuda-graph-trace=node` | Without it, CUDA Graph replay is opaque — you can't see individual kernels. This is the single most common nsys pitfall. Default is `graph` on driver ≥11.7, which hides everything inside the graph. |\n| `--export=sqlite` | Produces a `.sqlite` file that `nsys stats` reads directly |\n| `-o target\u002Fprofiling\u002F\u003Cname>` | Keep traces organized in one place |\n\n### Precise capture with cudaProfilerApi\n\nIf the `bench_serving` binary supports `--cuda-profiler-capture`, use it to capture only the measured iterations — this excludes model loading and warmup from the trace, producing a much cleaner profile:\n\n```bash\nnsys profile --force-overwrite=true --cuda-graph-trace=node \\\n  -c cudaProfilerApi --export=sqlite -o target\u002Fprofiling\u002F\u003Ctrace_name> \\\n  cargo run -r --bin bench_serving -- --cuda-profiler-capture request \\\n    --prompt-len \u003CPROMPT> --output-len \u003COUTPUT> --warmup 1 --iters 1\n```\n\nUse `--capture-range-end=repeat[:N]` to capture multiple separate ranges (e.g., N iterations) into the same trace. Each range produces its own segment.\n\n### Delay\u002Fduration capture (when cudaProfilerApi isn't available)\n\nFor workloads without profiler API integration, use `--delay` to skip startup and `--duration` to limit capture:\n\n```bash\nnsys profile --delay=10 --duration=30 --cuda-graph-trace=node \\\n  --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002F\u003Ctrace_name> \\\n  \u003Ccommand>\n```\n\n### Disabling CUDA Graph for cleaner kernel analysis\n\nCUDA Graph replay groups many kernels into a single opaque launch. Even with `--cuda-graph-trace=node`, profiler overhead inflates times differently than ungraphed launches. For precise per-kernel timing (e.g., comparing short vs long context), disable CUDA Graph:\n\n```bash\ncargo run -r --bin bench_serving -- --cuda-graph false request --prompt-len 2048 --output-len 32 ...\n```\n\nTradeoff: ungraphed traces show true individual kernel times but include ~5μs CPU launch overhead per kernel that the production (graphed) path avoids. Use graphed traces for real TPOT, ungraphed for kernel-level comparison.\n\n### Interactive profiling (long-running servers)\n\nFor profiling a server that's already running, use the launch\u002Fstart\u002Fstop workflow:\n\n```bash\n# Terminal 1: launch the server under nsys control, but don't start collecting yet\nnsys launch --trace=cuda,nvtx --cuda-graph-trace=node \\\n  --session-new=my_session -- cargo run -r --bin pegainfer-server -- ...\n\n# Terminal 2: start\u002Fstop collection on demand\nnsys start --session=my_session\n# ... send requests ...\nnsys stop --session=my_session\n```\n\nOr use `--start-later=true` with `nsys profile` to get the same deferred-start behavior in a single command.\n\n### Lighter traces\n\nIf the trace is too large or slow to process, limit tracing scope:\n\n```bash\n# Only CUDA + NVTX, skip OS scheduling noise — smaller trace, faster to open\nnsys profile --trace=cuda,nvtx --cuda-graph-trace=node ...\n```\n\nOther `--trace` options: `cublas` (trace cuBLAS API calls with parameters), `cudnn`, `osrt` (OS runtime like pthread), `mpi`, `python-gil`. Multiple values comma-separated.\n\n### Output naming tricks\n\nThe `-o` flag supports pattern substitution:\n- `%h` — hostname (useful for multi-node)\n- `%p` — PID of target process\n- `%n` — auto-incrementing number (avoids overwrite without `--force-overwrite`)\n- `%q{ENV_VAR}` — environment variable expansion\n- `%%` — literal `%`\n\nExample: `-o traces\u002F%h_rank%q{RANK}_%n` → `traces\u002Fgpu-node-01_rank0_1.nsys-rep`\n\n### Reproducible configs with `--command-file`\n\nStore nsys flags in a file to avoid long command lines:\n\n```bash\n# nsys_decode.cfg\n--force-overwrite=true\n--cuda-graph-trace=node\n--export=sqlite\n--trace=cuda,nvtx\n```\n\nThen: `nsys profile --command-file=nsys_decode.cfg -o target\u002Fprofiling\u002Ftrace \u003Ccommand>`\n\nCommand-line flags override file flags.\n\n### One-shot stats with `--stats=true`\n\nSkip the separate `nsys stats` step entirely:\n\n```bash\nnsys profile --stats=true --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nThis automatically generates all default summary reports (kern_sum, api_sum, mem_sum, etc.) to the console after collection finishes. Convenient for quick checks, but the output is long — better for scripted workflows.\n\n## Analyzing a trace\n\nRun these analyses in sequence. Each answers a different question.\n\n### Step 1: Kernel time breakdown\n\n```bash\nnsys stats --report cuda_gpu_kern_sum target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis tells you **which GPU kernels consume the most time**. Normal patterns:\n- MLP + GEMV typically >90% of decode time\n- Attention grows with sequence length (O(seq_len)) while MLP\u002FGEMV stay flat\n- RMSNorm and activation kernels are lightweight (\u003C3%)\n\n**Use `:base` to tame mangled names.** By default, kernel names are fully mangled C++ templates — unreadable. Append `:base` to strip template parameters:\n\n```bash\nnsys stats --report cuda_gpu_kern_sum:base target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis turns `std::enable_if\u003C!T7, void>::type internal::gemvx::kernel\u003Cint, int, __nv_bfloat16, ...>(T13)` into just `kernel`. Much easier to scan, but note that it merges kernels with different template instantiations. Use the full name when you need to distinguish shapes.\n\nThere's also `:mangled` if you need the raw linker symbol.\n\n### Step 2: CUDA API overhead\n\n```bash\nnsys stats --report cuda_api_sum target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis tells you **where the CPU\u002Fhost is spending time**. Watch for:\n- `cuMemcpyHtoDAsync` dominated by model loading (normal if one-time)\n- `cuStreamSynchronize` — if this is large during inference (not loading), the host is unnecessarily waiting for the GPU\n- `cuMemAllocAsync` \u002F `cuMemFreeAsync` — high counts or time means hot-path allocation churn\n- `cudaLaunchKernel` — large count × ~5μs each = significant CPU launch overhead\n\n### Step 3: Tail-aware stats\n\n```bash\npython3 tools\u002Fnsys_tail_stats.py target\u002Fprofiling\u002F\u003Ctrace>.sqlite --limit 15\n```\n\nThis is the team's custom tool that goes beyond nsys's built-in p50\u002Favg reporting. It shows `p50, p95, p99, max` and a tail score for each kernel. The p50 can hide serious problems:\n\n- **max\u002Fp50 > 2×**: Kernel has outlier invocations. Common causes: route imbalance in MoE, rank arrival skew in NCCL, allocator stalls.\n- **p99\u002Fp50 > 1.5×**: Systematic tail — not a one-off spike. Look for stream contention or variable work per invocation.\n- **High tail score but low total time**: The kernel is cheap on average but occasionally blocks everything. These are the sneakiest bottlenecks.\n\n### Step 4: Bandwidth efficiency (for GEMV-bound workloads)\n\nWhen GEMV dominates (>90% of GPU time, typical for bs=1 decode), compute memory bandwidth utilization to answer \"is there room to optimize, or are we already at the hardware limit?\"\n\n```\nBW_achieved = weight_bytes \u002F kernel_time\nBW_efficiency = BW_achieved \u002F GPU_peak_BW\n```\n\nFor each major GEMV shape, calculate weight size (M × K × sizeof(bf16)) and divide by the nsys avg kernel time. Compare against GPU peak memory bandwidth (e.g., RTX 5070 Ti = 896 GB\u002Fs, H100 = 3.35 TB\u002Fs). Large GEMVs (>20MB weights) typically reach 85-90% efficiency; small GEMVs (\u003C5MB) drop to 60-70% due to launch overhead and tail effects.\n\nIf overall efficiency is >80%, GEMV optimization has limited headroom — quantization or batching is the path forward. If \u003C70%, there's room for custom kernels.\n\n### Step 5 (optional): Auto-analysis\n\n```bash\nnsys analyze target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nRuns NVIDIA's built-in rules: `cuda_memcpy_async`, `cuda_memcpy_sync`, `cuda_memset_sync`, `cuda_api_sync`, `gpu_gaps`, `gpu_time_util`. It flags synchronous memcpy, unnecessary device syncs, and periods where the GPU is idle >500ms. Quick sanity check, but won't catch domain-specific problems.\n\nRun a specific rule: `nsys analyze --rule cuda_api_sync \u003Ctrace>.sqlite`\n\n## Advanced analysis techniques\n\n### Time-window filtering\n\nIsolate just the inference phase by filtering out model loading:\n\n```bash\n# Skip first 5 seconds (model loading), analyze everything after\nnsys stats --report cuda_gpu_kern_sum --filter-time \"5s\u002F\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n\n# Analyze only a specific 2-second window\nnsys stats --report cuda_gpu_kern_sum --filter-time \"5s\u002F7s\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nTime units are composable: `1s2ms3us4ns` = 1,002,003,004ns. This is much more precise than `--delay`\u002F`--duration` — it's post-hoc filtering on an already-captured trace.\n\nYou can also filter by NVTX range if the code has NVTX annotations:\n```bash\nnsys stats --report cuda_gpu_kern_sum --filter-nvtx \"decode_step@inference\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### Change time units\n\nDefault output is nanoseconds. Switch to milliseconds for readability:\n\n```bash\nnsys stats --report cuda_gpu_kern_sum --timeunit msec target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### Machine-readable output\n\nFor scripted analysis or piping into other tools:\n\n```bash\n# JSON — directly parseable by Python\u002Fjq\nnsys stats --report cuda_gpu_kern_sum --format json target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n\n# CSV — for spreadsheets or pandas\nnsys stats --report cuda_gpu_kern_sum --format csv --output . target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n\n# Pipe to grep for quick filtering\nnsys stats --report cuda_api_sum --format table --output @\"grep -E '(-|Name|cudaFree)'\" target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### Launch-to-execution correlation\n\n```bash\nnsys stats --report cuda_kern_exec_sum target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nThis report shows three time columns per kernel:\n- **AAvg** (API time) — how long the host-side launch API took\n- **QAvg** (queue time) — how long the kernel sat in the stream queue before the GPU picked it up\n- **KAvg** (kernel time) — actual GPU execution time\n\nIf QAvg is large compared to KAvg, kernels are queuing because the GPU is still busy with previous work, or the launch stream is congested.\n\n### Per-launch GPU trace\n\n```bash\nnsys stats --report cuda_gpu_trace target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\nShows every individual kernel launch with its timestamp, grid dimensions (`GrdX\u002FY\u002FZ`), block dimensions (`BlkX\u002FY\u002FZ`), register count (`Reg\u002FTrd`), and shared memory usage. Useful for:\n- Verifying launch configurations match what you expect\n- Finding specific outlier launches (sort by Duration)\n- Checking if grid dimensions vary across iterations (sign of dynamic shapes)\n\n### Multi-report in one command\n\nGenerate multiple reports in a single pass:\n\n```bash\nnsys stats --report cuda_gpu_kern_sum --report cuda_api_sum --report cuda_gpu_mem_time_sum \\\n  target\u002Fprofiling\u002F\u003Ctrace>.sqlite\n```\n\n### GPU memory tracking\n\nCapture GPU memory allocation\u002Fdeallocation patterns over time:\n\n```bash\nnsys profile --cuda-memory-usage=true --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nAdds overhead but shows memory watermarks. Useful when debugging OOM or fragmentation.\n\n### CUDA backtrace for slow API calls\n\nFind which host code is issuing slow CUDA API calls:\n\n```bash\nnsys profile --cudabacktrace=kernel:500 --force-overwrite=true --cuda-graph-trace=node \\\n  --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nCollects CPU backtraces for CUDA kernel launches that take >500ns (the threshold). The backtrace tells you exactly which Rust\u002FC++ function triggered the launch. Options: `all`, `kernel`, `memory`, `sync`, `other`, or comma-combined. Significant overhead — use selectively.\n\n### GPU hardware metrics\n\n```bash\nnsys profile --gpu-metrics-devices=all --gpu-metrics-frequency=10000 \\\n  --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nSamples SM utilization, memory throughput, and other hardware counters at 10kHz. Check available metric sets: `nsys profile --gpu-metrics-set=help`.\n\n**Caveat**: on consumer GPUs (GeForce), this requires setting GPU performance counter access. If you see `ERR_NVGPUCTRPERM`, run `nsys status -e` and check NVIDIA's permission guide.\n\n### Experimental plugins\n\n```bash\n# Power and temperature monitoring during profiling\nnsys profile --enable=nvml_metrics --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n\n# Network adapter metrics\nnsys profile --enable=network_interface --force-overwrite=true --export=sqlite -o target\u002Fprofiling\u002Ftrace \u003Ccommand>\n```\n\nList all plugins: `nsys profile --enable=help`\n\n### Export to other formats\n\n```bash\n# Parquet — for pandas\u002FSpark analysis on large traces\nnsys export --type=parquetdir -o target\u002Fprofiling\u002Ftrace_parquet target\u002Fprofiling\u002Ftrace.nsys-rep\n\n# Arrow — for Apache Arrow consumers\nnsys export --type=arrowdir -o target\u002Fprofiling\u002Ftrace_arrow target\u002Fprofiling\u002Ftrace.nsys-rep\n\n# Export only specific tables (faster for large traces)\nnsys export --type=sqlite --tables=CUPTI_ACTIVITY_KIND_KERNEL,StringIds \\\n  -o target\u002Fprofiling\u002Fkernels_only target\u002Fprofiling\u002Ftrace.nsys-rep\n\n# Time-filtered export — extract just the interesting window\nnsys export --type=sqlite --times=5s\u002F10s \\\n  -o target\u002Fprofiling\u002Finference_only target\u002Fprofiling\u002Ftrace.nsys-rep\n```\n\n### Recipes (multi-file analysis)\n\n`nsys recipe` runs higher-level analyses across one or more trace files. Requires `pandas` — install with:\n```bash\npython3 \u002Fopt\u002Fnvidia\u002Fnsight-systems\u002F*\u002Ftarget-linux-x64\u002Fpython\u002Fpackages\u002Fnsys_recipe\u002Finstall.py\n```\n\nUseful recipes for this project:\n\n| Recipe | What it does |\n|--------|-------------|\n| `diff` | Side-by-side comparison of two traces — kernel time, API time, memory ops |\n| `nccl_sum` | NCCL collective summary (multi-GPU\u002Fmulti-node) |\n| `nccl_gpu_overlap_trace` | NCCL\u002Fcompute overlap analysis — are collectives hiding behind compute? |\n| `cuda_gpu_kern_pace` | Kernel pacing — are kernels launching at a steady rate or bursty? |\n| `cuda_gpu_kern_hist` | Duration histogram per kernel — visualize the distribution, not just percentiles |\n| `cuda_gpu_time_util_map` | GPU utilization heatmap over time |\n\nExample:\n```bash\n# Diff two traces to see what changed\nnsys recipe diff -i trace_before.sqlite trace_after.sqlite\n\n# NCCL analysis for distributed profiling\nnsys recipe nccl_sum -i rank0.sqlite rank1.sqlite rank2.sqlite rank3.sqlite\n```\n\n## Available report types (quick reference)\n\n| Report | Purpose |\n|--------|---------|\n| `cuda_gpu_kern_sum[:base\\|:mangled]` | Kernel time totals — first thing to look at |\n| `cuda_api_sum` | Host-side CUDA API time |\n| `cuda_gpu_trace` | Per-launch kernel trace with grid\u002Fblock dims and registers |\n| `cuda_kern_exec_sum` | Launch → queue → execute timing per kernel |\n| `cuda_gpu_mem_size_sum` | Memory ops grouped by transfer size |\n| `cuda_gpu_mem_time_sum` | Memory ops grouped by duration |\n| `cuda_api_gpu_sum` | Combined API + kernel + memops view |\n| `nvtx_sum` \u002F `nvtx_pushpop_sum` | NVTX range summaries (if code has annotations) |\n| `nvtx_kern_sum` | Kernels grouped by NVTX range |\n| `osrt_sum` | OS runtime API summary (pthread, file I\u002FO) |\n\nFull list: `nsys stats --help-reports`\n\n## Interpreting results\n\n### What to report to the user\n\nAfter analysis, present:\n\n1. **Top 3-5 kernels by total time** — with percentage of total GPU time\n2. **Any tail anomalies** — kernels where max\u002Fp50 > 2×\n3. **Host overhead** — is `cuStreamSynchronize` or `cudaLaunchKernel` significant?\n4. **Diagnosis** — which kernel family is the optimization target?\n5. **Next action** — what to microbench or optimize\n\n### Mapping kernel names to model operations\n\nnsys kernel names are mangled C++ symbols. Map them to logical operations:\n- `gemvx::kernel` → cuBLAS GEMV (matrix-vector multiply for all linear projections)\n- `FusedAddRMSNormKernel` → residual connection + RMS normalization\n- `BatchDecodeWithPagedKVCacheKernel` → FlashInfer paged attention decode\n- `BatchPrefillWithPagedKVCacheKernel` → FlashInfer paged attention prefill\n- `AppendPagedKVCacheKernel` → KV cache page append\n- `RadixTopKKernel_Unified` → FlashInfer sampling (token selection)\n- `silu_mul_fused_kernel` → SiLU activation × gate (MLP)\n- `prefill_qk_norm_rope_kernel` → QK normalization + rotary position embedding\n- `embedding_batched_kernel` → token embedding lookup\n\nUse `:base` to get these short names directly: `nsys stats --report cuda_gpu_kern_sum:base`.\n\n### Diagnosing sequence-length degradation\n\nIf TPOT grows with context length, run two traces with different prompt lengths:\n\n```bash\n# Short context\nnsys profile ... -o target\u002Fprofiling\u002Fctx_short \\\n  cargo run -r --bin bench_serving -- request --prompt-len 1 --output-len 128 --warmup 1 --iters 1\n\n# Long context\nnsys profile ... -o target\u002Fprofiling\u002Fctx_long \\\n  cargo run -r --bin bench_serving -- request --prompt-len 2048 --output-len 128 --warmup 1 --iters 1\n```\n\nCompare `cuda_gpu_kern_sum` between the two. Only attention decode should scale with context — if other kernels also grow, investigate.\n\nIf `nsys recipe diff` is available (pandas installed), use it for an automated side-by-side:\n```bash\nnsys recipe diff -i target\u002Fprofiling\u002Fctx_short.sqlite target\u002Fprofiling\u002Fctx_long.sqlite\n```\n\n## Critical pitfalls (team knowledge)\n\nThese are real lessons from production profiling on this codebase:\n\n1. **`--cuda-graph-trace=node` inflates absolute times by 30-60%.** Always measure actual TPOT with `bench_serving` *without* nsys, and use nsys only for kernel time *proportions* and *composition*. Never quote a TPOT number from an nsys trace as ground truth.\n\n2. **Default `--cuda-graph-trace` is `graph`, not `node`.** On driver ≥11.7, nsys defaults to `graph` granularity — the entire CUDA Graph replay appears as a single opaque block. You MUST explicitly pass `--cuda-graph-trace=node` to see individual kernels. This is the #1 \"my trace is useless\" moment.\n\n3. **p50 is not enough.** A kernel can look cheap at p50 while dominating p99. In MoE workloads, NCCL all-reduce and routed expert kernels are especially prone to this — the p50 reflects pure compute but the p99 includes rank arrival skew. Always run `nsys_tail_stats.py` when tail behavior matters.\n\n4. **CPU launch overhead is multiplicative.** Each `cudaLaunchKernel` costs ~5μs on the host. A per-token loop launching 20,000 tiny kernels adds ~100ms of pure CPU overhead per decode step. If `cudaLaunchKernel` count is surprisingly high, look for kernel fusion or CUDA Graph opportunities. Use `cuda_kern_exec_sum` to see queue time vs execution time.\n\n5. **`cuStreamSynchronize` during inference is a red flag.** During model loading it's normal. During decode it means the host is blocking — look for unnecessary D2H copies or sync points. Use `--filter-time` to isolate inference-only API stats.\n\n6. **NCCL wait-inclusive time ≠ pure transfer time.** In distributed traces, if NCCL collectives look expensive, separate rank arrival skew from actual data movement. Two ranks may do the same collective but one \"takes 1.2ms\" only because the other arrived 1.18ms late. Use `nsys recipe nccl_gpu_overlap_trace` to analyze this.\n\n7. **CUDA toolkit version affects kernel performance.** Same code, same GPU: CUDA 13.1 generates ~3% faster code than 12.8 for custom CUDA kernels on sm_120. Always note the toolkit version in profiling reports. Check with `nvcc --version`.\n\n8. **Allocation churn on the hot path.** High `cuMemAllocAsync` \u002F `cuMemFreeAsync` counts during inference (not loading) indicate per-step allocations that should be pre-allocated or pooled. Check the CUDA API tail candidates table — allocation stalls show up as high max\u002Fp50 ratios.\n\n9. **PATH in non-interactive shells.** When profiling via SSH or detached tmux, `nsys` may not be in PATH because `~\u002F.bashrc` is only sourced for interactive shells. Either export PATH explicitly or use the full path `\u002Fusr\u002Flocal\u002Fcuda\u002Fbin\u002Fnsys`.\n\n10. **GPU metrics require privilege on consumer GPUs.** `--gpu-metrics-devices=all` fails on GeForce cards with `ERR_NVGPUCTRPERM`. On datacenter GPUs (A100\u002FH100\u002FB200), it works out of the box. Run `nsys status -e` to check.\n",{"name":5,"description":7},"main","2246b2ac1ed98ec92c408ff51ce7b33d4b255971",".claude\u002Fskills\u002Fnsys-profiling",120,"https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer","https:\u002F\u002Fpegainfer.org\u002F","https:\u002F\u002Fgithub.com\u002Fpegainfer-project","https:\u002F\u002Fopen-infer.org\u002F","2026-11-12 03:48:06",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add pegainfer-project\u002Fpegainfer --skill nsys-profiling",{"tool":71,"command":72},"bunx","bunx skills add pegainfer-project\u002Fpegainfer --skill nsys-profiling",{"tool":74,"command":75},"pnpm","pnpm dlx skills add pegainfer-project\u002Fpegainfer --skill nsys-profiling","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",true,[],[80,85,113,119,149],{"skillNo":81,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":51},"01M6CP47ZZ9G8N4YW1AEW11QRQ","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.codex\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",[],[24,23],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":92,"publishedAt":51},"01M6CP44T5R6WV38NVEEXAJ5RV","vllm-bench","Use vllm-bench to benchmark OpenAI-compatible or vLLM serving endpoints, especially multi-turn chat load tests. Use when the user asks for vllm bench, vllm-bench, multi-turn benchmark, chat serving benchmark, TTFT\u002FTPOT\u002Fthroughput measurement, concurrency sweep, load test, 压测, 多轮压测, or wants help installing, running --help, choosing flags, validating a local PegaInfer\u002FvLLM server, saving JSON results, or interpreting vllm-bench metrics.","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fvllm-bench\u002FSKILL.md",[91],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[93,94,95,96,97,98,99,49,100,101,102,103,104,28,31,105,106,107,108,109,110,37,111,25,40,112],"vllm bench","vLLM","json","concurrency","server","metrics","openai","benchmark","test","sweep","throughput","running","load","vllm server","measurement","openai-compatible","tests","local","endpoints","results",{"skillNo":114,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":118,"publishedAt":51},"01M6CP3YKSBD68R742QN3915C9","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fvllm-bench\u002FSKILL.md",[117],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},[93,94,104,28,31,105,106,107,108,103,109,110,96,37,111,25,40,112,102,95,97,98,99,49,100,101],{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":51},"01M6CP3VG0B2HGD6AM6BTNGG7Y","project-doc","Track any non-trivial task through a doc placed in the right domain directory under docs\u002F (models\u002F\u003Cline>\u002F, subsystems\u002F\u003Carea>\u002F, playbooks\u002F, lessons\u002F, etc.). Covers the full lifecycle: read docs first, write a plan, wait for approval, log execution, and capture lessons learned. Use this skill whenever the user gives a concrete task — not just Q&A or casual discussion. Even if the task seems straightforward, if it involves doing something (not just answering something), open or reuse a domain doc. Bias toward using this skill. When in doubt, use it.","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fproject-doc-bootstrap\u002FSKILL.md",[],[126,127,128,129,130,131,132,133,31,134,135,136,137,35,45,138,139,140,141,142,25,143,40,144,145,41,146,147,148,28],"project","project doc","doc","execution","wait for","directory","docs","task","q&a","wait","line","learned","domain","open","track","models","log","approval","write","read","plan","just","first",{"skillNo":150,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":151,"categories":152,"keywords":153,"publishedAt":51},"01M6CP3QVJ2MHXRM1SBXVGQNTK","https:\u002F\u002Fgithub.com\u002Fpegainfer-project\u002Fpegainfer\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fnsys-profiling\u002FSKILL.md",[],[24,23]]