[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69AHZ0M0Q90TP0KYMZ10HEP":3,"skill-same:01M69AHZ0M0Q90TP0KYMZ10HEP":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":50,"ownerHtmlUrl":51,"ownerBlogUrl":52,"ownerDescription":53,"updatedAt":54,"install":55,"resources":68},"01M69AHZ0M0Q90TP0KYMZ10HEP","ort-transformers-gpu-pytest",null,"Run the ONNX Runtime transformers Python tests (onnxruntime\u002Ftest\u002Fpython\u002Ftransformers) against a GPU wheel, and confirm real cuDNN\u002Fflash SDPA dispatch. Use when a transformers pytest fails with ModuleNotFoundError onnxruntime.capi, when torch's bundled CUDA\u002FcuDNN libs shadow the ORT-built ones (wrong-version dispatch or load errors), or when you must PROVE a test exercised the cuDNN SDPA decode tier instead of silently skipping\u002Ffalling back to MATH.","microsoft\u002Fonnxruntime","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69AGY0EW9AJXAH7HK223SQZ",22286,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fort-transformers-gpu-pytest\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"ort transformers gpu pytest","gpu","Transformers","pytest","when","python","decode","prove","load","errors","you","SDPA","tests","runtime","back","use","math","flash","run","onnx","test","2026-11-10 10:12:30",false,"---\nname: ort-transformers-gpu-pytest\ndescription: Run the ONNX Runtime transformers Python tests (onnxruntime\u002Ftest\u002Fpython\u002Ftransformers) against a GPU wheel, and confirm real cuDNN\u002Fflash SDPA dispatch. Use when a transformers pytest fails with ModuleNotFoundError onnxruntime.capi, when torch's bundled CUDA\u002FcuDNN libs shadow the ORT-built ones (wrong-version dispatch or load errors), or when you must PROVE a test exercised the cuDNN SDPA decode tier instead of silently skipping\u002Ffalling back to MATH.\n---\n\n# Running ONNX Runtime transformers GPU pytest (dispatch-verified)\n\nReusable, hard-won knowledge for running the Python transformers tests under\n`onnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F` against a **GPU-built wheel**, on a box where\n`torch` is also installed. Three gotchas below will each cost you an hour if\nrediscovered. See the `ort-test` skill for the general test taxonomy and the\nfalse-green modes; this skill is the GPU + transformers + SDPA-dispatch specialization.\n\n## 1. NEUTRAL-CWD gotcha — never run pytest from the repo root\n\n**Symptom:**\n\n```\nModuleNotFoundError: No module named 'onnxruntime.capi'\n# or an AttributeError deep inside onnxruntime import\n```\n\n**Cause:** the repo root contains a source directory `.\u002Fonnxruntime\u002F` (the C++\u002FPython\nsource tree). When pytest runs with the repo root on `sys.path[0]`, `import onnxruntime`\nresolves to that **source** package — which has no compiled `capi` extension — instead\nof the **installed wheel** in your venv's `site-packages`. The source dir *shadows* the\nwheel.\n\n**Fix:** run pytest from a **neutral, private working directory** — a fresh\n`mktemp -d`, not the repo root and not a bare shared `\u002Ftmp` — so the shadowing source\ndir is not on the path, and point at the test file by absolute path. Put the\ntransformers test-helper dir on `PYTHONPATH` so shared helpers still import.\n\n```bash\nWORKDIR=$(mktemp -d); cd \"$WORKDIR\"       # NEUTRAL + PRIVATE cwd — NOT repo root, NOT \u002Ftmp\nexport PYTHONPATH=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\npython -m pytest \u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F\u003Cfile>.py -v\n```\n\n**Why `mktemp -d` and not a bare `cd \u002Ftmp`:** pytest prepends the cwd to `sys.path`, and\nPython auto-imports `sitecustomize.py`\u002F`usercustomize.py` from it at startup — so on a\nshared box a co-tenant's planted `\u002Ftmp\u002Fsitecustomize.py` would execute as arbitrary code\nin your test process. A fresh per-run `mktemp -d` (private, `0700`) keeps the neutral-cwd\nsource-shadow protection while removing that injection vector.\n\nDo **not** `cd` into the repo and run `pytest onnxruntime\u002Ftest\u002F...` — that reintroduces\nthe shadowing. (This is the Python analogue of the C++ \"run from the build output dir\"\nrule in `ort-test`.)\n\n## 2. LD_PRELOAD lib-pinning gotcha — torch's CUDA\u002FcuDNN shadow ORT's\n\n**Symptom:** any of —\n- the SDPA decode tier silently routes to `MATH` instead of `CUDNN_FLASH_ATTENTION`\n  (wrong-version cuDNN loaded), or\n- `libcudnn.so.9: cannot open shared object file` \u002F `undefined symbol` \u002F cuDNN version\n  mismatch errors at first CUDA op, or\n- ORT loads a different CUDA runtime than it was built against.\n\n**Cause:** a pip-installed `torch` ships its **own** bundled CUDA runtime + cuDNN\n(e.g. cu124 → CUDA 12.4 \u002F cuDNN 9.1) under `site-packages\u002Fnvidia\u002F*\u002Flib`. If ORT was\nbuilt against a **different** CUDA\u002FcuDNN (e.g. CUDA 12.9 \u002F cuDNN 9.8), whichever set the\ndynamic loader resolves **first** wins. With torch imported (or its libs on the path),\ntorch's older libs can shadow the ones ORT `dlopen`s → wrong-version dispatch or load\nfailure.\n\n**Fix:** `LD_PRELOAD` the **system** CUDA runtime + cuDNN that ORT was built against so\nthey are loaded first, and add their dirs to `LD_LIBRARY_PATH`. Activate the venv that\nhas the ORT wheel. Concrete form used successfully (CUDA 12.9 + cuDNN 9.8; substitute\nyour absolute lib paths):\n\n```bash\nWORKDIR=$(mktemp -d); cd \"$WORKDIR\"       # neutral + private (see §1)\nsource \u002Fabs\u002Frepo\u002F.venv\u002Fbin\u002Factivate\nexport LD_PRELOAD=\u002Fabs\u002Fcuda12.9\u002Flib64\u002Flibcudart.so.12:\u002Fabs\u002Fcudnn9.8\u002Flib\u002Flibcudnn.so.9\nexport LD_LIBRARY_PATH=\u002Fabs\u002Fcuda12.9\u002Flib64:\u002Fabs\u002Fcudnn9.8\u002Flib\nexport PYTHONPATH=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\npython -m pytest \u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F\u003Cfile>.py -v\n```\n\nNotes:\n- `libcudart.so.12` is correct for **both** CUDA 12.4 and 12.9 (SONAME is major-only) —\n  pinning the 12.9 file forces the right minor.\n- `torch.cuda` still works fine under this preload — the bf16 IO-binding path that uses\n  `torch` tensors + `.data_ptr()` runs correctly.\n- Keep the two exports and the preload together; dropping `LD_LIBRARY_PATH` can still let\n  a transitive dependency resolve against torch's copy.\n\n## 3. Confirm REAL cuDNN SDPA dispatch (don't trust value-equality)\n\nA numerically-correct result does **not** prove the cuDNN SDPA path ran — the kernel has\na `MATH` fallback that produces the same answer (false-green mode 4 in `ort-test`). To\nprove the tier dispatched, **observe** ORT's routing rather than probing a version.\n\n### Observe-dispatch (the correct probe)\n\nORT's ONNX-domain Attention kernel emits a debug line when\n`ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1` is set **before** the `InferenceSession` is\ncreated (the option is read once at session creation). `AttentionKernelDebugInfo::Print`\nemits a token of the form:\n\n```\nSdpaKernel=CUDNN_FLASH_ATTENTION      # or =MATH, =FLASH_ATTENTION, =EFFICIENT_ATTENTION (non-exhaustive)\n```\n\nCapture stdout across a single `run()` and parse it. **Capture at the file-descriptor\nlevel, not `contextlib.redirect_stdout`:** the `SdpaKernel=` line is written to native\nfd-1 from C++, which Python-level stdout redirection never intercepts — you would get\n`dispatched=None`, a silent false-negative. Mirror ORT's own `_CaptureStdout`\n(`os.dup2` fd-1 to a temp file, run, restore, read it back):\n\n```python\nos.environ[\"ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO\"] = \"1\"   # BEFORE InferenceSession()\n\n# FD-level capture (see onnxruntime's _CaptureStdout for the exact idiom):\nsaved_fd = os.dup(1)\ntmp = tempfile.TemporaryFile()\nos.dup2(tmp.fileno(), 1)            # redirect native fd-1\ntry:\n    # ... create session, run once ...\nfinally:\n    os.dup2(saved_fd, 1)           # restore fd-1\n    os.close(saved_fd)\ntmp.seek(0)\ncaptured_text = tmp.read().decode()\n\nm = re.search(r\"SdpaKernel=(?P\u003Ckernel>[A-Z_]+)\", captured_text)\ndispatched = m.group(\"kernel\") if m else None\nassert dispatched == \"CUDNN_FLASH_ATTENTION\"\n```\n\nCaveat: `re.search` returns only the **first** `SdpaKernel=` token — correct for the\nsingle-node decode probe here. For a graph with **multiple** attention nodes use\n`re.findall` and check every token, or a later node's `MATH` fallback is masked by an\nearlier cuDNN hit.\n\nPrefer this over reading `torch.backends.cudnn.version()` or any library-version check:\na version probe reads **torch's** cuDNN, not the cuDNN ORT actually loaded\u002Fdispatched —\nthat mismatch is a real trustworthiness bug. Observe-dispatch reads ORT's own routing\ndecision, so it is correct across cuDNN versions with no hard-coded version table.\n\n### Non-skippable canary — `ORT_TEST_REQUIRE_CUDNN_SDPA`\n\nGating decode tests by observed dispatch has a failure mode: if the tier silently\nregresses (stops selecting cuDNN), the observation returns \"not dispatched\" and every\ndecode test **skips green**, hiding the regression as all-green skips.\n\nClose the hole with an env-gated canary: `ORT_TEST_REQUIRE_CUDNN_SDPA=1`. When set, the\ndispatch assertion becomes **non-skippable** — a `MATH` fallback \u002F non-dispatch on the\nminimal known-good config **FAILS LOUD** instead of skipping. When unset (dev boxes,\nunsupported cuDNN) it falls back to the normal skip guard so it never false-alarms.\n\nThe variable is *intended* for an operator to export on a known-good GPU CI leg once one\nexists. Note that today no ONNX Runtime pipeline definition exports it (there is no\nHopper+ GPU CI leg), so it has no effect in this project's CI and only matters for\nmanual\u002Flocal runs where a developer sets it explicitly — don't describe it in test\ndocstrings as an enforcement that CI already applies.\n\n```python\ndef require_cudnn_sdpa():\n    return os.environ.get(\"ORT_TEST_REQUIRE_CUDNN_SDPA\") == \"1\"\n\n# in the test:\nenforce = require_cudnn_sdpa()\nif not enforce and not cudnn_decode_supported(head_size):  # illustrative: your suite's own support predicate\n    self.skipTest(\"cuDNN SDPA decode tier not dispatched; set ORT_TEST_REQUIRE_CUDNN_SDPA=1 to enforce\")\n# then assert dispatch == CUDNN_FLASH_ATTENTION unconditionally\n```\n\nRun both ways to prove it works AND bites:\n\n```bash\npython -m pytest \u003Cfile>.py -v                              # normal: skips where unsupported\nORT_TEST_REQUIRE_CUDNN_SDPA=1 python -m pytest \u003Cfile>.py -v # enforced: fails if not cuDNN\n```\n\n**Prove the teeth.** A canary you never watched fail is not verified. Force `MATH`-only\nby setting the CUDA provider's `sdpa_kernel` **provider option** to the MATH bitmask\n(`16`) — a monkeypatch of the C++ selector is not reachable from Python — under\n`ORT_TEST_REQUIRE_CUDNN_SDPA=1`, and confirm it fails with, verbatim:\n\n```\nAssertionError: 'CUDNN_FLASH_ATTENTION' != 'MATH'\n```\n\nA run that never demonstrates this failure has not proven the canary has teeth (grounding\nrule: negative\u002Fteeth evidence must actually be observed, not asserted).\n\n## 4. Putting it together — one clean run block\n\n```bash\nWORKDIR=$(mktemp -d); cd \"$WORKDIR\"       # neutral + private (see §1)\nsource \u002Fabs\u002Frepo\u002F.venv\u002Fbin\u002Factivate\nexport LD_PRELOAD=\u002Fabs\u002Fcuda12.9\u002Flib64\u002Flibcudart.so.12:\u002Fabs\u002Fcudnn9.8\u002Flib\u002Flibcudnn.so.9\nexport LD_LIBRARY_PATH=\u002Fabs\u002Fcuda12.9\u002Flib64:\u002Fabs\u002Fcudnn9.8\u002Flib\nexport PYTHONPATH=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\nF=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F\u003Cfile>.py\n\npython -m pytest \"$F\" -v                               # A: normal\nORT_TEST_REQUIRE_CUDNN_SDPA=1 python -m pytest \"$F\" -v  # B: canary active, non-skippable\n# C: teeth — force MATH under the env var, expect the AssertionError above\n```\n\n**Check the passed count, not just the exit code.** `pytest -v` exits **0 even if every\ntest skipped** (no CUDA, or an unmet `@skipUnless(ml_dtypes)` guard) — the saved log then\nlooks like passing evidence but proves nothing. Require a **non-zero passed count and\nzero unexpected skips**, and note pytest **exit code 5 = \"no tests collected\"** (usually a\nwrong path or `-k` filter, not success). RUN B's canary only converts *dispatch-related*\nskips into failures — it does **not** rescue collection or environment skips, so still\nread the summary line.\n\nRedirect to a log (`... 2>&1 | tee \"$WORKDIR\u002Fgpu_run.log\"`) — the debug-info stdout and\npytest output are large, and a saved log is the evidence that the run happened and\ndispatched to cuDNN. Write it **inside** `$WORKDIR` (the `mktemp -d` above), not a\npredictable `\u002Ftmp\u002Fgpu_run.log` a co-tenant could pre-create as a symlink to clobber.\n\n## Gotcha quick-reference\n\n| Symptom | Root cause | Fix |\n|---|---|---|\n| `ModuleNotFoundError: onnxruntime.capi` | repo-root `.\u002Fonnxruntime\u002F` source shadows the wheel | run pytest from a private `mktemp -d` (not repo root, not bare `\u002Ftmp`); abs path + `PYTHONPATH` |\n| routes to `MATH` \u002F cuDNN load error | torch's bundled CUDA\u002FcuDNN shadow ORT's | `LD_PRELOAD` system `libcudart.so.12` + `libcudnn.so.9`, set `LD_LIBRARY_PATH` |\n| test passes but path unproven | `MATH` fallback gives same numbers | observe `SdpaKernel=` via `ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1` |\n| all tests skip green, regression hidden | dispatch-gated skip | `ORT_TEST_REQUIRE_CUDNN_SDPA=1` makes assertions non-skippable |\n","# Running ONNX Runtime transformers GPU pytest (dispatch-verified)\n\nReusable, hard-won knowledge for running the Python transformers tests under\n`onnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F` against a **GPU-built wheel**, on a box where\n`torch` is also installed. Three gotchas below will each cost you an hour if\nrediscovered. See the `ort-test` skill for the general test taxonomy and the\nfalse-green modes; this skill is the GPU + transformers + SDPA-dispatch specialization.\n\n## 1. NEUTRAL-CWD gotcha — never run pytest from the repo root\n\n**Symptom:**\n\n```\nModuleNotFoundError: No module named 'onnxruntime.capi'\n# or an AttributeError deep inside onnxruntime import\n```\n\n**Cause:** the repo root contains a source directory `.\u002Fonnxruntime\u002F` (the C++\u002FPython\nsource tree). When pytest runs with the repo root on `sys.path[0]`, `import onnxruntime`\nresolves to that **source** package — which has no compiled `capi` extension — instead\nof the **installed wheel** in your venv's `site-packages`. The source dir *shadows* the\nwheel.\n\n**Fix:** run pytest from a **neutral, private working directory** — a fresh\n`mktemp -d`, not the repo root and not a bare shared `\u002Ftmp` — so the shadowing source\ndir is not on the path, and point at the test file by absolute path. Put the\ntransformers test-helper dir on `PYTHONPATH` so shared helpers still import.\n\n```bash\nWORKDIR=$(mktemp -d); cd \"$WORKDIR\"       # NEUTRAL + PRIVATE cwd — NOT repo root, NOT \u002Ftmp\nexport PYTHONPATH=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\npython -m pytest \u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F\u003Cfile>.py -v\n```\n\n**Why `mktemp -d` and not a bare `cd \u002Ftmp`:** pytest prepends the cwd to `sys.path`, and\nPython auto-imports `sitecustomize.py`\u002F`usercustomize.py` from it at startup — so on a\nshared box a co-tenant's planted `\u002Ftmp\u002Fsitecustomize.py` would execute as arbitrary code\nin your test process. A fresh per-run `mktemp -d` (private, `0700`) keeps the neutral-cwd\nsource-shadow protection while removing that injection vector.\n\nDo **not** `cd` into the repo and run `pytest onnxruntime\u002Ftest\u002F...` — that reintroduces\nthe shadowing. (This is the Python analogue of the C++ \"run from the build output dir\"\nrule in `ort-test`.)\n\n## 2. LD_PRELOAD lib-pinning gotcha — torch's CUDA\u002FcuDNN shadow ORT's\n\n**Symptom:** any of —\n- the SDPA decode tier silently routes to `MATH` instead of `CUDNN_FLASH_ATTENTION`\n  (wrong-version cuDNN loaded), or\n- `libcudnn.so.9: cannot open shared object file` \u002F `undefined symbol` \u002F cuDNN version\n  mismatch errors at first CUDA op, or\n- ORT loads a different CUDA runtime than it was built against.\n\n**Cause:** a pip-installed `torch` ships its **own** bundled CUDA runtime + cuDNN\n(e.g. cu124 → CUDA 12.4 \u002F cuDNN 9.1) under `site-packages\u002Fnvidia\u002F*\u002Flib`. If ORT was\nbuilt against a **different** CUDA\u002FcuDNN (e.g. CUDA 12.9 \u002F cuDNN 9.8), whichever set the\ndynamic loader resolves **first** wins. With torch imported (or its libs on the path),\ntorch's older libs can shadow the ones ORT `dlopen`s → wrong-version dispatch or load\nfailure.\n\n**Fix:** `LD_PRELOAD` the **system** CUDA runtime + cuDNN that ORT was built against so\nthey are loaded first, and add their dirs to `LD_LIBRARY_PATH`. Activate the venv that\nhas the ORT wheel. Concrete form used successfully (CUDA 12.9 + cuDNN 9.8; substitute\nyour absolute lib paths):\n\n```bash\nWORKDIR=$(mktemp -d); cd \"$WORKDIR\"       # neutral + private (see §1)\nsource \u002Fabs\u002Frepo\u002F.venv\u002Fbin\u002Factivate\nexport LD_PRELOAD=\u002Fabs\u002Fcuda12.9\u002Flib64\u002Flibcudart.so.12:\u002Fabs\u002Fcudnn9.8\u002Flib\u002Flibcudnn.so.9\nexport LD_LIBRARY_PATH=\u002Fabs\u002Fcuda12.9\u002Flib64:\u002Fabs\u002Fcudnn9.8\u002Flib\nexport PYTHONPATH=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\npython -m pytest \u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F\u003Cfile>.py -v\n```\n\nNotes:\n- `libcudart.so.12` is correct for **both** CUDA 12.4 and 12.9 (SONAME is major-only) —\n  pinning the 12.9 file forces the right minor.\n- `torch.cuda` still works fine under this preload — the bf16 IO-binding path that uses\n  `torch` tensors + `.data_ptr()` runs correctly.\n- Keep the two exports and the preload together; dropping `LD_LIBRARY_PATH` can still let\n  a transitive dependency resolve against torch's copy.\n\n## 3. Confirm REAL cuDNN SDPA dispatch (don't trust value-equality)\n\nA numerically-correct result does **not** prove the cuDNN SDPA path ran — the kernel has\na `MATH` fallback that produces the same answer (false-green mode 4 in `ort-test`). To\nprove the tier dispatched, **observe** ORT's routing rather than probing a version.\n\n### Observe-dispatch (the correct probe)\n\nORT's ONNX-domain Attention kernel emits a debug line when\n`ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1` is set **before** the `InferenceSession` is\ncreated (the option is read once at session creation). `AttentionKernelDebugInfo::Print`\nemits a token of the form:\n\n```\nSdpaKernel=CUDNN_FLASH_ATTENTION      # or =MATH, =FLASH_ATTENTION, =EFFICIENT_ATTENTION (non-exhaustive)\n```\n\nCapture stdout across a single `run()` and parse it. **Capture at the file-descriptor\nlevel, not `contextlib.redirect_stdout`:** the `SdpaKernel=` line is written to native\nfd-1 from C++, which Python-level stdout redirection never intercepts — you would get\n`dispatched=None`, a silent false-negative. Mirror ORT's own `_CaptureStdout`\n(`os.dup2` fd-1 to a temp file, run, restore, read it back):\n\n```python\nos.environ[\"ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO\"] = \"1\"   # BEFORE InferenceSession()\n\n# FD-level capture (see onnxruntime's _CaptureStdout for the exact idiom):\nsaved_fd = os.dup(1)\ntmp = tempfile.TemporaryFile()\nos.dup2(tmp.fileno(), 1)            # redirect native fd-1\ntry:\n    # ... create session, run once ...\nfinally:\n    os.dup2(saved_fd, 1)           # restore fd-1\n    os.close(saved_fd)\ntmp.seek(0)\ncaptured_text = tmp.read().decode()\n\nm = re.search(r\"SdpaKernel=(?P\u003Ckernel>[A-Z_]+)\", captured_text)\ndispatched = m.group(\"kernel\") if m else None\nassert dispatched == \"CUDNN_FLASH_ATTENTION\"\n```\n\nCaveat: `re.search` returns only the **first** `SdpaKernel=` token — correct for the\nsingle-node decode probe here. For a graph with **multiple** attention nodes use\n`re.findall` and check every token, or a later node's `MATH` fallback is masked by an\nearlier cuDNN hit.\n\nPrefer this over reading `torch.backends.cudnn.version()` or any library-version check:\na version probe reads **torch's** cuDNN, not the cuDNN ORT actually loaded\u002Fdispatched —\nthat mismatch is a real trustworthiness bug. Observe-dispatch reads ORT's own routing\ndecision, so it is correct across cuDNN versions with no hard-coded version table.\n\n### Non-skippable canary — `ORT_TEST_REQUIRE_CUDNN_SDPA`\n\nGating decode tests by observed dispatch has a failure mode: if the tier silently\nregresses (stops selecting cuDNN), the observation returns \"not dispatched\" and every\ndecode test **skips green**, hiding the regression as all-green skips.\n\nClose the hole with an env-gated canary: `ORT_TEST_REQUIRE_CUDNN_SDPA=1`. When set, the\ndispatch assertion becomes **non-skippable** — a `MATH` fallback \u002F non-dispatch on the\nminimal known-good config **FAILS LOUD** instead of skipping. When unset (dev boxes,\nunsupported cuDNN) it falls back to the normal skip guard so it never false-alarms.\n\nThe variable is *intended* for an operator to export on a known-good GPU CI leg once one\nexists. Note that today no ONNX Runtime pipeline definition exports it (there is no\nHopper+ GPU CI leg), so it has no effect in this project's CI and only matters for\nmanual\u002Flocal runs where a developer sets it explicitly — don't describe it in test\ndocstrings as an enforcement that CI already applies.\n\n```python\ndef require_cudnn_sdpa():\n    return os.environ.get(\"ORT_TEST_REQUIRE_CUDNN_SDPA\") == \"1\"\n\n# in the test:\nenforce = require_cudnn_sdpa()\nif not enforce and not cudnn_decode_supported(head_size):  # illustrative: your suite's own support predicate\n    self.skipTest(\"cuDNN SDPA decode tier not dispatched; set ORT_TEST_REQUIRE_CUDNN_SDPA=1 to enforce\")\n# then assert dispatch == CUDNN_FLASH_ATTENTION unconditionally\n```\n\nRun both ways to prove it works AND bites:\n\n```bash\npython -m pytest \u003Cfile>.py -v                              # normal: skips where unsupported\nORT_TEST_REQUIRE_CUDNN_SDPA=1 python -m pytest \u003Cfile>.py -v # enforced: fails if not cuDNN\n```\n\n**Prove the teeth.** A canary you never watched fail is not verified. Force `MATH`-only\nby setting the CUDA provider's `sdpa_kernel` **provider option** to the MATH bitmask\n(`16`) — a monkeypatch of the C++ selector is not reachable from Python — under\n`ORT_TEST_REQUIRE_CUDNN_SDPA=1`, and confirm it fails with, verbatim:\n\n```\nAssertionError: 'CUDNN_FLASH_ATTENTION' != 'MATH'\n```\n\nA run that never demonstrates this failure has not proven the canary has teeth (grounding\nrule: negative\u002Fteeth evidence must actually be observed, not asserted).\n\n## 4. Putting it together — one clean run block\n\n```bash\nWORKDIR=$(mktemp -d); cd \"$WORKDIR\"       # neutral + private (see §1)\nsource \u002Fabs\u002Frepo\u002F.venv\u002Fbin\u002Factivate\nexport LD_PRELOAD=\u002Fabs\u002Fcuda12.9\u002Flib64\u002Flibcudart.so.12:\u002Fabs\u002Fcudnn9.8\u002Flib\u002Flibcudnn.so.9\nexport LD_LIBRARY_PATH=\u002Fabs\u002Fcuda12.9\u002Flib64:\u002Fabs\u002Fcudnn9.8\u002Flib\nexport PYTHONPATH=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\nF=\u002Fabs\u002Frepo\u002Fonnxruntime\u002Ftest\u002Fpython\u002Ftransformers\u002F\u003Cfile>.py\n\npython -m pytest \"$F\" -v                               # A: normal\nORT_TEST_REQUIRE_CUDNN_SDPA=1 python -m pytest \"$F\" -v  # B: canary active, non-skippable\n# C: teeth — force MATH under the env var, expect the AssertionError above\n```\n\n**Check the passed count, not just the exit code.** `pytest -v` exits **0 even if every\ntest skipped** (no CUDA, or an unmet `@skipUnless(ml_dtypes)` guard) — the saved log then\nlooks like passing evidence but proves nothing. Require a **non-zero passed count and\nzero unexpected skips**, and note pytest **exit code 5 = \"no tests collected\"** (usually a\nwrong path or `-k` filter, not success). RUN B's canary only converts *dispatch-related*\nskips into failures — it does **not** rescue collection or environment skips, so still\nread the summary line.\n\nRedirect to a log (`... 2>&1 | tee \"$WORKDIR\u002Fgpu_run.log\"`) — the debug-info stdout and\npytest output are large, and a saved log is the evidence that the run happened and\ndispatched to cuDNN. Write it **inside** `$WORKDIR` (the `mktemp -d` above), not a\npredictable `\u002Ftmp\u002Fgpu_run.log` a co-tenant could pre-create as a symlink to clobber.\n\n## Gotcha quick-reference\n\n| Symptom | Root cause | Fix |\n|---|---|---|\n| `ModuleNotFoundError: onnxruntime.capi` | repo-root `.\u002Fonnxruntime\u002F` source shadows the wheel | run pytest from a private `mktemp -d` (not repo root, not bare `\u002Ftmp`); abs path + `PYTHONPATH` |\n| routes to `MATH` \u002F cuDNN load error | torch's bundled CUDA\u002FcuDNN shadow ORT's | `LD_PRELOAD` system `libcudart.so.12` + `libcudnn.so.9`, set `LD_LIBRARY_PATH` |\n| test passes but path unproven | `MATH` fallback gives same numbers | observe `SdpaKernel=` via `ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1` |\n| all tests skip green, regression hidden | dispatch-gated skip | `ORT_TEST_REQUIRE_CUDNN_SDPA=1` makes assertions non-skippable |\n",{"name":5,"description":7},"main","4edd78a847b9d7755f32d6c99aa60cab2addd39b",".github\u002Fskills\u002Fort-transformers-gpu-pytest",4439,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime","https:\u002F\u002Fonnxruntime.ai","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-12 03:12:39",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add microsoft\u002Fonnxruntime --skill ort-transformers-gpu-pytest",{"tool":61,"command":62},"bunx","bunx skills add microsoft\u002Fonnxruntime --skill ort-transformers-gpu-pytest",{"tool":64,"command":65},"pnpm","pnpm dlx skills add microsoft\u002Fonnxruntime --skill ort-transformers-gpu-pytest","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fort-transformers-gpu-pytest\u002FSKILL.md",true,[69,75,79],{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fattribution.md","REFERENCE","参考文档",1932,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fsecops-detection-engineering\u002Freferences\u002Fattribution.md",{"path":76,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fyaral_compiler_and_udm_guardrails.md",20254,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fsecops-detection-engineering\u002Freferences\u002Fyaral_compiler_and_udm_guardrails.md",{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"scripts\u002Fvalidate_yaral_rule.py","SCRIPT","脚本",20350,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fcloud\u002Fsecops-detection-engineering\u002Fscripts\u002Fvalidate_yaral_rule.py",[86,116,147,168,184,208],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":97,"publishedAt":40},"01M69AJ65WPPT5DS4JA8YWN37B","workspace-estimation-shared-header","Use when editing a boundary-safe shared header meant to be included by both in-tree and shared-provider\u002Fplugin-bridge code (e.g. include\u002Fonnxruntime\u002Fcore\u002Fframework\u002Fworkspace_requirement.h, or any future header supporting OrtKernelWorkspaceEstimateFunc \u002F DeclareWorkspaceRequirements per issue #29775's Phase-A workspace-estimation roadmap), or when a workspace-estimation math helper needs to be callable from a future plugin EP DLL. Covers a Node forward-declaration ODR trap and the math-helper vs. graph-parsing-wrapper reuse boundary discovered while implementing PR #29811 (MatMulNBits pilot).","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fworkspace-estimation-shared-header\u002FSKILL.md",[92],{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000031","产品经理","PROFESSION","适用职业",[98,99,100,101,102,103,104,105,106,107,108,35,109,110,111,112,113,114,115,23],"workspace estimation shared header","workspace","estimation","shared","safe","pr","code","framework","graph","parsing","node","bridge","roadmap","Wrapper","issue","tree","implementing","phase",{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":127,"publishedAt":40},"01M69AJ3F3NXDS7QP76KZ9EX5H","webgpu-local-testing","Build and run ONNX Runtime WebGPU provider tests on Linux WITHOUT a real GPU, using a software Vulkan adapter (Mesa lavapipe). Use when you need to exercise WebGPU EP kernels off-Mac — the Linux webgpu CI leg is build-only, so software Vulkan is how you actually run WebGPU correctness tests locally. SCOPE - lavapipe only validates host-side enforce\u002Fshape bugs and MatMul-free kernels; any graph containing MatMul (including the expanded-Attention node tests) crashes lavapipe and runs ONLY on macOS-arm64 Metal, which is the source of truth for those. Covers install (dnf on Azure Linux), the --use_webgpu build flag, the onnxruntime_provider_test target, VK_ICD_FILENAMES, and the lavapipe MatMul crash gotcha.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fwebgpu-local-testing\u002FSKILL.md",[122],{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",[128,129,130,131,132,133,134,135,136,39,38,37,137,138,139,108,140,141,142,143,144,29,145,146,23],"testing","webgpu","webgpu local testing","local testing","local","macos","metal","vulkan","linux","ci","exercise","how","install","only","free","host","scope","vk","which",{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":151,"categories":152,"keywords":153,"publishedAt":40},"01M69AJ1ASRSCJ90HDWN05TW82","python-kwargs-setattr-security","When reviewing or fixing Python code that uses setattr() with user-controlled kwargs to configure C++ extension objects (SessionOptions, RunOptions, etc.) in ONNX Runtime. Use this to apply the allowlist pattern that prevents arbitrary file writes and other attacks via reflected property access.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fpython-kwargs-setattr-security\u002FSKILL.md",[],[24,154,155,38,104,156,157,158,34,159,160,161,32,162,163,164,165,166,167,23],"security","python kwargs setattr security","file","access","configure","objects","pattern","c++","extension","via","with","attacks","python code","apply",{"skillNo":169,"name":170,"nameZh":6,"description":171,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":172,"categories":173,"keywords":175,"publishedAt":40},"01M69AHW3EFVHBHB2S7103YA21","ort-test","Run ONNX Runtime tests. Use this skill when asked to run tests, debug test failures, or find and execute specific test cases in ONNX Runtime.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fort-test\u002FSKILL.md",[174],{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},[39,176,177,178,179,38,180,37,181,182,34,32,31,183,23],"ort test","run tests","debug","debug test","debug test failures","find","execute","test cases",{"skillNo":185,"name":186,"nameZh":6,"description":187,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":188,"categories":189,"keywords":190,"publishedAt":40},"01M69AHSKXBSERP5NGAJB5E6XD","ort-release-notes","Draft ONNX Runtime release notes using preset configurations for full ORT or scoped component releases. Use when generating highlights with PR links, compiling human contributor acknowledgments from compile_contributors.py output, and applying preset path filtering.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fort-release-notes\u002FSKILL.md",[],[191,192,193,194,195,196,38,103,197,198,199,200,201,202,32,203,204,205,206,207,23],"notes","ort release notes","release notes","release","component","preset","compile","compiling","contributors","releases","human","full","links","path","for","scoped","contributor",{"skillNo":209,"name":210,"nameZh":6,"description":211,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":212,"categories":213,"keywords":217,"publishedAt":40},"01M69AHQAPER0CRAZ0FTX0G2FT","ort-lint","Lint and format ONNX Runtime code. Use this skill when asked to lint, format, or check code style for C++ or Python files in ONNX Runtime.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fonnxruntime\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fort-lint\u002FSKILL.md",[214],{"categoryNo":215,"categoryName":216,"taxonomyType":125,"taxonomyTypeDesc":126},"01JBSEEDCAT000000000000007","代码审查",[218,219,24,220,38,104,221,222,34,223,161,32,224,205,23],"ort lint","lint","code style","format","check","files","style"]