ort-test

Running ONNX Runtime Tests

ONNX Runtime uses Google Test for C++ and unittest (preferred) / pytest for Python.

C++ tests

Test executables

Executable	What it tests
onnxruntime_test_all	Core framework, graph, optimizer, session tests
onnxruntime_provider_test	Operator/kernel tests (Conv, MatMul, etc.) across execution providers

Two attention_op_test.cc files — don't confuse them

There are two same-named files testing different operators. Both build into onnxruntime_provider_test:

Path	Operator	gtest suite
test/providers/cpu/llm/attention_op_test.cc	ONNX-domain Attention (opset 23/24)	AttentionTest.*
test/contrib_ops/attention_op_test.cc	contrib MultiHeadAttention / GroupQueryAttention	ContribOpAttentionTest.*

The MEA negative-offset regression tests (Attention_Causal_NonPadKVSeqLen_MEA_*, e.g. ..._MEA_NegOffset_ForceFlashDisabled_FP16_CUDA) live in the providers/cpu/llm file — the ONNX-domain op.

Use --gtest_filter to select specific tests:

./onnxruntime_provider_test --gtest_filter="*Conv3D*"

Running tests

Always run from the build output directory — tests may fail to find dependencies otherwise.

# Linux
cd build/Linux/Release
./onnxruntime_provider_test --gtest_filter="*TestName*"

# macOS
cd build/MacOS/Release
./onnxruntime_provider_test --gtest_filter="*TestName*"

# Windows
cd build\Windows\Release
.\onnxruntime_provider_test.exe --gtest_filter="*TestName*"

You can also run all tests via the build script (assumes a prior successful build):

./build.sh --config Release --test
.\build.bat --config Release --test    # Windows

Locating the build output directory

The default path follows the pattern build/<Platform>/<Config>/ where Platform is Linux, MacOS, or Windows. With Visual Studio multi-config generators on Windows, the config may appear twice (e.g., build/Windows/Release/Release/). The path can also be customized via --build_dir.

If you can't find a test binary, search for it:

# Windows
Get-ChildItem -Path build -Recurse -Filter "onnxruntime_provider_test.exe" | Select-Object -ExpandProperty FullName

# Linux/macOS
find build -name "onnxruntime_provider_test" -type f

Python tests

Use pytest as the test runner:

pytest onnxruntime/test/python/test_specific.py                          # entire file
pytest onnxruntime/test/python/test_specific.py::TestClass::test_method  # specific test
pytest -k "test_keyword" onnxruntime/test/python/                        # by keyword

Python test naming convention: test_<method>_<expected_behavior>_[when_<condition>]

Agent tips

• Activate a Python virtual environment before running tests. See "Python > Virtual environment" in AGENTS.md.
• Beware false-green results — a green run does not always prove anything. See the "False-green taxonomy" section below for the four ways a test can pass without testing your change.
• Redirect test output to a file (e.g., > test_output.txt 2>&1) — output can be large.
• For C++ tests, verify the build directory exists and a prior build completed before running.
• Use --gtest_filter to run a targeted subset when the full suite takes too long.
• Running WebGPU tests locally on Linux without a GPU — WebGPU op tests build into onnxruntime_provider_test and can run against a software Vulkan adapter (Mesa lavapipe). See the webgpu-local-testing skill.

False-green taxonomy — ways a test can "pass" without proving anything

A green result is not always a real pass. Watch for all five modes:

1. Zero-match filter. A --gtest_filter that matches no tests still exits 0 (green). Confirm the [==========] N tests ran line is non-zero — a zero-match run prints 0 tests from 0 test suites. Many operator/kernel gtests run only in onnxruntime_provider_test (CI runs this), NOT onnxruntime_test_all; the wrong binary matches nothing and looks green.
2. Stale binary from an incremental build. If the build did not actually recompile your change (e.g. a header not tracked by the compiler's depfile), the "passing" run executes the OLD code. A test that was failing cannot truly flip to passing without a real rebuild — treat an unexpected FAIL→PASS with suspicion and confirm the linked artifact's mtime advanced. CUDA/CUTLASS instance (nvcc depfiles don't track cutlass_fmha/*.h): see the cuda-cutlass-fmha-incremental-rebuild skill.
3. Checking the wrong artifact's freshness. With a dlopen'd shared provider (e.g. libonnxruntime_providers_cuda.so), the test executable is NOT relinked when the provider recompiles — its mtime stays old while the .so advances. Verify the artifact that actually links your change, not the test exe. Detail: cuda-cutlass-fmha-incremental-rebuild skill.
4. A correct fallback path masks the intended path. A value-only assertion can pass via a different, correct code path without ever exercising the one you meant to test (e.g. a test meant for MEA silently handled by the unfused fallback). Assert/verify which path ran, not just the output value — see "Verify which path/kernel actually executed" below.
5. Arch-portability false-green (verified on only one GPU arch). A CUDA kernel that launches on a large-dynamic-smem arch (e.g. sm90/H100, ~227KB) can fail to launch on a smaller opt-in cap (sm86/89 ~99KB, sm80 ~163KB) with CUDA failure 1: invalid argument — and a path with no fallback (e.g. ORT's MEA) turns that into a hard error, not a silent degrade. So a green run on your local GPU can mask a launch failure on CI's arch. Verify arch-portability, or pick a config whose shared-memory footprint fits every target arch (e.g. a small head_size). Concrete instance: CUTLASS MEA head_size=512 FP16 exceeds sm86's smem opt-in cap and dies at launch — live bug #28388 (the cuda-attention-kernel-patterns skill §1 has the dispatch detail).

Verify which path/kernel actually executed

Value equality alone does not prove the intended code path ran — a correct fallback can produce the right answer (false-green mode 4 above). When a test targets a specific kernel/path, confirm it actually dispatched there instead of trusting the output:

• Enable verbose logging and check the dispatch log line. ORT attention logs one of these exact strings (core/providers/cuda/llm/attention.cc):
  • ONNX Attention: using Flash Attention (:1400)
  • ONNX Attention: using Memory Efficient Attention (:1451)
  • Attention: using unified unfused path (:1482) — note: no ONNX prefix and it reads "unified unfused path", not "Unfused".
• Or force the path via the relevant env var / build config AND add a compile-time guard so the test SKIPs (not silently passes) when the target path is unavailable — e.g. SKIP_IF_MEA_NOT_COMPILED.

Operator-specific routing/forcing details: cuda-attention-kernel-patterns skill §1/§7.