tao-analyze-gaps-vlm-bcq
Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use after running VLM evaluation when you have a predictions JSON and need to identify failure cases for DEFT root cause analysis on a binary-classification VLM workflow.
- License 许可
- Apache-2.0
- 最近更新
- 2026-08-12
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- tao-analyze-gaps-vlm-bcq
- tags
- [ "gap-analysis", "rcca", "vlm", "evaluation", "false-positive", "false-negative" ]
- license
- Apache-2.0
- metadata
- { "author": "NVIDIA Corporation", "version": "0.1.0" }
- description
- Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use after running VLM evaluation when you have a predictions JSON and need to identify failure cases for DEFT root cause analysis on a binary-classification VLM workflow.
- allowed-tools
- Read Bash
- compatibility
- Requires docker + nvidia-container-toolkit.
02 资源文件 5
- 参考文档BENCHMARK.md4.2 KB
- 参考文档evals/evals.json830 B
- 参考文档references/skill_info.yaml534 B
- 参考文档skill-card.md3.6 KB
- 其他skill.oms.sig4.7 KB
03 SKILL.md 正文
VLM Binary Classification Gap Analysis
Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report.
Purpose
After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.
Usage
Invoke the vlm_bcq action inside the TAO Toolkit data services container with Hydra-style key=value overrides:
gap_analysis vlm_bcq \
predictions_json=/path/to/results.json \
results_dir=/path/to/output/gaps
Include videos_dir when video_id values in the predictions are relative paths:
gap_analysis vlm_bcq \
predictions_json=/path/to/results.json \
results_dir=/path/to/output/gaps \
videos_dir=/path/to/videos/root
After the run, surface the FP/FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.
Inputs
- predictions_json: Path to predictions JSON file. Must be a JSON array where each item has
video_id,response, andgtfields.responseandgtare parsed with word-boundary matching —'yes'or'no'anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning. - videos_dir (optional): Base directory for resolving relative
video_idpaths. If omitted,video_idvalues are used as absolute paths.
Predictions JSON format:
[
{
"video_id": "/path/to/video.mp4",
"response": "Yes, there is a collision.",
"gt": "B. No",
"question": "Is there a collision?"
}
]
Outputs
- kpi_gaps.jsonl: One JSON object per line for each FP/FN case. Fields:
video_id(absolute path),error_type(FPorFN),question,ground_truth,response. - kpi_gaps_report.txt: Human-readable table with total FP/FN counts.
If no gaps are found, no files are written and a message is logged.
Key Parameters
| Parameter | Required | Description |
|---|---|---|
| predictions_json | Yes | Path to predictions JSON file |
| results_dir | Yes | Output directory; created if it does not exist |
| videos_dir | No | Base directory for resolving relative video_id paths |
Error Patterns
| Error | Cause | Fix |
|---|---|---|
FileNotFoundError |
predictions_json does not exist |
Check the path |
ValueError: must be a JSON array |
Predictions file is not a list | Wrap predictions in [...] |
ValueError: missing 'gt'/'response'/'video_id' |
A prediction item is missing a required field | Inspect and fix the predictions JSON |
| Samples silently skipped | response or gt contains both or neither 'yes'/'no' |
Check logs for warnings; inspect those samples |
04 同仓库更多 Skills
xunit-skill
Generates xUnit.net tests in C#. Covers Fact/Theory, constructor injection, IClassFixture, and FluentAssertions. Use when user mentions "xUnit", "[Fact]", "[Theory]", "Assert.Equal", "C# xUnit". Triggers on: "xUnit", "[Fact]", "[Theory]", "Assert.Equal C#", "xUnit.net".
xcuitest-skill
Generates XCUITest UI tests for iOS/iPadOS apps in Swift. Apple's native testing framework for reliable, fast UI automation. Supports local simulators and TestMu AI cloud real devices. Use when user mentions "XCUITest", "XCTest", "iOS UI test", "Swift test", "XCUIApplication". Triggers on: "XCUITest", "XCTest UI", "iOS UI test", "Swift UI test", "XCUIApplication", "TestMu".
world-builder
Create and manage generative game zones for Dorothy's Pokemon-style world. Use this skill when creating, updating, or designing game worlds via MCP tools.
webdriverio-skill
Generates WebdriverIO (WDIO) automation tests in JavaScript or TypeScript. Supports local and TestMu AI cloud. Use when user mentions "WebdriverIO", "WDIO", "wdio.conf", "browser.url", "$", "$$". Triggers on: "WebdriverIO", "WDIO", "wdio", "browser.$".
web-design-guidelines
Web platform design and accessibility guidelines. Use when building web interfaces, auditing accessibility, implementing responsive layouts, or reviewing web UI code. Triggers on tasks involving HTML, CSS, web components, WCAG compliance, responsive design, or web performance.
watchos-design-guidelines
Apple Human Interface Guidelines for Apple Watch. Use when building watchOS apps, complications, or workout features. Triggers on tasks involving Watch UI, Digital Crown, glanceable interfaces, or wrist-based interactions.