AGENT SKILL

tao-analyze-gaps-vlm-bcq

Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use after running VLM evaluation when you have a predictions JSON and need to identify failure cases for DEFT root cause analysis on a binary-classification VLM workflow.

License 许可
Apache-2.0
最近更新
2026-08-12
Stars
★ 0
Forks
0

01 前言 Front Matter

name
tao-analyze-gaps-vlm-bcq
tags
[ "gap-analysis", "rcca", "vlm", "evaluation", "false-positive", "false-negative" ]
license
Apache-2.0
metadata
{ "author": "NVIDIA Corporation", "version": "0.1.0" }
description
Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use after running VLM evaluation when you have a predictions JSON and need to identify failure cases for DEFT root cause analysis on a binary-classification VLM workflow.
allowed-tools
Read Bash
compatibility
Requires docker + nvidia-container-toolkit.

02 资源文件 5

03 SKILL.md 正文

VLM Binary Classification Gap Analysis

Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report.

Purpose

After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.

Usage

Invoke the vlm_bcq action inside the TAO Toolkit data services container with Hydra-style key=value overrides:

gap_analysis vlm_bcq \
  predictions_json=/path/to/results.json \
  results_dir=/path/to/output/gaps

Include videos_dir when video_id values in the predictions are relative paths:

gap_analysis vlm_bcq \
  predictions_json=/path/to/results.json \
  results_dir=/path/to/output/gaps \
  videos_dir=/path/to/videos/root

After the run, surface the FP/FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.

Inputs

  • predictions_json: Path to predictions JSON file. Must be a JSON array where each item has video_id, response, and gt fields. response and gt are parsed with word-boundary matching — 'yes' or 'no' anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.
  • videos_dir (optional): Base directory for resolving relative video_id paths. If omitted, video_id values are used as absolute paths.

Predictions JSON format:

[
  {
    "video_id": "/path/to/video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]

Outputs

  • kpi_gaps.jsonl: One JSON object per line for each FP/FN case. Fields: video_id (absolute path), error_type (FP or FN), question, ground_truth, response.
  • kpi_gaps_report.txt: Human-readable table with total FP/FN counts.

If no gaps are found, no files are written and a message is logged.

Key Parameters

Parameter Required Description
predictions_json Yes Path to predictions JSON file
results_dir Yes Output directory; created if it does not exist
videos_dir No Base directory for resolving relative video_id paths

Error Patterns

Error Cause Fix
FileNotFoundError predictions_json does not exist Check the path
ValueError: must be a JSON array Predictions file is not a list Wrap predictions in [...]
ValueError: missing 'gt'/'response'/'video_id' A prediction item is missing a required field Inspect and fix the predictions JSON
Samples silently skipped response or gt contains both or neither 'yes'/'no' Check logs for warnings; inspect those samples

04 同仓库更多 Skills

headmktlucca-ui
headmktlucca-ui/neuroads-lp

xunit-skill

Generates xUnit.net tests in C#. Covers Fact/Theory, constructor injection, IClassFixture, and FluentAssertions. Use when user mentions "xUnit", "[Fact]", "[Theory]", "Assert.Equal", "C# xUnit". Triggers on: "xUnit", "[Fact]", "[Theory]", "Assert.Equal C#", "xUnit.net".

测试与质量Skill与智能体开发
★ 0
headmktlucca-ui
headmktlucca-ui/neuroads-lp

xcuitest-skill

Generates XCUITest UI tests for iOS/iPadOS apps in Swift. Apple's native testing framework for reliable, fast UI automation. Supports local simulators and TestMu AI cloud real devices. Use when user mentions "XCUITest", "XCTest", "iOS UI test", "Swift test", "XCUIApplication". Triggers on: "XCUITest", "XCTest UI", "iOS UI test", "Swift UI test", "XCUIApplication", "TestMu".

测试与质量Skill与智能体开发
★ 0
headmktlucca-ui
headmktlucca-ui/neuroads-lp

world-builder

Create and manage generative game zones for Dorothy's Pokemon-style world. Use this skill when creating, updating, or designing game worlds via MCP tools.

视觉设计Skill与智能体开发
★ 0
headmktlucca-ui
headmktlucca-ui/neuroads-lp

webdriverio-skill

Generates WebdriverIO (WDIO) automation tests in JavaScript or TypeScript. Supports local and TestMu AI cloud. Use when user mentions "WebdriverIO", "WDIO", "wdio.conf", "browser.url", "$", "$$". Triggers on: "WebdriverIO", "WDIO", "wdio", "browser.$".

测试与质量Skill与智能体开发
★ 0
headmktlucca-ui
headmktlucca-ui/neuroads-lp

web-design-guidelines

Web platform design and accessibility guidelines. Use when building web interfaces, auditing accessibility, implementing responsive layouts, or reviewing web UI code. Triggers on tasks involving HTML, CSS, web components, WCAG compliance, responsive design, or web performance.

设计师前端开发视觉设计
★ 0
headmktlucca-ui
headmktlucca-ui/neuroads-lp

watchos-design-guidelines

Apple Human Interface Guidelines for Apple Watch. Use when building watchOS apps, complications, or workout features. Triggers on tasks involving Watch UI, Digital Crown, glanceable interfaces, or wrist-based interactions.

设计师视觉设计
★ 0