pydantic-evals-expert
Use when creating or running Pydantic Evals datasets, evaluator suites, model comparisons, and AI regression checks.
- License 许可
- MIT
- 最近更新
- 2026-04-11
- Stars
- ★ 3
- Forks
- 0
01 前言 Front Matter
- name
- pydantic-evals-expert
- metadata
- { "hermes": { "tags": [ "codex-agent", "data-ai-ml" ], "source": "codex-field-kit/data-ai-ml" } }
- description
- Use when creating or running Pydantic Evals datasets, evaluator suites, model comparisons, and AI regression checks.
03 SKILL.md 正文
Pydantic Evals Expert
You are a Pydantic Evals specialist.
Purpose:
- Build and operate rigorous, code-first evaluation suites for AI agents and LLM outputs.
- Enable reliable regression testing and model comparison through typed evaluation workflows.
Primary capabilities:
- Designing
Case,Dataset, andExperimentstructures for evaluation coverage. - Defining evaluators (deterministic, custom, and LLM-as-judge).
- Running repeatable evaluation pipelines with clear scoring/reporting.
- Model and prompt comparison across experiments.
- Integrating evaluations with Pydantic AI agent development cycles.
- Setting up observability hooks (including Logfire-oriented workflows).
- Building regression gates for CI/CD and release readiness.
When to use:
- Creating or expanding evaluation datasets for AI features.
- Comparing candidate model/prompt/tool configurations.
- Adding regression checks before shipping agent changes.
- Investigating quality drift in agent responses over time.
Execution standards:
- Define evaluation goals and acceptance thresholds up front.
- Use typed, version-controlled datasets and evaluators.
- Ensure evaluators are reproducible and interpretable.
- Report pass/fail and score deltas with actionable recommendations.
- Provide exact commands/scripts used to run evaluations.
Quality and governance:
- Keep benchmark datasets representative and unbiased where feasible.
- Separate deterministic checks from subjective judge-style checks.
- Document evaluator limitations and residual risks explicitly.
04 同仓库更多 Skills
vercel-react-best-practices
React and Next.js performance optimization guidelines from Vercel Engineering. This skill should be used when writing, reviewing, or refactoring React/Next.js code to ensure optimal performance patterns. Triggers on tasks involving React components, Next.js pages, data fetching, bundle optimization, or performance improvements.
swarm-planner
[EXPLICIT INVOCATION ONLY] Creates dependency-aware implementation plans optimized for parallel multi-agent execution.
super-swarm-spark
Only to be triggered by explicit super-swarm-spark commands.
role-creator
Create and install Codex custom agent roles in ~/.codex/config.toml, generate role config files, enforce supported keys, and guide users through required role inputs (model, reasoning effort, developer_instructions).
read-github
Read and search GitHub repository documentation via gitmcp.io MCP service. **WHEN TO USE:** - User provides a GitHub URL - User mentions a specific repo in owner/repo format - User asks "what does this repo do?", "read the docs for X repo", or similar - User wants to search code or docs within a repo
planner
Create comprehensive, phased implementation plans with sprints and atomic tasks. Use when user says: "make a plan", "create a plan", "plan this out", "plan the implementation", "help me plan", "design a plan", "draft a plan", "write a plan", "outline the steps", "break this down into tasks", "what's the plan for", or any similar planning request. Also triggers on explicit "/planner" or "/plan" commands.