AGENT SKILL

m365-agent-evaluator

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

License 许可
MIT
所属仓库
microsoft/skills
最近更新
2026-11-14
Stars
★ 3.2k
Forks
368

01 前言 Front Matter

name
m365-agent-evaluator
description
Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

02 资源文件 17

03 SKILL.md 正文

M365 Agent Evaluator

Use this skill to help users evaluate Microsoft 365 Copilot declarative agents with @microsoft/m365-copilot-eval. The skill designs schema-compatible eval datasets, runs the public preview CLI, analyzes results, and recommends targeted fixes.

Default to Microsoft 365 Agents Toolkit (ATK) projects when detected, but do not hard-stop solely because the current directory is not ATK. The CLI can also evaluate deployed agents with an explicit M365_AGENT_ID or --m365-agent-id.

Always use this CLI invocation

npx -y --package @microsoft/m365-copilot-eval@latest runevals

Do not recommend the old private aka.ms installer, global installs, bare runevals, bare npx runevals, --input, or --html.

Activation workflow

  1. Identify the user goal: setup, dataset authoring, running evals, analyzing results, or updating an existing eval suite.
  2. Load only the reference needed for the current goal:
    • references/workflow.md for the end-to-end operator workflow and CLI commands.
    • references/azure-setup.md for prerequisites, env files, and secret handling.
    • references/eval-templates.md when creating or editing eval datasets.
    • references/pra-framework.md when deciding what scenarios to generate.
    • references/result-analysis.md after JSON/CSV/HTML results exist.
    • references/guardrails.md before writing files, handling secrets, clearing cache, signing out, or troubleshooting.
  3. Detect project shape:
    • ATK: .env.local, .env.local.user, env\.env.local.user, m365agents.yml, or appPackage\declarativeAgent.json.
    • Non-ATK: an eval dataset plus M365_AGENT_ID, --m365-agent-id, or a named environment file such as env\.env.dev.
  4. Verify prerequisites without exposing values:
    • Node.js 24.12.0 or newer.
    • Microsoft 365 Copilot license and a deployed M365 Copilot agent.
    • Tenant admin consent for the WorkIQ Client App.
    • TENANT_ID, Azure OpenAI in Foundry Models endpoint/key, and recommended/default gpt-4o-mini deployment.
  5. Choose the workflow:
    • No dataset: create evals\evals.json.
    • Existing dataset: run, analyze prior results, or propose changes.
    • Quick check: use inline prompts.
    • Exploration: use interactive mode.

Current dataset contract

Generate schema version 1.2.0 documents with a root items array. Do not generate the old PromptsObject or root prompts format.

Minimum shape:

{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Agent evaluation suite",
    "tags": ["starter"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent explains its supported scope without inventing unsupported capabilities."
    }
  ]
}

Use references\prompts-schema.json as the local schema source and references\eval-templates.md for copyable single-turn, multi-turn, evaluator, and threshold examples.

Public evaluator names

Evaluator names are case-sensitive. Use only the public configurable evaluator names unless a newer authoritative source proves otherwise.

Evaluator Semantics
Relevance LLM score from 1-5; default threshold 3.
Coherence LLM score from 1-5; default threshold 3.
Groundedness LLM score from 1-5 against context/expected evidence; default threshold 3.
Similarity LLM score from 1-5 against expected_response; default threshold 3.
Citations Count-based citation check; default threshold 1.
ExactMatch Boolean exact string match.
PartialMatch String similarity from 0.0-1.0; default threshold 0.5.

Treat ToolCallAccuracy as legacy/private for authoring. Do not add it to generated datasets unless current public CLI/schema documentation explicitly reintroduces it.

Common commands

# Version/help checks
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --help

# First-time setup / EULA
npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only

# Batch run with explicit JSON output
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.json

# Human-review HTML or spreadsheet-friendly CSV
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.html
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.csv

# Quick checks
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts "What can you help me with?" --expected "The agent describes its supported scope."

# Non-ATK or named environment
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --m365-agent-id <agent-id> --env dev

Use --concurrency only with values 1-5. Start with 1 for debugging and increase only after setup is stable.

Version and PATH safety

Before diagnosing agent behavior, confirm which executable is running:

Get-Command runevals -All
npm list -g @microsoft/m365-copilot-eval --depth=0
npm view @microsoft/m365-copilot-eval version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest where runevals

If bare runevals prints This version of the M365 Evals CLI has stopped working and must be updated, treat it as a stale PATH/global install. Re-run with the npx --package ...@latest command above, then ask before removing global shims with npm uninstall -g @microsoft/m365-copilot-eval.

File conventions

Path Purpose
.env.local Non-secret ATK config such as M365_TITLE_ID.
.env.local.user or env\.env.local.user Local secrets such as tenant ID and Azure OpenAI key.
env\.env.<environment> Named environment config for non-ATK or explicit --env workflows.
evals\evals.json Source-controlled eval dataset if the user wants it committed.
.evals\ Local run outputs; usually gitignored.

Never print or commit secrets, prompts containing sensitive data, retrieved content, debug logs, or raw result files unless the user explicitly asks and confirms the data is safe to share.

Generation guidance

Use PRA as a scenario-design framework:

  • Perceive: retrieval, grounding, and source coverage.
  • Reason: instruction adherence, synthesis, ambiguity handling, and refusal behavior.
  • Act: declared capability/action behavior. Score with public evaluators such as Relevance, Coherence, Similarity, ExactMatch, or PartialMatch; do not use legacy ToolCallAccuracy.

Ask before overwriting an existing dataset. When writing generated evals, write to a temporary file first and rename on success.

Result analysis guidance

Analyze only evaluator keys that are present. Missing score keys usually mean the evaluator was not configured for that item, not that it failed.

Use current score keys when present: relevance, coherence, groundedness, similarity, citations, exactMatch, and partialMatch. Group failures into likely root causes: instruction issue, grounding issue, citation issue, expected-answer mismatch, capability gap, auth/environment issue, or eval-quality issue.

Do not run real tenant-dependent evals unless the user has provided or approved the necessary tenant, agent, and Azure OpenAI configuration.

04 同仓库更多 Skills

microsoft
microsoft/skills

azure-advisor-cost-optimization

Analyze Azure Advisor Cost/IdleResources findings and prevalidate deletion safety, including Service Groups. WHEN: "find unused Azure resources", "what resources am I paying for but no longer using", "idle-resource savings", "historical idle-resource savings", "are these idle resources safe to delete". Generic Cost, other subcategories, and cross-category requests use azure-advisor.

★ 3.2k
microsoft
microsoft/skills

skill-creator

Guide for creating effective skills for AI coding agents working with Azure SDKs and Microsoft Foundry services. Use when creating new skills or updating existing skills.

Skill与智能体开发
★ 3.2k
microsoft
microsoft/skills

podcast-generation

Generate AI-powered podcast-style audio narratives using Azure OpenAI's GPT Realtime Mini model via WebSocket. Use when building text-to-speech features, audio narrative generation, podcast creation from content, or integrating with Azure OpenAI Realtime API for real audio output. Covers full-stack implementation from React frontend to Python FastAPI backend with WebSocket streaming.

前端开发后端与接口
★ 3.2k
microsoft
microsoft/skills

microsoft-docs

Understand Microsoft technologies by querying official documentation. Use whenever the user asks how something works, wants tutorials, needs configuration options, limits, quotas, or best practices for any Microsoft technology (Azure, .NET, M365, Windows, Power Platform, etc.)—even if they don't mention "docs." If the question is about understanding a concept rather than writing code, this is the right skill.

Skill与智能体开发
★ 3.2k
microsoft
microsoft/skills

mcp-builder

Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP), Node/TypeScript (MCP SDK), or C#/.NET (Microsoft MCP SDK).

Skill与智能体开发
★ 3.2k
microsoft
microsoft/skills

kql

KQL language expertise for writing correct, efficient Kusto Query Language queries. Covers syntax gotchas, join patterns, dynamic types, datetime pitfalls, regex patterns, serialization, memory management, result-size discipline, and advanced functions (geo, vector, graph). USE THIS SKILL whenever writing, debugging, or reviewing KQL queries — even simple ones — because the gotchas section prevents the most common errors that waste tool calls and cause expensive retry cascades. Trigger on: KQL, Kusto, ADX, Azure Data Explorer, Fabric Real-Time Intelligence, EventHouse, Log Analytics, log analysis, data exploration, time series, anomaly detection, summarize, where clause, join, extend, project, let statement, parse operator, extract function, any mention of pipe-forward query syntax.

数据分析师内容创作者写作与文案Skill与智能体开发
★ 3.2k