Web Development

GitHub Copilot Evals preview: repeatable tests for agent skills, MCP tools and coding workflows

Editorial Team

November 17, 2026

GitHub Copilot Evals entered public preview on November 17, 2026. It gives developers a way to evaluate a Copilot skill, custom agent, or connected Model Context Protocol tool with repeatable tasks instead of judging the result from a single impressive chat transcript. This is particularly useful when teams change agent instructions, switch models, or introduce tools that can act on a repository. Even a generally capable model may behave differently from one run to the next. A meaningful evaluation tests both the final answer and the steps the agent took to reach it.

Why evaluations matter for coding assistants

Imagine an agent assigned to identify the cause of a failing test suite. It might produce a convincing explanation without reading the relevant test, or it might patch an unrelated file to make a symptom disappear. Traditional outcome-only scoring can miss that behavior. An eval defines a realistic task, an expected result, and observable checks for the tool path. GitHub says Copilot Evals can evaluate skills, agents, and MCP tools, compare outcomes before and after changes, and inspect pass and fail results. The preview uses an extensible local evaluation engine called Vally.

Choose an evaluation scenario you can reproduce

Start in a sandbox repository containing a known, low-risk problem: a deliberately failing unit test, a broken dependency configuration, or a documented API misuse. Prepare a clean baseline revision and record the tests that establish success. Identify unacceptable side effects, such as modifying environment secrets, calling a production endpoint, or changing files outside the targeted feature. Ensure the evaluator has access only to the files and tools needed for the task. If the agent is allowed to edit code, prefer a disposable branch or isolated working tree rather than a live deployment workspace.

A premium conceptual view of repeatable skill and tool evaluations.

Run a controlled Copilot Evals experiment

1. Install the supported integration

GitHub lists several supported surfaces: a Visual Studio Code extension, the Copilot CLI plugin, and the GitHub Copilot app through the github/evals marketplace. Follow the current installation instructions for the surface you use. The feature is described as a public preview for Copilot Individual, Business, and Enterprise plans, so verify plan access and preview status before relying on it in a team's pipeline. Read the permissions required by the plugin and inspect the repository setup before enabling additional MCP servers.

2. Specify a task with measurable acceptance criteria

Use the natural-language creation flow or the /eval command described in GitHub's announcement. Define the initial repository state, what the agent should inspect, and the observable properties of a good result. For instance, a useful test can require a diagnosis linked to a failing test and a minimal fix that does not change authentication settings. Avoid scoring only whether the explanation contains a particular phrase. GitHub notes that users approve each eval run before it begins; keep the approval step and review the intended environment and tools.

3. Change one variable and rerun

Run the baseline first, then alter one skill instruction, model choice, or MCP tool description. Repeat the same evaluation and compare passing checks, failure explanations, and tool behavior. A better pass rate can still conceal undesirable side effects if the agent makes unnecessary edits or accesses sensitive context. Keep a small suite of boundary cases, including missing documentation, partial tool results, large files, and an attempted operation that should be refused. Record enough configuration metadata to make later comparisons meaningful.

Common evaluation mistakes

One mistake is building a test that rewards a polished explanation instead of validated repository behavior. Another is changing several variables together, which makes a performance improvement impossible to attribute. A third is overfitting a skill to a single canned task and mistaking that result for broader reliability. AI outputs can vary; rerun important scenarios and inspect outliers rather than treating a single green result as a statistically strong conclusion. Remember that an eval is not a substitute for unit tests, code review, security scanning, or release approval.

Turn the findings into an engineering decision

Use evaluations to identify recurring failure modes and guide concrete improvements to prompts, tools, permissions, and documentation. Agree on acceptance criteria that matter to maintainers: correctness, minimal changes, no secret exposure, and predictable recovery from missing context. Keep human sign-off for high-impact work. Copilot Evals offers a practical structure for reasoning about agent quality, but confidence should come from multiple well-designed tests and the normal software verification process, not from product labels or isolated demo success.

Sources and further reading

View 0 Comments & Join Discussion