Agent Evaluation¶
This page describes how the AI agents are tested before a release, what is covered and what is not. It is for the platform team and for anyone who changes agent instructions, skills or tools. The evaluation is a development-time activity of the code base. It does not run inside a customer environment.
Approach¶
The suites use Promptfoo with a simulated user. A language model plays a developer who asks the agent for a change. The agent under test is the production agent: it is built from the same agent configuration (instructions, model, temperature, tools) as in production and runs the same in-process agent loop. A change to an instruction file is therefore tested without a separate copy.
flowchart LR
su[Simulated user<br/>model, max 15 turns] <--> ag[Agent under test<br/>production config and loop]
ag -->|tool calls| mcp[Portal MCP tools<br/>local backend]
mcp --> repo[Test repository<br/>pull requests]
ag --> as[Assertions]
as --> res[Pass or fail<br/>and Promptfoo viewer]
repo -.->|cleanup hook<br/>abandons created PRs| repo| Element | Detail |
|---|---|
| Simulated user | A multi-turn provider with a maximum of 15 turns. The simulated user is a chat model deployed in Azure OpenAI. |
| Agent under test | Resolved with the production agent factory from agent_config/agents/<name>. Optional overrides allow a comparison run with another instruction file, model or temperature. |
| Tools | The agent calls the portal's MCP tools of a locally running backend |
| Conversation state | An in-memory message history for one episode. Nothing is created or deleted in Foundry. |
| Primary pass condition | The reply contains a valid pull request URL |
Suites¶
| Suite | Cases | Purpose |
|---|---|---|
| Spoke create | 22 | The agent creates a spoke configuration pull request. Groups: copy an existing spoke (8 cases), copy with schema descriptions (8 cases), create a new spoke (1 case), blocked requests that must not result in a pull request (5 cases). |
| Spoke QA | 4 | Question and answer cases, graded with a language-model rubric |
The QA cases check knowledge of the spoke documentation (resources a spoke deployment produces, the new-spoke workflow) and two agent rules (private link and network security group rules, and the pull request validation gate). A further create suite overlays customer-specific file name rules and an additional schema on the default configuration, to test the optional fileNameRules and additionalSchema settings. It is customer specific and not described here.
The integration agent has no evaluation suite in the source.
Assertions¶
Besides the pull request check, the suites use custom assertions:
| Assertion | Checks |
|---|---|
| Tool sequence | The agent called the expected tools in the expected order |
| Tool not executed | A forbidden tool, for example a pull request tool in a blocked case, was not called |
| Argument match | A tool was called with the expected arguments |
| Input diff | The configuration the agent submitted differs from the source spoke as expected |
| Spoke JSON validation status | The submitted JSON is valid against the schema |
A cleanup hook runs after each test of the create suite. It finds the pull request URL in the output and abandons the pull request, so repeated runs do not fill the test repository.
Running the suites¶
The suites run on a developer machine against a locally running portal (started with the local application host). The command opens a browser so that you sign in and the run captures a token; the token is not written to disk.
# from plugins/ai-agent-backend
yarn eval all --filter-metadata priority=smoke # smoke cases of both suites
yarn eval spoke-create --filter-pattern 'C01' # selected cases of the create suite
yarn eval spoke-qa # QA suite
yarn eval:view # open the Promptfoo viewer
Endpoints are read from environment variables. Do not commit tokens, endpoints or subscription IDs.
What is not covered¶
| Gap | Detail |
|---|---|
| No pipeline gate | No workflow runs the evaluation. A regression in an instruction file is found only if someone runs the suites. |
| No live evaluation | Production conversations are stored per user and telemetry goes to Azure Monitor, but no sampling, scoring or feedback pipeline exists in the code |
| One agent | Only the spoke agent has suites |
Status of the decision records¶
Three decision records describe the intended evaluation strategy: split evaluation tooling, Promptfoo for development time, and Foundry evaluation for live traffic. All three are still proposed and were written for the Foundry agent service. The accepted decision to run the agent loop in the portal (Decisions) replaced that runtime. The Promptfoo suites match the proposal for development-time evaluation. The live evaluation part is not implemented.
Needs verification
Confirm with the maintainers whether the live evaluation proposal is still planned, and whether an evaluation gate in the pipeline is intended.