Evals

Evals let you measure agent quality in a repeatable way. You can run an agent against a versioned test dataset before you deploy a change, and you can score live conversations as they happen.

Where to Find Evals

In the sidebar, under Quality & Governance, open Evals. It has four pages:

If Evals is not visible, ask your organization admin for evaluation permissions.

Recommended Workflow

1. Create an eval agent that knows how to judge the target agent's answers.
2. Create or choose the metrics the eval agent should report.
3. Build a test dataset for the target agent and deploy a dataset version.
4. Run a test with a specific agent version, eval agent version, and dataset version.
5. Review the report, fix the agent, and run the test again.
6. Optionally, attach the eval agent to the target agent for live evaluation.

To get started quickly, open the target agent and choose Auto-generate Eval Setup from its Evals menu.

Eval Agents

Click Create New on Eval Agents. The wizard has four steps:

1. Eval Agent Details — name and purpose.
2. Scoring Configuration — how responses are scored.
3. Inputs & Integrations — what the eval agent receives and uses.
4. Preview — review and save.

Deploy the eval agent before you use it in a test run or for live evaluation.

Metrics

Click Create New on Metrics. The wizard has three steps: Metric Details, Insights Configuration, and Preview.

Keep metric descriptions short and specific. Use the same judge model across runs you want to compare.

Test Datasets

1. Open Test Datasets and pick the target agent.
2. Create a dataset and choose a suite type: Regression, Benchmark, Safety, or Custom.
3. Click Add Test Case for each case, or Import JSON to load many at once. Each case holds the user input and the expected answer or criteria.
4. Click Deploy to create an immutable dataset version. Test runs use deployed versions, so results stay comparable even when you edit the dataset later.
5. Click Create Test Run to start a run from the dataset.

Running a Test

Choose Run Test from the agent's Evals menu, or Create Test Run from a dataset. In the dialog, select:

• Agent Version
• Eval Agent and its version
• Dataset and its version

When the run finishes, the report shows the Avg Score, Failed Test Cases, Tool Mismatch cases, and Recommendations. A case passes at a score of 70% or higher.

Open Test Run History to see earlier runs for an agent and compare results.

Live Evaluation

In the agent's Evals step, click Attach Eval Agent, then set:

• Triggers / Sampling strategy — Random, Errors first, or All sessions
• Verdict threshold — sessions that score below it are flagged
• Live evaluation — on or off

Flagged sessions appear under Live Escalated Sessions in the agent's Evals menu.

Observability

Eval sessions are kept separate from normal user conversations. In Logs, switch the view from Agent / App to Evals to review eval sessions, and use View QA on a session to see its scores. The Evals tab on the Observability dashboard shows evaluation trends.

Tips

• Keep datasets small and targeted for fast feedback. Use separate datasets for broad regression suites.
• Always deploy a dataset version before you share results.
• Add failed production sessions to a dataset when they represent a real regression.
• Change one thing at a time between runs: prompt, model, skills, or tools.

See Also

• Building & Testing Agents
• Observability & Analytics