Evals and the quality bar

In the dashboard

1. 1

   Datasets

   Eval > Datasets: add test cases (a question, plus the expected answer or the facts it must contain). You can also draft cases from the agent; drafts wait in Review until a person approves them.
2. 2

   Experiments and gates

   Eval > Experiments: run the agent on the dataset with the evaluators you choose, then Save gate (metric and threshold, for example pass rate at least 0.8). Check gate tells you whether the last run passed.
3. 3

   Make it the quality bar

   On the agent's Deploy tab, the Quality bar card: Choose an eval > Set as quality bar. Only an experiment on this agent with a gate qualifies. From then on, saving the agent creates an unpublished version; publishing it runs the experiment against that version, and a miss blocks the publish with the verdict.
4. 4

   Overrides are explicit

   Publish without passing the quality bar? needs the agent:publish:force permission and is recorded in the audit log.

As code and in CI

const goldens = new Dataset(stack, 'Goldens', {
  datasetName: 'docs-qa-goldens',
  agent,
  items: [
    DatasetItem.create({ itemName: 'export-limit', inputPrompt: 'How many rows can I export?', expectedFacts: ['100,000 rows'] }),
  ],
});

new Experiment(stack, 'Quality', {
  experimentName: 'docs-qa-quality',
  agent,
  dataset: goldens,
  evaluators: [Evaluator.llmJudge({ template: JudgeTemplate.FACTUALITY }), Evaluator.referenceMatch(), Evaluator.safety()],
  gate: EvalGate.create({ metric: EVAL_METRIC.PASS_RATE, threshold: 0.8 }),
  isQualityBar: true,
  // When the agent's tools forward the caller's identity, run the eval as a test user:
  // testEndUser: { endUserId: 'eval-user', customerGroupId: 'test-org' },
});

Agents whose tools forward the user's identity