Agent Evaluation Metric for multi-turn conversations | Amazon Web Services
Multi-turn agents fail in ways that single-turn evaluation misses: one early mistake quietly corrupts every later turn. This post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level way to measure agent quality. We apply it to its first dimension, correctness. We show how AEM pinpoints the one turn that caused a failure and separates it from the turns that merely inherited the problem.
The correctness challenge in multi-turn agentic conversations
Evaluating agent correctness in a multi-turn conversation is hard because the property itself is fragile, and holistic scores hide where it breaks. This section shows why cascading errors defeat outcome-level evaluation and motivates a decomposable, turn-level metric.
Why correctness matters
A single wrong tool call cascades into downstream failures across turns. Consider a five-turn conversation with an enterprise assistant. The user asks to create a sales report, then refine it. In turn 2, the agent selects the right action...
Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE
