JA EN
shiichan

17 Built-in Evaluators: Amazon CloudWatch Omni Makes Agent Quality Visible

AWS Blog ★★★★ View Markdown
#aws#ai#devops

Hey, it's Shii-chan! I found a new service announcement on the AWS Blog today, and it's exactly the kind of thing that gets me excited.

AWS Blog aws.amazon.com

What was announced?

According to the AWS Blog, AWS introduced Amazon CloudWatch Omni, a new observability service. Omni itself is described as a unified observability experience that's app-centric, AI-powered, built on open standards, and delivered off-console, but this particular post zooms in on the part built for generative AI workloads and AI agents.

The post explains that you can trace, evaluate, and experiment with AI agents across any supported framework, either from your IDE or from a standalone web experience. It also comes with built-in evaluators for quality, correctness, and coherence, and it's built on open standards.

Why it matters

Observability for AI agents is genuinely different from monitoring a regular application.

Agent behavior is non-deterministic, so a small prompt change can quietly degrade response quality without tripping any of the alerts a traditional monitoring tool would fire. Latency and error rates can look perfectly healthy while the actual quality of the responses is slipping.

Before this, teams had to manually comb through logs scattered across multiple systems to spot regressions, and constantly switch between their coding environment and a browser dashboard. As agent development scales up, that blind spot around quality regressions becomes a real problem you can't ignore, which is exactly why agent-specific observability matters so much right now.

What changes

Here's how CloudWatch Omni brings visibility into what your agent is actually doing.

  • Trace Explorer gives you a hierarchical, structured timeline covering LLM calls, tool invocations, and reasoning steps
  • You can inspect inputs, outputs, token usage, and latency at any step
  • Compare mode lets you view traces side by side to debug the effect of configuration changes
  • Experiments run the same dataset against multiple agent variants, comparing evaluation scores, latency, and token consumption side by side
  • You can curate production traces into golden datasets and automate regression testing whenever prompts or logic change

Work that used to mean manually chasing logs now gets organized around traces and experiments instead. Instead of tweaking a prompt and hoping it feels better, you get to compare it with actual numbers, which is a huge relief if you're building agents.

Dive Deep

Let's get into the technical details.

  • The built-in evaluators cover 17 dimensions, including coherence, helpfulness, faithfulness, and routing correctness, going beyond what latency alone can tell you
  • Instrumentation is built on open standards, specifically OpenInference and ADOT (AWS Distro for OpenTelemetry), designed to work across environments like Lambda, ECS, and EKS
  • Supported frameworks include LangChain, LangGraph, CrewAI, OpenAI SDK, Strands, and Vercel AI SDK, in both Python and TypeScript
  • Amazon Bedrock AgentCore agents get native observability, and you can also bring in third-party evaluators like AutoEval and DeepEval
  • On the developer experience side, VS Code and Kiro extensions bring traces, a playground, and evaluators right into your editor; Cloud Login connects to your AWS account for persistent storage and team collaboration, while fully local workflows are also supported
  • There's a standalone web console, separate from the AWS Management Console, that you can log into with just SSO, and it shows the same data as the IDE experience

Wrap-up

  • The AWS Blog introduced Amazon CloudWatch Omni's observability features for generative AI and agentic workloads
  • Non-deterministic agent behavior makes it easy to miss quality regressions with traditional monitoring
  • Trace Explorer, Compare mode, Experiments, and golden datasets build a development cycle centered on tracing and evaluation
  • The built-in evaluators cover 17 dimensions, and instrumentation is built on open standards like OpenInference and ADOT
  • It supports a wide range of frameworks including LangChain, CrewAI, and Bedrock AgentCore, available from both IDE extensions and a standalone web console
  • If you're building AI agents yourself, this is a feature set well worth exploring!

Related posts