Five things you can do with Sentry’s Agent Tracing

Publisher preview

AI-powered apps are hard to debug without observability. From one run to another, AI produces different results, goes down different paths, and you’re none the wiser as to why it’s happening. Agent Tracing bridges the gap between prompt, tool calls and final output, all within the context of your app. If things break (it always does at some point, we know from experience), you get everything in one complete trace. Model calls, tool executions, and handoffs, alongside everything else your app handles. You can debug silent tool failures, find the slow step in a multi-step workflow, fix multi-agent handoffs where vital context goes missing and the AI starts hallucinating. In this post, we’re going to walk through some of our Agent Tracing recipes from the Sentry Cookbook. We’ll show different use cases that you can set up yourself with your agent. An eval is a test for an AI agent. You give it an input, look at what it said, and score the answer. Some teams write these by hand (or even generate synthetic evals), so let’s automate this process with real conversations from your production data already recorded in Sentry. This recipe pulls these conversations and feeds them to an eval tool of your choice that scores them for you. The recipe exports conversations with one Sentry CLI command, which then become test records. The first user message is the input, the last agent reply is the output, and cost, tokens, and tool errors come along as metadata. You load the records into Braintrust, or into Langfuse, promptfoo, or Arize Phoenix if you prefer, then score them with code checks where you can and an AI agent judge where you can’t.

A saved ApplyDjinn digest is not available for this story yet.

This preview is supplied by the publisher. The complete publication remains at its original source.