Agents SDK Gets Pi 1.5: From 88 to 459 Agent Runs per Second

Hi everyone, Shii-chan here! Cloudflare's Agents SDK just got a sizable performance boost, so let's take a look!

Cloudflare Changelog developers.cloudflare.com

What was announced?

In the Cloudflare Changelog, Agents SDK v0.33.0 upgrades PiHarness to Pi 1.5.

There are two big parts: performance improvements around streaming, and updates to the session APIs.

The story so far

Until now, Pi wrote streamed answers and tool output to storage on every single update.

• Persisting on each update meant long answers piled up storage transactions
• The scheduler rechecked every conversation on each commit
• Streaming a large tool call copied all the text again whenever new text arrived

What changes

Here are the numbers from the original post:

• Storage writes: answers and tool output are now persisted in batches, within about a second of each update. Over SQLite, transactions drop by about 27x for long answers and 5.7x for turns with parallel tools, and CPU drops by about a quarter
• Throughput across sessions: the scheduler no longer rechecks every conversation. In a SQLite benchmark with 2,048 sessions, one Pi harness went from 88 to 459 agent runs per second
• Memory: streaming a 500 KB tool call used to take about 20 seconds of CPU and a 100 MB heap. Pi now appends new text instead

Streaming updates are still visible immediately. But because persistence is batched, a crash with the default settings can lose up to about one second of streamed progress. You can configure this interval in the Recovery docs.

Dive Deep

Let's go through the API changes too.

• session.watch(): replaces session.events(). It returns the newest transcript entries plus live state, such as the answer being streamed, running tools, and why the last run ended without an answer
• session.continue(): starts a new run from the existing transcript without adding a message. Use it after a model error or when a run hits its turn limit. The session must be idle, and if the transcript already ends in an answer it returns unanswered with the reason nothing_to_answer
• retry() and the parked status: removed. Tasks that hit their restart limit now end instead of being parked
• defaults.maxTurns: caps the tool rounds a run can take in new sessions
• Code mode: scripts can call browserTool and pass its screenshots to image(). browserTool also stops when the harness aborts its call

A few upgrade notes:

• Conversations run on Pi 1.5 can't be read by Pi 1.4, so if something goes wrong, roll forward rather than back
• Replace session.events() with session.watch(), and remove uses of retry() and the parked status
• The Pi Durable 1.5.0 changelog lists the breaking changes, and the Agents SDK v0.33.0 changelog has the upgrade steps

Wrap-up

• Agents SDK v0.33.0 moves PiHarness to Pi 1.5
• Batched persistence cuts storage transactions by about 27x and 5.7x, and CPU by about a quarter
• 2,048 sessions go from 88 to 459 runs per second, and large tool calls use far less memory
• session.watch() and session.continue() are new, while retry() and parked are gone
• There is no rollback to Pi 1.4, so plan to move forward

This one is especially for anyone running lots of agent sessions on Cloudflare!