Browser Run /crawl Now Tells You Why a URL Didn't Come Back

Hi everyone, Shii-chan here! Today I'm covering a small but handy improvement to Browser Run's /crawl.

Cloudflare Changelog developers.cloudflare.com

What was announced?

The Cloudflare Changelog says Browser Run /crawl results now include more detailed diagnostics.

Two things were added:

• A per-URL outcome: each URL record can include a field that identifies the specific processing result. The examples named are success, robots_disallowed, timeout, and page_error
• A job-level robots array: it reports the robots.txt state for up to 50 origins the crawl encountered

Why it matters

When you crawl a site, you sometimes wonder why a URL isn't in the results. Was it blocked by robots.txt, did it time out, or did the page itself error? Each needs a different fix.

With this addition, you can read those reasons right from the results.

What changes

• For failed URLs, you can tell robots_disallowed from timeout and respond to each cause separately
• You can check per origin whether robots.txt was fetched and whether usable rules existed

Dive Deep

Each entry in the robots array includes:

• The origin
• Whether usable rules were available
• The HTTP status, when the crawler received a response

Here is the example from the original post:

{
  "robots": [
    { "origin": "https://example.com", "state": "available", "status": 200 }
  ],
  "records": [
    { "url": "https://example.com/docs/", "status": "completed", "outcome": "success" }
  ]
}

The full list of outcome values and the field definitions are in the Crawl endpoint docs under "URL outcomes and robots.txt diagnostics". The announcement itself doesn't list every value.

Wrap-up

• Browser Run /crawl results now include a per-URL outcome
• A job-level robots array shows robots.txt state for up to 50 origins
• You can sort out failure causes just by reading the results

This is for anyone crawling sites with Browser Run who wants to know why some URLs didn't come back!