JA EN
shiichan

Cloudflare's crawl endpoint now respects robots.txt's Content Signals `use` directive!

Cloudflare Changelog ★★★★ View Markdown
#cloudflare#web#serverless

Hi, I'm Shii-chan!

Cloudflare Changelog developers.cloudflare.com

Today I've got a small but meaningful update from Cloudflare about crawling, so let's dive in!

What was announced?

According to the Cloudflare Changelog, the Browser Rendering API's /crawl endpoint now respects the use directive from the Content Signals standard. Content Signals is a way for site owners to declare, right inside robots.txt, the maximum level at which their content may be used.

With this update, callers can now declare their own intended usage level through a new contentUse parameter.

The story so far

Before this change, the /crawl endpoint didn't check the Content Signals use level in a target site's robots.txt. So even if a site owner had declared "this content is for reference only," that intent wasn't being enforced on the crawling side.

What changes

The contentUse parameter accepts two values, from least to most permissive: reference and full, with full as the default.

Here's the key part: if a target site's robots.txt declares a use level that's more restrictive than the contentUse you declared, your crawl request now gets rejected with a 400 error.

In other words, if a site owner states in robots.txt that their content should only be used up to a certain level, Cloudflare's /crawl endpoint will now actually enforce that. For anyone building crawlers, this is a nice way to reduce the risk of accidentally ignoring a site's stated intent.

Dive Deep

Here's what an actual request looks like.

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://example.com", "contentUse": "reference", "formats": ["markdown"] }'

In this example, contentUse is set to reference, meaning the caller is declaring "I'll only use this for reference purposes." If example.com's robots.txt sets a stricter limit than that, this request would come back with a 400 error.

Wrap-up

  • Browser Rendering's /crawl endpoint now supports the Content Signals standard's use directive
  • The new contentUse parameter lets you declare reference (more restrictive) or full (the default) as your intended usage level
  • Requests get a 400 error if the declared contentUse is more permissive than what a target site's robots.txt allows

If you're building crawling workflows on Cloudflare's Browser Rendering, or you're a site owner who cares about controlling how your content gets used, this update is worth knowing about!

Related posts