# Cloudflare's crawl endpoint now respects robots.txt's Content Signals `use` directive!

Hi, I'm Shii-chan!

Today I've got a small but meaningful update from Cloudflare about crawling, so let's dive in!

## What was announced?

According to the Cloudflare Changelog, the Browser Rendering API's `/crawl` endpoint now respects the `use` directive from the Content Signals standard. Content Signals is a way for site owners to declare, right inside robots.txt, the maximum level at which their content may be used.

With this update, callers can now declare their own intended usage level through a new `contentUse` parameter.

## The story so far

Before this change, the `/crawl` endpoint didn't check the Content Signals `use` level in a target site's robots.txt. So even if a site owner had declared "this content is for reference only," that intent wasn't being enforced on the crawling side.

## What changes

The `contentUse` parameter accepts two values, from least to most permissive: `reference` and `full`, with `full` as the default.

Here's the key part: if a target site's robots.txt declares a `use` level that's more restrictive than the `contentUse` you declared, your crawl request now gets rejected with a 400 error.

In other words, if a site owner states in robots.txt that their content should only be used up to a certain level, Cloudflare's `/crawl` endpoint will now actually enforce that. For anyone building crawlers, this is a nice way to reduce the risk of accidentally ignoring a site's stated intent.

## Dive Deep

Here's what an actual request looks like.

```bash
curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://example.com", "contentUse": "reference", "formats": ["markdown"] }'
```

In this example, `contentUse` is set to `reference`, meaning the caller is declaring "I'll only use this for reference purposes." If `example.com`'s robots.txt sets a stricter limit than that, this request would come back with a 400 error.

## Wrap-up

- Browser Rendering's `/crawl` endpoint now supports the Content Signals standard's `use` directive
- The new `contentUse` parameter lets you declare `reference` (more restrictive) or `full` (the default) as your intended usage level
- Requests get a 400 error if the declared `contentUse` is more permissive than what a target site's robots.txt allows

If you're building crawling workflows on Cloudflare's Browser Rendering, or you're a site owner who cares about controlling how your content gets used, this update is worth knowing about!