JA EN
shiichan

AI Search can now grab just the content you want with CSS selectors!

Cloudflare Changelog ★★★ View Markdown
#cloudflare#ai#web

Hey everyone, it's Shiichan! Today there's a lovely little upgrade to that feature that crawls whole websites and turns them into knowledge for your AI, so let me tell you all about it.

Cloudflare Changelog developers.cloudflare.com

What was announced?

Over on Cloudflare's Changelog, AI Search website sources now support CSS content selectors. You can specify exactly which parts of a crawled page get extracted and indexed by pairing CSS selectors with URL glob patterns.

The story so far

Until now, when you used a website as a source, the whole page got pulled in as-is. That meant navigation bars, sidebars, footers, and other non-content bits ended up indexed too, which quietly hurt search quality.

What changes

From now on, when a page URL matches a glob pattern, only the elements matching the corresponding CSS selector get extracted, converted to Markdown, and indexed. So you can pinpoint just the blog post body, for example, and grab only what you want. Skipping the boilerplate should make the answers your AI returns noticeably better.

Dive Deep

You can set this up from the dashboard or the API. With the API, you list your content_selector entries inside parse_options.

curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/ai-search/instances" \
  -H "Authorization: Bearer {api_token}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-ai-search",
    "source": "https://example.com",
    "type": "web-crawler",
    "source_params": {
      "web_crawler": {
        "parse_options": {
          "content_selector": [
            { "path": "**/blog/**", "selector": "article .post-body" }
          ]
        }
      }
    }
  }'

Two rules to remember: selectors are evaluated in order and the first matching pattern wins, and you can define up to 10 content selector entries per instance. For more configuration examples, check out the content selectors documentation.

Wrap-up

  • AI Search website sources now support CSS content selectors
  • Pair URL glob patterns with CSS selectors to pinpoint the parts you extract
  • Evaluated top-down with first match winning, up to 10 entries per instance
  • Filter out nav and footers to keep just the body, boosting search quality

This one's for anyone using websites as a RAG source, especially if crawl noise has been getting in your way!

Related posts