No More Wiring Workers AI and Vectorize by Hand! Cloudflare Launches Its New Search Engine, AI Search!

Hi everyone, it's Shii! Today I want to share some news I found on the Cloudflare blog. Apparently AI agents can now get their own dedicated search engine over your data. And even better, the pieces you used to have to wire together by hand are now bundled into a single service. Let's dive in!

Cloudflare Blog blog.cloudflare.com

What was announced?

Cloudflare's official blog announced a new managed search service called AI Search. You just point it at files or a website, and it builds a search engine over that data for you.

Until now, you had to combine several Cloudflare primitives yourself, Workers AI, AI Gateway, Vectorize, R2, and Browser Rendering, to build your own search or RAG pipeline. AI Search takes over that whole job as a single service. Once you feed it your data, you get more than just the search engine itself, you also get a /search endpoint and an /mcp endpoint out of the box.

The story so far

If you wanted an AI agent to search your own data, you used to have to wire together multiple primitives yourself.

• Generate embeddings with Workers AI
• Store the vectors in Vectorize
• Call the model through AI Gateway
• Keep files in R2
• Crawl web pages with Browser Rendering

That's the kind of manual plumbing you had to do. You could build something that worked, but the setup and ongoing maintenance overhead was substantial.

What changes

With AI Search, you can skip all that wiring and just point it at your data to get search working. Here's what's in it for agent developers.

• Hybrid search that combines semantic and keyword matching is available from the start
• Turning on public access for a namespace exposes /search and /mcp endpoints
• You can search across multiple websites or document sets at once (multi-instance search)
• You can brand it with a custom domain, like search.example.com/mcp
• Pairing it with Cloudflare Access lets you build authenticated, internal-only search endpoints
• There's also an EmDash CMS plugin that makes it easy to add semantic search over your site's content

If you're indexing a whole site, you don't need a sitemap: choosing the "Discover" parsing option lets it crawl and automatically find pages on its own. That said, right now website sources are limited to zones Cloudflare already manages, though that scope is expected to expand going forward.

Dive Deep

Let's look a bit closer at how it actually works.

Creating an instance is a single command. From the Wrangler CLI, you just specify a namespace, a source, and a crawl type.

npx wrangler ai-search instance create cloudflare-community \
  --namespace dev-stack \
  --source https://community.cloudflare.com \
  --type web-crawler \
  --parse-type discover

Binding it from a Worker is just as simple, you link a namespace in your Wrangler config.

{
  "ai_search_namespaces": [
    { "binding": "AI_SEARCH", "namespace": "cloudflare-stack" }
  ]
}

Here's an example of querying across multiple instances. Register it as an MCP tool, and your agent can call search_dev_stack to search across multiple document sets at once, with reranking enabled.

context.registerTool(
  'search_dev_stack',
  {
    description: 'Search current docs across the Cloudflare stack.',
    inputSchema: z.object({ query: z.string() }),
  },
  async ({ query }) => {
    const res = await context.env.AI_SEARCH.search({
      query,
      ai_search_options: {
        instance_ids: ['developers-cloudflare-com', 'astro'],
        retrieval: { max_num_results: 10 },
        reranking: { enabled: true },
      },
    })
    return { content: [{ type: 'text', text: format(res.chunks) }] }
  }
)

You can also use it directly as an MCP server, just add the URL to your client config.

{
  "mcpServers": {
    "dev-stack": { "url": "https://stack.mcp.cloudflare.com/mcp" }
  }
}

When crawling, it identifies itself as a bot named Cloudflare-AI-Search and properly respects robots.txt directives.

Cloudflare itself built an internal instance called "Dev Stack MCP" that indexes ten sources total, its Docs, Blog, API Docs, and Community, plus external framework docs for Astro, Vite, Vitest, Hono, Replicate, and OpenNext. The goal is for agents to cite current documentation instead of relying on stale training data when writing code.

A preview of the pricing model was also published. It's free during the beta, billing isn't enabled yet, and Cloudflare says it'll email customers well in advance before billing starts. Here's what the future usage-based pricing looks like.

• Base ingestion: $0.75 per 1M tokens, with 5M tokens free per month
• Image processing add-on: an extra $0.50 per 1M tokens, with 5M tokens free per month
• Storage: $2.00 per GB-month, with 10 GB free per month
• Semantic (hybrid/vector) queries: $0.75 per 1k queries, with 2,000 queries free per month
• Full-text queries: $0.10 per 1k queries, with 2,000 queries free per month
• Embedding and reranking: free when using AI Search's defaults or select models from the Workers AI catalog

As an example, if you index 20,000 documents (about 20M tokens) plus 1,000 images and run 30,000 semantic queries a month on the Workers Paid plan, the first month's total comes out to roughly $35. Since indexing is largely a one-time cost, later months are dominated by query charges and settle down to around $21.

Wrap-up

Here's a recap of today's news.

• AI Search launched, letting you get a search engine just by pointing at your data, no more manually combining Workers AI, AI Gateway, Vectorize, R2, and Browser Rendering
• It comes with hybrid search, multi-site search, MCP server support, custom domains, and Cloudflare Access integration
• Cloudflare itself uses it in production for its own "Dev Stack MCP," which indexes ten documentation sources
• It's free during the beta, and a preview of future usage-based pricing was also published

If you've been wanting your AI agents to search your own docs or site, this launch is probably right up your alley!