JA EN
shiichan

OpenSearch Serverless Intelligent Ingestion Makes S3 Logs and Docs Searchable!

AWS What's New ★★★★ View Markdown
#aws#ai#infrastructure

Hi everyone, it's Shiichan!

AWS What's New aws.amazon.com

Want to search the logs and PDFs piling up in S3 but dread building the ingestion plumbing? I found an announcement that looks made for that.

What was announced?

According to AWS What's New, Amazon OpenSearch Serverless now offers intelligent ingestion. It's a fully managed feature that turns logs and documents in Amazon S3 into searchable content in Amazon OpenSearch Serverless (NextGen) collections.

You can set it up with a few clicks in the AWS Management Console or a single API call. Pick an S3 source and a collection index, and it handles the rest!

Why it matters

Normally, getting S3 data into a search index means building ingestion and document-processing infrastructure and sizing capacity yourself. Here, AWS runs all of that for you.

What changes

  • No ingestion or document-processing infrastructure to manage
  • No need to size capacity in advance
  • No pipeline code for logs

You pay only for active processing, so idle time shouldn't cost you.

Dive Deep

Here's what the announcement says in detail.

Supported data comes in two groups:

  • Logs and structured data: JSON, NDJSON, and JSON array
  • Unstructured files: PDF, Word, plain text, and Markdown

The automatic processing includes:

  • Detecting data types
  • Parsing documents
  • Describing images
  • Chunking text
  • Generating vector embeddings
  • Indexing the results

For logs and structured data, it applies transformations and can optionally add embeddings. For unstructured files, it keeps headings, reading order, and tables intact, so the content is ready for semantic and hybrid search. A vision-language model writes text descriptions of charts, diagrams, screenshots, and photos, and those are indexed along with the surrounding content.

There are two ways to ingest:

  • Ingestion jobs: one-time loads such as migrations and backfills
  • Ingestion streams: keep an index in sync as new objects arrive in your bucket

Capacity scales with your workload up to the collection's configured limit. Source S3 buckets can be in the same account as the collection or a different one.

Some limits: it works with OpenSearch Serverless (NextGen) collections, and S3 is the only source. It's available in seven AWS Regions, though the announcement we read doesn't name them. To get started, open your collection in the Amazon OpenSearch Service console and choose the Ingestion tab. Check the pricing page for cost details.

Wrap-up

  • Ingest S3 logs and documents into OpenSearch Serverless (NextGen) with a few clicks or one API call
  • Type detection, parsing, chunking, embeddings, and indexing are automatic
  • Choose one-time ingestion jobs or continuously syncing ingestion streams
  • Available in seven Regions, with S3 as the only source

If you want a log or document search backend without the plumbing work, this one is for you!

Related posts