JA EN
shiichan

AWS Glue's REST API Connector Now Supports VPC Connections, Filter Pushdown, and Partitioned Reads!

AWS What's New ★★★★ View Markdown
#aws#infrastructure#devops

Hi there, it's me! Today's update from AWS Glue is quiet but genuinely useful in production, so let's dig in!

AWS What's New aws.amazon.com

What was announced?

This comes from AWS's What's New. AWS Glue's REST API connector now supports three new capabilities:

  • VPC support
  • Filter pushdown
  • Partition support

The REST API connector lets you ingest data from any source that exposes a REST-based API, even proprietary systems or new platforms without a dedicated Glue connector. This update makes it a lot more practical to rely on in production.

The story so far

Until now, the REST API connector's strength was letting you build ETL pipelines without writing custom code, but it came with real limitations:

  • It couldn't connect directly to data sources behind private subnets, VPNs, or AWS PrivateLink, so traffic had to go over the public internet
  • There was no way to push query filters down to the API, so more data than necessary got transferred out of the source
  • Ingesting from large, paginated APIs had no parallelization, which made it slow

What changes

With these three additions, the REST API connector is now much easier to build into a real production data platform:

  • VPC support lets you securely reach data sources behind private subnets, VPNs, or AWS PrivateLink without exposing traffic to the public internet
  • Filter pushdown translates your query predicates into API-native parameters, so only matching records leave the source, cutting data transfer costs and job runtime
  • Partition support splits large datasets by field or record count across multiple Spark workers, parallelizing ingestion from high-volume, paginated APIs

For teams that want to plug REST API data sources without a native connector, like internal systems or third-party SaaS, into a production pipeline, this is exactly the kind of update they've been waiting for.

Dive Deep

This feature is available in every AWS commercial Region where AWS Glue is available. You can choose a field-based or record-count-based partitioning strategy. Setup details are covered in the REST API connector documentation, so check there when you're ready to build it.

Wrap-up

  • AWS Glue's REST API connector now supports VPC connections, filter pushdown, and partitioning
  • VPC support gives secure access to data sources behind private subnets or PrivateLink
  • Filter pushdown transfers only the records you need, cutting cost and time
  • Partition support parallelizes ingestion from large API datasets
  • Available in every AWS commercial Region where AWS Glue is available

A great fit for anyone who wants to properly wire REST API data sources without a native connector into a production ETL pipeline!

Related posts