AWS DataSync now moves HDFS and Azure Blob data too!
Hi, it's me! Today I want to tell you about a data-migration update from AWS!
AWS What's New
aws.amazon.com
What was announced?
According to AWS What's New, DataSync's Enhanced mode now supports new transfer destinations through agent-based transfers, including:
- Hadoop Distributed File System (HDFS)
- Microsoft Azure Blob Storage
- Self-managed object storage
On top of that, Enhanced mode agents can now be deployed on Microsoft Hyper-V!
Why it matters
Until now, if you wanted to migrate data from an on-premises Hadoop cluster or Azure environment using DataSync, your choice of transfer destinations was more limited. With this update, the perks that come with Enhanced mode — parallelism, unlimited file counts, and detailed metrics — now apply to migrations from HDFS and Azure Blob as well.
What changes
The HDFS support is built with regulated industries in mind.
- High availability through multiple NameNode configurations
- Transparent Data Encryption (TDE) and Kerberos authentication
Together, these let you securely migrate petabyte-scale encrypted Hadoop data without sacrificing availability — great news for data migrations in industries like finance and healthcare.
Dive Deep
Another highlight is that agents can now be deployed on Hyper-V. If your on-premises environment already runs on Hyper-V, you can run Enhanced mode agents there without adding another virtualization layer.
This capability is available in every AWS Region where DataSync is offered. Get started from the AWS DataSync console, and check the AWS DataSync documentation for the details.
Wrap-up
- DataSync Enhanced mode now supports transfers to HDFS, Azure Blob Storage, and self-managed object storage
- Enhanced mode agents can now be deployed on Hyper-V
- HDFS support includes high availability via multiple NameNodes plus TDE and Kerberos authentication
- Especially relevant if you need to securely migrate petabyte-scale Hadoop data in a regulated industry
