How to Accelerate Data Uploads to Amazon S3

Imagine this: your company has offices in Europe and Argentina, and your Argentinian team regularly uploads large files to an Amazon S3 bucket located in Europe. Unfortunately, the uploads are very slow due to inconsistent internet quality, and some files occasionally fail to upload altogether. As a result, your data entry team spends hours each week trying to get the job done. To add to the challenge, Amazon S3 doesn’t allow you to upload files larger than 5GB in a single operation.

Today, we’ll explore two ways to improve upload performance to Amazon S3:

• Amazon S3 Multipart Upload: breaking down files into smaller parts to upload them more effectively.
• Amazon S3 Transfer Acceleration: using Amazon’s Global Network instead of the public internet to speed up uploads.

Amazon S3 Multipart Upload

Amazon S3 Multipart Upload is designed specifically to simplify the upload process for large files. Rather than uploading a large object all at once, multipart upload breaks the file into smaller parts, each of which can be uploaded independently.

This method offers several advantages, especially for files over 100MB:

• Faster uploads: Uploading smaller parts is quicker than uploading a single large file.
• Resilience: If one part fails, it doesn’t impact the others, so only the failed part needs to be re-uploaded.
• No expiration: Once the multipart upload process is started, there’s no specific deadline for completion.

This approach not only increases speed and reliability but also provides flexibility for handling large files, such as big CSVs, database backups, images, videos, and more.

Amazon S3 Transfer Acceleration

To understand Amazon S3 Transfer Acceleration and when to use it, consider our initial scenario: your company is located in Europe, but your team in South America uploads files daily to an Amazon S3 bucket hosted in the Europe region.

Let’s explore why the public internet can be inefficient in this situation:

• Slow connection speeds: Internet in some regions might not be fast or stable, especially if those areas lack the latest technology (like fiber or 5G).
• Distance and unpredictability: The further data has to travel, the longer it can take, as public internet data often takes indirect or unpredictable routes to its destination.
• Network congestion: During peak hours, heavy internet traffic can significantly slow down your uploads, forcing data to take “longer” routes to avoid congestion.

How Amazon S3 Transfer Acceleration Works

For users who are far from the S3 bucket’s AWS Region or who experience slow internet speeds, Amazon S3 Transfer Acceleration is a great solution. Here’s how it works:

1. User uploads data: The upload request is sent from the user to the nearest Amazon Edge Location (the closest AWS data center).
2. Data enters Amazon’s Global Network: From the edge location, the data flows over Amazon’s private, global network.
3. Optimized route to destination: The data travels through Amazon’s optimized pathways to the destination S3 bucket.

Using Transfer Acceleration, data can bypass the public internet for most of the journey, making it faster and more reliable for users far from their target S3 Region.

Note: AWS provides a tool to estimate the speed difference between Transfer Acceleration and your current internet connection. If Transfer Acceleration is unlikely to be faster, AWS won’t charge for using it.

Conclusion

• Use Multipart Upload if you’re uploading large files (typically over 100MB) and want to avoid network-related interruptions.
• Use S3 Transfer Acceleration if your users are located far from the AWS Region hosting your S3 bucket or if internet speeds are inconsistent.

With these two methods, you can minimize upload times and improve the reliability of data transfers to Amazon S3.