Faster File Uploads to AWS S3

Uploading files is a common requirement most of the applications have. And if you are on AWS cloud, you’re probably using S3 (Simple Storage Service) to store the files which is the cheapest option on the cloud. So now you should probably be wondering what’s the big deal with that? It is true that it’s very simple to implement file uploads with S3, but the way you implement it would directly affect the performance of your application and resource usage of your account.

Before moving on to the details, let’s see what would affect the performance and resource usage of file uploads.

  • Size of the file
  • Bandwidth of the server (available bandwidth)
  • Server concurrency ( how many connections it can handle )
  • Upload method ( multi parts or single part )

The traditional and most common way to implement file uploads is via a server-side api endpoint which accepts the file from the client (web or mobile) and uploads it to S3. The following diagram explains this model.

Let’s review the above model and see the downsides of using it?

  • Latency is hurt since there are two uploads (client -> server and server -> S3)
  • Server side bandwidth bottlenecks hurt the latency and throughput
  • Server side bandwidth usage cost has to be born by us
  • Multiple connections would add considerable pressure on to the server
  • Uploading file as a single part would hurt the throughput

And now comes the important question. How can we get rid of the above drawbacks? And what’s the alternative model we can use ?

Well, the simple answer would be, Let AWS handle it for us.

AWS S3 has provided a method to upload files via Pre-Signed URLs. What on earth does that mean ? Well, according to documentation,

A pre-signed URL gives you access to the object identified in the URL, provided that the creator of the pre-signed URL has permissions to access that object.

In simple terms, it’s a URL which has predefined permissions to modify a particular object. And using that URL, the client (web or mobile) can upload a file directly to AWS S3. The diagram below shows how this model works.

Let’s look at the steps involved in the above model.

  1. The client requests a pre-signed URL from the server.
  2. The server generates a pre-signed URL via the AWS SDK and sends it to the client.
  3. The client uploads the file directly to S3 using the pre-signed URL.

So what are the benefits we can achieve through this?

  • Direct upload to S3 from client reduces latency.
  • High bandwidth availability at AWS backends provides high throughput.
  • AWS backends offer high concurrency.
  • No pressure on our server.
  • The client doesn’t need to have security credentials to directly upload to S3.

You sometimes might feel that this is not that important, but remember this clearly has an impact on the performance of file uploads and reduces the resource usage by a considerable amount when we have fairly large amounts of users.

Leave a Comment

Your email address will not be published. Required fields are marked *