How to avoid having idle connection timeout while uploading large file? - django

Question

Consider our current architecture:
+---------------+
| Clients |
| (API) |
+-------+-------+
∧
∨
+-------+-------+ +-----------------------+
| Load Balancer | | Nginx |
| (AWS - ELB) +<-->+ (Service Routing) |
+---------------+ +-----------------------+
∧
∨
+-----------------------+
| Nginx |
| (Backend layer) |
+-----------+-----------+
∧
∨
----------------- +-----------+-----------+
File Storage | Gunicorn |
(AWS - S3) <-->+ (Django) |
----------------- +-----------------------+
When a client, mobile or web, try to upload large files (more than a GB) on our servers then often face idle connection timeouts. Either from their client library, on iOS for example, or from our load balancer.
When the file is actually being uploaded by the client, no timeouts occurs because the connection isn't "idle", bytes are being transferred. But I think when the file has been transferred into the Nginx backend layer and Django starts uploading the file to S3, the connection between the client and our server becomes idle until the upload is completed.
Is there a way to prevent this from happening and on which layer should I tackle this issue ?

Answer 1

I have faced the same issue and fixed it by using django-queued-storage on top of django-storages. What django queued storage does is that when a file is received it creates a celery task to upload it to the remote storage such as S3 and in mean time if file is accessed by anyone and it is not yet available on S3 it serves it from local file system. In this way you don't have to wait for the file to be uploaded to S3 in order to send a response back to the client.
As your application behind Load Balancer you might want to use shared file system such as Amazon EFS in order to use the above approach.

Answer 2

You can create an upload handler to upload file directly to s3. In this way you shouldn't encounter connection timeout.
https://docs.djangoproject.com/en/1.10/ref/files/uploads/#writing-custom-upload-handlers
I did some tests and it works perfectly in my case.
You have to start a new multipart_upload with boto for example and send chunks progressively.
Don't forget to validate the chunk size. 5Mb is the minimum if your file contains more than 1 part. (S3 Limitation)
I think this is the best alternative to django-queued-storage if you really want to upload directly to s3 and avoid connection timeout.
You'll probably also need to create your own filefield to manage file correctly and not send it a second time.
The following example is with S3BotoStorage.
S3_MINIMUM_PART_SIZE = 5242880
class S3FileUploadHandler(FileUploadHandler):
chunk_size = setting('S3_FILE_UPLOAD_HANDLER_BUFFER_SIZE', S3_MINIMUM_PART_SIZE)
def __init__(self, request=None):
super(S3FileUploadHandler, self).__init__(request)
self.file = None
self.part_num = 1
self.last_chunk = None
self.multipart_upload = None
def new_file(self, field_name, file_name, content_type, content_length, charset=None, content_type_extra=None):
super(S3FileUploadHandler, self).new_file(field_name, file_name, content_type, content_length, charset, content_type_extra)
self.file_name = "{}_{}".format(uuid.uuid4(), file_name)
default_storage.bucket.new_key(self.file_name)
self.multipart_upload = default_storage.bucket.initiate_multipart_upload(self.file_name)
def receive_data_chunk(self, raw_data, start):
buffer_size = sys.getsizeof(raw_data)
if self.last_chunk:
file_part = self.last_chunk
if buffer_size < S3_MINIMUM_PART_SIZE:
file_part += raw_data
self.last_chunk = None
else:
self.last_chunk = raw_data
self.upload_part(part=file_part)
else:
self.last_chunk = raw_data
def upload_part(self, part):
self.multipart_upload.upload_part_from_file(
fp=StringIO(part),
part_num=self.part_num,
size=sys.getsizeof(part)
)
self.part_num += 1
def file_complete(self, file_size):
if self.last_chunk:
self.upload_part(part=self.last_chunk)
self.multipart_upload.complete_upload()
self.file = default_storage.open(self.file_name)
self.file.original_filename = self.original_filename
return self.file

Answer 3

You can try to skip uploading the file to your server and upload it to s3 directly, then only get back an url for your application.
There is an app for that: django-s3direct you can give it a try.