Django, Heroku, boto: direct file upload to Google cloud storage - django

Question

In Django projects deployed on Heroku, I used to upload files to Google cloud storage via boto. However, recently I have to upload large files which will cause Heroku timeout.
I am following Heroku's documentation about direct file upload to S3, and customizing as follows:
Python:
conn = boto.connect_gs(gs_access_key_id=GS_ACCESS_KEY,
gs_secret_access_key=GS_SECRET_KEY)
presignedUrl = conn.generate_url(expires_in=3600, method='PUT', bucket=<bucketName>, key=<fileName>, force_http=True)
JS:
url = 'https://<bucketName>.storage.googleapis.com/<fileName>?Signature=...&Expires=1471451569&GoogleAccessId=...'; // "presignUrl"
postData = new FormData();
postData.append(...);
...
$.ajax({
url: url,
type: 'PUT',
data: postData,
processData: false,
contentType: false,
});
I got the following error message:
XMLHttpRequest cannot load http:/... Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8000' is therefore not allowed access.
EDIT:
The output of gsutil cors get gs://<bucketName>:
[{"maxAgeSeconds": 3600, "method": ["GET", "POST", "HEAD", "DELETE", "PUT"], "origin": ["*"], "responseHeader": ["Content-Type"]}]
It seems the CORS is OK. So, how do I solve the problem? Thanks.
EDIT 2:
The header of the OPTION request from Firefox:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.8,en-US;q=0.5,en;q=0.3
Access-Control-Request-Method: PUT
Connection: keep-alive
Host: <bucketName>.storage.googleapis.com
Origin: http://localhost:8000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0
The header of the OPTION request from Chrome:
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:zh-TW,zh;q=0.8,en;q=0.6,en-US;q=0.4,zh-CN;q=0.2
Access-Control-Request-Headers:
Access-Control-Request-Method:PUT
Connection:keep-alive
Host:directupload.storage.googleapis.com
Origin:http://localhost:8000
Referer:http://localhost:8000/
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
X-Client-Data:CIe2yQEIprbJAQjznMoB

Answer 1

The header issue is not coming from your app, I think it's coming from the cloud storage bucket. I had the same issue when setting up an api, the resource you are posting to is missing the header.
https://cloud.google.com/storage/docs/cross-origin
While useful for preventing malicious behavior, this security measure also prevents useful and legitimate interactions between known origins. For example, a script on a page hosted from Google App Engine at example.appspot.com might want to use static resources stored in a Cloud Storage bucket at example.storage.googleapis.com. However, because these are two different origins from the perspective of the browser, the browser won't allow a script from example.appspot.com to fetch resources from example.storage.googleapis.com using XMLHttpRequest because the resource being fetched is from a different origin.
So it looks like you need to configure the bucket to allow cors requests. The google documentation shows the following code to be run from the google cli.
https://cloud.google.com/storage/docs/cross-origin#Configuring-CORS-on-a-Bucket
gsutil cors set cors-json-file.json gs://example
[
{
"origin": ["http://mysite.heroku.com"],
"responseHeader": ["Content-Type"],
"method": ["GET", "HEAD", "DELETE", "PUT"],
"maxAgeSeconds": 3600
}
]
Which would allow you get, upload, and delete content. Hope that helps.

Answer 2

Based on the information in EDIT 2, something is wrong with the request. The preflight (OPTIONS) request includes the header ACCESS-CONTROL-REQUEST-HEADER. This is not a valid CORS header. The correct header is ACCESS-CONTROL-REQUEST-HEADERS, notice the 'S' at the end.
Even if the header was correct, it should not be requesting authorization for a access-control-allow-origin header. ACCESS-CONTROL-ALLOW-ORIGIN is not a header that is sent from the client. It is a header that will automatically be sent in the response from the server to the client when the server gets a preflight request. The client/browser will not allow a cross-origin PUT request unless it gets a ACCESS-CONTROL-ALLOW-ORIGIN header authorizing the browser document's current origin from the cross-origin server in the preflight request.
The presence of the bad header appears to correlate well with the error response you are receiving. However, it looks like that header was probably not in your original code, it looks like you added it later (based on your comments). Make sure to take that header config out, it is definitely not correct.
So I am a little confused about where that header is coming from, but I think it is the source of your problem.
It looks like you are using jQuery to make the AJAX PUT request. All I can really suggest is to make sure you haven't called $.ajaxSetup() somewhere in your JS code that might be configuring the bad header.

Answer 3

After so many trials and errors, I came up with the following. The programs worked, however, sometimes/some of the uploaded images are not visible; other times they are OK. I have no idea why this happened.
I'd like to solicit more ideas why file uploads are OK but some of the images are corrupted.
gsutil commands:
gsutil cors set cors.json gs://<bucketName>
gsutil defacl ch -u allUsers:R gs://<bucketName>
Content of cors.json file:
[
{
"origin": ["*"],
"responseHeader": ["Content-Type"],
"method": ["GET", "POST", "HEAD", "DELETE", "PUT"],
"maxAgeSeconds": 3600
}
]
HTML:
<p id=status>Choose your avatar:</p>
<input id=fileInput type=file>
JavaScript:
$(document).on('change', '#fileInput', function() {
var $this = $(this);
var file = $this[0].files[0];
$.ajax({
url: 'upload/sign/?fileName=' + file.name + '&contentType=' + file.type,
type: 'GET'
})
.done(function(data) {
var response = JSON.parse(data);
uploadFile(file, response.presignedUrl, response.url, response.contentType)
})
.fail(function() {
alert('Unable to obtain a signed URL.');
});
});
function uploadFile(file, presignedUrl, url, contentType) {
var postData = new FormData();
postData.append('file', file);
$.ajax({
url: presignedUrl,
type: 'PUT',
data: postData,
headers: {
'Content-Type': contentType,
},
processData: false,
contentType: false
})
.done(function() {
alert('File upload successful');
})
.fail(function() {
alert('Unable to upload the file.');
});
}
Django:
Project's urls.py:
urlpatterns = [
...
url(r'upload/', include('upload.urls', namespace='upload')),
]
App's urls.py:
urlpatterns = [
url(r'^$', views.upload, name='upload'),
url(r'^sign/', views.sign, name='sign'),
]
views.py:
def upload(request):
# ... render the template
def sign(request):
fileName = request.GET.get('fileName')
contentType = request.GET.get('contentType')
conn = boto.connect_gs(gs_access_key_id=GS_ACCESS_KEY,
gs_secret_access_key=GS_SECRET_KEY)
presignedUrl = conn.generate_url(3600, 'PUT', GS_BUCKET_NAME, fileName, headers={'Content-Type':contentType})
return HttpResponse(
json.dumps({
'presignedUrl': presignedUrl,
'url': GS_URL + fileName,
'contentType': contentType
})
)

Answer 4

I’m my experience I would like to note “It is not possible to by-pass heroku 30s timeout without using javascript AWS SDK”. Don’t use python AWS SDK (boto). You have to completely leave the back-end out of this. Now from your access origin error, the solution is your CORS. You should put this on your CORS Policy:
[
{
"AllowedHeaders": [
""
],
"AllowedMethods": [
"GET",
"PUT",
"POST",
"DELETE",
"HEAD"
],
"AllowedOrigins": [
""
],
"ExposeHeaders": [
"ETag"
]
}
]
Next for javascript AWS SDK. Follow my answer here: Upload file to s3 in front-end with JavaScript AWS SDK on django
There’s a lot missing from the answer I linked as I had to come up with a custom solution because Javascript AWS SDK also passes heroku 30s timeout. What I did was upload the video via javascript SDK than pass the videos ‘AWS url’ another view in a two step django form. With the changing of djnago views I reset heroku 30s timeout with the video already in my s3 bucket, and passed the fileKey to my url with the redirect. On the second part of the form I gain other information for my djnago object than submit it. This was so hard going through all documentation of direct upload to s3. If anyone is reading this and need help please comment for more. I’m on my phone now but I’ll kindly respond from my desktop to post code snippets ✌🏾