I am using boto3 to upload massive media files (2GB+) from my Django website to an s3 storage bucket. My issue is that when uploading a file larger than 2.5MB - the connection is immediately timed out and no debug information displays
I assume what is happening is that Django is using the temporary file upload handler, but the temp file handler won't work on the server the Django app is running on (pythonanywhere free tier).
This works flawlessly locally, because it just has to copy to the OS, not the server. I would like to skip the web server so I am not using all the storage/bandwidth
s3.upload_file needs the local path of the file in order to be able to upload it to S3, the only way I could find to do that is grab the temp_path of the file.
# method to chunk the upload process if file is over 2.4MB
def s3_multipart_upload(f, video_id):
# our server path
file_path = "media/"
new_folder = str(video_id)+"/"
file_name = f.name
server_path = file_path + new_folder + file_name
# the file gets copied to ~/tmp/ so grab that path
local_path = f.temporary_file_path()
# create s3 client connection
s3 = boto3.client('s3', settings.AWS_S3_REGION_NAME, **credentials)
config = TransferConfig(
multipart_threshold=1024 * 25,
max_concurrency=10,multipart_chunksize=1024 * 25,
use_threads=True
)
s3.upload_file(
local_path,
settings.AWS_STORAGE_BUCKET_NAME,
server_path,
ExtraArgs={'ACL': 'public-read'},
Config=config,
Callback = ProgressPercentage(local_path)
)
return
I would like to be able to upload directly to S3, straight from my website, without the files going through the server (python anywhere). If there is a cleaner way of doing this, a better server I should use, or if there is a way to upload the file without copying to temp first - I'm all ears.
(I am also new to Python/Django/Servers in general, so any help is appreciated)
I am not sure if you have found the answer but you can upload content directly to s3 using signed url mechanism.
Request Signed URL for s3 from django backend
Use frontend to upload content directory to s3
But I would suggest do this only if you have bigger media files to be uploaded.
I found this as secured approach.
https://www.serverless.com/blog/s3-one-time-signed-url
Related
I have been trying to upload the file received from a S3 trigger to an externally generated signed url. The key code to upload the file using HTTP PUT is as follows
source_filename = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
with open(source_filename , 'r') as object_file:
object_text = object_file.read()
response = requests.put(presigned_url, data=object_text)
the above method is recommended by AWS docs
This is not working since the open function is not able to recognize the "source_filename". Probably because open(a unix call) cannot recognize a S3 url(a custom file storage).
So what is the best method to upload S3 files directly to signed url
So what is the best method to upload S3 files directly to signed url
You can't upload directly from S3. You have to download the S3 object first, and then upload it using S3 pre-signed url to other destination.
Problem is i have hosted at pythonanywhere using django.Video is downloaded at pythonanywhere server and user/client system too.Thats why i used os. remove(path).After downloading it removes from server.
Is there any ways files donot write on pyhtonanywhere server. so that i donot use os.remove(path).
How to restrict to write at pythonanywhere server. Only to download at user system.
def fb_download(request):
link = request.GET.get('url')
html= requests.get(link)
try:
url= re.search('hd_src:"(.+?)"',html.text)[1]
except:
url= re.search('sd_src:"(.+?)"',html.text)[1]
path=wget.download(url, 'Video.mp4')
response=FileResponse(open(path, 'rb'), as_attachment=True)
os.remove(path)
return response
If I understand correctly, you're trying to get a request from a browser, which contains a URL. You then access the page at that URL and extract a further URL from it, and then you want to present the contents of that second URL -- a video -- to the browser.
The way you are doing that is to download the file to the server, and then to serve that up as a file attachment to the browser.
If you do it that way, then there is no way to avoid writing the file on the server; indeed, the way you are doing it right now might have problems because you are deleting the file before you've returned the response to the browser, so there may (depending on how the file deletion is processed and whether the FileResponse caches the file's contents) be cases where there is no file to send back to the browser.
But an alternative way to do it that might work would be to send a redirect response to the URL -- the one in your variable url -- like this, without downloading it at all:
def fb_download(request):
link = request.GET.get('url')
html= requests.get(link)
try:
url= re.search('hd_src:"(.+?)"',html.text)[1]
except:
url= re.search('sd_src:"(.+?)"',html.text)[1]
return redirect(url)
By doing that, the download happens on the browser instead of on the server.
I don’t understand javascript really good,
But i think if you download the file to the server
And then you can download the file to the use using JS
And i think you can use
I have a Rails 5 app using carrierwave, carrierwave-aws and the cloudfront-signer gems. I gave a default bucket configured for Carrierwave. I can upload and serve content fine (i.e. user avatars etc.).
I have another uploader which has a dynamic bucket (i.e. per client) to segregate data. I can upload no problem. The problem is that these files won't serve over cloudfront (403 Forbidden). I suspect it that the base CW config is trying to sign the buckets using the other bucket to access the files.
Here is my uploaded config that I tried so far:
def initialize(*)
super
# https://stackoverflow.com/questions/31589872/setup-for-an-uploader-carrierwave
self.aws_credentials = {
region: ENV['AWS_REGION']
}
self.aws_bucket = Apartment::Tenant.current
self.aws_signer = -> (unsigned_url, options) { Aws::CF::Signer.sign_url unsigned_url, options }
end
It sets the custom bucket but the URLs seem to fail. They are signed but all return 403.
I suspect I need to feed the aws_signer options the bucket or endpoint to make this work.
Any ideas?
Is it possible to upload a file to S3 from a remote server?
The remote server is basically a URL based file server. Example, using http://example.com/1.jpg, it serves the image. It doesn't do anything else and can't run code on this server.
It is possible to have another server telling S3 to upload a file from http://example.com/1.jpg
upload from http://example.com/1.jpg
server -------------------------------------------> S3 <-----> example.com
If you can't run code on the server or execute requests then, no, you can't do this. You will have to download the file to a server or computer that you own and upload from there.
You can see the operations you can perform on amazon S3 at http://docs.amazonwebservices.com/AmazonS3/latest/API/APIRest.html
Checking the operations for both the REST and SOAP APIs you'll see there's no way to give Amazon S3 a remote URL and have it grab the object for you. All of the PUT requests require the object's data to be provided as part of the request. Meaning the server or computer that is initiating the web request needs to have the data.
I have had a similar problem in the past where I wanted to download my users' Facebook Thumbnails and upload them to S3 for use on my site. The way I did it was to download the image from Facebook into Memory on my server, then upload to Amazon S3 - the full thing took under 2 seconds. After the upload to S3 was complete, write the bucket/key to a database.
Unfortunately there's no other way to do it.
I think the suggestion provided is quite good, you can SCP the file to S3 Bucket. Giving the pem file will be a password less authentication, via PHP file you can validate the extensions. PHP file can pass the file, as argument to SCP command.
The only problem with this solution is, you must have your instance in AWS. You can't use this solution if your website is hosted in other Hosting Providers and you are trying to upload files straight to S3 Bucket.
Technically it's possible, using AWS Signature Version 4, Assuming your remote server is the customer in the image below, you could prepare a form in the main server, and send the form fields to the remote server, for it to curl it. Detailed example here.
you can use scp command from Terminal.
1)using terminal, go to the place where there is that file you want to transfer to the server
2) type this:
scp -i yourAmazonKeypairPath.pem fileNameThatYouWantToTransfer.php ec2-user#ec2-00-000-000-15.us-west-2.compute.amazonaws.com:
N.B. Add "ec2-user#" before your ec2blablbla stuffs that you got from the Ec2 website!! This is such a picky error!
3) your file will be uploaded and the progress will be shown. When it is 100%, you are done!
this is my problem: I have some pdf files on a server, my Django web-application
is hosted on another server (not the same of the pdf files).
On my appplication i know the pdf files link on the other server. I want to download that pdf files through my application without read them on web server application.
I try to explane. If i click on download link, my browser shows the pdf into his internal pdf viewer. I don't want this, i want that on click on a button the user will download the file without open it on internal browser.
I looked here: http://docs.djangoproject.com/en/dev/ref/request-response/#telling-the-browser-to-treat-the-response-as-a-file-attachment
but this is not a good way for me, cause it requires that I read the file inside my web-application and after return it to the user.
Is it possible??
Hmm, sounds like the wrong tool for the job. You can't really "redirect" and modify the response header, which means using django just to set the Content-Disposition header would require you to stream the file through django, then have django stream it to the client.
Let a lighter weight web server handle that. If you happen to be using nginx, here's an awesome solution that fits your scenario 99% (the 1% being it's rails setting the header that nginx is waiting for).
If all you want is to set the header and the file doesn't need django processing, it would be even easier to proxy!
If you are not using nginx, I would change the title to a web server specific question about proxying a file & setting headers.
I had a similar problem recently. I have solved it downloading the file to my server and then writing it to the HttpResponse
Here is my code:
import requests
from wsgiref.util import FileWrapper
from django.http import Http404, HttpResponse
def startDownload():
url, filename, ext = someFancyLogic()
request = requests.get(url, stream=True)
# Was the request OK?
if request.status_code != requests.codes.ok:
return HttpResponse(status=400)
wrapper = FileWrapper(request.raw)
content_type = request.headers['content-type']
content_len = request.headers['content-length']
response = HttpResponse(wrapper, content_type=content_type)
response['Content-Length'] = content_len
response['Content-Disposition']
= "attachment; filename={0}.{1}".format(filename, ext)
return response