I want to upload a video file to AWS S3 bucket using Boto3. I've already created a bucket named 'django-test' and given the required permissions. I am using Django and working on Windows 10 machine.
I've created a function called store_in_s3 in views.py file in my Django app.
The expected file size is lower than 200mbs. I am a bit confused with the several approaches I've tried. Below is the existing code
def store_in_s3(request):
transfer = S3Transfer(boto3.client(
's3',
region_name = settings.AWS_S3_REGION_NAME,
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY
))
client = boto3.client('s3')
bucket = "django-test"
file = request.FILES["file"]
filename = file.name
transfer.upload_file(filename, bucket, "test.mov")
At this point, I am getting the following error: FileNotFoundError: [WinError 2] The system cannot find the file specified: 'test.mov'
But test.mov is the file I've uploaded using HTML form.
My code in HTML form is below:
<form method="post" enctype="multipart/form-data">
{% csrf_token %}
{{ form.file }}
<button type="submit">Submit</button>
</form>
Additional Information: I was successful at uploading the video file at one point in this development process but on S3 its size was ridiculously small - only 28 Bytes. That's why I restarted building the uploader.
I'll be greateful for any help. Please feel free if you need any more information on the question. Thank you.
As you mentioned the file size is greater than 2 MB, its getting stored in the temp location by Django. From the error message, it seems the filename can't be found. So, try it by passing the temp location path, in this case, i.e. file.temporary_file_path()
Related
I am fairly new to the Django and HTML world and and would like to be able to select a video file and then play it in an edited version (AI classification and some OpenCV editing).
At the moment, playing of a local file works in that way that the file path of my dummy video file is fixed in my script where I load the VideoStream.
However, I would like to be able to specify via HTML input which file to load.
I have the problem that I do not know how to pass the selected file.
Via urls, parameters can be passed, but I do not know how I can pass the path or URL of the video file as a parameter. Or can something like this be achieved via post requests and if so how?
Here are parts of my script (I don't use post request for this at the moment):
<form action="" method="POST" enctype="multipart/form-data">
{% csrf_token %}
<input id="myFileInput" type="text">
<input class="button1" id="inputfield" name="upload" type="file" accept="video/*" onchange="document.getElementById('myFileInput').src = window.URL.createObjectURL(this.files[0]); document.getElementById('buttonStart').className = 'button'; this.form.submit()" value="Select a File"/>
</form>
The file path is stored in 'myFileInput'. But I could also get the file path through post request. Then I have the problem, that no class instance that I know where I could save that path and load it later.
<img id="demo" src="">
and the function for
function video_on() {
document.getElementById("demo").src = "{% url 'mask_feed' urlpath='path_I_would_like_to_pass' %}"; //
document.getElementById("wait").innerHTML = "Turning on. . . . .";
setTimeout(showText, 3000);
}
and the urls.py file:
path('mask_feed/<path:urlpath>', views.mask_feed, name='mask_feed'),
And of course I have a mask_feed method in my views.py that takes urlpath as argument.
How can I "insert" the file path I get with html insert into the Django template "{% url 'mask_feed' urlpath='path_I_would_like_to_pass' %}"?
I would like to do something like this:
document.getElementById("demo").src = "{% url 'mask_feed' urlpath='document.getElementById('myFileInput').src' %}"
But it doesn't work because of the quotation marks.
And if this is not the way to go, how should I do it?
My webpage has a functionality that lets users request fairly large json files (3MB+) to be generated. The generation process takes some time and I send them an email with a download link once it is ready.
My problem is that clicking the download URL will open the content of the json file in the browser instead of starting a download dialog.
I have found this question: Serving .json file to download but the solution reserialize the json file into the response:
mixed_query = list(invoices) + list(pcustomers)
json_str = serializers.serialize('json', mixed_query))
response = HttpResponse(json_str, content_type='application/json')
response['Content-Disposition'] = 'attachment; filename=export.json'
As the JSON file already exists, I do not want to rewrite it entirely within the response. I just want users to download the already generated file.
How can I force the URL to start a download when reached?
I am using Django 3.1, with Python 3.6
Here is the email template sent to users once the download link is ready:
{% extends "emails/email_css.html" %}
{% block mail_content %}
<p>Hi, {{ username }}</p>
<p>Your export for {{ export_name }} is ready!</p>
<p>The file will be available for download for 2 days, or until you request a new export.</p>
<a class="download_button" type="button" href={{ download_url }}>Download</a>
{% endblock %}
Edit:
I also tried this solution How to download file using anchor tag <a> but the json file still gets shown in the browser instead of being downloaded.
I eventually found how to do it, by returning a FileResponse with as_attachment=True.
from django.http import FileResponse
def download_export(request):
response = FileResponse(open("path/to/file.json", 'rb'), as_attachment=True,
filename="file_name.json")
return response
Try adding the download property to the <a> tag. You can optionally specify a download filename or not.
Without filename:
<a class="download_button" type="button" href={{ download_url }} download>Download</a>
With filename:
<a class="download_button" type="button" href={{ download_url }} download="someJsonFile.json">Download</a>
See docs here for the download property. Also the related StackOverflow post here.
Im trying to get images from an s3 bucket, and show them on a web page using flask (and boto3 to access the bucket).
I currently have a list of all the pictures from the bucket, but cant get the html to show them(gives me 404 error).
How do I do this without downloading the files?
this is what I have so far:
def list_files(bucket):
contents = []
for image in bucket.objects.all():
contents.append(image.key)
return contents
def files():
list_of_files = list_files(bucket)
return render_template('index.html', my_bucket=bucket, list_of_files=list_of_files)
and this is the html snippet:
<table class="table table-striped">
<br>
<br>
<tr>
<th>My Photos</th>
{% for f in list_of_files %}
<td> <img src="{{ f }}"></td>
{% endfor %}
Thanks a lot!
since loading an image to a html page requires a real image which exists in the directory. images from AWS S3 can be loaded onto a html page if you download them first in the directory, then use its url as a source in html <image> tag.
i found a solution to this but you need to modify it as your needs.
define a function that loads the image from S3 as:
import matplotlib.image as mpimg
import numpy as np
import boto3
import tempfile
s3 = boto3.resource('s3', region_name='us-east-2')
bucket = s3.Bucket('bucketName')
object = bucket.Object('dir/subdir/2015/12/7/img01.jpg')
tmp = tempfile.NamedTemporaryFile()
def imageSource(bucket, object, tmp):
with open(tmp.name, 'wb') as f:
object.download_fileobj(f)
src = tmp.name #dir/subdir/2015/12/7/img01.jpg
retrun src
Just ran into this problem as well, seems like this hasn't been updated for a while so will try to add it.
Your current approach below is right. The only issue is that in order to render an image that is not going to be downloaded to your server, you have to have a direct url to your S3 file. Currently, you only have the image name, not the full url.
def list_files(bucket):
contents = []
for image in bucket.objects.all():
contents.append(image.key)
return contents
def files():
list_of_files = list_files(bucket)
return render_template('index.html', my_bucket=bucket, list_of_files=list_of_files)
Currently, your items in the list of files will look like this:
['file_name1', 'file_name2', 'file_name3']
In order for them to render in your browser directly you need them to look like this:
['file_url1', 'file_url2', 'file_url3']
s3 file urls look something like this: https://S3BUCKETNAME.s3.amazonaws.com/file_name1.jpg
Therefore, instead of the line below
contents.append(image.key)
you need to replace the image.key with something that makes the URL
contents.append(f'https://{S3BUCKETNAME}.s3.amazonaws.com/{image.key})
That should do it, the html you have should work correctly as is. The only other big risk is the files you uploaded are not public, for that you'll need to look at the settings of your bucket on AWS.
Additional Resources and Sources:
Adding a public policy to your AWS S3 Bucket: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html
Uploading and downloading files with Flask & S3: https://stackabuse.com/file-management-with-aws-s3-python-and-flask/
I was following LPTHW ex51 by Zed shaw http://learnpythonthehardway.org/book/ex51.html, and was doing his study drills on web.py , i am a web.py beginner and was successful in uploading an image on Webpage form and then storing it in a local folder. The issue is each image I store , replaces the earlier one. Also I cant figure out how to upload multiple images on server and store them all.
Here is my class Upload in app.py:
class Upload(object):
def GET(self):
web.header("Content-Type","text/html; charset=utf-8")
return render.upload()
def POST(self):
x= web.input(myfile={})
filedir= "C:/Users/tejas/Documents/filesave"
if 'myfile' in x:
fout = open(filedir + '/' + 'myfile.jpg', 'wb') # creates the file where the uploaded file should be stored
fout.write(x.myfile.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete
return "Success! Your image has been saved in the given folder."
raise web.seeother('/upload')
and my upload form- upload.html :
<html>
<head><title>
<div id="header" <h1 style="color:blue;">Upload image file</h1><div/>
</title></head>
<body background-color=light-blue,font-family=verdana,font-size=100%;>
<form method="POST" enctype="multipart/form-data" action="">
<input type="file" name="myfile"/>
<br/> <br/><br/>
<input type="submit"/>
</form>
</body>
</html>
I tried searching a lot for similar questions but all in PHP , and so I try something similar with the code but I could not get it working. Any suggestions to improve the code?
the reason your code is replacing the earlier on is because your hardcoding the path to save the image
fout = open(filedir + '/' + 'myfile.jpg', 'wb')
everytime you upload the file to be altered is the same this could be corrected by adding an new name each time your uploading new file or extracting the name from web input
fout = open(filedir + '/' + x.myfile.filename, 'wb');
according to python when opening for writing files with the same name will be erased
make sure each new file you upload has different name from the previous uploaded
I'm uploading a file directly to an S3 bucket using a multipart form upload and a signed policy (with AWS Signature Version 2), as explained here and here.
The upload is successful (I get redirected to the success_action_redirect URL) but the file is not visible in the bucket, under the key it should be. Though the ACL of the uploaded file was set to public-read, I thought it might be a permission issue, but even the owner of the bucket does not see the file.
Does someone have a hint at might be wrong?
Thank you.
Turns out that all I needed to do was to make sure that the uploaded filename is included in the key that was being uploaded to S3.
If you have a form like this:
<form action="http://johnsmith.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
<input type="input" name="key" value="user/eric/" /><br />
(...)
</form>
Then the file will be uploaded to user/eric. What tripped me up is that the key defined this way was an existing S3 folder. AWS made it seem like the upload was successful but probably just dropped the uploaded files as the key already existed. The solution was to include the filename in the key thusly:
<form action="http://johnsmith.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
<input type="input" name="key" value="user/eric/${filename}" /><br />
(...)
</form>
Also see the Upload examples docs.
Whenever we are uploading the small parts of file using presigned url, at that time it will upload that parts in temp location of AWS.
Once successfully uploaded all parts of file, perform the CompleteMultipartUploadRequest and it will store the your file in s3 bucket.
I hope it will work for you.
CompleteMultipartUploadResult multipartCompleteResult = null;
List<PartETag> partETags = new new ArrayList<>();
partETags.add(new new PartETag(partNumber1, eTag1));
partETags.add(new new PartETag(partNumber2, eTag2));
partETags.add(new new PartETag(partNumber3, eTag3));
CompleteMultipartUploadRequest multipartCompleteRequest =
new CompleteMultipartUploadRequest(getAmazonS3BucketName(), objectKey, uploadId, partETags);
multipartCompleteResult = getAmazonS3Client().completeMultipartUpload(multipartCompleteRequest);