Im currently working an e-shop. So my idea is to store images with Django models in PgAdmin4. As i saw in older posts methods like bytea('D:\image.jpg') and so on just converts the string constant to its binary representation.
So my question is if there is a newer method to store the actual image, or if it is possible to grab the image via a path?
models.py
image = models.ImageField(null=True, blank=True)
PgAdmin4
INSERT INTO product_images(
id, image)
VALUES (SERIAL, ?);// how to insert image?
There are several options for keeping images. The first is to use a storage service like S3, which I recommend. You can read this article for more detailed information. I can also recommend that I have used a third party package ready to use S3 with Django. If you use this option, imagefield will keep the path in S3.
Another option is if you are using only one server, you can keep the pictures in that server's local. Again imagefield will keep the path.
If you say I want to keep it directly in the database, you can follow this link. Currently, there is no newer method for it.
But I have to say that I think using a storage service like S3 is the best way under all circumstances.
Related
I have a general understanding question. I am building a flutter app that relies on a content library containing text files, latex equations, images, pdfs, videos etc.
The content lies on an aws amplify backend. Depending on the navigation of the user in the app, the corresponding data is fetched and displayed.
I am not sure about the correct way of fetching the data. The current method (which works) is that the data is stored in an S3 bucket. When data is requested, the data is downloaded to a temporary directory and then opened and processed in the app. This is actually not slow, but I feel that it is not the way it should be done.
When data is downloaded a file transfer notification pops up, which bothers me because it is shown all the time. Also I would like to read the data directly with something like a get request, without downloading the file first (specially for text files, which I would like to read directly into a String). But here I don't know how it works, because I don't see that you can save data in a file system with the other amplify services like data store or the rest api. Also, the S3 bucket is an intuitive way of storing data that is easy to use for the content creators of my company, for me it seems that the S3 bucket is the way to go. However with S3 I have only figured out the download method to fetch data.
Could someone give me a hint on what is the correct approach for this use case? Thank you very much!
I’m trying to build application with backend in java that allows users to create a text with images in it (something like a a personal blog). I’m planning to store these images to s3 bucket. When uploading image files to bucket i’m hashing the original name and store the hashed one in the bucket. Images are for display purpose only, no user will be able to download them. Frontend displays these images by getting a path to them from the server. So the question is, is there any need to store original name of the image file in the database? And what are the reasons, if any, of doing so?
I guess in general it is not needed because what is more important is how these resources are used or managed in the system.
Assuming your service is something like data access (similar to google drive), I don't think it's necessary to store it in DB, unless you want to make faster search queries.
ok this is a mix of architecture design and code question.. I've had this problem on a couple of applications and I'm not sure how to handle this.
The problem is fairly basic, an app which allows users to attach files/images to an object.
Let's say I have the following Django models: Document and Attachment.
class Document(models.Model):
body = models.TextField(blank=True, help_text='Plain text.')
class Attachment(models.Model):
document = models.ForeignKey(Document, on_delete=models.CASCADE, related_name='attachments')
attachment = models.FileField(upload_to=get_attachment_path, max_length=255, null=False)
filesize = models.IntegerField(default=0)
filetype = models.CharField(blank=True, max_length=100)
orientation = models.IntegerField(default=0)
The get_attachment_path callable simply builds a path based on the document pk and the user.
When a user creates a document, he can attach files to it. As it's a modern world, you want to be able to upload the files before creating the document, so I have a TempUpload object and a direct connection from the web application to S3 (using pre-signed URL).
When the user clicks save on the document form, I get an array of all the TempUpload objects that I need to attach to the new document.
Now here's the problem... within the heroku 30s timeout constraint, I need to:
create the document (quite fast)
iterate the array of TempUpload objects, and create the Attachment
copy the file to its final destination (using boto3 copy_object)
get the filetype (using the magic library, I only query the first 128 bytes, but still.. one roundtrip to S3)
get the file size (another roundtrip to S3)
get the orientation (only if the attachment is an image, also based on the EXIF data and using streaming)
I've already moved the thumbnail generation to an external service. But with bigger files (standard camera pictures can easily be 6-10Mb big), I can have processing delay of almost 1 sec per file, meaning that if the user uploads more than 20 images, it's getting very very close to the heroku timeout...
I'm currently using celery and redis to move most of the processing outside the response lifecycle, but it's not really nice... it can happen that the document is requested before the async task complete. in which case some info are not available yet (size/orientation).
I think that should actually be a fairly standard feature, I'd be interested to know how do you implement this ?
Edit:
Just to add some strategies I'm thinking of:
get the info at the initial upload instead of when saving the document. This would work but the problem is that I don't know when the upload is completed (this happens directly between the web app and S3)
use lambda fonction ?
use S3 tags instead of moving the file to easily differentiate the "saved" files and the ones that should be deleted ?
For testing and dev I'd like to keep as much logic as possible within the django app.. but hey, maybe that's not possible...
Background
I'm doing fairly big file uploads on Django. File size is generally 10MB-100MB.
I'm on Heroku and I've been hitting the request timeout of 30 seconds.
The Beginning
In order to get around the limit, Heroku's recommendation is to upload from the browser DIRECTLY to S3.
Amazon documents this by showing you how to write an HTML form to perform the upload.
Since I'm on Django, rather than write the HTML by hand, I'm using django-uploadify-s3 (example). This provides me with an SWF object, wrapped in JS, that performs the actual upload.
This part is working fine! Hooray!
The Problem
The problem is in tying that data back to my Django model in a sane way.
Right now the data comes back as a simple URL string, pointing to the file's location.
However, I was previously using S3 Boto from django-storages to manage all of my files as FileFields, backed by the delightful S3BotoStorageFile.
To reiterate, S3 Boto is working great in isolation, Uploadify is working great in isolation, the problem is in putting the two together.
My understanding is that the only way to populate the FileField is by providing both the filename AND the file content. When you're uploading files from the browser to Django, this is no problem, as Django has the file content in a buffer and can do whatever it likes with it. However, when doing direct-to-S3 uploads like me, Django only receives the file name and URL, not the binary data, so I can't properly populate the FieldFile.
Cry For Help
Anyone know a graceful way to use S3Boto's FileField in conjunction with direct-to-S3 uploading?
Else, what's the best way to manage an S3 file just based on its URL? Including setting expiration, key id, etc.
Many thanks!
Use a URLField.
I had a similar issue where i want to store file to s3 either directly using FileField or i have an option for the user to input the url directly. So to circumvent that, i used 2 fields in my model, one for FileField and one for URLField. And in the template i could use 'or' to see which one exists and to use that like {{ instance.filefield or instance.url }}.
This is untested, but you should be able to use:
from django.core.files.storage import default_storage
f = default_storage.open('name_you_expect_in_s3', 'r')
#f is an instance of S3BotoStorageFile, and can be assigned to a field
obj, created = YourObject.objects.get_or_create(**stuff_you_know)
obj.s3file_field = f
obj.save()
I think this should set up the local pointer to s3 and save it, without over writing the content.
ETA: You should do this only after the upload completes on S3 and you know the key in s3.
Checkout django-filetransfers. Looks like it plays nice with django-storages.
I've never used django, so ymmv :) but why not just write a single byte to populate the content? That way, you can still use FieldFile.
I'm thinking that writing actual SQL may be the easiest solution here. Alternatively you could subclass S3BotoStorage, override the _save method and allow for an optional kwarg of filepath which sidesteps all the other saving stuff and just returns the cleaned_name.
I want to allow users upload an image through the Django admin, crop and scale that image in memory (probably using PIL), and save it to Amazon S3 without saving the image on the local filesystem. I'll save the image path in my database, but that is the only aspect of the image that is saved locally. I'd like to integrate this special image upload widget into the normal model form on the admin edit page.
This question is similar, except the solution is not using the admin interface.
Is there a way that I can intercept the save action, do manipulations and saving of the image to S3, and then save the image path and the rest of the model data like normal? I have a pretty good idea of how I would crop and scale and save the image to S3 if I can just get access to the image data.
See https://docs.djangoproject.com/en/dev/topics/http/file-uploads/#changing-upload-handler-behavior
If images are smaller than a particular size, the will already be stored only in memory, so you can likely tune the FILE_UPLOAD_MAX_MEMORY_SIZE parameter to suit your needs. Additionally, you'll have to make sure that you don't access the .path field of these uploaded images, because that will cause them to be written out to a file. Instead, use (for example) the .read() method. I haven't tested this, but I believe this will work:
image = PIL.Image(request.FILES['my_file'])
Well if you don't want to touch the Admin part of Django then you can define scaling in the models save() method.
But when using the ImageField in Django. Django can actually do the saving for you. It has height and width options available.
https://docs.djangoproject.com/en/dev/ref/models/fields/#imagefield
For uploading to S3 I really suggest using django-storages backends from:
https://bitbucket.org/david/django-storages/src (preferably S3-boto version)
That way you basically will not have to write any code yourself. You can just use available libraries and solutions that people have tested.