Django storage S3 - Save File Path only without file using ModelSerializer - django

I'm using boto3 to upload files to S3 and save their path in the FileField.
class SomeFile(models.Model):
file = models.FileField(upload_to='some_folder', max_length=400, blank=True, null=True)
For the above model the following code works to create a record.
ff = SomeFile(file='file path in S3')
ff.full_clean()
ff.save()
Now, when I use ModelSerializer to do the same.
class SomeFileSerializer(serializers.ModelSerializer):
class Meta:
model = SomeFile
fields = ('file')
I get this error after running the code below
rest_framework.exceptions.ValidationError: {'file': [ErrorDetail(string='The submitted data was not a file. Check the encoding type on the form.', code='invalid')]}
serializer = SomeFileSerializer(data={'file': 'file path to S3'})
serializer.is_valid(raise_exception=True)
I need help in setting up the serializer to accept file path without actually having the file.

I was really in the same situation, and it was hard to find the solution on the web.
We have two options to solve this problem.
1. Passing data directly to save method
Read action: use serializer's read only ImageField
Write action: pass kwargs to save method
serializers.py
from rest_framework import serializers
class SomeFileSerializer(serializers.ModelSerializer):
file = serializers.ImageField(read_only=True)
class Meta:
model = SomeFile
fields = ('file')
views.py
serializer = SomeFileSerializer(data={'file': 'file path to S3'})
serializer.is_valid(raise_exception=True)
# for put method
serializer.save(file=request.data.get('file'))
# for patch method (if partial=True in serializer)
if request.data.get('file'):
serializer.save(file=request.data.get('file'))
else:
serializer.save()
2. Using CharField instead of ImageField
Read action: override to_representation function to response absolute url
Write action: use CharField to avoid ImageField's validation and action
serializers.py
from rest_framework import serializers
class SomeFileSerializer(serializers.ModelSerializer):
file = serializers.CharField(max_length=400)
class Meta:
model = SomeFile
fields = ('file')
def to_representation(self, instance):
representation = super().to_representation(instance)
if instance.file:
# update filename to response absolute url
representation['file'] = instance.file_absolute_url
return representation
models.py
class SomeFile(models.Model):
file = models.FileField(upload_to='some_folder', max_length=400, blank=True, null=True)
#property
def file_absolute_url(self):
return self.file.url if self.file else None
Although I chose the 2nd solution because of drf_spectacular for documentation, the 1st solution would be easy to implement.

Related

Django - problem with saving data in Createview with ModelForm to non-default database

I've got problem with saving data to non-default database.
In models.py I've got:
grid_fs_storage = GridFSStorage(collection='tab_userinquiries', base_url='mydomain.com/userinquiries/',database='mongo_instance')
class DocsUserInquiry(models.Model):
query_pk = models.CharField(blank=False, null=False, unique=True, max_length=150, primary_key=True) # auto - calculated
query_file_md5 = models.CharField(blank=False, null=True, unique=False, max_length=200) # auto - calculated
query_file = models.FileField(upload_to='userinquiries',storage=grid_fs_storage,null=True) # auto from form
In views.py:
class UploadInquiryFileView(CreateView,LoginRequiredMixin):
model=DocsUserInquiry
template_name ='new_inquiry.html'
success_message = "You've added your new Token successfully"
form_class = UploadInquiryFileForm
def post(self, request, *args, **kwargs):
form = self.form_class(request.POST, request.FILES)
if form.is_valid():
file_name = request.FILES['query_file'].name
print(f'file...{file_name}')
q_pk = random_id()
file_in = self.request.FILES.get('query_file')
f_md5 = calculate_md5(file_in)
form.instance.query_pk = q_pk
form.instance.query_file_md5 = f_md5
form.save()
return HttpResponse(self.success_message)
The problem is every time when I submit form I've got
Exception Type: TypeError
Exception Value: database must be an instance of Database
I've tried added this to post method:
instance = form.save(commit=False)
instance.save(using='mongo_instance')
but the error is the same.
Any ideas how to resolve this issue?
NOTE:
This issue is related only with modelform or when I use custom list of fields in view. When I'm using CreateView without ModelForm but with fields = 'all' and additionally with the logic passed to form_valid method of the view instead of post everything works fine. Then files are added to my mongo db.
After some tests I've figured out the problem is in gridfs. After making small change in FileField I got expected result.
I had to delete upload_to and storage from this field in models.py. This saves data from my form into mongo but without the file itself. To do that I had to add in post method of my CreateView connection to mongo via MongoClient and save file in such separate way.
Completely don't know what's the issue with gridfs storage pointed directly in the model's FileField.

Django 4.1: Can't customize PasswordChangeForm for Generic Views

I'm trying to create a view for resetting the password, But I'm trying to use the django.contrib.auth.forms.PasswordChangeForm to use the form on the view, But there's a problem!
In django.contrib.auth.forms.SetPasswordForm form's __init__ function a user is needed for the form to work, And I don't know how to implement or remove that.
Basically what I understood about this form was that it's a global url, But my urls follow this path: 'u/<slug:slug>/account/password/
I would like to know how to rewrite the __init__ function in a way that it doesn't mess up the parent(forms.Form)'s `init function or either provide the user to the form somehow.
Right now if I try to run the code I get the error:
SetPasswordForm.init() missing 1 required positional argument: 'user'
forms.py:
class UserPasswordChangeForm(PasswordChangeForm):
class Meta:
model = User #Not sure if this even does anything with PasswordChangeForm
fields = ('old_password', 'new_password1', 'new_password2')
views.py:
class UserPasswordChangeView(UpdateView):
model = User
form_class = UserPasswordChangeForm
template_name = 'auth/password.html'
urls.py:
path('u/<slug:slug>/account/password/', UserPasswordChangeView.as_view(), name='password'),
user model used(if needed):
class User(AbstractUser):
slug = models.SlugField(blank=False, null=False)
def get_absolute_url(self):
return reverse('profile', kwargs={'slug': self.slug})

Upload temporary file in django rest framework

I'm trying to create a way to upload xlsx files and then use celery to perform some actions.
I'm thinking this:
A view to upload the file and save it temporarily
Use celery to execute what I want in the file and then delete it.
I'm trying to do something like this:
class ImportMyFileView(APIView):
parser_classes = (FileUploadParser, )
def post(self, request, filename, format=None):
my_file = request.data["file"]
with open(f"/tmp/{my_file.name}", "wb+") as destination:
for chunk in my_file.chunks():
destination.write(chunk)
# call_celery_here()
...
Return something
I can generate the file where I want, but the problem is that when I open xlsx. I get this here:
--X-INSOMNIA-BOUNDARY
Content-Disposition: form-data
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
PK<q^Q_rels/.rels���J1��}��{w�Dd���ЛH}���a7�0u}{�Z���I~��7C��f�G�Fo�Z+���{�����kW�#�VJ$cʪ��l� �n�0�\Q�X^:�`���d�d{�m]_�d����h��V����F�w�^F9��W��-�(F/3�O�DSU�N�l/w�{N(�[��q��T����u<��r�?焮�s9�F����M��h���'h?PKf����
Is there any detail missing?
Here is how I would do it, relying on DRF's built in functionality:
import os
from rest_framework import serializers
from django.core.files.storage import FileSystemStorage
class UploadSerializer(serializers.Serializer):
file = serializers.FileField()
class UploadView(APIView):
...
def post(self, request):
ser = UploadSerializer(request.data)
ser.is_valid(raise_exception=True)
fs = FileSystemStorage(tempfile.gettempdir())
file_name = fs.save(content=ser.validated_data['file'])
full_path = os.path.join(fs.location, file_name)
celery_func.delay(file_name=full_path)
return Response("Upload OK")
A more robust way to do this would be to create a model representing your uploads to be processed, and use the django model's FileField.
class Todo(models.Model):
xlsx_file = models.FileField(...) # read the docs on this
created_at = models.DateTimeField(auto_now_add=True)
is_complete = models.BooleanField(default=False)
class UploadView(APIView):
def post(self, request):
...
todo = Todo.objects.create(
xslx_file = ser.validated_data['file']
)
celery_func.delay(todo_id=todo.pk)
return Response("Upload OK")
Once this works you can look into using a ModelSerializer either alone, or paired with a ModelViewSet. Thats a bigger learning curve though.

Django dynamic FileField upload_to

I'm trying to make dynamic upload path to FileField model. So when user uploads a file, Django stores it to my computer /media/(username)/(path_to_a_file)/(filename).
E.g. /media/Michael/Homeworks/Math/Week_1/questions.pdf or /media/Ernie/Fishing/Atlantic_ocean/Good_fishing_spots.txt
VIEWS
#login_required
def add_file(request, **kwargs):
if request.method == 'POST':
form = AddFile(request.POST, request.FILES)
if form.is_valid():
post = form.save(commit=False)
post.author = request.user
post.parent = Directory.objects.get(directory_path=str(kwargs['directory_path']))
post.file_path = str(kwargs['directory_path'])
post.file_content = request.FILES['file_content'] <-- need to pass dynamic file_path here
post.save()
return redirect('/home/' + str(post.author))
MODELS
class File(models.Model):
parent = models.ForeignKey(Directory, on_delete=models.CASCADE)
author = models.ForeignKey(User, on_delete=models.CASCADE)
file_name = models.CharField(max_length=100)
file_path = models.CharField(max_length=900)
file_content = models.FileField(upload_to='e.g. /username/PATH/PATH/..../')
FORMS
class AddFile(forms.ModelForm):
class Meta:
model = File
fields = ['file_name', 'file_content']
What I have found was this, but after trial and error I have not found the way to do it. So the "upload/..." would be post.file_path, which is dynamic.
def get_upload_to(instance, filename):
return 'upload/%d/%s' % (instance.profile, filename)
class Upload(models.Model):
file = models.FileField(upload_to=get_upload_to)
profile = models.ForeignKey(Profile, blank=True, null=True)
You can use some thing like this(i used it in my project):
import os
def get_upload_path(instance, filename):
return os.path.join(
"user_%d" % instance.owner.id, "car_%s" % instance.slug, filename)
Now:
photo = models.ImageField(upload_to=get_upload_path)
Since the file_path is an attribute on the File model, can you not build the full path something like this:
import os
def create_path(instance, filename):
return os.path.join(
instance.author.username,
instance.file_path,
filename
)
And then reference it from your File model:
class File(models.Model):
...
file_content = models.FileField(upload_to=create_path)
Link to docs
The other answers work flawlessly; however, I want to point out the line in the source code that allows such functionality. You can view the function, generate_filename, here, in Django's source code.
The lines that make the magic happen:
if callable(self.upload_to):
filename = self.upload_to(instance, filename)
When you pass a callable to the upload_to parameter, Django will call the callable to generate the path. Note that Django expects your callable to handle two arguments:
instance
the model that contains the FileField/ImageField
filename
the name of the uploaded file, including the extension (.png, .pdf, ...)
Also note that Python does not force your callable's arguments to be exactly 'instance' and 'filename' because Django passes them as positional parameters. For example, I prefer to rename them:
def get_file_path(obj, fname):
return os.path.join(
'products',
obj.slug,
fname,
)
And then use it like so:
image = models.ImageField(upload_to=get_file_path)

Django Admin: Show duplicate file in Storage

I have a models.FileField on my Admin-Page in a Model and would like to show an error to the user when he tries to upload an already existing file.
I already tried overriding get_available_name() on FileSystemStorage, but if I throw a ValidationError, it doesn't get displayed nicely.
Is there any way to do this (easily)?
Provide a custom form to your ModelAdmin:
class FileModel(models.Model):
name = models.CharField(max_length=100)
filefield = models.FileField()
class CustomAdminForm(forms.ModelForm):
# Custom validation: clean_<fieldname>()
def clean_filefield(self):
file = self.cleaned_data.get('filefield', None):
if file:
# Prepare the path where the file will be uploaded. Depends on your project.
# In example:
file_path = os.path.join( upload_directory, file.name )
# Check if the file exists
if os.path.isfile(file_path):
raise ValidationError("File already exists")
return super(CustomAdminForm, self).clean_filefield()
# Set the ModelAdmin to use the custom form
class FileModelAdmin(admin.ModelAdmin):
form = CustomAdminForm