I'm having a Django stack process issue and am wondering if there's a way to request user input.
To start with, the user is loading Sample data (Oxygen, CHL, Nutrients, etc.) which typically comes from an excel file. The user clicks on a button indicating what type of sample is being loaded on the webpage and gets a dialog to choose the file to load. The webpage passes the file to python via VueJS/Django where python passes the file down to the appropriate parser to read that specific sample type. The file is processed and sample data is written to a database.
Issues (I.E sample ID is outside an expected range of IDs because it was keyed in wrong when the sample was taken) get passed back to the webpage as well as written to the database to tell the user when something happened:
(E.g "No Bottle exists for sample with ID 495619, expected range is 495169 - 495176" or "356 is an unreasonable salinity, check data for sample 495169"). Maybe there's fifty samples without bottle IDs because the required bottle file wasn't loaded before the sample data. Generally you have one big 10L bottle with water in it, the ocean depth (pressure) and bottle ID where the bottle was closed is in a bottle file, and samples are placed into different vials with that bottle's unique id and the vials are run thought different machines and tests to produce the sample files.
My issue occurs when the user picks a file that contains data that has already been loaded. If the filename matches the existing file data was loaded from I clear data associated with that file and reload the database with the new data, sometimes data is spread over several files that were already loaded and uploader will merge all the files, including some that weren't uploaded, together.
A protocol for uploading data is for the uploader to append/prepend their initials onto a copy of a file if corrections were made and not to modify the original file; a chain of custody. Sometimes a file can be modified by multiple people and each person will create a copy and append/prepend their initials so people will know who all touched the data. (I don't make the rules I just work with what I have)
So we get all the way back to the parser and it's discovered the data already exists (for a given sample ID), but the filename is different. At this point I want to ask the user, do you want to reload all the data loaded from the other file, update existing data with the new file or ignore existing data and only append new data.
Is there a way for Django to make a request to the webpage to ask the user how it should handle this data without having to terminate the current request? - which the webpage is waiting for a response from the server to say the data was loaded and what errors with the data might have been found -
My current thoughts are to:
Ask the user before every file upload how a collision should be handled, if it happens
Or
Abort the data load, pass an error with a code back to the webpage, the error code indicates to the webpage that the user has to decide what to do. Upon the user answering, the load process is restarted with the same file, but with a flag to tell the parser what to do when the issue is eventually encountered again.
Nothing is written to the database until a whole file is read so no problem aborting the process and restarting if the parser doesn't know what to do, but I feel like there might be a better way.
Related
I'm building an app where the user uploads 1-4 files to shiny through fileinput. An issue arrises where, should the user not drag/select multiple files in one go, then the app will not be able to access them. For example, say the user has 4 files saved locally in 4 different folders and they try uploading them one by one, the app will not function. This happens because when the files are uploaded, fileinput creates a dataframe where one column (datapath) contains the path to a temp file which you can then reference in the server. In the documentation it states...
datapath
The path to a temp file that contains the data that was uploaded. This file may be deleted if the user performs another upload operation.
https://shiny.rstudio.com/reference/shiny/1.6.0/fileInput.html
Is there any way around this problem to prevent this datapath being deleted or perhaps find a way to store the temp file so it won't be lost should a user upload another file?
I had considered multiple fileinput boxes but that just makes the app messy.
There is a reproducible example in the example section of the documentation above.
I have prepared some reports based on the files I prepared. I am wondering is it possible to save this report (measures and visualizations) and also the steps I made while transforming data? I want to be able to load new files (which in the structure are the same as the ones I used creating my report) and the data transformation and report done automatically on this updated data.
Is it possible?
You can save it as a template - file extension pbit. It saves only the structure of the file, without actual data. When opening the report it refreshes it, and if there are parameters in the report (e.g. folder/file path or server address) it will refresh it considering the input values
you can read more here
https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-templates
You can simply save your file as a template.
I am trying to use Cloudinary as a CDN and am having some trouble getting file uploads to work properly. I have followed their blog posts and website, but am running into a consistent and very annoying error.
I have a model associated with both an image (a cover photo) and a media object (a PDF or ebook, like a .mobi or .epub). I have a model form set up to create an object:
class NewMediaObjectForm(forms.ModelForm):
class Meta:
model = MediaObject
fields = ('cover_photo', ...)
cover_photo = CloudinaryJsFileField(options={'tags': 'cover_photo'})
Now, I've read this tutorial from Cloudinary and I know that their form looks like this:
class NewMediaObjectForm(forms.ModelForm):
class Meta:
model = MediaObject
fields = ('cover_photo', ...)
cover_photo = CloudinaryJsFileField(options={'tags': 'cover_photo'})
Here's my problem: if I match their level of indention, the generated HTML shows that the input field is getting all of the right Cloudinary stuff attached to it - but the upload itself doesn't work. The page simply refreshes with an error message stating that no image was selected. Importantly, I can see from the generated HTML that the tags I've specified are coming through.
If I use method #1, with my indention, the file uploads to Cloudinary but none of my tags are applied. It also treats everything as an image, giving me a "invalid image file" error when trying to upload anything other than images (such as the ebook files I mentioned earlier).
I want both - how can I get this upload field to work AND get it to respect the options I'm trying to define?
(I do have cloudinary.config called in the appropriate views; I do have cloudinary_includes and cloudinary_js_config in the appropriate templates; I've imported everything and am calling cl_init_js_callbacks on the form in the view.)
One more addition - I am running this on a local machine using manage.py's runserver rather than deploying, if that has any impact on the configuration.
UPDATE:
I found something I was doing wrong. I've fixed it and made things worse.
The necessary jQuery was not being loaded. Now I am loading it, but the upload button simply doesn't function. I press the button, I select the file, the select dialog disappears, and it shows that no file has been selected. However, I can see that the tags are being passed to the generated HTML, so it's a step in the right direction. Any ideas?
Please forgive me. This was a PEBCAK issue; I will leave this visible for anyone else who may make the same mistakes I've made in the future.
Make sure the proper jQuery scripts are being loaded - open the Chrome developer console, Firebug, whatever and double check. Then, make sure they're being loaded after the DOM so that there are elements for the script to attach to.
Cloudinary will expect html/cloudinary_cors.htmlto be accessible in your static directory.
Either I have broken something, or the default behavior for this particular type of ModelField is to simply IMMEDIATELY upload the selected file and continue displaying "No File Chosen." I thought that nothing was happening and was very surprised when I saw 50+ images successfully uploaded.
I forgot to printto my log and it filled up. I tried to choose the 'save' option from the popup window. I entered a name+path for the log and it didn't accept it. I tried entering just a name and it accepted it, but I have no idea where it saved to.
I've searched around and found many answers on how to avoid this situation, but no answers on where the save option puts your log.
It is saved as a catalog entry in SASUSER.PROFILE with whatever name you gave it. FILE is the way to get it saved as a text file rather than a catalog entry.
Note, this only works if you have write access to SASUSER; if you regularly run SAS with rsasuser or if you share user profiles for whatever reason (or if you're running a second+ copy of the same SAS session), the save option will not work at all (and should give you an error).
You can also specify -altlog="<path to log>" on the SAS exe invocation/shortcut. This writes the entire session log to file as you go along.
I have the following class:
class VideoFile(models.Model):
media_file = models.FileField(upload_to=update_filename, null=True)
And when I try to upload large files to it (from 100mb up to 2Gb) using the following request, it can take quite a long time after the upload process, and during the VideoFile.save() process.
def upload(request):
video_file = VideoFile.objects.create(uploader=request.user.profile)
video_file.media_file = uploaded_file
video_file.save()
On my Macbook Pro Core i7, 8Gb RAM, a 300mb uploaded file can take around 20 seconds or so to run video_file.save()
I suspect this delay is relating to a disk copy operation from /tmp to the files permanent location? I've proven this by running watch ls -l on the target directory, and as soon as video_file.save() runs, I can see the file appear and grow throughout the delay.
Is there a way to eliminate this file transfer delay? Either by uploading the file directly to the target filename or just by moving the original file instead of copying? This is not the only upload operation across the site however so any solution needs to be localized to this model.
Thanks for any advice!
UPDATE:
Just further evidence to support a copy instead of a move, i can watch lsof during the upload and see a file within /private/var/folders/... written from python which maps exactly to upload progress. After upload is complete, another lsof entry appears for the ultimate file location which grows over time. After that's complete, both entries disappear.
Ok after a bit of digging I've come up with a solution. It turns out Django's default storage already attempts to move the file instead of copy, which it first tests:
hasattr(content, 'temporary_file_path')
This attribute exists for the class TemporaryUploadedFile which is the object returned to the Upload View, however the field itself is created as the class specified by FileField.attr_class
So instead I decided to subclass FieldFile and FileField and slot in the temporary_file_path attribute:
class VideoFieldFile(FieldFile):
_temporary_file_path = None
def temporary_file_path(self):
return self._temporary_file_path
class VideoFileField(FileField):
attr_class = VideoFieldFile
Finally in the view, before saving the model I manually assigned the temp path:
video_file.media_file._temporary_file_path = uploaded_file.temporary_file_path()
This now means my 1.1Gb test file becomes available in about 2-3 seconds rather than the 50 seconds or so I was seeing before. It also comes with the added benefit that if the files exist on different file systems, it appears to fall back to the copy operation.
As a side note however, my site is not utilizing MemoryFileUploadHandler which some sites may use to handle smaller file uploads, so I'm not sure how nice my solution might work with that, but I'm sure it would be simple enough to detect the uploaded file's class and act accordingly.
I would caution that there are quite a few reasons why uploading to /tmp and then cping over is best practice, and that uploading large files directly to their target is a potentially dangerous operation.
But, what you're asking is absolutely possible. Django defines upload handlers:
You can write custom handlers that customize how Django handles files. You could, for example, use custom handlers to enforce user-level quotas, compress data on the fly, render progress bars, and even send data to another storage location directly without storing it locally.