Flask custom error handling for upload function - flask

I have developed an upload form to take specific .xlsx file as upload. The requirement is to handle any exceptions for upload of non xlsx (for e.g. zip, exe file). I am using pyexcel library for reading the upload. I tried creating following code to handle this exception:
enter image description here
enter image description here
The error handling code is as follows:
class FILE_TYPE_NOT_SUPPORTED_FMT(Exception):
pass
#app.errorhandler(FILE_TYPE_NOT_SUPPORTED_FMT)
def custom_handler(errrors):
app.logger.error('Unhandled Exception: %s', (errrors))
return render_template('400.html'), 400
and the upload code is as follows:
#users.route("/oisdate_upload", methods=['GET', 'POST'])
#login_required
def doimport_ois_date():
msg=None
if request.method == 'POST':
def OIS_date_init_func(row):
#c.id = row['id']
c = Ois_date(row['date'],row['on'],row['m1'],row['m2'],row['m3'],row['m6'],row['m9'],row['y1'],row['y2'],row['y3'],row['y4'],row['y5'],row['y7'],row['y10'])
return c
request.save_book_to_database(
field_name='file', session=db.session,
tables=[Ois_date],
initializers=[OIS_date_init_func])
msg = "Successfully uploaded"
#return redirect(url_for('users.doimport_ois_date'), code=302)
if((Ois_date.query.order_by(Ois_date.date.desc()).first()) is not None):
date_query = Ois_date.query.order_by(Ois_date.date.desc()).first()
start_date = date_query.date
date_query1 = Ois_date.query.order_by(Ois_date.date.asc()).first()
end_date = date_query1.date
return render_template('OISdate_upload.html',msg=msg, start_date=start_date,end_date=end_date)
I am unable to figure out how to correctly capture the error and handle it, any feedback would be appreciated.

You have two options to handle this exception.
1) you import the exception directly from the pyexcel package and use it as the error:
e.g.
from pyexcel.exceptions import FileTypeNotSupported
...
#app.errorhandler(FileTypeNotSupported)
...
2) Or, you can wrap the code where you want to load the spreadsheet in a try-except block and throw a custom error.
from pyexcel.exceptions import FileTypeNotSupported
class CustomError(Exception)
pass
#app.errorhandler(CustomError)
# do something
pass
#app.route('/upload_excel')
def upload_excel():
try:
function_where_you_load_excel()
except FileTypeNotSupported:
raise CustomError

Related

Dajngo CSV FIle not download ? When we have a large CSV file download its takes some time?Django 502 bad gateway nginx error Django

How can I download a large CSV file that shows me a 502 bad gateway error?
I get this solution I added in below.
Actually, in this, we use streaming references. In this concept for example we download a movie it's will download in the browser and show status when complete this will give the option to show in a folder same as that CSV file download completely this will show us.
There is one solution for resolving this error to increase nginx time but this is will affect cost so better way to use Django streaming. streaming is like an example when we add a movie for download it's downloading on the browser. This concept is used in Django streaming.
Write View for this in Django.
views.py
from django.http import StreamingHttpResponse
503_ERROR = 'something went wrong.'
DASHBOARD_URL = 'path'
def get_headers():
return ['field1', 'field2', 'field3']
def get_data(item):
return {
'field1': item.field1,
'field2': item.field2,
'field3': item.field3,
}
class CSVBuffer(object):
def write(self, value):
return value
class Streaming_CSV(generic.View):
model = Model_name
def get(self, request, *args, **kwargs):
try:
queryset = self.model.objects.filter(is_draft=False)
response = StreamingHttpResponse(streaming_content=(iter_items(queryset, CSVBuffer())), content_type='text/csv', )
file_name = 'Experience_data_%s' % (str(datetime.datetime.now()))
response['Content-Disposition'] = 'attachment;filename=%s.csv' % (file_name)
except Exception as e:
print(e)
messages.error(request, ERROR_503)
return redirect(DASHBOARD_URL)
return response
urls.py
path('streaming-csv/',views.Streaming_CSV.as_view(),name = 'streaming-csv')
For reference use the below links.
https://docs.djangoproject.com/en/4.0/howto/outputting-csv/#streaming-large-csv-files
GIT.
https://gist.github.com/niuware/ba19bbc0169039e89326e1599dba3a87
GIT
Adding rows manually to StreamingHttpResponse (Django)

google-ml-engine custom prediction routine error responses

I have a custom prediction routine in google-ml-engine. Works very well.
I now am doing input checking on the instance data, and want to return error responses from my predict routine.
The example: https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines
Raises exceptions on input errors, etc. However, when this happens the response body always has {'error': Prediction failed: unknown error}. I can see the correct errors are being logged in google cloud console, but the https response is always the same unknown error.
My question is:
How to make the Custom prediction routine return a proper error code and error message string?
Instead of returning a prediction, I can return an error string/code in prediction -but it ends up in the prediction part of the response which seems hacky and doesn't get any of the google errors eg based on instance size.
root:test_deployment.py:35 {'predictions': {'error': "('Instance does not include required sensors', 'occurred at index 0')"}}
What's the best way to do this?
Thanks!
David
Please take a look at the following code, I created a _validate function inside predict and use a custom Exception class.
Basically, I validate instances, before I call the model predict method and handle the exception.
There may be some overhead to the response time when doing this validation, which you need to test for your use case.
requests = [
"god this episode sucks",
"meh, I kinda like it",
"what were the writer thinking, omg!",
"omg! what a twist, who would'v though :o!",
99999
]
api = discovery.build('ml', 'v1')
parent = 'projects/{}/models/{}/versions/{}'.format(PROJECT, MODEL_NAME, VERSION_NAME)
parent = 'projects/{}/models/{}'.format(PROJECT, MODEL_NAME)
response = api.projects().predict(body=request_data, name=parent).execute()
{'predictions': [{'Error code': 1, 'Message': 'Invalid instance type'}]}
Custom Prediction class:
import os
import pickle
import numpy as np
import logging
from datetime import date
import tensorflow.keras as keras
class CustomModelPredictionError(Exception):
def __init__(self, code, message='Error found'):
self.code = code
self.message = message # you could add more args
def __str__(self):
return str(self.message)
def isstr(s):
return isinstance(s, str) or isinstance(s, bytes)
def _validate(instances):
for instance in instances:
if not isstr(instance):
raise CustomModelPredictionError(1, 'Invalid instance type')
return instances
class CustomModelPrediction(object):
def __init__(self, model, processor):
self._model = model
self._processor = processor
def _postprocess(self, predictions):
labels = ['negative', 'positive']
return [
{
"label":labels[int(np.round(prediction))],
"score":float(np.round(prediction, 4))
} for prediction in predictions]
def predict(self, instances, **kwargs):
try:
instances = _validate(instances)
except CustomModelPredictionError as c:
return [{"Error code": c.code, "Message": c.message}]
else:
preprocessed_data = self._processor.transform(instances)
predictions = self._model.predict(preprocessed_data)
labels = self._postprocess(predictions)
return labels
#classmethod
def from_path(cls, model_dir):
model = keras.models.load_model(
os.path.join(model_dir,'keras_saved_model.h5'))
with open(os.path.join(model_dir, 'processor_state.pkl'), 'rb') as f:
processor = pickle.load(f)
return cls(model, processor)
Complete code in this notebook.
If it is still relevant to you, I found a way by using google internal libraries (not sure if it would be endorsed by Google though).
AI platform custom prediction wrapping code only returns custom error message if the Exception thrown is a specific one from their internal library.
It might also not be super reliable as you would have very little control in case Google wants to change it.
class Predictor(object):
def predict(self, instances, **kwargs):
# Your prediction code here
# This is an internal google library, it should be available at prediction time.
from google.cloud.ml.prediction import prediction_utils
raise prediction_utils.PredictionError(0, "Custom error message goes here")
#classmethod
def from_path(cls, model_dir):
# Your logic to load the model here
You would get the following message in your HTTP response
Prediction failed: Custom error message goes here

How to zip multiple uploaded file in Django before saving it to database?

I am trying to compress a folder before saving it to database/file storage system using Django. For this task I am using ZipFile library. Here is the code of view.py:
class BasicUploadView(View):
def get(self, request):
file_list = file_information.objects.all()
return render(self.request, 'fileupload_app/basic_upload/index.html',{'files':file_list})
def post(self, request):
zipfile = ZipFile('test.zip','w')
if request.method == "POST":
for upload_file in request.FILES.getlist('file'): ## index.html name
zipfile.write(io.BytesIO(upload_file))
fs = FileSystemStorage()
content = fs.save(upload_file.name,upload_file)
data = {'name':fs.get_available_name(content), 'url':fs.url(content)}
zipfile.close()
return JsonResponse(data)
But I am getting the following error:
TypeError: a bytes-like object is required, not 'InMemoryUploadedFile'
Is there any solution for this problem? Since I may have to upload folder with large files, do I have to write a custom TemporaryFileUploadHandler for this purpose? I have recently started working with Django and it is quite new to me. Please help me with some advice.
InMemoryUploadedFile is an object that contains more than just file you should open file and read it content ( InMemoryUploadedFile.file is the file)
InMemoryUploadedFile.open()
You should open file with open() and then read() it's content, also you should check if you have uploaded files correctly also you could use with syntax for both zip and file
https://www.pythonforbeginners.com/files/with-statement-in-python

Python user defined exception handling

i need a little bit of help understanding a problem that i have with user defined exceptions in python 2.7.11.
I have two files main.py and myErrors.py .in main i post data and receive a response and and in myErrors i handle the errors.
What i'm trying to do is execute the version error in the try:except statement, but it doesn't get executed even thought it should be. what i'm doing is that i pass the response to myErrors and update that data to a dictionary in the errors file.-
my question was badly phrased. What I want to do is, is pass the response to the error handler, but i don't want to execute it, until we get to the Try:except clause in on_response method. So when we get the response and if it's not successful, then check the error code and raise the exception. Now what i'm doing is checking first for errors and then executing the check for success (error code)
Here is the main
def send_messages(self):
response = cm.postData(url=simulateSasServer, jsondata=json_data)
self.on_response(response)
def on_response(self, response):
myERRORS.myERRORS(response)
# if registration is succesful change state to REGISTERED.
if 'registrationResponse' in response:
try:
responseObjects = response['registrationResponse']
for responseObject in responseObjects:
if responseObject['error']['errorCode'] == 0:
do_action
except myErrors.Version():
raise ("version_message")
Here is the myErrors
class myERRORS(Exception):
error_code = {'SUCCESS': 0,
'VERSION': 100,
}
response_data = {}
def __init__(self, response):
self.response_data.update(response)
class Version(myERRORS):
def __init__(self):
self.name = "VERSION"
self.err_code = self.error_code['VERSION']
self.msg = "SAS protocol version used by CBSD is not supported by SAS"
self.version_error()
if self.version_error() is True:
print (self.name, self.err_code, self.msg)
raise Exception(self.name, self.err_code, self.msg)
def version_error(self):
response_objects = self.response_data.values()[0]
if 'registrationResponse' in self.response_data:
for r_object in response_objects:
if r_object['error']['errorCode'] == self.error_code['VERSION']:
return True
Any help is much appreciated.
There isn't really anything special about exceptions. They are classes. What you did is create an instance of a class. You did not raise it. Change:
myERRORS.myERRORS(response)
to:
raise myERRORS.myERRORS(response)

Flask Sqlalchemy model working in python shell but not in web interface

I am writing a simple web application in flask(python) which will ask students to fill experimental data in a form and that data will be submitted on a server.
My user class looks like:
class User(db.Model):
id=db.Column(db.Integer,primary_key=True)
email=db.Column(db.String,unique=True)
name=db.Column(db.String)
password=db.Column(db.String)
data = db.relationship('ExpData', backref='student', lazy='dynamic')
and ExpData class, which will store data looks like:
class ExpData(db.Model):
id=db.Column(db.Integer,primary_key=True)
exp_1_data=db.Column(db.String)
submission_date=db.Column(db.DateTime)
user_id=db.Column(db.Integer, db.ForeignKey('user.id'))
Now User model works perfectly, whide ExpData models does not put anything in database. I used the following lines to achieve it:
u=models.User(email=usern,name=nameu,password=userp,rollno=rollno)
db.session.add(u)
db.session.commit()
Then in required function:
user=g.user
dat=models.ExpData(exp_1_data='dataStr',submit=1,submission_date=datetime.now(),student=user)
db.session.add(dat)
db.session.commit()
in Python shell it works perfectly. But for some reason through web interface its not working.
Other details:
Users are logged in via a login_user() function provided by Flask-Login
Then user is assigned to g.user through another decorator
#app.before_request
def before_request():
g.user = current_user
Then Database entry is done whenever following function is executed upon visiting the url
#app.route('/exp1echo', methods=['GET','POST'])
#login_required
def exp1echo():
exp1_data={}
exp1_data["ans"]=float(request.args.get("ans")) #getting some data through ajax 'get' call
user=g.user
filename = user.name+" simplePrint.pdf" # save file with user name in file name
pdf=render_template('exp1_post.html',exp1_data=exp1_data)
pisa.CreatePDF(pdf.encode("ISO-8859-1"), file(filename, "wb"))
pdf.close() # Just fyi this call does return an error
#"No handlers could be found for logger "xhtml2pdf"" but all works fine
abc={'a':1,'b':2,'c':3}
dataStr=str(abc)
dat=ExpData(exp_1_data='dataStr',submit=1,submission_date=datetime.now(),student=g.user)
db.session.add(dat)
db.session.commit()
return jsonify({'base_vl':1}) # return some value to ajax call
User loader is defined as:
lm = LoginManager()
lm.init_app(app)
#lm.user_loader
def load_user(id):
return models.User.query.get(int(id))
Users are registered through following URL:
#app.route('/register', methods=['GET','POST'])
def register():
if request.method == 'POST':
user_pass=request.form["password"]
user_name=request.form["username"]
name_=request.form["name"]
u=models.User(email=user_name,name=name_,password=user_pass)
db.session.add(u)
db.session.commit()
return redirect(url_for('login'))
return render_template('register_form.html')
Well this is embarrassing! Found out the error. For some reason pdf.close() function in exp1echo() gives an error (as I mentioned earlier). That error does not show any message during debugging nor it exits the programme. Hence I ignored it. But it terminates the execution of that function. So it was not sending any call to data base. So I commented it out and voila! It worked.