I have some server side processing of some data (client-side library = jQuery DataTables)
I am using POST as my ajax method. In my Flask webapp, I can access the POST data with request.values
The data type / structure of request.values is werkzeug.datastructures.CombinedMultiDict
If the user wants to sort a column, the request contains a key called action with a value of filter (note the below printouts are obtained with for v in request.values: print v, request.values[v])
...
columns[7][data] role
columns[8][search][regex] false
action filter
columns[10][name]
columns[3][search][value]
...
all the column names are also contained in the request as keys. The columns that have search terms will have the search string as a value for the column name key (as opposed to empty for columns with no search term entered. So, If I want to search for firstname containing bill, I would see the following in my request
columns[7][searchable] true
...
columns[6][name]
firstname bill
columns[0][search][value]
columns[2][searchable] true
...
columns[5][data] phone
role
columns[10][data] registered_on
...
columns[0][searchable] true
email
columns[7][orderable] true
...
columns[2][search][value]
Notice how role and email are empty. So my code below is very non-DRY
rv = request.values
if rv.get('action') == 'filter':
if len(rv.get('firstname')):
q = q.filter(User.firstname.ilike('%{0}%'.format(rv.get('firstname'))))
if len(rv.get('lastname')):
q = q.filter(User.lastname.ilike('%{0}%'.format(rv.get('lastname'))))
if len(rv.get('username')):
q = q.filter(User.username.ilike('%{0}%'.format(rv.get('username'))))
if len(rv.get('email')):
q = q.filter(User.email.ilike('%{0}%'.format(rv.get('email'))))
if len(rv.get('phone')):
q = q.filter(User.phone.ilike('%{0}%'.format(rv.get('phone'))))
if len(rv.get('region')):
q = q.filter(User.region.name.ilike('%{0}%'.format(rv.get('region'))))
if len(rv.get('role')):
q = q.filter(User.role.name.ilike('%{0}%'.format(rv.get('role'))))
if len(rv.get('is_active')):
q = q.filter(User.is_active_ == '{0}'.format(rv.get('is_active')))
if len(rv.get('is_confirmed')):
q = q.filter(User.is_confirmed == '{0}'.format(rv.get('is_confirmed')))
if len(rv.get('registered_on_from')):
fdate = datetime.strptime(rv.get('registered_on_from'), '%Y-%m-%d')
q = q.filter(User.registered_on > fdate)
if len(rv.get('registered_on_to')):
tdate = datetime.strptime(rv.get('registered_on_to'), '%Y-%m-%d')
q = q.filter(User.registered_on < tdate)
I was building the sorting functionality, and I found the following statement that greatly simplified my life (see this answer)
q = q.order_by('{name} {dir}'.format(name=sort_col_name, dir=sort_dir))
I was wondering if there was a way to simplify this set of filtering queries like the above sorting code since I will have to do this for many other models.
This should help:
from sqlalchemy import inspect
from sqlalchemy.sql.sqltypes import String,Boolean
def filter_model_by_request(qry,model,rv):
if rv.get('action') == 'filter':
mapper = inspect(model).attrs # model mapper
col_names = list(set([c.key for c in mapper]) & set(rv.keys()))
# col_names is a list generated by intersecting the request values and model column names
for col_name in col_names:
col = mapper[col_name].columns[0]
col_type = type(col.type)
if col_type == String: # filter for String
qry = qry.filter(col.ilike('%{0}%'.format(rv.get(col_name))))
elif col_type == Boolean: # filter for Boolean
qry = qry.filter(col == '{0}'.format(rv.get(col_name)))
return qry
Example call (I used it with a #app.before_request and a cURL call to verify):
qry = db.session.query(User)
print filter_model_by_request(qry,User,request.values).count()
The date range filtering is not included in the function, add this feature if you wish, your code is fine for that purpose.
side note: be careful with the bigger/smaller operators for the dates. You're excluding the actual requested dates. Use <= or >= to include dates in filtering action. It's always a pitfall for me..
Related
Im having this code to create and add students to database.
I need to make validation of count which must be integer, only positive, equal or less 100.
Please help.
def generate_students(request):
count = request.GET.get('count')
studentslist = []
for student in range(0, int(count)):
student = Student.objects.create(first_name = fake.first_name(), last_name = fake.last_name(), age = random.randint(18,100))
studentslist.append(student)
output = ', '.join(
[f"id = {student.id} {student.first_name} {student.last_name}, age = {student.age};" for student in studentslist]
)
return HttpResponse(str(output))
The best way is likely to work with a form, since a form has a lot of validation inplace, can clean the object, and print sensical errors.
We thus can work with a simple form:
from django import forms
class CountForm(forms.Form):
count = forms.IntegerField(min_value=1, max_value=100)
then we can validate the input with:
def generate_students(request):
form = CountForm(request.GET)
if form.is_valid():
count = form.cleaned_data['count']
studentslist = [
Student.objects.create(first_name = fake.first_name(), last_name = fake.last_name(), age = random.randint(18,100))
for _ in range(count)
]
output = ', '.join(
[f'id = {student.id} {student.first_name} {student.last_name}, age = {student.age};'
for student in studentslist]
)
else:
return HttpResponse(str(form.errors))
return HttpResponse(str(output))
Note: Section 9 of the HTTP protocol
specifies that requests like GET and HEAD should not have side-effects, so you
should not change entities with such requests. Normally POST, PUT, PATCH, and
DELETE requests are used for this. In that case you make a small <form> that
will trigger a POST request, or you use some AJAX calls.
I did the code for insert records from so_parts table to so_bo table using Query...How can I use ORM method to do this kind of job. Is there any other way(best)to do that? Here is my code`
`
#api.multi
def save_rapair_parts(self, vals):
#get todays date and convert it to string
created_date = datetime.datetime.today().strftime("%m/%d/%Y")
str_date = str(created_date)
so_p_id = self.so_p_id.id
bo_status = self.bo_status
so_part_t = self.so_part_t
so_part_sno = self.so_part_sno
product = self.so_part_product
product_str = 'Repair '+str(product)
part_id = self.id
bench_order_table.search(['id','bo_sno','created_date','bo_number','rep_description','bo_status'])
#insert details intoso bench orders
`
if so_part_t=='r_b':
try:
sequence = self.env['ir.sequence'].next_by_code('so.benchorder') or '/'
str_sequence = str(sequence)
query = """SELECT so_work_authorization FROM my_depots_so WHERE id=%d """ % (so_p_id)
self.env.cr.execute(query)
result = self.env.cr.fetchall()
result_number = json.dumps(result, ensure_ascii=False)
strip_number = result_number.strip('\' \" [] ')
work_auth_no = str(strip_number)
work_auth_no += "-"
work_auth_no += str_sequence
insert ="""INSERT INTO my_depots_so_bo(id,so_bo_id,bo_sno,created_date,bo_number,rep_description,bo_status) values %s """
parameters = (part_id,so_p_id,so_part_sno,str_date,work_auth_no,product_str,bo_status)
self.env.cr.execute(insert,(parameters,))
my_depots_bo(id,bo_sno,created_date,bo_number,rep_description,bo_status) values %s """
# self.env.cr.execute(insert_query, (parameters,))
except Exception:
print "Error in inserting values"`
yes there is a better way because when you use ORM
method you also checks access right for user to:
for your select query:
rec = self.env['my.depots.so'].search_read(['id', '=', so_p_id], ['so_work_authorization'])
if rec:
rec = rec[0] # search_read return a list of dictionary
so_work_authorization = rec['so_work_authorization']
# and do what ever you want with the result
# to create
# call create method witch accept a dictionary
# field_name : value
new_rec = self.env['my.depots.so.bo'].create({
'so_bo_id': so_p_id, # many2one must be an integer value
'bo_sno': bo_nso_value,
'bo_number': value_of_number,
# ....
# ....
# add al field
}) # create return the new created record as model object
for inserting use: self.env['model.name'].create(vals)
for updating use : self.env['model.name'].write(vals)
using ORM method makes sure that user don't pass the security access rigths
Hope you get the idea
I am attempting to alter a previous script that utilizes biopython to fetch information about a species phylum. This script was written to retrieve information one species at a time. I would like to modify the script so that I can do this for 100 organisms at a time.
Here is the initial code
import sys
from Bio import Entrez
def get_tax_id(species):
"""to get data from ncbi taxomomy, we need to have the taxid. we can
get that by passing the species name to esearch, which will return
the tax id"""
species = species.replace(" ", "+").strip()
search = Entrez.esearch(term = species, db = "taxonomy", retmode = "xml")
record = Entrez.read(search)
return record['IdList'][0]
def get_tax_data(taxid):
"""once we have the taxid, we can fetch the record"""
search = Entrez.efetch(id = taxid, db = "taxonomy", retmode = "xml")
return Entrez.read(search)
Entrez.email = ""
if not Entrez.email:
print "you must add your email address"
sys.exit(2)
taxid = get_tax_id("Erodium carvifolium")
data = get_tax_data(taxid)
lineage = {d['Rank']:d['ScientificName'] for d in
data[0]['LineageEx'] if d['Rank'] in ['family', 'order']}
I have managed to modify the script so that it accepts a local file that contains one of the organisms I am using. But I need to extend this to a 100 organisms.
So the idea was to generate a list from the file of my organisms and somehow separately fed each item generated from the list into the line taxid = get_tax_id("Erodium carvifolium") and replace "Erodium carvifolium" with my organisms name. But I have no idea how to do that.
Here is the sample version of the code with some of my adjustments
import sys
from Bio import Entrez
def get_tax_id(species):
"""to get data from ncbi taxomomy, we need to have the taxid. we can
get that by passing the species name to esearch, which will return
the tax id"""
species = species.replace(' ', "+").strip()
search = Entrez.esearch(term = species, db = "taxonomy", retmode = "xml")
record = Entrez.read(search)
return record['IdList'][0]
def get_tax_data(taxid):
"""once we have the taxid, we can fetch the record"""
search = Entrez.efetch(id = taxid, db = "taxonomy", retmode = "xml")
return Entrez.read(search)
Entrez.email = ""
if not Entrez.email:
print "you must add your email address"
sys.exit(2)
list = ['Helicobacter pylori 26695', 'Thermotoga maritima MSB8', 'Deinococcus radiodurans R1', 'Treponema pallidum subsp. pallidum str. Nichols', 'Aquifex aeolicus VF5', 'Archaeoglobus fulgidus DSM 4304']
i = iter(list)
item = i.next()
for item in list:
???
taxid = get_tax_id(?)
data = get_tax_data(taxid)
lineage = {d['Rank']:d['ScientificName'] for d in
data[0]['LineageEx'] if d['Rank'] in ['phylum']}
print lineage, taxid
The question marks refer to places where I am stumped as what to do next. I don't see how I can connect my loop to replace the ? in get_tax_id(?). Or do I need to somehow append each of the items in the list so that they are modified each time to contain get_tax_id(Helicobacter pylori 26695) and then find some way to place them in the line containing taxid =
Here's what you need, place this below your function definitions, i.e. after the line that says: sys.exit(2)
species_list = ['Helicobacter pylori 26695', 'Thermotoga maritima MSB8', 'Deinococcus radiodurans R1', 'Treponema pallidum subsp. pallidum str. Nichols', 'Aquifex aeolicus VF5', 'Archaeoglobus fulgidus DSM 4304']
taxid_list = [] # Initiate the lists to store the data to be parsed in
data_list = []
lineage_list = []
print('parsing taxonomic data...') # message declaring the parser has begun
for species in species_list:
print ('\t'+species) # progress messages
taxid = get_tax_id(species) # Apply your functions
data = get_tax_data(taxid)
lineage = {d['Rank']:d['ScientificName'] for d in data[0]['LineageEx'] if d['Rank'] in ['phylum']}
taxid_list.append(taxid) # Append the data to lists already initiated
data_list.append(data)
lineage_list.append(lineage)
print('complete!')
I'm trying to get the next and previous objects of a comic book issue. Simply changing the id number or filtering through date added is not going to work because I don't add the issues sequentially.
This is how my views are setup and it WORKS for prev_issue and does return the previous object, but it returns the last object for next_issue and I do not know why.
def issue(request, issue_id):
issue = get_object_or_404(Issue, pk=issue_id)
title = Title.objects.filter(issue=issue)
prev_issue = Issue.objects.filter(title=title).filter(number__lt=issue.number)[0:1]
next_issue = Issue.objects.filter(title=title).filter(number__gt=issue.number)[0:1]
Add an order_by clause to ensure it orders by number.
next_issue = Issue.objects.filter(title=title, number__gt=issue.number).order_by('number').first()
I know this is a bit late, but for anyone else, django does have a nicer way to do this, see https://docs.djangoproject.com/en/1.7/ref/models/instances/#django.db.models.Model.get_previous_by_FOO
So the answer here would be something something like
next_issue = Issue.get_next_by_number(issue, title=title)
Django managers to do that with a bit of meta class cleaverness.
If it's required to find next and previous objects ordered by field values that can be equal and those fields are not of Date* type, the query gets slightly complex, because:
ordering on objects with same values limiting by [:1] will always produce same result for several objects;
object can itself be included in resulting set.
Here's are querysets that also take into account the primary keys to produce a correct result (assuming that number parameter from OP is not unique and omitting the title parameter as it's irrelevant for the example):
Previous:
prev_issue = (Issue.objects
.filter(number__lte=issue.number, id__lt=instance.id)
.exclude(id=issue.id)
.order_by('-number', '-id')
.first())
Next:
next_issue = (Issue.objects
.filter(number__gte=issue.number, id__gt=instance.id)
.exclude(id=issue.id)
.order_by('number', 'id')
.first())
from functools import partial, reduce
from django.db import models
def next_or_prev_instance(instance, qs=None, prev=False, loop=False):
if not qs:
qs = instance.__class__.objects.all()
if prev:
qs = qs.reverse()
lookup = 'lt'
else:
lookup = 'gt'
q_list = []
prev_fields = []
if qs.query.extra_order_by:
ordering = qs.query.extra_order_by
elif qs.query.order_by:
ordering = qs.query.order_by
elif qs.query.get_meta().ordering:
ordering = qs.query.get_meta().ordering
else:
ordering = []
ordering = list(ordering)
if 'pk' not in ordering and '-pk' not in ordering:
ordering.append('pk')
qs = qs.order_by(*ordering)
for field in ordering:
if field[0] == '-':
this_lookup = (lookup == 'gt' and 'lt' or 'gt')
field = field[1:]
else:
this_lookup = lookup
q_kwargs = dict([(f, get_model_attr(instance, f))
for f in prev_fields])
key = "%s__%s" % (field, this_lookup)
q_kwargs[key] = get_model_attr(instance, field)
q_list.append(models.Q(**q_kwargs))
prev_fields.append(field)
try:
return qs.filter(reduce(models.Q.__or__, q_list))[0]
except IndexError:
length = qs.count()
if loop and length > 1:
return qs[0]
return None
next_instance = partial(next_or_prev_instance, prev=False)
prev_instance = partial(next_or_prev_instance, prev=True)
note that do not use object.get(pk=object.pk + 1) these sorts of things, IntegrityError occurs if object at that pk is deleted, hence always use a query set
for visitors:
''' Useage '''
"""
# Declare our item
store = Store.objects.get(pk=pk)
# Define our models
stores = Store.objects.all()
# Ask for the next item
new_store = get_next_or_prev(stores, store, 'next')
# If there is a next item
if new_store:
# Replace our item with the next one
store = new_store
"""
''' Function '''
def get_next_or_prev(models, item, direction):
'''
Returns the next or previous item of
a query-set for 'item'.
'models' is a query-set containing all
items of which 'item' is a part of.
direction is 'next' or 'prev'
'''
getit = False
if direction == 'prev':
models = models.reverse()
for m in models:
if getit:
return m
if item == m:
getit = True
if getit:
# This would happen when the last
# item made getit True
return models[0]
return False
original author
Usage
# you MUST call order by to pass in an order, otherwise QuerySet.reverse will not work
qs = Model.objects.all().order_by('pk')
q = qs[0]
prev = get_next_or_prev(qs, q, 'prev')
next = get_next_or_prev(qs, q, 'next')
next_obj_id = int(current_obj_id) + 1
next_obj = Model.objects.filter(id=next_obj_id).first()
prev_obj_id= int(current_obj_id) - 1
prev_obj = Model.objects.filter(id=prev_obj_id).first()
#You have nothing to loose here... This works for me
I've got this complex filtering mechanism...
d = copy(request.GET)
d.setdefault('sort_by', 'created')
d.setdefault('sort_dir', 'desc')
form = FilterShipmentForm(d)
filter = {
'status': ShipmentStatuses.ACTIVE
}
exclude = {}
if not request.user.is_staff:
filter['user__is_staff'] = False
if request.user.is_authenticated():
exclude['user__blocked_by__blocked'] = request.user
if form.is_valid():
d = form.cleaned_data
if d.get('pickup_city'): filter['pickup_address__city__icontains'] = d['pickup_city']
if d.get('dropoff_city'): filter['dropoff_address__city__icontains'] = d['dropoff_city']
if d.get('pickup_province'): filter['pickup_address__province__exact'] = d['pickup_province']
if d.get('dropoff_province'): filter['dropoff_address__province__exact'] = d['dropoff_province']
if d.get('pickup_country'): filter['pickup_address__country__exact'] = d['pickup_country']
if d.get('dropoff_country'): filter['dropoff_address__country__exact'] = d['dropoff_country']
if d.get('min_price'): filter['target_price__gte'] = d['min_price']
if d.get('max_price'): filter['target_price__lte'] = d['max_price']
if d.get('min_distance'): filter['distance__gte'] = d['min_distance'] * 1000
if d.get('max_distance'): filter['distance__lte'] = d['max_distance'] * 1000
if d.get('available_on'): # <--- RELEVANT BIT HERE ---
filter['pickup_earliest__lte'] = d['available_on'] # basically I want "lte OR none"
filter['pickup_latest__gte'] = d['available_on']
if d.get('shipper'): filter['user__username__iexact'] = d['shipper']
order = ife(d['sort_dir'] == 'desc', '-') + d['sort_by']
shipments = Shipment.objects.filter(**filter).exclude(**exclude).order_by(order) \
.annotate(num_bids=Count('bids'), min_bid=Min('bids__amount'), max_bid=Max('bids__amount'))
And now my client tells me he wants pickup/drop-off dates to be 'flexible' as an option. So I've updated the DB to allow dates to be NULL for this purpose, but now the "available for pickup on" filter won't work as expected. It should include NULL/None dates. Is there an easy fix for this?
Flip the logic and use exclude(). What you really want to do is exclude any data that specifies a date that doesn't fit. If pickup_latest and pickup_earliest are NULL it shouldn't match the exclude query and wont be removed. Eg
exclude['pickup_latest__lt'] = d['available_on']
exclude['pickup_earliest__gt'] = d['available_on']
Most database engines don't like relational comparisons with NULL values. Use <field>__isnull to explicitly check if a value is NULL in the database, but you'll need to use Q objects to OR the conditions together.
Don't think that's actually a django-specific question. Variable 'd' is a python dictionary, no? If so, you can use this:
filter['pickup_latest__gte'] = d.get('available_on', None)