Concurrency and django scaling horizontal - django

Hello I would like to ask you for advice for an application, I currently use Django cookie cutter for the project, I use as a library, Django MPTT in this model it must respect a certain hierarchy and that some objects do not find themselves in the same place as another to do this I need to isolate the objects to then perform my calculation and create my new objects. First of all since it could take a while I used Cellery to make my additions to the database I tried several concurrent accesses and I never had a rollback, I thought instead of using Cellery since I know how to scale the docker into several Django, I would integrate this creation into my view and use the #atomic decorator. The problem is that when we create objects at the same time on each different server (django1,django2,django3) I have a lot of rollback while with celeriac I have no problem. I don't understand why competition is more difficult in Django than in celery? Do you have any advice for me? For a better management of different Django servers at the same time, thank you in advance example :
Work in celery norollback :
#celery_app.task()
def create_user_in_matrix(user_id):
.....
new_human = HumanFaction.objects.select_for_update().filter(id=int(user_id))
with transaction.atomic():
new_human = new_human.first()
alderol = HumanFaction.objects.get(user_id=alderol_user_id)
print('selection utilisateur', new_human)
level = 1
level_max = HumanFaction.objects.aggregate(Max('level'))['level__max']
if level_max == 0:
level_max = 1
print('level maximum', level_max)
while level <= level_max and user_whole:
users = HumanFaction.objects.select_for_update().filter(level=level)
HumanFaction.objects.create(user=new_human, parent=aledol)
.....
Operates every third time in a django view during a colision test frequent rollback (django,1,2,3)
#transaction.atomic
def human_group(request):
......
new_human = HumanFaction.objects.select_for_update().filter(id=int(user_id))
with transaction.atomic():
new_human = new_human.first()
alderol = HumanFaction.objects.get(user_id=alderol_user_id)
print('selection utilisateur', new_human)
level = 1
level_max = HumanFaction.objects.aggregate(Max('level'))['level__max']
if level_max == 0:
level_max = 1
print('level maximum', level_max)
while level <= level_max and user_whole:
users = HumanFaction.objects.select_for_update().filter(level=level)
HumanFaction.objects.create(user=new_human, parent=aledol)
.....

Related

Allowing users to select which flow to roll back to django-viewflow

Hey all i have been using viewflow as the workflow engine in my django project. I wanted to know if it was possible to allow users to 'select' which flow they wish to roll back to if lets say an approval was rejected.
Here , the director chooses 'Reject' , however it doesn't make sense to end the flow here , instead it should be a be a selectable 'roll back' , so that the the people down the line do not need to restart the entire process again.
Here's what i have done so far :
flows.py
#director will approve or dont approve
approve_by_director = flow.View(
UpdateProcessView,
form_class=DirectorApproveForm,
task_title="Approval By Director"
).Permission("cash.director"
).Next(this.check_director)
check_director = flow.If(
cond=lambda act: act.process.director,
task_title="Processing"
).Then(this.send).Else(this.justification)
#justifications for the roll back.
justification = flow.View(
JustificationView,
task_title="Justifications for Roll Back"
).Assign(lambda act: self.request.user
).Permission(auto_create=True
).Next(this.roll_back)
roll_back = flow.Handler(this.roll_back_call).Next(this.approve_by_preparer) ##<---- here , i am just sending it back to the 'preparer' , however it would be great if this could be dynamic!
end = flow.End()
def roll_back_call(self, activation):
esig = ESignatures.objects.filter(paymentVoucherProcess = activation.process).filter(voided = False)
docu = Attachment.objects.filter(paymentVoucherProcess = activation.process).filter(voided = False)
if len(esig) > 0 :
for sig in esig:
sig.voided = True
sig.save()
if len(docu) > 0 :
for doc in docu:
doc.voided = True
doc.save()
activation.process.preparer = False
activation.process.verifier = False
activation.process.treasury = False
activation.process.director = False
The problem here is that since the .next() node , i hard coded the phase in which i wish to roll back to, however this is not ideal as it would be optimal for the user to be able to 'select' which phase they were to send it back...
Therefore , I have two questions :
1. Is my method of the roll back correct? (maybe there is a better way to do the roll back instead of calling the process and hard code refreshing the fields)
2. Is there a way to select which part of the flow the user wish to roll back to?
Thanks and i would greatly appreciate anybody's advise
Actually, the good BPMN practice pattern is to finish flow as soon as possible, and do not involve complex cycles. That makes code simple, clean and easy to support. Simplifies reporting and keeps all revisions of data inside different process instances
With Viewflow flow-restart decision could be implemented as flow.View that records user decision and following flow.Handler that starts new flow instance using one of additional flow.StartFunction
Another option, is to use flow.View to record user decision and flow.Switch to step into another task.
Generally, BPMN separates user tasks and gateways, that lead to all decisions been recorded, before used.

How to reuse template in Flask-appbuilder with exposed custom handlers?

It is a very specific question regarding Flask-appbuilder. During my development, I found FAB's ModelView is suitable for admin role, but need more user logic handlers/views for complex designs.
There is a many to many relationship between devices and users, since each device could be shared between many users, and each user could own many device. So there is a secondary table called accesses, describes the access control between devices and users. In this table, I add "isHost" to just if the user owns the device. Therefore, we have two roles: host and (regular) user. However, these roles are not two roles defined as other applications, since one man can be either host or user in same time. In a very simple application, enforce the user to switch two roles are not very convinient. That makes things worse.
Anyway, I need design some custom handlers with traditional Flask/Jinja2 templates. For example:
class PageView(ModelView):
# FAB default URL: "/pageview/list"
datamodel = SQLAInterface(Page)
list_columns = ['name', 'date', 'get_url']
#expose("/p/<string:url>")
def p(self, url):
title = urllib.unquote(url)
r = db.session.query(Page).filter_by(name = title).first()
if r:
md = r.markdown
parser = mistune.Markdown()
body = parser(md)
return self.render_template('page.html', title = title, body = body)
else:
return self.render_template('404.html'), 404
Above markdown page URL is simple, since it is a seperate UI. But if I goes to DeviceView/AccountView/AccessView for list/show/add/edit operations. I realized that I need a unique styles of UI.
So, now how can I reuse the existing templates/widgets of FAB with custom sqlalchemy queries? Here is my code for DeviceView.
class DeviceView(ModelView):
datamodel = SQLAInterface(Device)
related_views = [EventView, AccessView]
show_template = 'appbuilder/general/model/show_cascade.html'
edit_template = 'appbuilder/general/model/edit_cascade.html'
#expose('/host')
#has_access
def host(self):
base_filters = [['name', FilterStartsWith, 'S'],]
#if there is not return, FAB will throw error
return "host view:{}".format(repr(base_filters))
#expose('/my')
#has_access
def my(self):
# A pure testing method
rec = db.session.query(Access).filter_by(id = 1).all()
if rec:
for r in rec:
print "rec, acc:{}, dev:{}, host:{}".format(r.account_id, r.device_id, r.is_host)
return self.render_template('list.html', title = "My Accesses", body = "{}".format(repr(r)))
else:
return repr(None)
Besides sqlalchemy code with render_template(), I guess base_filters can also help to define custom queries, however, I have no idea how to get query result and get them rendered.
Please give me some reference code or example if possible. Actually I have grep keywords of "db.session/render_template/expoaw"in FAB's github sources. But no luck.

FK validation within Django

Good afternoon,
I have my django server running with a REST api on top to serve my mobile devices. Now, at some point, the mobile device will communicate with Django.
Let's say the device is asking Django to add an object in the database, and within that object, I need to set a FK like this:
objectA = ObjectA.objects.create(title=title,
category_id = c_id, order = order, equipment_id = e_id,
info_maintenance = info_m, info_security = info_s,
info_general = info_g, alphabetical_notation = alphabetical_notation,
allow_comments = allow_comments,
added_by_id = user_id,
last_modified_by_id = user_id)
If the e_id and c_id is received from my mobile devices, should I check before calling this creation if they actually still exists in the DB? That is two extra queries... but if they can avoid any problems, I don't mind!
Thanks a lot!
It think that Django creates constraint on Foreign Key by default ( might depend on database though ). This means that if your foreign keys point to something that does not exist, then saving will fail ( resulting in Exception on Python side ).
You can reduce it to a single query (it should be a single query at least, warning I haven't tested the code):
if MyObject.objects.filter(id__in=[e_id, c_id]).distinct().count() == 2:
# create the object
ObjectA.objects.create(...)
else:
# objects corresponding e_id and c_id do not exist, do NOT create ObjectA
You should always validate any information that's coming from a user or that can be altered by a determined user. It wouldn't be difficult for someone to sniff the traffic and start constructing their own REST requests to your server. Always clean and validate external data that's being added to the system.

Optimizing database queries in Django

I have a bit of code that is causing my page to load pretty slow (49 queries in 128 ms). This is the landing page for my site -- so it needs to load snappily.
The following is my views.py that creates a feed of latest updates on the site and is causing the slowest load times from what I can see in the Debug toolbar:
def product_feed(request):
""" Return all site activity from friends, etc. """
latestparts = Part.objects.all().prefetch_related('uniparts').order_by('-added')
latestdesigns = Design.objects.all().order_by('-added')
latest = list(latestparts) + list(latestdesigns)
latestupdates = sorted (latest, key = lambda x: x.added, reverse = True)
latestupdates = latestupdates [0:8]
# only get the unique avatars that we need to put on the page so we're not pinging for images for each update
uniqueusers = User.objects.filter(id__in = Part.objects.values_list('adder', flat=True))
return render_to_response("homepage.html", {
"uniqueusers": uniqueusers,
"latestupdates": latestupdates
}, context_instance=RequestContext(request))
The query that causes the most time seem to be:
latest = list(latestparts) + list(latestdesigns) (25ms)
There is another one that's 17ms (sitewide annoucements) and 25ms (adding tagged items on each product feed item) respectively that I am also investigating.
Does anyone see any ways in which I can optimize the loading of my activity feed?
You never need more than 8 items, so limit your queries. And don't forget to make sure that added in both models is indexed.
latestparts = Part.objects.all().prefetch_related('uniparts').order_by('-added')[:8]
latestdesigns = Design.objects.all().order_by('-added')[:8]
For bonus marks, eliminate the magic number.
After making those queries a bit faster, you might want to check out memcache to store the most common query results.
Moreover, I believe adder is ForeignKey to User model.
Part.objects.distinct().values_list('adder', flat=True)
Above line is QuerySet with unique addre values. I believe you ment exactly that.
It saves you performing a subuery.

How to test Models in Django with Foreign Keys

I want to make sure I am testing Models/Objects in isolation and not as one huge system.
If I have an Order object and it has Foreign Keys to Customers, Payments, OrderItems, etc. and I want to test Order functionality, I need to create fixtures for all of that related data, or create it in code. I think what I really need to be doing is mocking out the other items, but I don't see an easy (or possible) solution for that if I am doing queries on these Foreign Keys.
The common solutions (fixtures) don't really let me test one Object at a time. I am sure this is partly caused by my app being way over coupled.
I am trying my darndest to adopt TDD as my main method of working, but the way things work with Django, it seems you can either run very trivial unit tests, or these massive integration tests.
[Edit] Better explicit example and a some more humility
What I mean is that I seem to be able to only run trivial unit tests. I have seen people with very well tested and granular modules. I am certain some of this can be followed back to poor design.
Example:
I have a model call Upsell which is linked to a Product model. Then I have a Choices model which are children of Upsell (do you want what's behind door #1, #2, #3).
The Upsell model has several methods on it that derive items necessary to render the template from their choices. The most important one is that it creates a URL for each choice. It does this through some string mangling etc. If I wanted to test the Upsell.get_urls() method, I want to have it not depend on the values of choices in the fixtures, and I want to not have it depend on the value of Product in the fixtures.
Right now I populate the db in the setUp method for the tests, and that works well with the way Django backs out the transaction every time, but only outside of setUp and tearDown. This works fairly well except some of the Models are fairly complex to set up, while I actually only need to get one attribute for it.
I can't give you an example of that, since I can't accomplish it, but here is the type of thing I am doing now. Basically I input an entire order, create the A/B experiment it was attached to, etc. And that's not counting Product, Categories, etc. all set up by fixtures. It's not the extra work I am concerned as I can't even test one database-based object at a time. The Tests below are important, but they are integration tests. I would like to be building up to something like this by testing each item separately. As you pointed out, maybe I shouldn't have chosen a framework so closely tied to the db. Does any sort of dependency injection exist with something like this? (beyond my testing, but the code itself as well)
class TestMultiSinglePaySwap(TestCase):
fixtures = ['/srv/asm/fixtures/alchemysites.json','/srv/asm/fixtures/catalog.json','/srv/asm/fixtures/checkout_smallset.json','/srv/asm/fixtures/order-test-fixture.json','/srv/asm/fixtures/offers.json']
def setUp(self):
self.o = Order()
self.sp = SiteProfile.objects.get(pk=1)
self.c = Customer.objects.get(pk=1)
signals.post_save.disconnect(order_email_first, sender=Order)
self.o.customer = self.c
p = Payment()
p.cc_number = '4444000011110000'
p.cc_exp_month = '12'
p.cc_type = 'V'
p.cc_exp_year = '2020'
p.cvv2 = '123'
p.save()
self.o.payment = p
self.o.site_profile = self.sp
self.o.save()
self.initial_items = []
self.main_kit = Product.objects.get(pk='MOA1000D6')
self.initial_items.append(self.main_kit)
self.o.add_item('MOA1000D6', 1, False)
self.item1 = Product.objects.get(pk='MOA1041A-6')
self.initial_items.append(self.item1)
self.o.add_item('MOA1041A-6', 1, False)
self.item2 = Product.objects.get(pk='MOA1015-6B')
self.initial_items.append(self.item2)
self.o.add_item('MOA1015-6B', 1, False)
self.item3 = Product.objects.get(pk='STP1001-6E')
self.initial_items.append(self.item3)
self.o.add_item('STP1001-6E', 1, False)
self.swap_item1 = Product.objects.get(pk='MOA1041A-1')
def test_single_pay_swap_wholeorder(self):
o = self.o
swap_all_skus(o)
post_swap_order = Order.objects.get(pk = o.id)
swapped_skus = ['MOA1000D','MOA1041A-1','MOA1015-1B','STP1001-1E']
order_items = post_swap_order.get_all_line_items()
self.assertEqual(order_items.count(), 4)
pr1 = Product()
pr1.sku = 'MOA1000D'
item = OrderItem.objects.get(order = o, sku = 'MOA1000D')
self.assertTrue(item.sku.sku == 'MOA1000D')
pr2 = Product()
pr2.sku = 'MOA1015-1B'
item = OrderItem.objects.get(order = o, sku = 'MOA1015-1B')
self.assertTrue(item.sku.sku == 'MOA1015-1B')
pr1 = Product()
pr1.sku = 'MOA1041A-1'
item = OrderItem.objects.get(order = o, sku = 'MOA1041A-1')
self.assertTrue(item.sku.sku == 'MOA1041A-1')
pr1 = Product()
pr1.sku = 'STP1001-1E'
item = OrderItem.objects.get(order = o, sku = 'STP1001-1E')
self.assertTrue(item.sku.sku == 'STP1001-1E')
Note that I have never actually used a Mock framework though I have tried. So I may also just be fundamentally missing something here.
Look into model mommy. It can automagically create objects with Foreign Keys.
This will probably not answer your question but it may give you food for thought.
In my opinion when you are testing a database backed project or application there is a limit to what you can mock. This is especially so when you are using a framework and an ORM such as the one Django offers. In Django there is no distinction between the business model class and the persistence model class. If you want such a distinction then you'll have to add it yourself.
Unless you are willing to add that additional layer of complexity yourself it becomes tricky to test the business objects alone without having to add fixtures etc. If you must do so you will have to tackle some of the auto magic vodoo done by Django.
If you do choose to grit your teeth and dig in then Michael Foord's Python Mock library will come in quite handy.
I am trying my darndest to adopt TDD as my main method of working, but the way things work with Django, it seems you can either run very trivial unit tests, or these massive integration tests.
I have used Django unit testing mechanism to write non-trivial unit tests. My requirements were doubtless very different from yours. If you can provide more specific details about what you are trying to accomplish then users here would be able to suggest other alternatives.