How do you ensure a middleware is not applied twice?
My application is using bidi and I have my routes and handlers separate. So my final ring handler has some middlewares (such as wrap-params and wrap-session) and some of my handlers have their own custom middlewares. I can manually watch over my main handler's constructor and my handlers file, but I would rather not rely on manual vigilance.
Is there a design that helps manage middlewares better? Or some library?
My structure looks like this:
1. routes 2. handlers
\ /
\ /
3. (bidi.ring/make-handler routes handlers)
It's possible to wrap something in 2 and then again in 3.
3 is actually a component and make-handler is called in its start. handler & routes come from 3's component's dependencies. I have considered adding another protocol to 1 & 2 with a wrap-middlewares method, but the order of middlewares is significant. For example (buddy-auth's) wrap-authentication depends on wrap-session & wrap-params.
for middleware you control it's fairly straightforward, you can just have each middleware add a key with it's name to the request and bail (or noop if you're feeling lenient) if that key is in any request it recieves. for middleware you don't control i suppose were stuck with staring at the code and thinking really hard, or as you describe it "manual vigilance".
Related
It seems an obvious combination to use django's signalling architecture together with channels: a model is changed, a signal is fired, and a consumer is notified (via the signal handler and channel layer) to update a client via, for instance, a WebSocket.
However, very often a model is updated as part of a transaction, and django's pre_save and post_save signals are sent in the middle of the transaction, meaning the change you notify the client about may never happen at all. Even more annoyingly, when you send the information across the channel layer you're looking at the database from another thread and, since the transaction is not complete, you can't get the new data.
django.db.transaction.on_commit provides a way to schedule (part of) the signal handler to run after the transaction is done, providing a workaround, but this seems to be such an obvious problem with absolutely no documentation or stackoverflow chatter that I'm a bit perturbed (one can of course find discussions about each pair of the three mentioned topics!) So is this architecture: signal -> handler -> channel layer -> consumer [ -> client ] actually a bad one? Is there some alternative that doesn't involve wrapping each handler in a on_commit hook, or is there some common idiom for doing that an DRY way?
Imagine you have a User model in your web app, and that you need to keep this user in sync with an external service via an API. Thus, when you create a user locally, you need to create it remotely as well.
You have all your operations under transaction.atomic() and you try to keep all your 3rd-party API calls after the atomic block, which is reasonable.
But, a system being a system, it grows in complexity until the point you have some really hard to remove 3rd-party calls within an update call.
That said, is there a way to extend Django's transaction mechanism, kind of adding some callback functions, like rollback.add_callback(clean_3rdparty_user(user_id=134))?
That way I can guarantee that all necessary rollback actions are taken and my system is in sync?
The author of Django's transaction hook code has this to say about why there is on_commit() but not on_rollback():
A rollback hook is even harder to implement robustly than a commit hook, since a variety of things can cause an implicit rollback. For instance, your database connection was dropped because your process was killed without a chance to shutdown gracefully: your rollback hook will never run.
Since rollbacks are typically triggered by an exception, a simple approach is to just catch any exceptions and run your undo code there.
try:
with transaction.atomic():
# Do database stuff
# Do external stuff
except:
# We know the database stuff has rolled back, so...
# Undo external stuff
raise
This is not particularly elegant. I agree with the following from the same source:
The solution is simple: instead of doing something during the atomic block (transaction) and then undoing it if the transaction fails, use on_commit to delay doing it in the first place until after the transaction succeeds. It’s a lot easier to undo something you never did in the first place!
But it sounds like you already agree with that as well.
I'm trying to set up python-telegram-bot library in webhook mode with Django. That should work as follows: on Django startup, I do some initial setting of python-telegram-bot and get a dispatcher object as a result. Django listens to /telegram_hook url and receives updates from Telegram servers. What I want to do next is to pass the updates to the process_update method of the dispatcher created on startup. It contains all the parsing logic and invokes callbacks specified during setup.
The problem is that the dispatcher object needs to be saved globally. I know that global states are evil but that's not really a global state because the dispatcher is immutable. However, I still don't know where to put it and how to ensure that it will be visible to all threads after setup phase is finished. So the question is how do I properly save the dispatcher after setup to invoke it from Django's viewset?
P.S. I know that I could use a built-in web server or use polling or whatever. However, I have reasons to use Django and I anyway would like to know how to deal with cases like that because it's not the only situation I can imagine when I need to store an immutable object created on startup globally.
It looks like you need thread safe singleton like this one https://gist.github.com/werediver/4396488 or http://alacret.blogspot.ru/2015/04/python-thread-safe-singleton-pattern.html
import threading
# Based on tornado.ioloop.IOLoop.instance() approach.
# See https://github.com/facebook/tornado
class SingletonMixin(object):
__singleton_lock = threading.Lock()
__singleton_instance = None
#classmethod
def instance(cls):
if not cls.__singleton_instance:
with cls.__singleton_lock:
if not cls.__singleton_instance:
cls.__singleton_instance = super(SingletonMixin, cls).__new__(cls)
return cls.__singleton_instance
By the definition of CQRS command can/should be validated and at the end even declined (if validation does not pass). As a part of my command validation I check if state transition is really needed. So let take a simple, dummy example: actor is in state A. A command is send to actor to transit to state B. The command gets validated and at the end event is generated StateBUpdated. Then the exact same command is send to transit to state B. Again command gets validated and during the validation it is decided that no event will be generated (since we are already in state B) and just respond back that command was processed and everything is ok. It is kind of idempotency thingy.
Nevertheless, I have hard time (unit) testing this. Usual unit test for persistent actor looks like sending a command to the actor and then restarting actor and check that state is persisted. I want to test if I send a command to the actor to check how many events were generated. How to do that?
Thanks
We faced this problem while developing our internal CQRS framework based on akka persistence. Our solution was to use Persistence Query(https://doc.akka.io/docs/akka/2.5/scala/persistence-query.html). In case you haven't used it, it is a query interface that journal plugins can optionally implement, and can be used as the read side in a CQRS system.
For your testing purposes, the method would be eventsByPersistenceId, which will give you an akka streams Source with all the events persisted by an actor. The source can be folded into a list of events like:
public CompletableFuture<List<Message<?>>> getEventsForIdAsync(String id, FiniteDuration readTimeout) {
return ((EventsByPersistenceIdQuery)readJournal).eventsByPersistenceId(id, 0L, Long.MAX_VALUE)
.takeWithin(readTimeout)
.map(eventEnvelope -> (Message<?>)eventEnvelope.event())
.<List<Message<?>>>runFold(
new ArrayList<Message<?>>(),
(list, event) -> {
list.add(event);
return list;
}, materializer)
.toCompletableFuture();
}
Sorry if the above seems bloated, we use Java, so if you are used to Scala it is indeed ugly. Getting the readJournal is as easy as:
ReadJournal readJournal = PersistenceQuery.lookup().get(actorSystem)
.getReadJournalFor(InMemoryReadJournal.class, InMemoryReadJournal.Identifier())
You can see that we use the akka.persistence.inmemory plugin since it is the best for testing, but any plugin which implements the Persistence Query API would work.
We actually made a BDD-like test API inside our framework, so a typical test looks like this:
fixture
.given("ID1", event(new AccountCreated("ID1", "John Smith")))
.when(command(new AddAmount("ID1", 2.0)))
.then("ID1", eventApplied(new AmountAdded("ID1", 2.0)))
.test();
As you see, we also handle the case of setting up previous events in the given clause as well a potentially dealing with multiple persistenceIds(we use ClusterSharding).
From you description it sounds like you need either to mock your persistence, or at lest be able to access it's state easily. I was able to find two projects that will do that:
akka-persistence-mock which is designed for use in testing, but not actively developed.
akka-persistence-inmemory
which is very useful when testing persistent actors, persistent FSM and akka cluster.
I would recommend the latter, since it provides the possibility of retrieving all messages from the journal.
My web app is based on (embedded) Jetty 9. The code that runs inside Jetty (i.e. from the *.war file) has the need to, at times, execute an HTTP request back into Jetty and itself, completely asynchronously to "real" HTTP requests coming from the network.
I know what you will say, but this is the situation I ended up with after merging multiple disparate products into one container and presently cannot avoid it. A stop-gap measure is in place - it actually sends a network HTTP request back to itself (presently using Jetty client, but that does not matter). However, not only that adds more overhead, it also does not allow us to pass actual object references we'd like to be able to pass via, say, request attributes.
Desire is to be able to do something like constructing new HttpServletRequest and HttpServletResponse pair and use a dispatcher to "include" (or similar) the other servlet we presently can only access via the network. We've built "dummy" implementations of those, but the this fails in Jetty's dispatcher line 120 with a null pointer exception:
public void include(ServletRequest request, ServletResponse response) throws ServletException, IOException
{
Request baseRequest=(request instanceof Request)?((Request)request):HttpChannel.getCurrentHttpChannel().getRequest();
... because this is not an instance of Jetty's Request class and getCurrentHttpChannel() returns null because the thread is a worker thread, not an http serving one and does not have Jetty's thread locals set up.
I am contemplating options, but would like some guidance if anyone can offer it. Some things I am thinking of:
Actually use Jetty's Request class as a base. Currently not visible to the web app (a container class, would have to play with classpath and class loaders perhaps. May still be impossible (don't know what to expect there).
Play with Jetty's thread locals, attempt to tell Jetty to set up current thread as necessary. Don't know where to begin. UPDATE Tried to setServerClasses([]) and then set the current HttpChannel to the one I 'stole' from another thread. Failed misearably: java.lang.IllegalAccessError: tried to access method org.eclipse.jetty.server.HttpChannel.setCurrentHttpChannel(Lorg/eclipse/jetty/server/HttpChannel;)V from class ...
Ideally, find a better/proper way of feeding a "top" request in without going via the network. Ideally would execute on the same thread, but I would be less concerned with that.
Remember that, unfortunately, I cannot avoid this at this time. I would much rather invoke code directly, but I cannot, as the code I had to add into mine is too big to handle at this time and too dependent on some third party filters I can't even modify (and only work as filters, on real requests).
Please help!