I'm writing PHP for fairly simple workflow for Amazon SWF. I've found myself starting to write a library to check if certain actions have been started or completed. Essentially looping over the event list to check how things have progressed, and then starting an appropriate activity if its needed. This can be a bit faffy at times as the activity type and input information isn't in every event, it seems to be in the ActivityTaskScheduled event. This sort of thing I've discovered along the way, and I'm concerned that I could be missing subtle things about event lists.
It makes me suspect that someone must have already written some sort of generic library for finding the current state of various activities. Maybe even some sort of more declarative way of coding up the flowcharts that are associated with SWF. Does anything like this exist for PHP?
(Googling hasn't come up with anything)
I'm not aware of anything out there that does what you want, but you are doing it right. What you're talking about is coding up the decider, which necessarily has to look at the entire execution state (basically loop through the event list) and decide what to do next.
Here's an example written in python
( Using Amazon SWF To communicate between servers )
that looks for events of type 'ActivityTaskCompleted' to then decide what to do next, and then, yes, looks at the previous 'ActivityTaskScheduled' entry to figure out what the attributes for the previous task were.
If you write a php framework that specifies the workflow in a declarative way then a generic decider that implements it, please consider sharing it :)
I've since found https://github.com/cbalan/aws-swf-fluent-php which looks promising, but not really used it, so can't speak to the whether it works or not.
I've forked it and started a bit of very light refactoring to allow some testing, available at https://github.com/michalc/aws-swf-fluent-php
Related
We have adopted an AWS powered microservice architecture where different sorts of payloads enter the system with a UUID and type via mysql.lambda_async from our database.
The problem is, that we've noticed that messages can come out of order. Imagine the scenario with the following type of message:
DEASSIGN_ROLE
ASSIGN_ROLE
When the actual intention was a quick toggle:
ASSIGN_ROLE
DEASSIGN_ROLE
Now we have a user with the wrong (elevated) permissions.
I've done some cursory research and for example answers like Handling out of order events in CQRS read side suggest using sequence numbers.
Introducing a sequence number would be quite hard as we have many different types of messages. A sequence number would require a syncronous counter, where we have gone great pains to be simply asynchronous. Bear in mind, our system that generates the message is an SQL trigger ultimately.
Are there simpler solutions I am missing?
I would say there is an unsolvable problem :
you want to be full asynchronous
you need sequentiality in your results
We had the same problem as yours, and we ended by setting sequences by type of messages.
Trying to be asynchronous and parallel when possible (according to message types/topics)
I have given a project to develop Algorithmic trading system using c++ and quickFix library, I search on google about quickFix library but didn't find any useful information.
Can anybody give me some information , from where should I start?
You provide very little detail in your question, so I can only guess at a helpful approach. I have done what you are starting, in Python, and can give you some orientation. All the links Karl mentioned are crucial, (you should pay special attention to the quickfix documentation on the config file) to which I would add FIXIMATE.
To do something like this in QF you need to answer a number of questions.
Logon. Figure out how to logon. Try to get a data dictionary from your counterparty. You don't want to be forced to modify your DD too much.
Interface. How will you tell QF to logon, logoff, exit terrible positions, and so forth? I use a command line tool (cmd2) that gives me this ability. Other people code GUI windows.
Message Cracking. Some versions of QF come with a cracker but if you don't have it in C++ you will have to write your own so you can parse the incoming messages.
Data Management. How will you save incoming market data, both in RAM and to disk for analysis later? How will you represent and monitor your positions, your working orders, your audit trail? Familiarize yourself with the ScreenLogFactory and FileLogFactory in QF.
Auxiliary Functions. You will need a lot of functions you will write yourself to help at all stages. Save them all in one place and organize them into categories for easy access.
Monitoring. How will you know if something goes wrong (or right) when you are not in front of your computer monitoring the algo? I launch a completely separate process which consumes messages via a queue and sends me texts and emails.
Risk. You don't want your machine sending 1000 orders to market in the blink of an eye. You need to code some checks that will veto bad orders as a final stage before they go out. Also some code that will tell you if you are in a position when you are supposed to be flat. This part is very important.
Strategy. You will need the ability to quickly ingest data, analyze it, and generate signals. For flexibility you should not design your strategy into your system, but you should design a strategy object which can support any strategy you come up with. Then you deploy those objects within your system.
Order handling. Your algo needs to know when and how to enter orders, cancel them, move stops, etc. It will need to deal with partial fills, and be able to support multiple order types.
This is just the beginning, off the top of my head. It is a long road to do all by yourself with no help. Very interesting though, and rewarding.
You can find the QuickFIX downloads on the quickfixengine.org website here: http://www.quickfixengine.org/. From there, you can download either the source code or download pre-built packages for Visual Studio 2010, 2012 and 2013.
Documentation for QuickFIX can be found in their documentation area here: http://www.quickfixengine.org/quickfix/doc/html/. The documentation includes compilation/installation instructions and a "Getting Started" section which discusses setting up a project and writing your first QuickFIX application.
If you wish to know more about the FIX protocol, you are advised to look at the FIX website here: http://www.fixtradingcommunity.org. There are specifications on that website that will give you information on the types of messages supported by FIX and how they should be used.
Since there's no complete BPM framework/solution in ColdFusion as of yet, how would you model a workflow into a ColdFusion app that can be easily extensible and maintainable?
A business workflow is more then a flowchart that maps nicely into a programming language. For example:
How do you model a task X that follows by multiple tasks Y0,Y1,Y2 that happen in parallel, where Y0 is a human process (need to wait for inputs) and Y1 is a web service that might go wrong and might need auto retry, and Y2 is an automated process; follows by a task Z that only should be carried out when all Y's are completed?
My thoughts...
Seems like I need to do a whole lot of storing / managing / keeping
track of states, and frequent checking with cfscheuler.
cfthread ain't going to help much since some tasks can take days
(e.g. wait for user's confirmation).
I can already image the flow is going to be spread around in multiple UDFs,
DB, and CFCs
any opensource workflow engine in other language that maybe we can port over to CF?
Thank you for your brain power. :)
Study the Java Process Definition Language specification where JBoss has an execution engine for it. Using this Java based engine may be your easiest solution, and it solves many of the problems you've outlined.
If you intend to write your own, you will probably end up modelling states and transitions, vertices and edges in a directed graph. And this as Ciaran Archer wrote are the components of a State Machine. The best persistence approach IMO is capturing versions of whatever data is being sent through workflow via serialization, capturing the current state, and a history of transitions between states and changes to that data. The mechanism probably needs a way to keep track of who or what has responsibility for taking the next action against that workflow.
Based on your question, one thing to consider is whether or not you really need to represent parallel tasks in your solution. Where instead it might be possible to en-queue a set of messages and then specify a wait state for all of those to complete. Representing actual parallelism implies you are moving data simultaneously through several different processes. In which case when they join again you need an algorithm to resolve deltas, which is very much a non trivial task.
In the context of ColdFusion and what you're trying to accomplish, a scheduled task may be necessary if the system you're writing needs to poll other systems. Consider WDDX as a serialization format. JSON, while seductively simple, I recall has some edge cases around numbers and dates that can cause you grief.
Finally see my answer to this question for some additional thoughts.
Off the top of my head I'm thinking about the State design pattern with state persisted to a database. Check out the Head First Design Patterns's Gumball Machine example.
Generally this will work if you have something (like a client / order / etc.) going through a number of changes of state.
Different things will happen to your object depending on what state you are in, and that might mean sitting in a database table waiting for a flag to be updated by a user manually.
In terms of other languages I know Grails has a workflow module available. I don't know if you would be better off porting to CF or jumping ship to Grails (right tool for the job and all that).
It's just a thought, hope it helps.
This question is a refinement of my question Different ways of observing data changes.
I still have a lot of classes in my C++ application, which are updated (or could be updated) frequently in complex mathematical routines and in complex pieces of business logic.
If I go for the 'observer' approach, and send out notifications every time a value of an instance is changed, I have 2 big risks:
sending out the notifications itself may slow down the applications seriously
if user interface elements need to be updated by the change, they are updated with every change, resulting in e.g. screens being updated thousends of times while some piece of business logic is executing
Some problems may be solved by adding buffering-mechanisms (where you send out notifications when you are going to start with an algorith, and when the algorithm is finished), but since the business logic may be executed on many places in the software, we end up adding buffering almost everywhere, after every possible action chosen in the menu.
Instead of the 'observer' aproach, I could also use the 'mark-dirty' approach, only marking the instances that have been altered, and at the end of the action telling the user interface that it should update itself.
Again, business logic may be executed from everywhere within the application, so in practice we may have to add an extra call (telling all windows they should update themselves) after almost every action executed by the user.
Both approaches seem to have similar, but opposite disadvantages:
With the 'observer' approach we have the risk of updating the user-interface too many times
With the 'mark-dirty' approach we have the risk of not updating the user-interface at all
Both disadvantages could be solved by embedding every application action within additional logic (for observers: sending out start-end notifications, for mark-dirty: sending out update-yourself notifications).
Notice that in non-windowing applications this is probably not a problem. You could e.g. use the mark-dirty approach and only if some calculation needs the data, it may need to do some extra processing in case the data is dirty (this is a kind of caching approach).
However, for windowing applications, there is no signal that the user is 'looking at your screen' and that the windows should be updated. So there is no real good moment where you have to look at the dirty-data (although you could do some tricks with focus-events).
What is a good solution to solve this problem? And how have you solved problems like this in your application?
Notice that I don't want to introduce windowing techniques in the calculation/datamodel part of my application. If windowing techniques are needed to solve this problem, it must only be used in the user-interface part of my application.
Any idea?
An approach I used was with a large windows app a few years back was to use WM_KICKIDLE. All things that are update-able utilise a abstract base class called IdleTarget. An IdleTargetManager then intercepts the KICKIDLE messages and calls the update on a list of registered clients. In your instance you could create a list of specific targets to update but I found the list of registered clients enough.
The only gotcha I hit was with a realtime graph. Using just the kick idle message it would spike the CPU to 100% due to constant updating of the graph. Use a timer to sleep until the next refresh solved that problem.
If you need more assistance - I am available at reasonable rates...:-)
Another point I was thinking about.
If you are overwhelmed by the number of events generated, and possibly the extra-work it is causing, you may have a two phases approach:
Do the work
Commit
where notifications are only sent on commit.
It does have the disadvantage of forcing to rewrite some code...
You could use the observer pattern with coalescing. It might be a little ugly to implement in C++, though. It would look something like this:
m_observerList.beginCoalescing();
m_observerList.notify();
m_observerList.notify();
m_observerList.notify();
m_observerList.endCoalescing(); //observers are notified here, only once
So even though you call notify three times, the observers aren't actually notified until endCoalescing when the observers are only notified once.
I would like to be able to measure the features in our application that are being used.
For example how much certain windows are opened, certain controls are clicked.
I can imagine a tool that measures this and sends a report to a web server, that can further process it to create meaningful data from it
First question : should you do it ? People don't like when their software phones home without their consent. But assuming they are ok with it then:
It's technically possible, with two approaches: automatic or manual. Of course, given your question, I assume that you are using Qt.
Automatic:
give a proper name to all the QObject that you want to trace
install an event filter on your application to catch all the ChildEvent about objects that are created and destroyed.
from the ChildEvent, you can extract the object's name
then you can already log how often that object is created. You can also use the opportunity to add an event listener to that specific object, to be notified when it is shown or hidden or track other kind of usage
log everything to a log file
Manual :
add log statements to relevant part of your code that you want to track.
Final :
send the log file on a regular basis
I guess, your answer is "No". I don't think there are such libraries.
I also think, the best solution here is logging, meaning you should manually introduce some log functions into your main program features and send back the log file. When it comes to logging, you may consider using aspect-oriented programming (and there are such tools for C++), it may simplify your task...