I am doing some tests on the FIX engine sample of FXCM. The complete code is available here.
There is a function void FixApplication::SubscribeMarketData() that allow to continuously receive update of a particular symbol of the Market. Here is what it look like :
// Subscribes to the EUR/USD trading security
void FixApplication::SubscribeMarketData()
{
// Subscribe to market data for EUR/USD
string request_ID = "EUR_USD_Request_";
FIX44::MarketDataRequest request;
request.setField(MDReqID(request_ID));
request.setField(SubscriptionRequestType(
SubscriptionRequestType_SNAPSHOT_PLUS_UPDATES));
request.setField(MarketDepth(0));
request.setField(NoRelatedSym(1));
// Add the NoRelatedSym group to the request with Symbol
// field set to EUR/USD
FIX44::MarketDataRequest::NoRelatedSym symbols_group;
symbols_group.setField(Symbol("EUR/USD"));
request.addGroup(symbols_group);
// Add the NoMDEntryTypes group to the request for each MDEntryType
// that we are subscribing to. This includes Bid, Offer, High, and Low
FIX44::MarketDataRequest::NoMDEntryTypes entry_types;
entry_types.setField(MDEntryType(MDEntryType_BID));
request.addGroup(entry_types);
entry_types.setField(MDEntryType(MDEntryType_OFFER));
request.addGroup(entry_types);
entry_types.setField(MDEntryType(MDEntryType_TRADING_SESSION_HIGH_PRICE));
request.addGroup(entry_types);
entry_types.setField(MDEntryType(MDEntryType_TRADING_SESSION_LOW_PRICE));
request.addGroup(entry_types);
Session::sendToTarget(request, sessionID(true));
}
Is there a way to tell the FIX server that I only want to receive updates every 5min ?
Or should I implement a function that catch the continuous flow of data and output a data every 5 min?
I already tried to search for a parameter in the FIX engine that I could modify to return a T periodic flow of data but I didn't find anything. If it exist I prefer to use it rather than create a function to handle the ticks flow.
The feature you are suggesting would be have to be a counterparty-specific feature implemented with probably custom fields. I don't believe the standard FIX dictionary provides fields that would support this.
So, yes, your hypothetical client-side solution would be the way to go.
Related
Lets assume that I have an input DataStream and want to implement some functionality that requires "memory" so I need ProcessFunction that gives me access to state. Is it possible to do it straight to the DataStream or the only way is to keyBy the initial stream and work in keyed-context?
I'm thinking that one solution would be to keyBy the stream with a hardcoded unique key so the whole input stream ends up in the same group. Then technically I have a KeyedStream and I can normally use keyed state, like I'm showing below with keyBy(x->1). But is this a good solution?
DataStream<Integer> inputStream = env.fromSource(...)
DataStream<Integer> outputStream = inputStream
.keyBy(x -> 1)
.process(...) //I've got acess to state heree
As I understand that's not a common usecase because the main purpose of flink is to partition the stream, process them seperately and then merge the results. In my scenario thats exactly what I'm doing, but the problem is that the merge step requires state to produce the final "global" result. What I actually want to do is something like this:
DataStream<Integer> inputStream = env.fromElements(1,2,3,4,5,6,7,8,9)
//two groups: group1=[1,2,3,4] & group2=[5,6,7,8,9]
DataStream<Integer> partialResult = inputStream
.keyBy(val -> val/5)
.process(<..stateful processing..>)
//Can't do statefull processing here because partialResult is not a KeyedStream
DataStream<Integer> outputStream = partialResult
.process(<..statefull processing..>)
outputStream.print();
But Flink doesnt seem to allow me do the final "merge partial results operation" because I can't get access to state in process function as partialResult is not a KeyedStream.
I'm beginner to flink so I hope what I'm writing makes sense.
In general I can say that I haven't found a good way to do the "merging" step, especially when it comes to complex logic.
Hope someone can give me some info, tips or correct me if I'm missing something
Thank you for your time
Is "keyBy the stream with a hardcoded unique key" a good idea? Well, normally no, since it forces all data to flow through a single sub-task, so you get no benefit from the full parallelism in your Flink cluster.
If you want to get a global result (e.g. the "best" 3 results, from any results generated in the preceding step) then yes, you'll have to run all records through a single sub-task. So you could have a fixed key value, and use a global window. But note (as the docs state) you need to come up with some kind of "trigger condition", otherwise with a streaming workflow you never know when you really have the best N results, and thus you'd never emit any final result.
I'm giving the AWS' Step Functions a try and I'm interested in them for implementing long-running procedures. One functionality I would like to provide to my users is the possibility of showing execution's progress. Using describeExecution I can verify if some execution is still running or done. But progress is a logical measure and Step Functions itself has no way to tell me how much of the process is left.
For that, I need to provide the logic myself. I can measure the progress in the tasks of the state machine knowing the total number of steps needed to take and counting the number of steps already taken. I can store this information in the state of the machine which is passed among steps while the machine is running. But how can I extract this state using API? Of course, I can store this information is an external storage like DynamoDb but that's not very elegant!
The solution I have found my self (so far this is the only), is using getExecutionHistory API. This API returned a list of events that are generated for the Step Functions and it can include input or output (or neither) based on whether the event is for a starting a lambda function or is it for the time a lambda function has exited. You can call the API like this:
var params = {
executionArn: 'STRING_VALUE', /* required */
maxResults: 10,
reverseOrder: true
};
stepfunctions.getExecutionHistory(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
By reversing the order of the list of events, we can get the latest ones first. Then we can look for the latest output in the list. The first one you'll find will be the latest version of the output which is the current state of the Step Functions.
We are building some plugins in Microsoft Dynamics CRM by inheriting from IPlugin. We have these configured so they fire whenever an Account is updated.
The problem is the plugins are calling our services, which causes our service to respond with an update. We are doing some pretty hacky things right now to prevent these cyclical updates from happening.
We were wondering if there was a way to pass a value to the IOrganizationService service (the web service) that a plugin can look at. Our other system could send a flag ("hey, don't bothing sending an update!") and the plugin could skip calling back.
Can we pass parameters from web service to the plugins?
Good idea could be usage of custom flag-field. For example you add bit field and call it CallFromExternalSystem. So when you make an update from your external system through IOranizationService you just fill this flag with true field and in plugin you can check condition that this field is present in fields list so you have no need to call external system endpoint again.
We decided the correct solution was to use the value found in IPluginExecutionContext.InputParameters["Target"]. In the case of an Update, this returns an Entity containing attributes for all the attributes that were updated.
We basically have a list of attribute names we cared about. We loop through names and see if any of them appear in the entity attribute list. If so, we send an update to our other system. The good news is, Dynamics CRM ignores updates where the values don't actually change, so trying to update a value to itself is no-op.
public void Execute(IServiceProvider serviceProvider)
{
IPluginExecutionContext context = serviceProvider.GetService(typeof(IPluginExecutionContext));
Entity entity = (Entity)context.InputParameters["Target"];
string[] fields = new string[] { "name", "statecode", "address1_line1" };
bool hasUpdates = fields.Where(f => entity.Attributes.Contains(f)).Any();
if (!hasUpdates)
{
return;
}
}
The IB API reqHistoricalData() method offers a whatToShow argument which can take values to denote you seek data on TRADES, MIDPOINT, BID, ASK etc...
However, the API's historicalData callback, provided to asynchronously receive the requested historical data, doesn't return the relevant whatToShow making it impossible to ascertain what one is looking at. Is it the line for the TRADES, the BIDS or the ASKS I requested???
I get around this the obvious way, namely by requesting TRADES first, waiting for the entirety of the messages to come back and then requesting BIDS then wait again and request ASKS.
Does anyone have a better solution?
Please use the tickerId field properly which is the first parameter in reqHistoricalData() method. When you get the historical data with callbacks, you will be receiving this id back as the first parameter with historicalData().
You just need to keep track which tickerId is associated with which kind of data (bid, ask or trade) to identify that on the callback.
Example:
While requesting:
reqHistoricalData(1, ..whatToShow = Bid,...);
reqHistoricalData(2, ..whatToShow = Ask,...);
Callback handling:
historicalData(int reqId,....)
if(reqId == 1)
//This is the data built of bids as per request1
else if(reqId == 2)
//This is the data built of asks as per request2
It seems that c++ drivers doesn't accept mongodb connection uri format.
There's no documentation on how i should create connection string, any guess?
I need to connect to a replica set with 3 servers, and set readPreference options.
Create a connection to a replica set in MongoDB C++ client
Until the problems explained in #acm's answer are resolved, I have found a workaround to the bad Connection Strings of the C++ driver. You can create a DBClientReplicaSet using a vector of hosts and ports this way:
//First create a vector of hosts
//( you can ignore port numbers if yours are default)
vector<HostAndPort> hosts;
hosts.push_back(mongo::HostAndPort("YourHost1.com:portNumber1"));
hosts.push_back(mongo::HostAndPort("YourHost2.com:portNumber2"));
hosts.push_back(mongo::HostAndPort("YourHost3.com:portNumber3"));
//Then create a Replica Set DB Client:
mongo::DBClientReplicaSet connection("YourReplicaSetName",hosts,0);
//Connect to it now:
connection.connect();
//Authenticate to the database(s) if needed
std::string errmsg;
connection.auth("DB1Name","UserForDB1","pass1",errmsg);
connection.auth("DB2Name","UserForDB2","pass2",errmsg);
Now, you can use insert, update, etc. just as you did with DBClientConnection. For a quick fix, you can replace your references to DBClientConnection with DBClientBase (which is a parent to both DBClientConnection and DBClientReplicaSet)
Last pitfall: if you are using getLastError(), you must use it with the aimed database name like this:
connection.getLastError(std::string("DBName"));
cause otherwise it will always return "command failed: must log in" as described in this JIRA ticket.
Set the read preferences for every request
You have two ways to do that:
SlaveOK option
It lets your read queries be directed to secondary servers.
It takes place in the query options, which are at the end of the parameters of DBClientReplicaSet.query(). The options are listed in Mongo's official documentation
The one you would look for is mongo::QueryOption_SlaveOk, which will allow you to have reads made on secondary instances.
This is how you should call query();
connection.query("Database.Collection",
QUERY("_id" << id),
n,
m,
BSON("SomeField" << 1),
QueryOption_SlaveOk);
where n is the number of documents to return (0 if you don't want any limit), m the number to skip (defaults to 0), the next field is your projection and the last your query option.
To use several query option, you can use bitwise or | like this :
connection.query("Database.Collection",
QUERY("_id" << id),
n,
m,
BSON("SomeField" << 1),
QueryOption_SlaveOk | QueryOption_NoCursorTimeout | QueryOption_Exhaust);
Query::readPref option
The Query object has a readPref method which sets read preferences for a special query. It should be called for each query.
You can pass different arguments for more control. They are listed here.
So here's what you should do (I did not test that one cause I can't right now but it should work just fine)
/* you should pass an array for the tags. Not sure if this is required.
Anyway, let's create an empty array using the builder. */
BSONArrayBuilder bab;
/* if any, add your tags here */
connection.query("Database.Collection",
QUERY("_id" << id).readPref(ReadPreference_SecondaryPreferred, bab.arr()),
n,
m,
BSON("SomeField" << 1),
QueryOption_NoCursorTimeout | QueryOption_Exhaust);
Note: if any readPref option is used, it should override the slaveOk option.
Hope this helped.
Please see the connection string documentation for details on the connection string format.
(code links below are to 2.2.3 files)
To use a connection string with the C++ driver, you should use the ConnectionString class. You first call the ConnectionString::parse static method with a connection string to obtain a ConnectionString object. You then call ConnectionString::connect to obtain a DBClientBase object which you can then use to send queries.
As for read preference, at the moment I do not see a way to set the read preference in the connection string for the C++ driver, which would preclude a per-connection setting.
However, the implementation of DBClientBase returned by calling ConnectionString::parse with a string that identifies a replica set will return you an instance of DBClientReplicaSet. That class honors $readPreference in queries, so you can set your read preference on a per-query basis.
Since the current C++ drivers still do not accept the standard mongodb connection URIs, I've opened a ticket:
https://jira.mongodb.org/browse/CXX-2
Please vote for it to help get this fixed.
it seems like you can set read Preference before send a read request by call "readPref" method of your Query object. I'v not found a way to set read Preference on mongo collection object yet.