I'm implementing some kind of distributed database. In the nodes there are "Agent"-programs running, which get queries by a "Router" and send them to the local database. After that the results should be send back to the router.
How can I send a MYSQL_RES structure over network? What is the best way to do this? At the moment I just build a protobuf-Object with the data, but the deserialization of that object is quiet slow. And building the object needs a run over all rows.
Is there a possibility to send the mysql-binary-result to the router directly and interpret it on the router? I need the agent, as the resultsets must be examined locally, too. I'm using C/c++.
Regards
Related
I am writing an application where client nodes access a postgres database via an intermediary application. The clients cannot query the database directly; instead the intermediary application receives queries from the clients over a network, runs them, and relays the results.
libpqxx's pqxx::result, pqxx::row and pqx::field classes provides many conversion routines that allow the results of queries to be translated into c++ variables. I'd like my client application to make use of these, but that requires somehow either transferring the pqxx objects over the network, or sending the underlying "flexible textual format", and constructing pqxx objects from this on the client.
'Textual data' is generally simple enough to send over a network, but I can't see any way to access it from the pqxx::result objects, or any way to construct pqxx objects from that data were I able to access and transfer it. Is there any way to do this?
Is there a way for a function in PSQL to call a webservice that returns a JSON and use this JSON to perform something in the database within a trigger call ?
psql is the command line client application. So no.
If you mean pl/pgsql, the server side function language, then ... still no. It, by design, cannot make network connections, access files, etc.
You could use PL/Python or PL/Perl to do this... but you shouldn't. If the web service is very slow, there are DNS problems, etc your whole database could slow to a nonfunctioning crawl.
Instead use LISTEN and NOTIFY with an external client to process a work queue and store the results in the database. With PostgreSQL 9.5's SKIP LOCKED this can be done concurrently very easily. With 9.4 and older it's easiest to do it serially, or using one worker managing multiple asynchronous requests.
This is much like the "how do I send email from a stored procedure" questions. You can use plperl or plpython, but you shouldn't.
I am writing some code for a client and server using c++.
I am using boost serialize to serialize an object on the client side and then send it to a server.
The server then unserializes the stream and recreates the object.
The server then calls the run function of the object.
The server then serializes the object and sends it back to the client.
The reason I am doing the above is for instance the client needs to know what files are in the /home/dataIncoming folder on the server.
This is just one simple example
My question is that it seems that for the objects that are serialized I need the same code on the server AND the client. Or how would the server know how to unserialize the object sent.
Thus if I change the server code i need to make sure I also get the code over to the client program.
How do programmers easily solve this problem of duplicate code on the client and the server?
Or is there some way I could serialize so this duplicate code did not need to exist on the client and the server?
or do i simply send a protocol stream over to the server and have the server read in that protocol stream to reconstruct commands to be run and info to be sent back.
It seems easiest to just construct an object on the client side, have it run on the server, then sent it back to the client with results.
Thanks for all your ideas!
For generic client server operations:
You compile your code into a library, and use the library on both the client and server.
Your client and server both need to understand the data format. There is no way around this.
You could have a more abstract layer that allows the client to submit the format of new data types, but this abstract layer itself would need to be a data format understood by both the client and server.
It's unclear to me how your example differs (it at all) from the generic case.
The client and the server have to share data (and the data format), but not the code.
It sounds like your object encapsulates three things:
The data
Some client code
Some server code
A lot of object oriented programming encourages putting data and the code that operates on it into the same object, but that's not always the best solution. Why not separate those things out?
Put your data into a data object and serialize that. The server would have something that operated on that data (presumably some object, not necessarily though) and the client would have something that operated on that data. So, I think you just need to reconsider how your programs are structured.
The architecture you describe also has another downside. If you want to modify the behavior of the server, you have to also update the client. If these concerns were separate, this wouldn't be a problem. The only reason you would have to update both the client and the server is if the data format changes.
Usually the Encoding and Decoding is taken care by the serialization library that you use. (google prorpbuf, nano-pb, boost etc)
However Since the encoded from and decoded to classes/structures would be essentially be the same.
So essentially we need the same information on both ends (server and the client)
A good idea is to create a dll or an .so for this functional module which when changed can be reflected on both the server and the client, without recompilation.
I'm trying to modify a game engine so it records events (like key presses), and store these in a MySQL database on a remote server. The game engine is written in C++, and I currently have the following straightforward architecture, using mysql++ to directly INSERTrecords into appropriate databases:
Unfortunately, there's a very large overhead when connecting to the MySQL server, and the game stops for a significant amount of time. Pushing a batch of Xs worth of events to the server causes a significant delay in gameplay (60s worth of events can take 12s to synchronise). There are also apparently security concerns with leaving the MySQL port accessible publicly.
I was considering an alternative option, instead sending commands to the server, which can interact with the database in its own time:
Here the game would only send the necessary information (e.g. the table to update and the data to insert). I'm not sure whether the speed increase would be sufficient, or what system would be appropriate for managing the commands sent from the game.
Someone else suggested Log4j, but obviously I need a C++ solution. Is there an appropriate existing framework for accomplishing what I want?
Most applications gathering user-interface interaction data (in your case keystrokes) put it into a local file of some sort.
Then at an appropriate time (for example at the end of the game, or the beginning of another game), they POST that file, often in compressed form, to a publicly accessible web server. The software on the web server decompresses the data and loads it into the analytics system (the MySQL server in your case) for processing.
So, I suggest the following.
stop making your MySQL server's port available to people you don't know and trust.
get your game to gather keystrokes locally somehow.
get it to upload that data in big bunches when your game is not in realtime mode.
write a web service to receive and interpret these files.
That way you'll build a more secure analytics system and a more responsive game.
The question is a little general, so to help narrow the focus, I'll share my current setup that is motivating this question. I have a LAMP web service running a RESTful API. We have two client implementations: one browser-based javascript client (local storage store) and one iOS-based client (core data store). Obviously these two clients store data very differently, but the data itself needs to be kept in two-way sync with the remote server as often as possible.
Currently, our "sync" process is a little dumb (as in, non-smart). Conceptually, it looks like:
Client periodically asks the server for ALL of the most-recent data.
Server sends down the remote data, which overwrites the current set of local data in the client's store.
Any local creates/updates/deletes after this point are treated as gold, and immediately sent to the server.
The data itself is stored relationally, and updated occasionally by client users. The clients in my specific case don't care too much about the relationships themselves (which is why we can get away with local storage in the browser client for now).
Obviously this isn't true synchronization. I want to move to a system where, conceptually, a "diff" of the most recent changes are sent to the server periodically, and the server sends back a "diff" of the most recent changes it knows about. It seems very difficult to get to this point, but maybe I just don't understand the problem very well.
REST feels like a good start, but REST only talks about the way two data stores talk to each other, not how the data itself is synchronized between them. (This sync process is left up to the implementer of each store.) What is the best way to implement this process? Is there a modern set of programming design patterns that apply to inform a specific solution to this problem? I'm mostly interested in a general (technology agnostic) approach if possible... but specific frameworks would be useful to look at too, if they exist.
Multi-master replication is always (and will always be) difficult and bespoke, because how conflicts are handled will be specific to your application.
IMO A more robust approach is to use Master-slave replication, with your web service as the master and the clients as slaves. To keep the clients in sync, use an archived atom feed of the changes (see event sourcing) as per RFC5005. This is the closest you'll get to a modern standard for this type of replication and it's RESTful.
When the clients are online, they do not update their replica directly, instead they send commands to the server and have their replica updated via the atom feed.
When the clients are offline things get difficult. Your clients will need to have a model of how your web service behaves. It will need to have an offline copy of your replica, which should be copied on write from the online replica (the online replica is the one that is updated by the atom feed). When the client executes commands that modify the data, it should store the command (for later replay against the web service), the expected result (for verification during replay) and update the offline replica.
When the client goes back online, it should replay the commands, compare the result with the expected result and notify the client of any variances. How these variances are handled will vary based on your application. The offline replica can then be discarded.
CouchDB replication works over HTTP and does what you are looking to do. Once databases are synced on either end it will send diffs for adds/updates/deletes.
Couch can do this with other Couch machines or with a mobile framework like TouchDB.
https://github.com/couchbaselabs/TouchDB-iOS
I've done a fair amount of it, but you can always set up CouchDB on one machine, set up TouchDB on a mobile device and then watch the HTTP traffic go back and forth to get an idea of how they do it.
Or read this: http://guide.couchdb.org/draft/replication.html
Maybe something from the link above will help you get an idea of how to do your own diffs for your REST service. (Since they are both over HTTP thought it could be useful.)
You may want to look into the Dropbox Datastore API:
https://www.dropbox.com/developers/datastore
It sounds like it might be a very good fit for your purposes. They have iOS and javascript clients.
Lately, I've been interested in Meteor.
The platform sets up Mongo on the server and minimongo in the browser. The client subscribes to some data and when that data changes, the platform automatically sends down the new data to the client.
It's a clever solution to the syncing problem, and it solves several other problems as well. It will be interesting to see if more platforms do this in the future.