we have an application which utilizes a UDP service , and observe that 75% of calls to this UDP service is repeated.
Hence would wish to apply an In-Memory Cache , so as to avoid the costly network call and improve the application's turn-around time.
Never used caching before , hence any pointers on suitable cache libraries in C++/Unix environ would greatly help.
Also would like to share this cache across multiple processes.
The cache is required to store key value pairs of string type.
Have a look at redis, it's a noSQL key-value database. Here you have an interactive tutorial. We use it in several of our applications successfully.
Gemfire (no relation) is a commercial distributed caching system. Servers are written in Java but native support exists for C++ (among others).
Related
I am writing a time tracking Windows application in C++ that uses sqlite3 engine to store its data. For my purpose it would be nice to share the database file across the local network (in a Windows network share folder) among several copies of my application, so that multiple users of the software could share data.
Is there a mechanism to do that with SQLite?
"nice to share the database file across the local network" You really don't want to do that. It will end up being more trouble than it's worth. In ideal circumstances it works, although the performance sucks a bit. In non-ideal circumstances, it will block forever without giving you any idea why and what's at fault.
It's much easier to partition your system into a server and a client. They can both run within the same application. When the application starts, it checks if there are any servers on the local network, and if there aren't, it starts one. It then connects to the server.
That's what Filemaker at least used to do 20 years ago, and it worked pretty well. Should be a breeze to implement using modern frameworks today (say Qt or boost).
In my Visual C++ application, I want to allocate a lot of objects, which will use up all available memory in the system. To solve this problem, I decide to store the objects in database. I just have 3 candidates: MySQL, PostgreSQL, and SQLite. But don’t know which one is more appropriate.
What I need is:
Store objects in the database instead of memory.
Fast to find the objects via a key.
Light-weight so the RDBMS will not require a lot of system resources, including both the memory and disk spaces.
No server required.
Easy to deploy.
Which one should be best for my needs? Of course, if you have any other better alternatives, then just tell me.
SQLite provides a detailed doc how when it should be used. But MySQL and PostgreSQL does not so it is a little difficult to choose as I am not familiar with these two. Thanks.
I'd use SQLite. It doesn't require a service and is cross platform. It is easy to deploy and is light-weight. It supports transaction. It's in the public domain.
Your questions:
Store objects in the database instead of memory.
Any database can do this, that's the definition of a database.
Fast to find the objects via a key.
Also standard functionality, if you can't find your data, what's the point of using a database.
Light-weight so the RDBMS will not require a lot of system
resources, including both the memory and disk spaces.
That's mostly in your hands, bad queries generate a lot of overhead. No matter what brand of database (or software language) you use.
No server required.
Do you mean "hardware" or "client-server model" ? Both MySQL and PostgreSQL are services in a client-server model. SQLite works best for a single client.
Easy to deploy.
All 3 databases are easy to deploy, but SQLite is the easiest one. It's not a server like the others.
It looks like SQLite is the best fit, but also check your other requirements, the ones you didn't mention: performance, reliability, backup, failover, etc. etc. And do you needs an RDBMS for this kind of work? A C++ object in memory is very different from a bunch of records in a couple of databases that can be accessed by using SQL.
I want to create an application that, when executed, has runtime functions that are accessible by other applications.
For example, a C++ application that stores values in files and retrieves this information. While this application is running, any other C++ applications could access it's save and retrieve functionality to save and retrieve data, but it should have no other connection to this system.
Sounds like a simple job for web services, or a remote database, or even an LDAP server.
Store and retrieve are operations common to all of these.
If the goal is to learn some specific technology, then ask a more specific question. Otherwise, don't reinvent any wheels. There are plenty of things out there for store and retrieve.
One of the simplest "store and retrieve" APIs I know of is Berkeley DB or Sleepycat.
We built a giant, clustered, simple key based database for a major telecom company using LDAP on top of Berkeley DB (aka Sleepycat). All open-source software and commodity hardware and it supports mission critical operations for millions of customers.
A more modern rendition of this might use memcached as well.
If you go HTTP based, you can use something simple as libcurl against an Apache web server to implement "RESTful" services with GET and PUT commands.
If you run it locally (same server), and access via localhost (127.0.0.1) then there is very little latency in the TCP stack, and it amounts to little more than memcpys at the kernel level.
simple message passing would do, say, JSON over ØMQ, or i.e. all in all, msgpack-rpc or protobuf-remote or Cap'n Proto RPC
I want to know whether I can use Infinispan for cached data synchronization with Oracle database. This is my scenario. I have two main applications. One is highly concurrent use app and the second is used as admin module. Since it is highly concurrent I want to reduce database connections (Load the entities into cache (read write enable) and use it from this place without calling database). But meantime I want to update database according to the cache changes because admin module is using database directly. Can that update process (cache to database) handle in entity level without involving application? Please let me know whether Infinispan supports this scenario or not. If supports please share ideas.
Yes, it is possible. Infinispan supports this use case.
This should be simple configuration "problem" only. All you need to use is properly configured CacheStore with passivation disabled. It will keep your cache (used by highly concurrent application) synchronized with database.
What does it exactly cause?
When passivation is disabled, whenever an element is modified, added
or removed, then that modification is persisted in the backend store
via the cache loader. There is no direct relationship between eviction
and cache loading. If you don't use eviction, what's in the persistent
store is basically a copy of what's in memory.
By memory is meant a cache here.
If you want to know even more about this and about other interesting options please see: https://docs.jboss.org/author/display/ISPN/Cache+Loaders+and+Stores#CacheLoadersandStores-cachepassivation
Maybe it's worth to consider aforementioned eviction. Whether to disable or enable it. It depends mainly on load generated by your highly concurrent application.
Actually, this only works when you use Infinispan in the same cluster for the admin module. If you load A in memory with Infinispan, change A to something else in the database directly with the admin module, then Infinispan will not know A has been updated.
So if for example I am trying to implement something that looks like Facebook's Graph API that needs to be very quick and support millions of users, what is the disadvantage of just using Redis instead of a RDBMS?
Thanks!
Jonathan
There are plenty of potential benefits and potential drawbacks of using Redis instead of a classical RDBMS. They are very different beasts indeed.
Focusing only on the potential drawbacks:
Redis is an in-memory store: all your data must fit in memory. RDBMS usually stores the data on disks, and cache part of the data in memory. With a RDBMS, you can manage more data than you have memory. With Redis, you cannot.
Redis is a data structure server. There is no query language (only commands) and no support for a relational algebra. You cannot submit ad-hoc queries (like you can using SQL on a RDBMS). All data accesses should be anticipated by the developer, and proper data access paths must be designed. A lot of flexibility is lost.
Redis offers 2 options for persistency: regular snapshotting and append-only files. None of them is as secure as a real transactional server providing redo/undo logging, block checksuming, point-in-time recovery, flashback capabilities, etc ...
Redis only offers basic security (in term of access rights) at the instance level. RDBMS all provide fine grained per-object access control lists (or role management).
A unique Redis instance is not scalable. It only runs on one CPU core in single-threaded mode. To get scalability, several Redis instances must be deployed and started. Distribution and sharding are done on client-side (i.e. the developer has to take care of them). If you compare them to a unique Redis instance, most RDBMS provide more scalability (typically providing parallelism at the connection level). They are multi-processed (Oracle, PostgreSQL, ...) or multi-threaded (MySQL, Microsoft SQL Server, ... ), taking benefits of multi-cores machines.
Here, I have only described the main drawbacks, but keep in mind there are also plenty of benefits in using Redis (very fast, good concurrency support, low latency, protocol pipelining, good to easily implement optimistic concurrent patterns, good usability/complexity ratio, excellent support from Salvatore and Pieter, pragmatic no-nonsense approach, ...)
For your specific problem (graph), I would suggest to have a look at neo4J or OrientDB which are specifically designed to store graph-oriented data.
I have some additions:
There is a value length limitations in redis. When using redis, you always think about your redis K,V size, especially in redis cluster