I have a distributed system that processes sessions (the definition of session is not important for this problem except to note that its a process that has a duration that is larger than a second, usually much larger), where I want to identify what is the largest number of sessions processed concurrently during a given time period.
The basic setup is a Redis database where I increment a counter for every session start and decrement it for every session end. The counter value thereby represents the current concurrency at any given point in time.
My problem is how to generate accurate metric of the peak (max) concurrency at given time slices (say, what was the max concurrency in a given day).
I would like to hear how other people would solve this problem, but my current approach is as follows:
Session start
INCR counter-name to increase the current value of the counter
The result of the increment command is the current value of the counter
ZADD collector-name NX <counterval> <uniqueid> to store the currently known concurrency value in an ordered set. Flake-id can be used for fast id generations, but if the session already has a unique ID - which often is the case - we can just use that.
Session end
DECR counter-name to reduce the current concurrency value
Each report time period
RENAME collector-name tempkey to take a snapshot of the status and allow workers to start a new collector.
ZREVRANGEBYSCORE tempkey +inf -inf WITHSCORES LIMIT 0 1 is run, returning the peak value of the counter since the last check (and the unique id of the session that caused the peak, if it is of any relevance).
DEL tempkey as we don't need it anymore.
Notes:
The final max calculation is done offline from the counter, and its also only O(log(n)).
The data entry is also O(log(n)), which might be a problem under high load, but n here is the number of entries in the current period so we can just increase our reporting frequency to improve performance (nice side effect - lets increase performance by generating more data!)
Are there any flaws in this setup that I've missed?
I'm not detecting any major flaws in the flow, but the choice of data structure could be improved.
Sorted Sets are comparatively expensive in terms of space and time, and your scenario doesn't utilize their special ability (i.e. ordering). More optimal structures would be a Hash of counters, or the highly-compressed BITFIELD.
Related
I want time points picked in the same running thread to never be equal. That's because I use time points to differentiate between different calculation results.
Pseudocode:
StampedResult fn() {
auto result = Calculations();
auto time_stamp = std::chrono::steady_clock::now();
return {time_stamp, result);
}
Now, if Calculations() was always complex, that would be auto-solved. But sometimes, Calculations() may return immediately.
So, I thought I should check if 2 sequential calls to steady_clock::now() can return the same value, like this:
https://onlinegdb.com/BkiDAZRe8
On onlinegdb.com and on my laptop Intel® Core™ i7-8750H CPU # 2.20GHz I never get the same value returned. But could some other super-high frequency processor actually return the same values given steady_clock's nanosecond accuracy?
Is it ensured that 2 sequential std::chrono::steady_clock::now() will not be equal?
It is not guaranteed by the standard. Quote from latest draft:
[time.clock.steady]
Objects of class steady_clock represent clocks for which values of time_point never decrease as physical time advances and for which values of time_point advance at a steady rate relative to real time.
That is, the clock may not be adjusted.
Staying the same satisfies "never decrease" requirement.
Clocks have limited granularity and if the granularity is less than frequency of calls to now, then it is feasible in theory that the value remains same between two calls. Complexity of the call is a practical limitation for the same value occurring again, but that is incidental.
If you wish to avoid duplicate value, then you can feasibly protect against the possibility by storing the last time stamp. If the new one is equal or less than the old, then nudge the new one up by one unit of measurement. The "less" part becomes possibility in case there are three equal values and second one was therefore nudged beyond the third.
steady_clock is required to not go backwards, and it is required to progress forwards at regular intervals. That's it. It isn't even required to have nanosecond accuracy (or for the clock's actual accuracy to match the precision of its time points).
For what you're doing, making your own internal counter that goes up every time you do a calculation is a better alternative.
No. There's no guarantee that two calls will not return the same value.
If you manage to query the clock multiple times within its resolution, you will get the same result multiple times.
How likely this is to happen depends on the clock you query, but it is always a possibility.
In short; you cannot use a timestamp as a unique identifier. You probably could use "timestamp + thread_id" if thread IDs are not re-used. Maybe better to sort on timestamp first, then secondly on a monotonic incrementing id assigned to each thread upon its creation. That would guarantee uniqueness and establish a fixed sort order.
Apologies in advance as I think I need to give a background of my problem.
We have a proprietary database engine written in native c++ built for 32bit runtime, the database records are identified by their record number (basically offset in the file where a record is written) and a "unique id" (which is nothing more than -100 to LONG_MIN).
Previously the engine limits a database to only 2gb (where block of record could be a minimum size of 512bytes up to 512*(1 to 7)). This effectively limits the number of records to about 4 million.
We are indexing this 4 million records and storing the index in a hashtable (we implemented an extensible hashing for this) and works brilliantly for 2gb database. Each of the index is 24bytes each. each record's record number is indexed as well as the record's "unique id" (the indexes reside in the heap and both record number and "unique id" can point to the same index in heap). The indexes are persisted in memory and stored in the file (however only the record number based indexes are stored in file). While in memory, a 2gb database's index would consume about 95mb which still is fine in a 32bit runtime (but we limited the software to host about 7 databases per database engine for safety measure)
The problem begins when we decided to increases the size of the database from 2gb to 32gb. This effectively increased the number of records to about 64 million, which would mean the hashtable would contain 1.7gb worth of index in heap memory for a single 32gb database alone.
I ditched the in memory hashtable and wrote the index straight to a file, but I failed to consider the time it would take to search for an index in the file, considering I could not sort the indexes on demand (because write to the database happens all the time which means the indexes must be updated almost immediately). Basically I'm having problems with re-indexing, that is our software needs to check if a record exist and it does so by checking the current indexes if it resides there, but since I now changed it from in-memory to file I/O index, its now taking forever just to finish 32gb indexing (2gb indexing as I have computed it will apparently take 3 days to complete).
I then decided to store the indexes in order based on record number so I dont have to search them in file, and structure my index as such:
struct node {
long recNum; // Record Number
long uId; // Unique Id
long prev;
long next;
long rtype;
long parent;
}
It works perfectly if I use recNum to determine where in file the index record is stored and retrieves it using read(...), but my problem is if the search based on "unique id".
When I do a search on the index file based on "unique id", what I'm doing essentially is loading chunks of the 1.7gb index file and checking the "unique id" until I get a hit, however this proves to be a very slow process. I attempted to create an Index of the Index so that I could loop quicker but it still is slow. Basically, there is a function in the software that will eventually check every record in the database by checking if it exist in the index first using the "unique id" query, and if this function comes up, to finish the 1.7gb index takes 4 weeks in my calculation if I implement a file based index query and write.
So I guess what 'm trying to ask is, when dealing with large databases (such as 30gb worth of database) persisting the indexes in memory in a 32bit runtime probably isn't an option due to limited resource, so how does one implement a file based index or hashtable with out sacrificing time (at least not so much that its impractical).
It's quite simple: Do not try to reinvent the wheel.
Any full SQL database out there is easily capable of storing and indexing tables with several million entries.
For a large table you would commonly use a B+Tree. You don't need to balance the tree on every insert, only when a node exceeds the minimum or maximum size. This gives a bad worst case runtime, but the cost is amortized.
There is also a lot of logic involved in efficiently, dynamically caching and evicting parts of the index in memory. I strongly advise against trying to re-implement all of that on your own.
I am learning C++ on my own and I would like to work a tiny project to improve my C++ skills.
I am implementing a program to have the capability of recording a number of different experiment tasks, each of which has a pre-defined time limit represented by integer numbers in time unit.
Different kinds of tasks can start at the same time or sequentially.
One kind of task can only run one at a time.
The tasks run normally with a duration of 24 hours at max.
The computer running the program have to suffer rebooting during the tasks within one day, for some reason. (so have to write data to disk)
One task can start and stop multiple times, so the time_elapsed is accumulated.
For example:
tasks, time_limit(time unit), time_elapsed(time unit).
t0, 500, 0
t1, 1000, 700
t2, 700, 700
As you see, the project is just a timer. But I am not sure how to store the three parameters to my program; I have
string: task names;
integer: time limit, and
integer: time elapsed.
Basically there are three variants for the project asking for different ways (your suggestions) to store the above data:
total task numbers are 10-100; start-stop cycle AND query interval can be minutes apart, and no memory requirement;
total task numbers are 10^100; start-stop cycle AND query interval can be millisecond scale, and no memory requirement;
total task numbers are 100!; start-stop cycle AND query interval can be milisecond scale, and with as minimum mem as possible.
I have several solutions:
zeroth, I just have all the three variables in three primitive [] arrays, read / write files whenever updating it via array position-based indexing.
first, using vectors instead of arrays in the above case.
second, with unordered_map, set task-name as key and set time_limit, time_elapsed pair<int, int> as the value. whenever updating the record, input a task name(string) to update the corresponding integer value in the pair (C++ STL).
third, using a tuple <string, int, int> to store <task_name, time_limit, time_elapsed> respectively.
fourth, using C struct arrays, to have something like
struct {
string task name;
int time_limit
int time_elapsed;};
Or any other types of data structure.
EDIT:
To query the data structure, current status at any time need to be transmitted to some other class objects of the program to trigger corresponding events(which is omitted in this post), thus a list of each unfinished (time_elapsed is less then time_limit) tasks' task_name, elapsed time and time left are read as input for the other class objects.
To think this way, it will help me learn a lot.
I would like to hear comments / suggests how to choose an efficient data structures based on different requirement listed above.
Thanks
Ok so the problem is that you have a collection of triples {task_name, time_limit, time_elapsed}. You frequently need to find a triple given its task_name, so as to update time_elapsed. You need to insert new triples. And when a task expires (time_elapsed >= time_limt) you should remove it from the collection (and possibly add to another collection), so as to be able to print the non-expired tasks.
You could do something like this:
#include <set>
#include <string>
struct Task {
std::string task_name;
size_t time_limit;
size_t time_elapsed;
};
struct TaskLess {
bool operator()(const Task& task1, const Task& task2) {
return task1.task_name < task2.task_name;
}
};
std::set<Task, TaskLess> tasks;
To find/update/insert/delete a task given its task_name is a logarithmic search. To print non-expired tasks you iterate through the set.
You could also consider further optimizations: (a) use an std::unordered_map instead of a set; (b) use an integer as a task key instead of a string.
["Hitting a moving target" edit]
The answer below is offered for the scenario 1 - samples in the thousands or so.
Samples in the 10^100 or 100! ranges - unless you have a multiverse-sized computer better forget about it (number of atoms in the our observable universe is something around 10^80 - 10^81). Besides, I don't think you'd have time to accumulate such a high number of samples, given that the age of our universe is 13.8 billion years (which gives a magnitude order of 10^20 milliseconds count since the BigBang).
As of 2015, the entire CERN data size is a meagre 25 petabytes - that's less than 10^16.
Based on the scenario of usage (collection time at minutes apart, query for incomplete tasks), I'd go with your fourth solution, storing these structs in a std:vector - the number of entries to be stored/processed can't be that big so no need for special containers or a database (20 experiments running a month with 10 mins collection period will result in a bit over 3600 entries).
The query side, C++11 style, artificially blown up code (i.e. not the most compact form) to make evident what is happening:
std::vector<MyStruct> allSamples;
// assuming a code that fills allSamples up
// this is a lambda function, used as a predicate
auto unfinishedTest=[](const MyStruct& s) {
return s.time_elapsed < s.time_limit;
}
auto unfinishedIterator=std::find_if(
allSamples.begin(), allSamples.end(), // range of lookup
unfinishedTest // filtering predicate
);
// now you have an iterator which only gives you unfinished tests
while(unfinishedIterator != allSamples.end()) {
const MySample& unfinishedJobAtHand = *unfinishedIterator;
// do what you please
}
References:
std:: containers with std::vector in particular
find_if specification
lamda functions syntax
Hi imagine I have such code:
0. void someFunction()
1. {
2. ...
3. if(x>5)
4. doSmth();
5.
6. writeDataToCard(handle, data1);
7.
8. writeDataToCard(handle, data2);
9.
10. incrementDataOnCard(handle, data);
11. }
The thing is following. If step 6 & 8 gets executed, and then someone say removes the card - then operation 10 will not be completed successfully. But this will be a bug in my system. Meaning if 6 & 8 are executed then 10 MUST also be executed. How to deal with such situations?
Quick Summary: What I mean is say after step 8 someone may remove my physical card, which means that step 10 will never be reached, and that will cause a problem in my system. Namely card will be initialized with incomplete data.
You will have to create some kind of protcol, for instance you write to the card a list of operatons to complete:
Step6, Step8, Step10
and as you complete the tasks you remove the entry from the list.
When you reread the data from the disk, you check the list if any entry remains. If it does, the operation did not successfully complete before and you restore a previous state.
Unless you can somehow physically prevent the user from removing the card, there is no other way.
If the transaction is interrupted then the card is in the fault state. You have three options:
Do nothing. The card is in fault state, and it will remain there. Advise users not to play with the card. Card can be eligible for complete clean or format.
Roll back the transaction the next time the card becomes available. You need enough information on the card and/or some central repository to perform the rollback.
Complete the transaction the next time the card becomes available. You need enough information on the card and/or some central repository to perform the completion.
In all three cases you need to have a flag on the card denoting a transaction in progress.
More details are required in order to answer this.
However, making some assumption, I will suggest two possible solutions (more are possible...).
I assume the write operations are persistent - hence data written to the card is still there after card is removed-reinserted, and that you are referring to the coherency of the data on the card - not the state of the program performing the function calls.
Also assumed is that the increment method, increments the data already written, and the system must have this operation done in order to guarantee consistency:
For each record written, maintain another data element (on the card) that indicates the record's state. This state will be initialized to something (say "WRITING" state) before performing the writeData operation. This state is then set to "WRITTEN" after the incrementData operation is (successfully!) performed.
When reading from the card - you first check this state and ignore (or delete) the record if its not WRITTEN.
Another option will be to maintain two (persistent) counters on the card: one counting the number of records that began writing, the other counts the number of records that ended writing.
You increment the first before performing the write, and then increment the second after (successfully) performing the incrementData call.
When later reading from the card, you can easily check if a record is indeed valid, or need to be discarded.
This option is valid if the written records are somehow ordered or indexed, so you can see which and how many records are valid just by checking the counter. It has the advantage of requiring only two counters for any number of records (compared to 1 state for EACH record in option 1.)
On the host (software) side you then need to check that the card is available prior to beginning the write (don't write if its not there). If after the incrementData op you you detect that the card was removed, you need to be sure to tidy up things (remove unfinished records, update the counters) either once you detect that the card is reinserted, or before doing another write. For this you'll need to maintain state information on the software side.
Again, the type of solution (out of many more) depends on the exact system and requirements.
Isn't that just:
Copy data to temporary_data.
Write to temporary_data.
Increment temporary_data.
Rename data to old_data.
Rename temporary_data to data.
Delete the old_data.
You will still have a race condition (if a lucky user removes the card) at the two rename steps, but you might restore the data or temporary_data.
You haven't said what you're incrementing (or why), or how your data is structured (presumably there is some relationship between whatever you're writing with writeDataToCard and whatever you're incrementing).
So, while there may be clever techniques specific to your data, we don't have enough to go on. Here are the obvious general-purpose techniques instead:
the simplest thing that could possibly work - full-card commit-or-rollback
Keep two copies of all the data, the good one and the dirty one. A single byte at the lowest address is sufficient to say which is the current good one (it's essentially an index into an array of size 2).
Write your new data into the dirty area, and when that's done, update the index byte (so swapping clean & dirty).
Either the index is updated and your new data is all good, or the card is pulled out and the previous clean copy is still active.
Pro - it's very simple
Con - you're wasting exactly half your storage space, and you need to write a complete new copy to the dirty area when you change anything. You haven't given enough information to decide whether this is a problem for you.
... now using less space ... - commit-or-rollback smaller subsets
if you can't waste 50% of your storage, split your data into independent chunks, and version each of those independently. Now you only need enough space to duplicate your largest single chunk, but instead of a simple index you need an offset or pointer for each chunk.
Pro - still fairly simple
Con - you can't handle dependencies between chunks, they have to be isolated
journalling
As per RedX's answer, this is used by a lot of filesystems to maintain integrity.
Pro - it's a solid technique, and you can find documentation and reference implementations for existing filesystems
Con - you just wrote a modern filesystem. Is this really what you wanted?
Here's a piece of code I've spent the last 2 days optimizing and profiling because it was taking too much time:
{
mongo::ScopedDbConnection _dbConnection (DbHost);
_dbConnection->insert(TokensDB, tokensArray );
_dbConnection.done();
}
{
mongo::ScopedDbConnection _dbConnection (DbHost);
_dbConnection->insert(IdxDB, postingsArray);
_dbConnection.done();
}
Here postingsArray is std::vector<BSON (int64_t, int64_t, int64_t, int)>, 20 000 elements. This insert always takes only a couple of milliseconds. tokensArray is std::vector<BSON (int64_t, std::string)>, 5000 elements. This is the odd insert.
If I do it exactly as in the code fragment above, it takes 45-50 ms. But if I switch the two blocks around as it initially was (insert to IdxDB first and TokensDB second) it takes 400-500 ms. What is going on here? Why does order matter? Why is inserting 5000 2-field records taking much longer than inserting 20k 4-field objects?
My initial idea is it's because of std::string field (it holds single english word, so about 5-7 symbols on average). I've replaced it with random int64_t number - no noticeable change in insert completion time.
All the profiling is done on a clean database and with exactly the same data every time, I don't believe it's my error in organizing the measurements.
MongoDB performs a lot of things in the background so it is normal that the insertion of the large postingsArray takes little time but affects the performance after that. When you measure the postingsArray insert alone you are only measuring the time it takes for the MongoDB driver to accept the insert. But when you measure the consequent operations you begin to notice the background workload started by the postingsArray insert.
See point 6 there: http://article.gmane.org/gmane.comp.db.mongodb.user/818
BTW, the way your example written I would suspect MongoDB gives you the same connection for the inserts. (E.g. you might be taking a connection from the pool, inserting the postingsArray with it, releasing it, then taking the same connection from the pool again and inserting the tokensArray with it). In that case the TCP/IP socket might still be busy with the postingsArray insert and what you're seeing might be hitting the limit on the TCP/IP buffer.
P.S. You might want to change the write concern in order to measure the actual time it takes for the MongoDB to perform the insert: http://article.gmane.org/gmane.comp.db.mongodb.user/68288