Random function call from multiple threads in Qt/C++ - c++

I have a multi-thread QT application that sometimes need a random alphanumeric string from one of its threads (some threads start at application startup, others start or die during lifetime), and I would like to obtain that by calling a function defined in a common header, to avoid code replication.
Here there's a code snippet:
QString generateRandomAlphanumericString(int length)
{
qsrand(static_cast<uint>(QTime::currentTime().msec())); //bad
QString randomAS = QString();
static const char alphanum[] =
"0123456789"
"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
"abcdefghijklmnopqrstuvwxyz";
for (int i = 0; i < length; ++i)
randomAS[i] = alphanum[qrand() % (sizeof(alphanum) - 1)];
return randomAS;
}
I initially did some mistakes.
At the beginning I called qsrand(static_cast<uint>(QTime::currentTime().msec())); in the main function, but I've learned that it should be done per-thread.
Then I put the qsrand call in the function above, but it's not correct.
Please consider that at program startup many threads start "together", so if I initialize the seed with current time in msec the seed is the same among them.
Is there a way to change that function accordingly without modify all points in my application where a thread starts its life?
Any implementation done in pure C++ (without the use of QT) is fine. Could the new random C++11 library help in some way to achieve my task?

void InitSeedForThread(uint globalSeed, int myThreadIndex)
{
qsrand(globalSeed);
for (int i = 0; i < myThreadIndex; ++i)
qrand();
}
auto GetRandom(int numThreads)
{
for (int i = 0; i < numThreads - 1)
qrand();
return qrand();
}
Given an ordered list of numbers A, B, C, D, E, F, G, H, ...
splits it into n lists. If n was 4, you would get
1. A, E, I, ...
2. B, F, J, ...
3. C, G, K, ...
4. D, H, L, ...
Con: Doing RNG is somewhat expensive, and you're repeating a lot of work. However, since you're doing QT (UI-bound) I'm assuming that performance isn't an issue.
Alternatively, you could do a global random function with a mutex, but that ain't free either.

I finally found a good solution (thanks everybody who has contributed with comments):
enum ThreadData {TD_SEED};
static QThreadStorage<QHash<ThreadData, uint> *> cache;
inline void insertIntoCache(ThreadData data, uint value)
{
if (!cache.hasLocalData())
cache.setLocalData(new QHash<ThreadData, uint>);
cache.localData()->insert(data, value);
}
inline void removeFromCache(ThreadData data)
{
if (cache.hasLocalData())
cache.localData()->remove(data);
}
inline bool hasInCache(ThreadData data)
{
if (!cache.hasLocalData()) return false;
return cache.localData()->contains(data);
}
inline uint getCachedData(ThreadData data)
{
if (cache.hasLocalData() && cache.localData()->contains(data))
return cache.localData()->value(data);
return 0;
}
inline int getThRandom()
{
uint seed = 0;
if (!hasInCache(TD_SEED))
{
seed = QDateTime::currentMSecsSinceEpoch() % 100000000;
#ifdef Q_OS_WIN
seed += GetCurrentThreadId();
#else
seed += QThread::currentThreadId();
#endif
qsrand(static_cast<uint>(seed));
insertIntoCache(TD_SEED, seed);
}
else {
seed = getCachedData(TD_SEED);
}
return qrand();
}
Basically, as suggested by Igor I've made use of QThreadStorage to store a seed for each thread. I've used an hash for future extensions.
Then, I've made use of QDateTime::currentMSecsSinceEpoch() instead of QTime::currentTime().msec() to have a different number across multiple application starts (if for example the random generated value is stored in a file/db and should be different).
Then, I've add an offset, as suggested by UKMonkey, using the thread ID.
So, my original function will be:
QString generateRandomAlphanumericString(int length)
{
QString randomAS = QString();
static const char alphanum[] =
"0123456789"
"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
"abcdefghijklmnopqrstuvwxyz";
for (int i = 0; i < length; ++i)
randomAS[i] = alphanum[getThRandom() % (sizeof(alphanum) - 1)];
return randomAS;
}
I've run some tests, producing from different threads thousand of alphanumeric strings, storing them to multiple files and double checked for duplicates among them and between multiple application run.

Related

OpenMP: how to optimize a thread-safe hash-table

I'm trying to make a hash table (unordered_map) thread-safe, for a performance-critical application. I'm using OpenMP to handle multithreading. The first solution I tried was to make all access to the hash-table critical, but this ended up lowering the performance of threads so much that the parallel version wasn't faster anymore. I then came with the following idea: make 100 separate hash-tables, initialized to empty set, and store them in an array. Use some hash function to map the key to the range [0, 1, ..., 99], and store the value in the corresponding hash table. Here is a pseudo-code to demonstrate this idea:
auto hash_vec = vector<unordered_map<int,int> >(100);
int prime = 534161472029; // some big prime
int rand_int = rand(); // random int used for
int simple_hash(int v) {
return (rand_int * v) % prime;
}
void safe_write(int key, int val) {
int h = simple_hash(key) % 100;
hash_vec[h][key] = val; // must be made safe
}
int safe_read() {
int h = simple_hash(key) % 100;
int val = hash_vec[h][key]; // must be made safe
return val;
}
so the idea is that if value of h is not the same, there won't be any race condition. For example, if simple_hash(k1) is not equal to simple_hash(k2), a simultaneous call to safe_write(k1,v1) and safe_write(k2,v2) won't cause any thread to wait. Is such a thing possible? If not, is there a better way to ensure performance is not too compromised?

Searching for an exact string match in a (arbitrary large) stream - C++

I am building a simple multi-server for string matching. I handle multiple clients at the same time by using sockets and select. The only job that the server does is this: a client connects to a server and sends a needle (of size less than 10 GB) and a haystack (of arbitrary size) as a stream through a network socket. Needle and haystack are an arbitrary binary data.
Server needs to search the haystack for all occurrences of the needle (as an exact string match) and sends a number of needle matches back to the client. Server needs to process clients on the fly and be able to handle any input in a reasonable time (that is a search algorithm have to have a linear time complexity).
To do this I obviously need to split the haystack into a small parts (possibly smaller than the needle) in order to process them as they are coming through the network socket. That is I would need a search algorithm that is able to handle a string, that is split into parts and search in it, the same way as strstr(...) does.
I could not find any standard C or C++ library function nor a Boost library object that could handle a string by parts. If I am not mistaken, algorithms in strstr(), string.find() and Boost searching/knuth_morris_pratt.hpp are only able to handle the search, when a whole haystack is in a continuous block of memory. Or is there some trick, that I could use to search a string by parts that I am missing? Do you guys know of any C/C++ library, that is able to cope with such a large needles and haystacks resp. that is able to handle haystack streams or search in haystack by parts?
I did not find any useful library by googling and hence I was forced to create my own variation of Knuth Morris Pratt algorithm, that is able to remember its own state (shown bellow). However I do not find it to be an optimal solution, as a well tuned string searching algorithm would surely perform better in my opinion, and it would be a less worry for a debugging later.
So my question is:
Is there some more elegant way to search in a large haystack stream by parts, other than creating my own search algorithm? Is there any trick how to use a standard C string library for this? Is there some C/C++ library that is specialized for a this kind of task?
Here is a (part of) code of my midified KMP algorithm:
#include <cstdlib>
#include <cstring>
#include <cstdio>
class knuth_morris_pratt {
const char* const needle;
const size_t needle_len;
const int* const lps; // a longest proper suffix table (skip table)
// suffix_len is an ofset of a longest haystack_part suffix matching with
// some prefix of the needle. suffix_len myst be shorter than needle_len.
// Ofset is defined as a last matching character in a needle.
size_t suffix_len;
size_t match_count; // a number of needles found in haystack
public:
inline knuth_morris_pratt(const char* needle, size_t len) :
needle(needle), needle_len(len),
lps( build_lps_array() ), suffix_len(0),
match_count(len == 0 ? 1 : 0) { }
inline ~knuth_morris_pratt() { free((void*)lps); }
void search_part(const char* haystack_part, size_t hp_len); // processes a given part of the haystack stream
inline size_t get_match_count() { return match_count; }
private:
const int* build_lps_array();
};
// Worst case complexity: linear space, linear time
// see: https://www.geeksforgeeks.org/kmp-algorithm-for-pattern-searching/
// see article: KNUTH D.E., MORRIS (Jr) J.H., PRATT V.R., 1977, Fast pattern matching in strings
void knuth_morris_pratt::search_part(const char* haystack_part, size_t hp_len) {
if(needle_len == 0) {
match_count += hp_len;
return;
}
const char* hs = haystack_part;
size_t i = 0; // index for txt[]
size_t j = suffix_len; // index for pat[]
while (i < hp_len) {
if (needle[j] == hs[i]) {
j++;
i++;
}
if (j == needle_len) {
// a needle found
match_count++;
j = lps[j - 1];
}
else if (i < hp_len && needle[j] != hs[i]) {
// Do not match lps[0..lps[j-1]] characters,
// they will match anyway
if (j != 0)
j = lps[j - 1];
else
i = i + 1;
}
}
suffix_len = j;
}
const int* knuth_morris_pratt::build_lps_array() {
int* const new_lps = (int*)malloc(needle_len);
// check_cond_fatal(new_lps != NULL, "Unable to alocate memory in knuth_morris_pratt(..)");
// length of the previous longest prefix suffix
size_t len = 0;
new_lps[0] = 0; // lps[0] is always 0
// the loop calculates lps[i] for i = 1 to M-1
size_t i = 1;
while (i < needle_len) {
if (needle[i] == needle[len]) {
len++;
new_lps[i] = len;
i++;
}
else // (pat[i] != pat[len])
{
// This is tricky. Consider the example.
// AAACAAAA and i = 7. The idea is similar
// to search step.
if (len != 0) {
len = new_lps[len - 1];
// Also, note that we do not increment
// i here
}
else // if (len == 0)
{
new_lps[i] = 0;
i++;
}
}
}
return new_lps;
}
int main()
{
const char* needle = "lorem";
const char* p1 = "sit voluptatem accusantium doloremque laudantium qui dolo";
const char* p2 = "rem ipsum quia dolor sit amet";
const char* p3 = "dolorem eum fugiat quo voluptas nulla pariatur?";
knuth_morris_pratt searcher(needle, strlen(needle));
searcher.search_part(p1, strlen(p1));
searcher.search_part(p2, strlen(p2));
searcher.search_part(p3, strlen(p3));
printf("%d \n", (int)searcher.get_match_count());
return 0;
}
You can have a look at BNDM, which has same performances as KMP:
O(m) for preprocessing
O(n) for matching.
It is used for nrgrep, the sources of which can be found here which containts C sources.
C source for BNDM algo are here.
See here for more information.
If I have well understood your problem, you want to search if a large std::string received part by part contains a substring.
If it is the case, I think you can store for each iteration the overlapping section between two contiguous received packets. And then you just have to check for each iteration that either the overlap or the packet contains the desired pattern to find.
In the example below, I consider the following contains() function to search a pattern in a std::string:
bool contains(const std::string & str, const std::string & pattern)
{
bool found(false);
if(!pattern.empty() && (pattern.length() < str.length()))
{
for(size_t i = 0; !found && (i <= str.length()-pattern.length()); ++i)
{
if((str[i] == pattern[0]) && (str.substr(i, pattern.length()) == pattern))
{
found = true;
}
}
}
return found;
}
Example:
std::string pattern("something"); // The pattern we want to find
std::string end_of_previous_packet(""); // The first part of overlapping section
std::string beginning_of_current_packet(""); // The second part of overlapping section
std::string overlap; // The string to store the overlap at each iteration
bool found(false);
while(!found && !all_data_received()) // stop condition
{
// Get the current packet
std::string packet = receive_part();
// Set the beginning of the current packet
beginning_of_current_packet = packet.substr(0, pattern.length());
// Build the overlap
overlap = end_of_previous_packet + beginning_of_current_packet;
// If the overlap or the packet contains the pattern, we found a match
if(contains(overlap, pattern) || contains(packet, pattern))
found = true;
// Set the end of previous packet for the next iteration
end_of_previous_packet = packet.substr(packet.length()-pattern.length());
}
Of course, in this example I made the assumption that the method receive_part() already exists. Same thing for the all_data_received() function. It is just an example to illustrate the idea.
I hope it will help you to find a solution.

Function always returns 1

I´m trying to write a simple branch predictor that should output either TAKEN (1) or NOT_TAKEN (0) depending on history stored in int. However it always outputs TAKEN instead of dynamicaly changing the prediction.
#define PHT_CTR_MAX 3
#define PHT_CTR_INIT 2
class PREDICTOR{
private:
UINT32 counter;
public:
PREDICTOR(void);
bool GetPrediction(UINT64 PC);
void UpdatePredictor(UINT64 PC, OpType opType, bool resolveDir, bool predDir, UINT64 branchTarget);
};
PREDICTOR::PREDICTOR(void){
counter = PHT_CTR_INIT;
}
bool PREDICTOR::GetPrediction(UINT64 PC){
if(counter > (PHT_CTR_MAX/2)){
return TAKEN;
}else{
return NOT_TAKEN;
}
}
void PREDICTOR::UpdatePredictor(UINT64 PC, OpType opType, bool resolveDir, bool predDir, UINT64 branchTarget){
if(resolveDir == TAKEN){
SatIncrement(counter, PHT_CTR_MAX);
}else{
SatDecrement(counter);
}
}
PREDICTOR::PREDICTOR is used to "build" the predictor (create arrays, set initial values...), it is called right in the beginning.
PREDICTOR::GetPrediction should return either TAKEN (when counter = 3 or 2) or NOT_TAKEN (when counter = 0 or 1).
PREDICTOR::UpdatePredictor is called after GetPrediction. It updates the predictor via resolveDir - resolveDir is the actual direction of the branch.
If resolveDir = 1 it does saturated increment of counter (saturated means it never exceeds PHT_CTR_MAX).
If resolveDir = 0 it decrements the counter.
Although this predictor is really simple it does not work. It throws out exactly same results as if I just did GetPrediction{return TAKEN} which is obviously wrong. My coding skills aren´t really great so I might have done something wrong - probably in the GetPrediction or UpdatePredictor function.
Here is an example of predictor that works just fine, although this one is little bit more complex:
#define PHT_CTR_MAX 3
#define PHT_CTR_INIT 2
#define HIST_LEN 17
class PREDICTOR{
private:
UINT32 ghr; // global history register
UINT32 *pht; // pattern history table
UINT32 historyLength; // history length
UINT32 numPhtEntries; // entries in pht
public:
PREDICTOR(void);
bool GetPrediction(UINT64 PC);
void UpdatePredictor(UINT64 PC, OpType opType, bool resolveDir, bool predDir, UINT64 branchTarget);
PREDICTOR::PREDICTOR(void){
historyLength = HIST_LEN;
ghr = 0;
numPhtEntries = (1<< HIST_LEN);
pht = new UINT32[numPhtEntries];
for(UINT32 ii=0; ii< numPhtEntries; ii++){
pht[ii]=PHT_CTR_INIT;
}
}
bool PREDICTOR::GetPrediction(UINT64 PC){
UINT32 phtIndex = (PC^ghr) % (numPhtEntries);
UINT32 phtCounter = pht[phtIndex];
if(phtCounter > (PHT_CTR_MAX/2)){
return TAKEN;
}
else{
return NOT_TAKEN;
}
}
void PREDICTOR::UpdatePredictor(UINT64 PC, OpType opType, bool resolveDir, bool predDir, UINT64 branchTarget){
UINT32 phtIndex = (PC^ghr) % (numPhtEntries);
UINT32 phtCounter = pht[phtIndex];
if(resolveDir == TAKEN){
pht[phtIndex] = SatIncrement(phtCounter, PHT_CTR_MAX);
}else{
pht[phtIndex] = SatDecrement(phtCounter);
}
// update the GHR
ghr = (ghr << 1);
if(resolveDir == TAKEN){
ghr++;
}
}
This predictor works in the same way as my simple one, except that it uses an array of counters instead of single one. When GetPrediction is called the array is indexed by last 17 bits of resolveDir (branch history, global history register or ghr) that are XORed with PC (adress of current branch). This selects the appropriate counter from array that is then used to do the prediction. UpdatePredictor works the same way, array is indexed and then counter is choosen. Counter is updated with information from resolveDir. Lastly the global history buffer (ghr, branch history, call it what you want) is also updated.
Code of the SatIncrement and SatDecrement functions:
static inline UINT32 SatIncrement(UINT32 x, UINT32 max)
{
if(x<max) return x+1;
return x;
}
static inline UINT32 SatDecrement(UINT32 x)
{
if(x>0) return x-1;
return x;
}
Thanks for help.
The reason the code doesn't work as expected is that SatIncrement and SatDecrement take arguments by-value and return the new value, which then must be assigned back to the variable that is supposed to be incremented/decremented.
SatIncrement(counter, PHT_CTR_MAX);
will pass the value of counter but will not modify counter itself. The return value with the new value is not used and so effectively this line does nothing. The same is true for SatDecrement(counter);.
Therefore your branch predictor never changes state and always returns the same prediction.
Fix it by following the other code example:
counter = SatIncrement(counter, PHT_CTR_MAX);
and
counter = SatDecrement(counter);
Given that this is an exercise you probably cannot change SatIncrement and SatDecrement, however in practice one would probably let these functions take arguments by-reference, so that they can modify the passed variable directly, avoiding the repetition of counter at the call site:
static inline void SatIncrement(UINT32& x, UINT32 max)
{
if(x<max) x++;
}
If the original signature were chosen, then since C++17 one can add the [[nodiscard]] attribute to the function to make the compiler print a warning if the return value is not used:
[[nodiscard]] static inline UINT32 SatIncrement(UINT32 x, UINT32 max)
{
if(x<max) return x+1;
return x;
}
It would have warned you here and made the problem clearer.

c++ oop beginner - return the output sum of each created objects in the vector within one function call

I have a simple class
#ifndef _VOICE
#define _VOICE
#include "maximilian.h"
class voice{
public:
voice(int waveType);
void setNote(double pitch,double amplitude);
void triggerNote();
void releaseNote();
double getOutput();
bool getState();
private:
maxiOsc op;
convert mtof;
double amp;
double freq;
double out;
int type;
bool state;
};
#endif // _VOICE
#include "voice.h"
voice::voice(int waveType){
type=waveType;
state=false;
out=0;
}
void voice::setNote(double pitch,double amplitude){
freq=pitch;
amp=amplitude;
}
void voice::triggerNote(){
if(type==1)
out=op.sinewave(mtof.mtof(freq))*amp;
if(type==2)
out=op.triangle(mtof.mtof(freq))*amp;
if(type==3)
out=op.saw(mtof.mtof(freq))*amp;
if(type==4)
out=op.square(mtof.mtof(freq))*amp;
state=true;
}
void voice::releaseNote(){
out=0;
state=false;
}
bool voice::getState(){
return state;
}
double voice::getOutput(){
return out;
}
I then store these voices in a vector
I wish to find a more effective way to acces the output sum of all objects within the vector
it looks something like this:
double PreMaster;
PreMaster=voices[0].getOutput()+voices[1].getOutput()+voices[2].getOutput()+voices[3].getOutput()....(n-object)...;
(in my program it goes to 34)
(for) does not work since I need to output the value simultaneously in each objects...
there should be a way inside the class to have a global function that will return the output sum of each created objects within one function call...
I am a beginner at oop so please pardon me if solution seems obvious.
And thanks in advance if you can point me in a useful direction.
EDIT:
I have a callback that gets called at sampling rate (44100hz).
I have a vector of voice object (they output simple synth waveforms)
to form a sounding chord I need to sum the output of each voice "simultaneously".
I then feed the sum to my soundcard every sample at sampling rate frequency.
void testApp::audioOut (float * output, int bufferSize, int nChannels){
for (int i = 0; i < bufferSize; i++){
for(int j=0;j<voices.size();j++){
if(keys[j]==true){
voices[j].triggerNote();
}
else if(keys[j]==false)
voices[j].releaseNote();
}
PreMaster=voices[0].getOutput()+voices[1].getOutput()+voices[2].getOutput()+voices[3].getOutput()
+voices[4].getOutput()+voices[5].getOutput()+voices[6].getOutput()+voices[7].getOutput()
+voices[8].getOutput()+voices[9].getOutput()+voices[10].getOutput()+voices[11].getOutput()
+voices[12].getOutput()+voices[13].getOutput()+voices[14].getOutput()+voices[15].getOutput()
+voices[16].getOutput()+voices[17].getOutput()+voices[18].getOutput()+voices[19].getOutput()
+voices[20].getOutput()+voices[21].getOutput()+voices[22].getOutput()+voices[23].getOutput()
+voices[24].getOutput()+voices[25].getOutput()+voices[26].getOutput()+voices[27].getOutput()
+voices[28].getOutput()+voices[29].getOutput()+voices[30].getOutput()+voices[31].getOutput()
+voices[32].getOutput()+voices[33].getOutput();
MasterOut.stereo(PreMaster, outputs, 0.5);
lAudioOut[i] = output[i*nChannels ] = outputs[0]; /* You may end up with lots of outputs. add them here */
rAudioOut[i] = output[i*nChannels + 1] = outputs[1];
}
}
hope this clarify a bit...
Yeah I forgot about += so I tested it and it stop my program form outputing sounds.when I do it with my ugly method it works though...
You can perform a simple loop, since there is no reason to believe that a for does not work, given your code example:
double sum = 0;
for (const auto& v : voices) {
sum += v.getOutput();
}
Another option is to use std::accumulate and a suitable functor:
// function to sum voice's getOutput() to get running total
double addOutput(double sum, const voice& v) { return sum + voice.getOutput(); }
....
double sum = std::accumulate(voices.begin(), voices.end(), 0.0, addOutput);
An example:
int sum = 0;
for(std::vector<voice>::iterator i = voices.begin(); i != voices.end(); i++)
sum += i->getOutput();

CPPUnit how do I write a test?

Okay I basically want to get the ball rolling and write some CPPUnit tests but I have no idea how to go about it. Here I have some code that basically gets a pointer to the Menu Button for the associated button group and position arguments, how would I go about creating a test for this?
CMenuButton* CMenuContainer::GetButton(const enumButtonGroup argGroup, const int32_t argPosition)
{
CMenuButton* pButton = NULL;
if (argGroup < MAX_GROUP_BUTTONS)
{
pButton = m_ButtonGroupList[argGroup].GetButton(argPosition);
}
return pButton;
In reply to #Fabio Ceconello, would it be possible to set some tests for some code like this?
unsigned long CCRC32::Reflect(unsigned long ulReflect, const char cChar)
{
unsigned long ulValue = 0;
// Swap bit 0 for bit 7, bit 1 For bit 6, etc....
for(int iPos = 1; iPos < (cChar + 1); iPos++)
{
if(ulReflect & 1)
{
ulValue |= (1 << (cChar - iPos));
}
ulReflect >>= 1;
}
return ulValue;
}
CppUnit isn't well suited for creating automated tests for user interface. It's more for processing-only units. For instance, let's say you created a replacement for std::vector and want to make sure it behaves like the original one, you could write tests that add elements to both your and the standard implementation, then do some more handling (removing, changing elements, etc.) and after each step check if the two have a consistent result.
For UI I'm not aware of good open source/free tools, but one good commercial tool is TestComplete from Smart Bear, among others.
For the second example you gave, the first thing is to define a validity check for the Reflect() method. You can, for instance, calculate the result of some values by hand to check if the returned value for each of them is what was expected. Or you could use an inverse function that's known to be fully working.
Assuming the first option, you could write the test like this:
class CRC32Test : public CppUnit::TestCase
{
public:
CRC32Test( std::string name ) : CppUnit::TestCase( name ) {}
void runTest()
{
struct Sample {unsigned long ulReflect; char cChar; unsigned long result};
static Sample samples[] =
{
// Put here the pre-calculated values
};
int count = sizeof(samples) / sizeof(Sample);
for (int i = 0; i < count; ++i)
CPPUNIT_ASSERT(subject.Reflect(samples[i].ulReflect, samples[i].cChar) == samples[i].result);
}
private:
CRC32 subject;
};
int main(void)
{
CppUnit::TextUi::TestRunner runner;
runner.addTest(new CppUnit::TestCaller<CRC32Test>("test CRC32", &CRC32::runTest));
runner.run();
}