C++ SQLite Serialized Mode Questions - c++

I have been through Quite a few pages, and have an ok Idea of whats happening it think, but I have a few Questions just to be sure....
my program uses the -DTHREADSAFE=1 compile options, forks on receiving a database request (Select, Delete, Insert, Update) from a user or my network, then the child process handles the various database tasks, and relaying of messages should that be required and so on,
at the moment my database is not setup for concurrency which I wont lie is a major design flaw, but that's beside the point at the moment, let's say I have a function that prints all the entries in my table LEDGER as follows...
void PersonalDataBase::printAllEntries()
{
//get all entries
const char query [] = "select * from LEDGER";
sqlite3_stmt *stmt;
int error
try
{
if ((error = sqlite3_prepare(publicDB, query, -1, &stmt, 0 )) == SQLITE_OK)
{
int ctotal = sqlite3_column_count(stmt);
int res = 0;
while ( 1 )
{
res = sqlite3_step(stmt);
if ( res == SQLITE_ROW )
{
Entry *temp = loadBlockRow(stmt);
string from, to;
from = getNameForHash(temp -> from);
to = getNameForHash(temp -> to);
temp -> setFromOrTo(from, 0);
temp -> setFromOrTo(to, 1);
temp -> printEntry();
printlnEnd();
delete temp;
}
else if ( res == SQLITE_DONE || res==SQLITE_ERROR)
{
if (res == SQLITE_ERROR) { throw res; }
sqlite3_finalize(stmt);
break;
}
}
}
//problems
else
{
throw error;
}
}
catch (int err)
{
sqlite3_finalize(stmt);
setupOutput();
cout << "Database Error: " << sqlite3_errmsg(publicDB) << ", Error Code: " << (int) error << endl;
cout << "Did Not Find Values Try Again After Fixing Problems Above." << endl;
printlnEnd();
}
println("Done!");
}
my setupOutput(), printlnEnd(), println(), all help with my use of 'non-blocking' keyboard i/o, they work as I want lets not worry about them here, and think of them as just a call to cout
ok so now at this point I figure there are 4 options...
A while around my try/catch, then in catch check if err = 5, if so I need to setup a sqlite3_busy_handler and have it wait for whatever is blocking the current operation (once it returns SQLITE_OK and have cleaned up all my old variables I reiterate through the while/try again), now as only one of these can be setup at a time, let's say for instance Child1 is doing a large write and child2 and child3 are trying to say read and update concurrently on top of the first child's write, so if a SQLITE_BUSY is returned by this function I print out an error, then restart my while loop (restarting the function), of course after I have finalized my old statement, and cleared up any local objects that may have been created, if this a correct line of thinking?
Should I setup a recursive mutex, say screw it to SQLites own locking mechanism, set it up to be shared across processes then only allow one operation on a database at a time? for using my app on a small scale this doesn't seem to bad of an option, however I'm reading a lot of warnings on using a recursive mutex and am wondering if this is is the best option, as many posts say handle mutual exclusion yourself. however then I cannot have concurrent reads, which is a bit of a pain
Use option 1 but instead of using the SQLite busy handler, just call usleep on a random number, clean up data, and restart while?
before/after any function involving my database use sqlite3_exec() with "BEGIN IMMEDIATE"/"COMMIT" respectively, Locking the database for the duration of the code in between those 2 statements. So that nothing enclosed within can(or at least should) return SQLITE_BUSY, then if my "BEGIN IMMEDIATE" returns BUSY (it should be the only one so long as everything is set up correctly), I use the sqlite3_busy_handler which honestly, if only one process can use it at a time seems annoying... or a random number with usleep(), (presumably at this number is rather large 1mil = 1 second the chance of overlap between 1-20 processes is pretty slim) so each process will constantly try to re lock the database at random intervals for their own purposes
Is there a better way? or which one of these is best?

SQLite's internal busy handler (installed with sqlite3_busy_timeout()) already sleeps a more-or-less random number of times; there is no need to write your own handler.
Using your own locking mechanism would be more efficient than random waiting, but only if you have reader/writer locks.
BEGIN or BEGIN IMMEDIATE ensure that no other statement in the same transaction can run into a lock, but only if IMMEDIATE is used for transactions that write.
To allow concurrent readers and writers, consider using WAL mode. (But this does not allow multiple writers either.)

Related

How to stop one thread from two parallel threads?

I have a program in which we can monitor 2 objects at same time.
myThread = new thread (thred1, id);
vec.push_back (myThread);
In thred1 function,i use Boolean function to read the stored values from a different vector and it runs parallely like this:
element found 2 -- hj
HUMIDITY-1681692777 DISPLAYED IN RH
element found 1 -- hj
TEMPERATURE--1714636915 IN DEGREE CELSIUS
This keeps on running as that is what my program should do.
I have a case where I need to get ID from the user and stop that particular thread and the other should keep running till I stop it.Can someone help me with that?
void thred1 (int id)
{
bool err = false;
while (stopThread == false)
{
for (size_t i = 0; i < v.size (); i++)
{
if (id == v[i]->id)
{
cout << "element found " << v[i]->id << " -- " << v[i]->name << endl;
v[i]->Read ();
this_thread::sleep_for (chrono::seconds (4));
err = true;
break;
}
}
if (!err)
{
cout << "element not found" << endl;
break;
}
}
}
Suspension
1. Assuming you want to suspend the monitor thread but only temporarily (i.e making any changes) then you can just use a mutex. Lock it before accessing the shared vector and unlock it when you're done, ensuring that only one thread can access the data at a time.
2. You can actively suspend the thread using OS support such as SuspendThread and ResumeThread, in the case of Windows, when it's ready.
Termination
1. You could use an event for each monitor thread, name being linked to the ID would work. At each iteration of the monitor check for the termination event, ending the thread if it's active.
2. Pass some variable to each thread, store them in a map with the thread handle being the key, and similar to the previous option just check the value for each iteration.
3. Store all threads in a map with the handle as key, terminating it directly with OS support.
Honestly there are a ton of ways to do this, the best implementation depends on why exactly you want to stop the monitor thread. Any sort of synchronization object like a mutex should be fine if you're reading from one thread and writing from another. Otherwise, just storing all threads with the internal ID as key and the thread as the value should be fine for terminating monitor threads on demand.

Threading and Mutex

I'm working on a program that simulates a gas station. Each car at the station is it's own thread. Each car must loop through a single bitmask to check if a pump is open, and if it is, update the bitmask, fill up, and notify other cars that the pump is now open. My current code works but there are some issues with load balancing. Ideally all the pumps are used the same amount and all cars get equal fill-ups.
EDIT: My program basically takes a number of cars, pumps, and a length of time to run the test for. During that time, cars will check for an open pump by constantly calling this function.
int Station::fillUp()
{
// loop through the pumps using the bitmask to check if they are available
for (int i = 0; i < pumpsInStation; i++)
{
//Check bitmask to see if pump is open
stationMutex->lock();
if ((freeMask & (1 << i)) == 0 )
{
//Turning the bit on
freeMask |= (1 << i);
stationMutex->unlock();
// Sleeps thread for 30ms and increments counts
pumps[i].fillTankUp();
// Turning the bit back off
stationMutex->lock();
freeMask &= ~(1 << i);
stationCondition->notify_one();
stationMutex->unlock();
// Sleep long enough for all cars to have a chance to fill up first.
this_thread::sleep_for(std::chrono::milliseconds((((carsInStation-1) * 30) / pumpsInStation)-30));
return 1;
}
stationMutex->unlock();
}
// If not pumps are available, wait until one becomes available.
stationCondition->wait(std::unique_lock<std::mutex>(*stationMutex));
return -1;
}
I feel the issue has something to do with locking the bitmask when I read it. Do I need to have some sort of mutex or lock around the if check?
It looks like every car checks the availability of pump #0 first, and if that pump is busy it then checks pump #1, and so on. Given that, it seems expected to me that pump #0 would service the most cars, followed by pump #1 serving the second-most cars, all the way down to pump #(pumpsInStation-1) which only ever gets used in the (relatively rare) situation where all of the pumps are in use simultaneously at the time a new car pulls in.
If you'd like to get better load-balancing, you should probably have each car choose a different random ordering to iterate over the pumps, rather than having them all check the pumps' availability in the same order.
Normally I wouldn't suggest refactoring as it's kind of rude and doesn't go straight to the answer, but here I think it would help you a bit to break your logic into three parts, like so, to better show where the contention lies:
int Station::acquirePump()
{
// loop through the pumps using the bitmask to check if they are available
ScopedLocker locker(&stationMutex);
for (int i = 0; i < pumpsInStation; i++)
{
// Check bitmask to see if pump is open
if ((freeMask & (1 << i)) == 0 )
{
//Turning the bit on
freeMask |= (1 << i);
return i;
}
}
return -1;
}
void Station::releasePump(int n)
{
ScopedLocker locker(&stationMutex);
freeMask &= ~(1 << n);
stationCondition->notify_one();
}
bool Station::fillUp()
{
// If a pump is available:
int i = acquirePump();
if (i != -1)
{
// Sleeps thread for 30ms and increments counts
pumps[i].fillTankUp();
releasePump(i)
// Sleep long enough for all cars to have a chance to fill up first.
this_thread::sleep_for(std::chrono::milliseconds((((carsInStation-1) * 30) / pumpsInStation)-30));
return true;
}
// If no pumps are available, wait until one becomes available.
stationCondition->wait(std::unique_lock<std::mutex>(*stationMutex));
return false;
}
Now when you have the code in this form, there is a load balancing issue which is important to fix if you don't want to "exhaust" one pump or if it too might have a lock inside. The issue lies in acquirePump where you are checking the availability of free pumps in the same order for each car. A simple tweak you can make to balance it better is like so:
int Station::acquirePump()
{
// loop through the pumps using the bitmask to check if they are available
ScopedLocker locker(&stationMutex);
for (int n = 0, i = startIndex; n < pumpsInStation; ++n, i = (i+1) % pumpsInStation)
{
// Check bitmask to see if pump is open
if ((freeMask & (1 << i)) == 0 )
{
// Change the starting index used to search for a free pump for
// the next car.
startIndex = (startIndex+1) % pumpsInStation;
// Turning the bit on
freeMask |= (1 << i);
return i;
}
}
return -1;
}
Another thing I have to ask is if it's really necessary (ex: for memory efficiency) to use bit flags to indicate whether a pump is used. If you can use an array of bool instead, you'll be able to avoid locking completely and simply use atomic operations to acquire and release pumps, and that'll avoid creating a traffic jam of locked threads.
Imagine that the mutex has a queue associated with it, containing the waiting threads. Now, one of your threads manages to get the mutex that protects the bitmask of occupied stations, checks if one specific place is free. If it isn't, it releases the mutex again and loops, only to go back to the end of the queue of threads waiting for the mutex. Firstly, this is unfair, because the first one to wait is not guaranteed to get the next free slot, only if that slot happens to be the one on its loop counter. Secondly, it causes an extreme amount of context switches, which is bad for performance. Note that your approach should still produce correct results in that no two cars collide while accessing a single filling station, but the behaviour is suboptimal.
What you should do instead is this:
lock the mutex to get exclusive access to the possible filling stations
locate the next free filling station
if none of the stations are free, wait for the condition variable and restart at point 2
mark the slot as occupied and release the mutex
fill up the car (this is where the sleep in the simulation actually makes sense, the other one doesn't)
lock the mutex
mark the slot as free and signal the condition variable to wake up others
release the mutex again
Just in case that part isn't clear to you, waiting on a condition variable implicitly releases the mutex while waiting and reacquires it afterwards!

fsync() and write() in different threads

I am trying to write program using fsync() and write() but fsync need time to sync data but i haven't this time to wait. I made one more thread for fsync()
Here is my code:
#include <thread>
void thread_func(int fd) {
while (1) {
if(fsync(fd) != 0)
std::cout << "ERROR fsync()\n";
usleep(100);
}
}
int main () {
int fd = open ("device", O_RDWR | O_NONBLOCK);
if (fd < 0) {
std::cout << "ERROR: open()\n";
return -1;
}
std::thread *thr = new std::thread (thread_func, fd);
if (thr == nullptr) {
std::cout << "Cannot create thread\n";
close (fd);
return -1;
}
while (1) {
if (write (fd, 'x', 1) < 1)
std::cout << "ERROR write()\n";
}
close(fd);
}
Question is:
is it need to lock different thread when i use file descriptor to fsync in other thread than main?
when i test my program without mutex it have no problem. and when i read man description for fsync it have nothing for different thread.
If the fact that fsync takes time and even sometimes blocks for a very short time is a problem, then you are most probably doing something wrong.
Normally, you do not want to call fsync at all, ever. It is a serious anti-optimization to do so, and one will only ever want to do it if it must be assured that data has been written out1. In this case, however, you absolutely want fsync to block, this is not only works-as-intended, but necessary.
Only when fsync has returned, you know that it has done its task. You know that the OS has done its best to assure that data has been written, and only then it is safe to proceed. If you offload this to a background thread, you can just as well not call fsync, because you don't know when it's safe to assume data has been written.
If initiating writes is your primary goal, you use sync_file_range under Linux (which runs asynchronously) followed by a call to fsync some time later. The reason for following up with fsync is both to ensure that writes are done, and the fact that sync_file_range does not update metadata, so unless you are strictly overwriting already allocated data within the file, your writes may not be visible in case of a crash even though data is on disk (I can't imagine how that might happen since allocating more sectors to a file necessarily means metadata must be modified, but the manpage explicitly warns that this can happen).
1The fsync function still does not (and cannot) guarantee that data is on a permanent storage, it might still be somewhere in the cache hierarchy, such as a controller's or disk's write cache.
Unless you require the thread for something else I would suggest you use the asynchronous I/O aio library:
struct aiocb fsync_cb = {
.aio_fildes = fd
, .aio_sigevent = {
.sigev_notify = SIGEV_NONE
}
}
aio_fsync(O_SYNC, &fsync_cb);
There is also an equivalent variant for write.
struct aiocb write_cb = {
.aio_fildes = fd
, .aio_buf = buffer
, .aio_nbytes = nbytes
, .aio_offset = offset
, .aio_sigevent = {
.sigev_notify = SIGEV_NONE
}
}
aio_write(&write_cb);
If you choose not to have any notificaton of success then you will have to check/wait at some point for completion:
while (aio_error(&write_cb) == EINPROGRESS);

how can i test synchronize function in C++

i have a synchronize function that i want to test if it ends.
i want to be able to run code for X time, and if the time ends to continue.
here what i want:
bool flag = false;
some_function_that_run_the_next_block_for_x_sec()
{
my_sync_func_that_i_want_to_test();
flag = true;
}
Assert::IsTrue(flag);
is there a simple way to do this?
SynchronizationContext
thanks.
The link you posted gives me little insight on how that class would be used (maybe Microsoft is saving up bytes on sample code to pay for Ballmer's golden parachute next year?) so pardon me for completely ignoring it.
Something like this:
auto result = async(launch::async, my_sync_func_that_i_want_to_test);
future_status status = result.wait_for(chrono::milliseconds(100));
if (status == future_status::timeout)
cout << "Timed out" << endl;
if (status == future_status::ready)
cout << "Finished on time" << endl;
Need inclusion of the <future> and <chrono> headers.
If my_sync_func_that_i_want_to_test() never finishes you'll have another problem. The future object (result) will block until the thread launched by async() finishes. There's no portable way to recover from "killed/canceled/aborted" threads, so this will probably require some platform-specific code, even if you roll out your own async_that_detaches_the_thread() (which is not hard to find, here's one example).

How to reduce cpu usage during data transfer on TCP ports realtime

I have a socket program which acts like both client and server.
It initiates connection on an input port and reads data from it. On a real time scenario it reads data on input port and sends the data (record by record ) on to the output port.
The problem here is that while sending data to the output port CPU usage increases to 50% while is not permissible.
while(1)
{
if(IsInputDataAvail==1)//check if data is available on input port
{
//condition to avoid duplications while sending
if( LastRecordSent < LastRecordRecvd )
{
record_time temprt;
list<record_time> BufferList;
list<record_time>::iterator j;
list<record_time>::iterator i;
// Storing into a temp list
for(i=L.begin(); i != L.end(); ++i)
{
if((i->recordId > LastRecordSent) && (i->recordId <= LastRecordRecvd))
{
temprt.listrec = i->listrec;
temprt.recordId = i->recordId;
temprt.timestamp = i->timestamp;
BufferList.push_back(temprt);
}
}
//Sending to output port
for(j=BufferList.begin(); j != BufferList.end(); ++j)
{
LastRecordSent = j->recordId;
std::string newlistrecord = j->listrec;
newlistrecord.append("\n");
char* newrecord= new char [newlistrecord.size()+1];
strcpy (newrecord, newlistrecord.c_str());
if ( s.OutputClientAvail() == 1) //check if output client is available
{
int ret = s.SendBytes(newrecord,strlen(newrecord));
if ( ret < 0)
{
log1.AddLogFormatFatal("Nice Send Thread : Nice Client Disconnected");
--connected;
return;
}
}
else
{
log1.AddLogFormatFatal("Nice Send Thread : Nice Client Timedout..connection closed");
--connected; //if output client not available disconnect after a timeout
return;
}
}
}
}
// Sleep(100); if we include sleep here CPU usage is less..but to send data real time I need to remove this sleep.
If I remove Sleep()...CPU usage goes very high while sending data to out put port.
}//End of while loop
Any possible ways to maintain real time data transfer and reduce CPU usage..please suggest.
There are two potential CPU sinks in the listed code. First, the outer loop:
while (1)
{
if (IsInputDataAvail == 1)
{
// Not run most of the time
}
// Sleep(100);
}
Given that the Sleep call significantly reduces your CPU usage, this spin-loop is the most likely culprit. It looks like IsInputDataAvail is a variable set by another thread (though it could be a preprocessor macro), which would mean that almost all of that CPU is being used to run this one comparison instruction and a couple of jumps.
The way to reclaim that wasted power is to block until input is available. Your reading thread probably does so already, so you just need some sort of semaphore to communicate between the two, with a system call to block the output thread. Where available, the ideal option would be sem_wait() in the output thread, right at the top of your loop, and sem_post() in the input thread, where it currently sets IsInputDataAvail. If that's not possible, the self-pipe trick might work in its place.
The second potential CPU sink is in s.SendBytes(). If a positive result indicates that the record was fully sent, then that method must be using a loop. It probably uses a blocking call to write the record; if it doesn't, then it could be rewritten to do so.
Alternatively, you could rewrite half the application to use select(), poll(), or a similar method to merge reading and writing into the same thread, but that's far too much work if your program is already mostly complete.
if(IsInputDataAvail==1)//check if data is available on input port
Get rid of that. Just read from the input port. It will block until data is available. This is where most of your CPU time is going. However there are other problems:
std::string newlistrecord = j->listrec;
Here you are copying data.
newlistrecord.append("\n");
char* newrecord= new char [newlistrecord.size()+1];
strcpy (newrecord, newlistrecord.c_str());
Here you are copying the same data again. You are also dynamically allocating memory, and you are also leaking it.
if ( s.OutputClientAvail() == 1) //check if output client is available
I don't know what this does but you should delete it. The following send is the time to check for errors. Don't try to guess the future.
int ret = s.SendBytes(newrecord,strlen(newrecord));
Here you are recomputing the length of the string which you probably already knew back at the time you set j->listrec. It would be much more efficient to just call s.sendBytes() directly with j->listrec and then again with "\n" than to do all this. TCP will coalesce the data anyway.