NET-SNMP and multithreading - c++

I am writing a C++ SNMP server using a NET-SNMP library. I read the documentation and still got one question. Can multiple threads sharing single snmp session and using it in procedures like snmp_sess_synch_response() simultaneously, or I must init and open new session in each thread?

Well, when I am trying to snmp_sess_synch_response() from two different threads using the same opaque session pointer simultaneously, one of three errors always occures. The first is memory access violation, the second is endless WaitForSingleObject() in both threads and the third is heap allocation error.
I suppose I can treat this as an answer, thus sharing single session between multiple threads is unsafe, because using it in procedures like snmp_sess_synch_response() simultaneously will cause an errors.
P.S. Here is the piece of code of described before:
void* _opaqueSession;
boost::mutex _sessionMtx;
std::shared_ptr<netsnmp_pdu> ReadObjectValue(Oid& objectID)
{
netsnmp_pdu* requestPdu = snmp_pdu_create(SNMP_MSG_GET);
netsnmp_pdu* response = 0;
snmp_add_null_var(requestPdu, objectID.GetObjId(), objectID.GetLen());
void* opaqueSessionCopy;
{
//Locks the _opaqueSession, wherever it appears
boost::mutex::scoped_lock lock(_sessionMtx);
opaqueSessionCopy = _opaqueSession;
}
//Errors here!
snmp_sess_synch_response(opaqueSessionCopy, requestPdu, &response);
std::shared_ptr<netsnmp_pdu> result(response);
return result;
}
void ExecuteThread1()
{
Oid sysName(".1.3.6.1.2.1.1.5.0");
try
{
while(true)
{
boost::thread::interruption_pont();
ReadObjectValue(sysName);
}
}
catch(...)
{}
}
void ExecuteThread2()
{
Oid sysServices(".1.3.6.1.2.1.1.7.0");
try
{
while(true)
{
boost::thread::interruption_pont();
ReadObjectValue(sysServices);
}
}
catch(...)
{}
}
int main()
{
std::string community = "public";
std::string ipAddress = "127.0.0.1";
snmp_session session;
{
SNMP::snmp_sess_init(&session);
session.timeout = 500000;
session.retries = 0;
session.version = SNMP_VERSION_2c;
session.remote_port = 161;
session.peername = (char*)ipAddress.c_str();
session.community = (u_char*)community.c_str();
session.community_len = community.size();
}
_opaqueSession = snmp_sess_open(&session);
boost::thread thread1 = boost::thread(&ExecuteThread1);
boost::thread thread2 = boost::thread(&ExecuteThread2);
boost::this_thread::sleep(boost::posix_time::seconds::seconds(30));
thread1.interrupt();
thread1.join();
thread2.interrupt();
thread2.join();
return 0;
}

Related

Reason for losing messeges over NNG sockets in raw mode

Some context to my problem:
I need to establish an inter-process communication using C++ and sockets and I picked NNG library for that along with nngpp c++ wrapper. I need to use push/pull protocol so no contexts handling is available to me. I wrote some code based on raw example from nngpp demo. The difference here is that, by using push/pull protocol I split this into two separate programs. One for sending and one for receiving.
Problem descripion:
I need to receive let's say a thousand or more messages per second. For now, all messages are captured only when I send about 50/s. That is way too slow and I do believe it can be done faster. The faster I send, the more I lose. At the moment, when sending 1000msg/s I lose about 150 msgs.
Some words about the code
The code may be in C++17 standard. It is written in object-oriented manner so in the end I want to have a class with "receive" method that would simply give me the received messages. For now, I just print the results on screen. Below, I supply some parts of the project with descriptions:
NOTE msgItem is a struct like that:
struct msgItem {
nng::aio aio;
nng::msg msg;
nng::socket_view itemSock;
explicit msgItem(nng::socket_view sock) : itemSock(sock) {}
};
And it is taken from example mentioned above.
Callback function that is executed when message is received by one of the aio's (callback is passed in constructor of aio object). It aims at checking whether everything was ok with transmission, retrieving my Payload (just string for now) and passing it to queue while a flag is set. Then I want to print those messages from the queue using separate thread.
void ReceiverBase<Payload>::aioCallback(void *arg) try {
msgItem *msgItem = (struct msgItem *)arg;
Payload retMsg{};
auto result = msgItem->aio.result();
if (result != nng::error::success) {
throw nng::exception(result);
}
//Here we extract the message
auto msg = msgItem->aio.release_msg();
auto const *data = static_cast<typename Payload::value_type *>(msg.body().data());
auto const count = msg.body().size()/sizeof(typename Payload::value_type);
std::copy(data, data + count, std::back_inserter(retMsg));
{
std::lock_guard<std::mutex> lk(m_msgMx);
newMessageFlag = true;
m_messageQueue.push(std::move(retMsg));
}
msgItem->itemSock.recv(msgItem->aio);
} catch (const nng::exception &e) {
fprintf(stderr, "server_cb: %s: %s\n", e.who(), e.what());
} catch (...) {
fprintf(stderr, "server_cb: unknown exception\n");
}
Separate thread for listening to the flag change and printing. While loop at the end is for continuous work of the program. I use msgCounter to count successful message receival.
void ReceiverBase<Payload>::start() {
auto listenerLambda = [](){
std::string temp;
while (true) {
std::lock_guard<std::mutex> lg(m_msgMx);
if(newMessageFlag) {
temp = std::move(m_messageQueue.front());
m_messageQueue.pop();
++msgCounter;
std::cout << msgCounter << "\n";
newMessageFlag = false;
}}};
std::thread listenerThread (listenerLambda);
while (true) {
std::this_thread::sleep_for(std::chrono::microseconds(1));
}
}
This is my sender application. I tweak the frequency of msg sending by changing the value in std::chrono::miliseconds(val).
int main (int argc, char *argv[])
{
std::string connection_address{"ipc:///tmp/async_demo1"};
std::string longMsg{" here normally I have some long test text"};
std::cout << "Trying connecting sender:";
StringSender sender(connection_address);
sender.setupConnection();
for (int i=0; i<1000; ++i) {
std::this_thread::sleep_for(std::chrono::milliseconds(3));
sender.send(longMsg);
}
}
And this is receiver:
int main (int argc, char *argv[])
{
std::string connection_address{"ipc:///tmp/async_demo1"};
std::cout << "Trying connecting receiver:";
StringReceiver receiver(connection_address);
receiver.setupConnection();
std::cout<< "Connection set up. \n";
receiver.start();
return 0;
}
Nothing speciall in those two applications as You see. the setup method from StringReciver is something like that:
bool ReceiverBase<Payload>::setupConnection() {
m_connected = false;
try {
for (size_t i = 0; i < m_parallel; ++i) {
m_msgItems.at(i) = std::make_unique<msgItem>(m_sock);
m_msgItems.at(i)->aio =
nng::aio(ReceiverBase::aioCallback, m_msgItems.at(i).get());
}
m_sock.listen(m_adress.c_str());
m_connected = true;
for (size_t i = 0; i < m_parallel; ++i) {
m_msgItems.at(i)->itemSock.recv(m_msgItems.at(i)->aio);
}
} catch (const nng::exception &e) {
printf("%s: %s\n", e.who(), e.what());
}
return m_connected;
}
Do You have any suggestions why the performance is so low? Do I use lock_guards properly here? What I want them to do is basically lock the flag and queue so only one side has access to it.
NOTE: Adding more listeners thread does not affect the performance either way.
NOTE2: newMessageFlag is atomic

How to handle condition variable missed signal from another thread

So in the compilable code below, I'm sending a Query message to be handled by another thread and I want to wait for a response or timeout if it hits a certain timeout. I don't know why the wait_until is missing the signal and hitting the timeout period when it should not be doing that. It only happens if the handler is returning a response REALLY fast. How do you propose I fix the code below?
#include <mutex>
#include <memory>
#include <condition_variable>
#include <atomic>
#include <thread>
#include <iostream>
#include <queue>
#include <zconf.h>
class Question
{
};
class Answer
{
public:
bool isAnswered = false;
};
class Query
{
std::condition_variable _cv;
std::mutex _mutex;
std::atomic_bool _questionAnswered;
std::atomic_bool _questionSet;
std::shared_ptr<Question> _question;
std::shared_ptr<Answer> _answer;
public:
void setQuestion(std::shared_ptr<Question> & question)
{
if(!_questionSet)
{
_question = question;
_questionSet = true;
}
};
void setAnswer(std::shared_ptr<Answer> answer)
{
std::unique_lock<std::mutex> lock(_mutex);
if(!_questionAnswered)
{
// Set the answer and notify the getAnswerWithTimeout() to unlock if holding
_answer = answer;
_questionAnswered = true;
lock.unlock();
_cv.notify_all();
}
};
std::shared_ptr<Answer> getAnswerWithTimeout(uint64_t micros)
{
std::unique_lock<std::mutex> lock(_mutex);
if(!_questionAnswered)
{
auto now = std::chrono::system_clock::now();
// When timeout occurs, lock down this class, set the answer as null, and set error to timeout
if (!_cv.wait_until(lock, now + std::chrono::microseconds(micros), [&]() { return (bool)_questionAnswered; }) )
{
_answer = nullptr;
_questionAnswered = true;
}
}
return _answer;
};
};
void function_to_run(std::shared_ptr<Query> query)
{
// Respond to query and set the answer
auto answer = std::make_shared<Answer>();
answer->isAnswered = true;
// Set the response answer
query->setAnswer(answer);
}
std::queue<std::shared_ptr<Query>> queryHandler;
bool keepRunning = true;
std::mutex queryHandlerMutex;
std::condition_variable queryHandlerCv;
void handleQueryHandler()
{
while (true)
{
std::shared_ptr<Query> query;
{
std::unique_lock<std::mutex> lock(queryHandlerMutex);
queryHandlerCv.wait(lock, [&] { return !keepRunning || !queryHandler.empty(); });
if (!keepRunning) {
return;
}
// Pop off item from queue
query = queryHandler.front();
queryHandler.pop();
}
// Process query with function
function_to_run(query);
}
}
void insertIntoQueryHandler(std::shared_ptr<Query> & query)
{
{
std::unique_lock<std::mutex> lock(queryHandlerMutex);
// Insert into Query Handler
queryHandler.emplace(query);
}
// Notify query handler to start if locked on empty
queryHandlerCv.notify_one();
}
std::shared_ptr<Answer>
ask(std::shared_ptr<Query> query, uint64_t timeoutMicros=0)
{
std::shared_ptr<Answer> answer = nullptr;
// Send Query to be handled by external thread
insertIntoQueryHandler(query);
// Hold for the answer to be returned with timeout period
answer = query->getAnswerWithTimeout(timeoutMicros);
return answer;
}
int main()
{
// Start Up Query Handler thread to handle Queries
std::thread queryHandlerThread(handleQueryHandler);
// Create queries in infinite loop and process
for(int i = 0; i < 1000000; i++)
{
auto question = std::make_shared<Question>();
auto query = std::make_shared<Query>();
query->setQuestion(question);
auto answer = ask(query, 1000);
if(!answer)
{
std::cout << "Query Timed out after 1000us" << std::endl;
}
}
// Stop the thread
{
std::unique_lock<std::mutex> lock(queryHandlerMutex);
keepRunning = false;
}
queryHandlerCv.notify_one();
queryHandlerThread.join();
return 0;
}
As discussed in the comments, the main issue here is the timeout period you're using (1ms), in this interval:
auto now = std::chrono::system_clock::now();
.... another thread may sneak in here ....
if (!_cv.wait_until(lock, now + std::chrono::microseconds(micros), [&]() { return (bool)_questionAnswered; }) )
{
another thread can sneak in and consume a timeslice (e.g. 10ms) and the wait_until would timeout immediately. Furthermore there are reports of unexpected behaviour with wait_until as described here:
std::condition_variable wait_until surprising behaviour
Increasing the timeout to something in the order of several timeslices will fix this. You can also adjust thread priorities.
Personally I advocate polling a condition variable with wait_for which is efficient and also bails in a timely fashion (as opposed to polling a flag and sleeping).
Time slices in non-RTOS systems tend to be in the order of 10ms, so I would not expect such short timeouts to work accurately and predictably in general-purpose systems. See this for an introduction to pre-emptive multitasking:
https://www.geeksforgeeks.org/time-slicing-in-cpu-scheduling/
as well as this:
http://dev.ti.com/tirex/explore/node?node=AL.iEm6ATaD6muScZufjlQ__pTTHBmu__LATEST
As jtbandes points out, it's worth using tools such as Clang's thread sanitiser to check for potential logic races: https://clang.llvm.org/docs/ThreadSanitizer.html

How to do IPC using Unix Domain Socket in D?

Here I have a program that wants to
detect whether if it's the only instance
1.1. it does that by trying to create a Unix Domain Socket
and trying to binding it to a specific address.
if a duplicate program is not running, establish an UDS
and then listen to the socket.
2.1. if any message comes through that socket, the program will log the incoming message
2.2. otherwise it should keep listening to the socket forever
if there's a duplicate program it should send a message and then exit.
Here's what I have:
import std.socket, std.experimental.logger;
immutable string socketAddress = "\0/tmp/com.localserver.myapp";
void main()
{
auto socket = new std.socket.Socket(std.socket.AddressFamily.UNIX,
std.socket.SocketType.STREAM);
auto addr = new std.socket.UnixAddress(socketAddress);
auto isUnique = () {
bool result;
scope (success)
log("returns: ", result);
try
{
socket.bind(addr);
result = true;
}
catch (std.socket.SocketOSException e)
result = false;
// else throw error
return result;
}();
if (isUnique)
{
log("Unique instance detected. Listening...");
// works upto now
char[] buffer = [];
while (1)
{
socket.listen(0);
socket.receive(buffer);
if (buffer != []) {
log("Received message: ", buffer);
}
buffer = [];
}
}
else
{
log("Duplicate instance detected.");
socket.connect(addr);
import std.stdio;
stdout.write("Enter your message:\t");
socket.send(readln());
log("Message has been sent. Exiting.");
}
}
The documentation does not seem very friendly to those who does not have any experience in socket programming. How can I send and receive message with std.socket.Socket?
After binding, you actually need to accept. It will return a new Socket instance which you can actually receive from. Your client side branch looks ok. I think that is your key mistake here.
I also have a code sample in my book that shows basic functionality of std.socket which can help as an example:
http://arsdnet.net/dcode/book/chapter_02/03/
it is tcp, but making it unix just means changing the family, like you already did in your code.
You can also look up socket tutorials for C and so on, the D socket is just a thin wrapper around those same BSD style socket functions.
As Adam pointed out I had use listen() method first and then apply the accept() method which returns a socket that can receive message. Then the receiver socket takes a char[N] buffer.
import std.socket, std.experimental.logger;
class UDSIPC
{
private:
static immutable string socketAddress = "\0/tmp/com.localserver.myapp";
static immutable size_t messageBufferSize = 64;
static immutable string socketAddressName = "\0/tmp/com.localserver.myapp";
Socket socket;
UnixAddress uaddr;
public:
this(in string socketAddressName = socketAddressName)
{
socket = new Socket(AddressFamily.UNIX, SocketType.STREAM);
uaddr = new UnixAddress(socketAddress);
}
bool getUniqueness()
{
bool result;
scope (success)
log("returns: ", result);
try
{
socket.bind(uaddr);
result = true;
}
catch (SocketOSException e)
result = false;
// else throw error
return result;
}
string getMessage()
{
socket.listen(0);
auto receiverSocket = socket.accept();
char[messageBufferSize] buffer;
auto amount = receiverSocket.receive(buffer);
import std.string;
return format!"%s"(buffer[0 .. amount]);
}
void sendMessage(in string message)
{
socket.connect(uaddr);
socket.send(message);
}
}
void main()
{
auto ipc = new UDSIPC();
if (ipc.getUniqueness())
{
while (true)
{
log(ipc.getMessage());
}
}
else
{
import std.stdio, std.string;
ipc.sendMessage(readln().chomp());
}
}

Returning code from pthread creation in C++ is 11

I have thread creation problem using Pthread. My code is as follows. I show only some portion due to space constraints.
Main.c create Detectdirection instance and send to the function.
d = new Detectdirection();
while(run)
{
int ret = d->run_parallel(d);
if(ret == -1)
run = false;
}
My Detectdirection Class has two functions to run in parallel:
class Detectdirection{
public:
int run_parallel(void*p);
void *Tracking(void *p);
static void *Tracking_helper(void * p);
void *ReadImage(void *p );
static void *ReadImage_helper(void *p );
private:
pthread_t thread[2];
}
void *Detectdirection::ReadImage(void *p){
Detectdirection *app = (Detectdirection*)p;
while(run){
}
pthread_exit(NULL);
}
void *Detectdirection::Tracking(void *p){
Detectdirection *app = (Detectdirection*)p;
while(run){
}
pthread_exit(NULL);
}
void *Detectdirection::Tracking_helper(void *p){
Detectdirection *app = (Detectdirection*)p;
return ((Detectdirection*)p)->Tracking(app);
}
void *Detectdirection::ReadImage_helper(void *p ){
Detectdirection *app = (Detectdirection*)p;
return ((Detectdirection*)p)->ReadImage(app);
}
int Detectdirection::run_parallel(void* p){
Detectdirection *app = (Detectdirection*)p;
int rc = pthread_create(&thread[0], NULL, app->ReadImage_helper, app);
if (rc) {
printf("ERROR; return code from pthread_create() is %d\n", rc);
return -1;
}
rc = pthread_create(&thread[1], NULL, app->Tracking_helper, app);
if (rc) {
printf("ERROR; return code from pthread_create() is %d\n", rc);
return -1;
}
return 0;
}
Compile is ok and when I run, I have thread creation error. That sort of return type 11 happens only when many threads are created. But now I create only two thread and I have that error. What could be wrong?
I believe your are getting EAGAIN (based on the error code 11). That (obivously) means your system doesn't have enough resources to create threads anymore.
POSIX documentation says:
[EAGAIN] The system lacked the necessary resources to create another
thread, or the system-imposed limit on the total number of threads in
a process {PTHREAD_THREADS_MAX} would be exceeded.
I am not quite sure the following is true.
But now I create only two thread and I have that error. What could be wrong?
Here,
while(run)
{
int ret = d->run_parallel(d);
if(ret == -1)
run = false;
}
You are creating in a loop and each call d->run_parallel() creates two threads. So, you are potentially creating infinite number of threads
as the loop only breaks when pthread_create() fails. So, you may want to look at this loop carefully whether you really want to do as it is right now.
You don't seem to join with the threads you create. So, you could detach the threads so that thread-specific resources are released immediately when the thread(s) exit.
You can do:
pthread_detach(pthread_self());
in both ReadImage_helper() and Tracking_helper() functions to detach them. This could potentially solve your resource issue.
If it's still present then you have to look at ways to limit the number of threads that are simultaneously running on your system. One possible option is to use thread pools -- create a fixed number of threads and assign them new tasks as the threads complete their current task(s).

memory leak with sockets and map

I have a socket server, everytime a new connection is made, a XClient class is instantiated and I am inserting it into a map. I am watching the memory usage through task manager. everytime a new connection is made, lets assume, the memory usage of my program increases by 800kb for example. Inside that class, there is a connected variable, which will tell me wheter this client is active or not. I created a thread to run endlessly and iterate through all the elements of my map and I'm checking if the connected variable is true or false. if it is false, I am (at least I think I am...) releasing the memory used by the previously instantiated XClient class. BUT, the memory usage is being decreased only half of the 800kb (for example, no precise values). So, when a client connects: +800kb. when client disconnects: -400kb. I think I have a memory leak? If I have 100 clients connected, that 400kb that is not being released would turn into 4000kb of non-used(?) memory, and that would be a problem.
So, here is my code.
The thread to iterate through all elements:
DWORD Update(XSockets *sockets)
{
while(true)
{
for(sockets->it = sockets->clients.begin(); sockets->it != sockets->clients.end(); sockets->it++)
{
int key = (*sockets->it).first;
if(sockets->clients[key]->connected == false) // remove the client, releasing memory
{
delete sockets->clients[key];
}
}
Sleep(100);
}
return true;
}
The code that is adding new XClients instances to my map:
bool XSockets::AcceptConnections()
{
struct sockaddr_in from;
while(true)
{
try
{
int fromLen = sizeof(from);
SOCKET client = accept(this->loginSocket,(struct sockaddr*)&from,&fromLen);
if(client != INVALID_SOCKET)
{
srand(time(NULL));
int clientKey = rand();
XClient* clientClass = new XClient(inet_ntoa(from.sin_addr),clientKey,client);
this->clients.insert(make_pair(clientKey,clientClass));
}
Sleep(100);
}
catch(...)
{
printf("error accepting incoming connection!\r\n");
break;
}
}
closesocket(this->loginSocket);
WSACleanup();
return true;
}
And the declarations:
map<int,XClient*> clients;
map<int,XClient*>::iterator it;
You've got several problems, but the chief one is that you appear to be sharing a map between threads without any synchronization at all. That can lead to all kinds of trouble.
Are you using c++11 or Boost? To avoid memory leak nightmares like this, you could create a map of shared pointers. This way, you can let the structure clean itself up.
This is how I would do it:
#include <memory>
#include <map>
#include <algorithm>
#include <functional>
#include <mutex>
typedef std::shared_ptr<XClient> XClientPtr;
std::map<int, XClientPtr> client;
std::mutex the_lock;
bool XSockets::AcceptConnections()
{
/* snip */
auto clientClass = std::make_shared<XClient>(/*... params ...*/);
the_lock.lock();
clients[clientKey] = clientClass;
the_lock.unlock();
/* snip */
}
bool client_is_connected(const std::pair<int, XClientPtr> &p) {
return p.second->connected;
}
DWORD Update(XSockets *sockets) {
while(true) { /* You should probably have some kind of
exit condition here. Like a global "running" bool
so that the thread will eventually stop. */
the_lock.lock();
auto it = sockets->clients.begin(), end = sockets->clients.end();
for(; it != end; ) {
if (!it->second->connected)
//Clients will be destructed here if their refcount goes to 0
sockets->clients.erase(it++);
else
++it;
}
the_lock.unlock();
Sleep(100);
}
return 1;
}
Note: Above code is untested. I haven't even tried to compile it.
See What happens to an STL iterator after erasing it in VS, UNIX/Linux?. In your case, you are not deleting everything, so you will want to not use a for loop.
sockets->it = sockets->clients.begin();
while (sockets->it != sockets->clients.end())
{
int key = (*sockets->it).first;
if(sockets->clients[key]->connected == false) // remove the client, releasing memory
{
delete sockets->clients[key];
sockets->clients.erase(sockets->it++);
}
else
{
sockets->it++;
}
}