protobuf SerializeToArray rasie segmentation fault at InternalSerializeWithCachedSizesToArray - c++

To reuse protobuf messages, I firstly allocate it(msg_test,msg_proto), and then send it in an loop(for every loop do something like clear,set,send).
protobuf syntax = "proto3";
Call:
int i=0;
while(true){
SendTestImprove(i++);
usleep(1000*1000);
}
Function:
int SendTestImprove(int count)
{
msg_test->Clear();
msg_test->set_count(count);
msg_proto->Clear();
msg_proto->set_allocated_count(msg_test);
msg_proto->set_tick_count(GetTickCount());
zmq_msg_t zmsg;
int size = msg_proto->ByteSize();
int rc = zmq_msg_init_size(&zmsg,size);
if(rc==0){
try
{
rc = msg_proto->SerializeToArray(zmq_msg_data(&zmsg),size)?0:-1;
}
catch (google::protobuf::FatalException fe)
{
LOGFMTE("PbToZmq error: %s",fe.message().c_str());
}
}
int zsize = zmq_msg_size(&zmsg);
rc = zmq_msg_send(&zmsg,m_pub,0);
zmq_msg_close(&zmsg);
LOGFMTD("zmq_msg_send,size=%d,zsize=%d",rc,zsize);
return 0;
}
The error occurred when i=1, at the line :
rc = msg_proto->SerializeToArray(zmq_msg_data(&zmsg),size)?0:-1;
error info is like:
stopped:segmentation fault,at
InternalWriteMessageNoVirtualToArray
,InternalSerializeWithCachedSizesToArray
Can anyone help?

The likely problem is that you're assigning ownership of msg_test to msg_proto in each iteration of the loop, by calling msg_proto->set_allocated_count(msg_test). The first time you do that it's fine, but then the next time through the loop when you call set_allocated_count() a second time, the proto will delete msg_test and reassign the now dangling pointer. The simplest solution is to avoid calling set_allocated_count() and instead just assign a copy of msg_test, like this:
*msg_proto->mutable_count() = msg_test;

Related

Do I need to delete objects passed to google protocol buffer (protobuf)?

I have simple messages:
message SmallValue {
int32 val = 1;
}
message Value {
int32 val1 = 1;
int32 val2 = 2;
SmallValue val3 = 3;
}
message SendMessage {
int32 id = 1;
oneof message {
Value value= 2;
}
My piece of code:
// create new pointer for smallValue
SmallValue* smallValue = new SmallValue();
smallValue->set_val3(3);
// create new object value and set_allocated_val3
Value value;
value.set_val1(1);
value.set_val2(2);
value.set_allocated_val3(smallValue);
// create new object message and set_allocated_value
SendMessage message;
message.set_id(0);
message.set_allocated_value(&value);
// after some work, release value from message
message.release_value();
And my questions are:
1. After calling message.release_value() is it OK not to call delete &value; as I didn't create new pointer?
2. Will memory of smallValue will be deleted automatically along with value as I didn't call value.release_smallValue();?
// I'm a newbie to C++ as well as protobuf. Please do tell if something odd about my code.
Thanks!
It is usually best to avoid using the set_allocated_* and release_* methods; those provide advanced memory-management features that you should not need unless you are really trying to optimize some performance-critical code.
You could rewrite your code like this to avoid having to worry about memory management as much:
SendMessage message;
message.set_id(0);
Value* value = message.mutable_value();
value->set_val1(1);
value->set_val2(2);
value->mutable_val3()->set_val(3);

MySQL Connector/C++ error in while loop

void dataProcess::loginProcess(byte* packet, uint16_t length)
{
unique_ptr<ResultSet> res;
unique_ptr<Statement> stmt;
byte idPacket[4] = {packet[9], packet[8], packet[7], packet[6]};
uint32_t id, account_id;
string account_pass, account_type;
id = decodeToInt(idPacket, 4);
account_pass = decodeToChar(packet, 14, length - 1);
account_type = decodeToChar(packet, 10, 11);
stmt.reset(this->con->createStatement());
res.reset(stmt->executeQuery("SELECT * FROM account_info"));
while(res->next())
{
account_id = res->getUInt("id");
if(lowerCase(account_type).compare("ts") == 0)
{
if((account_id == id) && (account_pass.compare(res->getString("password")) == 0))
{
sendPacket("F4440300010300");
}
else
{
wrongPass();
}
}
else
{
wrongPass();
}
}
}
so the above function is being called once in each iteration of an infinite while loop and the function always failed at the 2nd iteration. I think this is somehow related to the deletion of the ResultSet object at the first iteration because when I try using a normal pointer and using the delete statement the same problem occurs but if I remove the delete it can get through the 2nd iteration but I'm quite sure that the ResultSet needs to be deleted according to the example on MySQL website. I am quite new to mysql Connector/C++ so I am very unsure to what is causing this problem. From debugging it appears to fail at this line on the second iteration.
res.reset(stmt->executeQuery("SELECT * FROM account_info"));
This is the error that appears after the program enters the second iteration
Error in `./server': free(): invalid size: 0x00007f8214004ca0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x790cb)[0x7f8219aa70cb]
/lib/x86_64-linux-gnu/libc.so.6(+0x8275a)[0x7f8219ab075a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f8219ab418c]
/usr/lib/x86_64-linux-gnu/libmysqlcppconn.so.7(_ZN5boost20checked_array_deleteIc EEvPT_+0x1f)[0x7f821a9397df]
/usr/lib/x86_64-linux-gnu/libmysqlcppconn.so.7(_ZN5boost12scoped_arrayIcED1Ev+0x 1b)[0x7f821a9390b5]
/usr/lib/x86_64-linux-gnu/libmysqlcppconn.so.7(_ZN3sql5mysql15MySQL_ResultSetC2E N5boost10shared_ptrINS0_9NativeAPI22NativeResultsetWrapperEEENS2_8weak_ptrINS4_2 3NativeConnectionWrapperEEENS_9ResultSet9enum_typeEPNS0_15MySQL_StatementERNS3_I NS0_17MySQL_DebugLoggerEEE+0x205)[0x7f821a98cde3]
/usr/lib/x86_64-linux-gnu/libmysqlcppconn.so.7(_ZN3sql5mysql15MySQL_Statement12e xecuteQueryERKNS_9SQLStringE+0x11b)[0x7f821a99443f]
./server(+0x326e)[0x56192448126e]
./server(+0x2f40)[0x561924480f40]
./server(+0x2da3)[0x561924480da3]
./server(+0x2a45)[0x561924480a45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ca)[0x7f8219dfc6ca]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x5f)[0x7f8219b360af]
class and the constructor:
class dataProcess : public thread
{
public:
dataProcess(int sock);
virtual ~dataProcess();
protected:
virtual void thread_handler();
private:
void wrongPass();
void loginProcess(byte* packet, uint16_t length);
void sendPacket(string packet);
int sock;
player* p;
Driver* driver;
unique_ptr<Connection> con;
};
dataProcess::dataProcess(int sock)
{
this->sock = sock;
driver = get_driver_instance();
con.reset(driver->connect("localhost", "root", "password"));
con->setSchema("ts_server");
thread::startThread();
}
Ok guys I have solved this problem and I have learned a very important lesson as a newbie. The problem occurs due to the undefined behaviour caused by my carelessness to ensure I have "deleted all my dynamically allocated variables" Once I have added delete in the right place everything works perfectly. so anyone facing these kind of random problems make sure to check these :)

Memory usage of C++ program grows, (shown in Debian's "top"), until it crashes

I'm working on a C++ program that should be able to run for several days, so it is a bit of a hassle that its memory consumption seems to grow really fast.
The full code of the program is a little long, so I'll post just the related things. The structure is the following:
int main (void){
//initialization of the global variables
error = 0;
state = 0;
cycle = 0;
exportcycle = 0;
status = 0;
counter_temp_ctrl = 0;
start = 0;
stop = 0;
inittimer();
mysql_del ("TempMeas");
mysql_del ("TempMeasHist");
mysql_del ("MyControl");
mysql_del ("MyStatus");
initmysql();
while(1){
statemachine();
pause();
}
}
The timer function that is initialized above is the following:
void catch_alarm (int sig)
{
//Set the statemachine to state 1 (reading in new values)
start = readmysql("MyStatus", "Start", 0);
stop = readmysql("MyStatus", "Stop", 0);
if (start == 1){
state = 1;
}
if (stop == 1){
state = 5;
}
//printf("Alarm event\n");
signal (sig, catch_alarm);
return void();
}
So basically, since I'm not setting the start bit in the webinterface that modifies the MyStatus Tab the program just calls the readmysql function twice every second (the timer's interval). The readmysql function is given below:
float readmysql(string table, string row, int lastvalue){
float readdata = 0;
// Initialize a connection to MySQL
MYSQL_RES *mysql_res;
MYSQL_ROW mysqlrow;
MYSQL *con = mysql_init(NULL);
if(con == NULL)
{
error_exit(con);
}
if (mysql_real_connect(con, "localhost", "user1", "user1", "TempDB", 0, NULL, 0) == NULL)
{
error_exit(con);
}
if (lastvalue == 1){
string qu = "Select "+ row +" from "+ table +" AS a where MeasTime=(select MAX(MeasTime) from "+ table;
error = mysql_query(con, qu.c_str());
}
else{
string qu = "Select "+ row +" from "+ table;
error = mysql_query(con, qu.c_str());
}
mysql_res = mysql_store_result(con);
while((mysqlrow = mysql_fetch_row(mysql_res)) != NULL)
{
readdata = atoi(mysqlrow[0]);
}
//cout << "readdata "+table+ " "+row+" = " << readdata << endl;
// Close the MySQL connection
mysql_close(con);
//delete mysql_res;
//delete mysqlrow;
return readdata;
}
I thought that the variables in this function are stored on the stack and are freed automaticaly when leaving the function. However it seems that some part of the memory is not freed, because it just grows after all. As you can see I have tried to use the delete function on two of the variables. Seems to have no effect. What am i doing wrong in terms of memory-management and so on?
Thanks for your help!
Greetings Oliver.
At least mysql_store_result is leaking. From documentation:
After invoking mysql_query() or mysql_real_query(), you must call mysql_store_result() or mysql_use_result() for every statement that successfully produces a result set (SELECT, SHOW, DESCRIBE, EXPLAIN, CHECK TABLE, and so forth). You must also call mysql_free_result() after you are done with the result set.
If your program continuously consumes memory (without ever releasing it), then you have a memory leak.
A good way to detect memory leaks, is to run it through a memory debugger, e.g. valgrind:
$ valgrind /path/to/my/program
Once your program started eating memory, stop it and valgrind will give you a nice summary about where your program allocated memory that was never freed.
There is no need to let the system run out of memory and crash; just wait until it has eaten some memory that has not been freed. Then fix your code. Then repeat until no more memory errors can be detected.
Also note that valgrind intercepts your systems memory management. This usually results in a (severe) performance penalty.

Segmentation fault(core dumped) in multi threading using boost threads

When try to run my program with up to 1 thread, it works fine for a while (some seconds or minutes) but finally get segmentation fault(core dumped) or double free(faststop ) error.
Here are the function which the threads run.
//used in the Function
[Added] typedef folly::ProducerConsumerQueue<std::string*> PcapTask;
struct s_EntryItem {
Columns* p_packet; //has some arbitrary method and variables
boost::mutex _mtx;
};
//_buffersConnection.wait_and_pop()
Data wait_and_pop() {
boost::mutex::scoped_lock lock(the_mutex);
while (the_queue.empty()) {
the_condition_variable.wait(lock);
}
Data popped_value = the_queue.front();
the_queue.pop();
return popped_value;
}
struct HandlerTask {
std::string year;
folly::ProducerConsumerQueue<std::string*> queue = NULL;
};
-----------------------------------------
//The function which threads run
void Connection() {
std::string datetime, year;
uint32_t srcIPNAT_num, srcIP_num;
std::string srcIP_str, srcIPNAT_str, srcIPNAT_str_hex;
int counter = 0;
while (true) {
//get new task
HandlerTask* handlerTask = _buffersConnection.wait_and_pop();
PcapTask* pcapTask = handlerTask->queue;
year = handlerTask->year;
counter = 0;
do {
pcapTask->popFront();
s_EntryItem* entryItem = searchIPTable(srcIP_num);
entryItem->_mtx.lock();
if (entryItem->p_packet == NULL) {
Columns* newColumn = new Columns();
newColumn->initConnection(srcIPNAT_str, srcIP_str, datetime, srcIP_num);
entryItem->p_packet = newColumn;
addToSequanceList(newColumn);
} else {
bool added = entryItem->p_packet->addPublicAddress(srcIPNAT_str_hex, datetime);
if (added == false) {
removeFromSequanceList(entryItem->p_packet);
_bufferManager->addTask(entryItem->p_packet);
Columns* newColumn = new Columns();
newColumn->initConnection(srcIPNAT_str, srcIP_str, datetime, srcIP_num);
//add to ip table
entryItem->p_packet = newColumn;
addToSequanceList(newColumn);
}
}
entryItem->_mtx.unlock();
++_totalConnectionReceived;
} while (true);
delete pcapTask;
delete handlerTask;
}
}
You can use Valgrind, its very easy. Build your app in debug config and pass program executable to valgrind. It can tell you wide spectre of programming errors occuring in your app in runtime. The price of using Valgrind is that program runs considerably slower (some times tens times slower) than without Valgrind. Specically, for example, Valgrind will tell you where your your programs' memory was free'ed first when it tried to free it second time when it happens.
I'm not sure that it's the problem, but...
Are you sure that you must call delete over pcapTask?
I mean: you delete it but queue in struct HandlerTask is a class member, not a pointer to a class.
Suggestion: try to comment the line
delete pcapTask;
at the end of Connection()
--- EDIT ---
Looking at you added typedef, I confirm that (if I'm not wrong) there is something strange in your code.
pcapTask is defined as a PcapTask pointer, that is a folly::ProducerConsumerQueue<std::string*> pointer; you initialize it with a folly::ProducerConsumerQueue<std::string*> (not pointer)
I'm surprised that you can compile your code.
I think you should, first of all, resolve this antinomy.
p.s.: sorry for my bad English.

Pthread and void* attempt to de-reference a generic pointer

When I debug my PRJ I get this error:
args Error: Multiple errors reported.\ Failed to execute MI command: -var-create -
args Error message from debugger back end: Attempt to dereference a generic pointer.\ Unable to create variable object
the error comes when casting from void* args to Mapper* arg.
UPDATE 1
KMaster, KMapper implements respectively Master, Mapper but they do not add nothing rilevant. Effectively is KMapper that call the method work(). Here is the code:
int main(){
int np=1,K=4;
string path="lucca.gps";
KMaster* master=new KMaster(path,np,K);
KMapper* mapper[np];
master->splitting();
for(int i=0;i<np;i++){
mapper[i]=new KMapper(master,master->mData[i].key,master->mData[i].value);
while(mapper[i]->work()!=0){
cout<<"failed creating mapper, retry..."<<endl;
sleep(1000);
}
}
}
int KMaster::splitting(){
cout<<"start splitting"<<endl;
fstream is(path.c_str());
string s="";
getline(is,s);
while(!is.eof()){
for(int i=0;i<nProc;i++){
pair<double,double> res;
is>>res.first;
is>>res.second;
is>>s;
mapData[i].push_back(res);
Data.push_back(res);
if(is.eof()) break;
}
}
list<pair<double,double> >::iterator it=Data.begin();
int increment=Data.size()/K;
for(int i=0;i<K;i++){
Klusters.push_back(*it);
advance(it,increment);
}
for(int i=0;i<nProc;i++){
mData[i].key=&Klusters;
mData[i].value=&mapData[i];
}
cout<<"splitting completed"<<endl;
return 0;
}
int Mapper::work(){
Mapper* m=this;
void* p=m;
return pthread_create(&thread,NULL,start,p);
}
void* start(void* args){
cout<<"start()"<<endl;
Mapper* arg= reinterpret_cast<Mapper*>(args);
arg->mapResult=arg->map(arg->k,arg->v);
cout<<"Mapper finish, sending result..."<<endl;
arg->send(arg->mapResult);
}
Hope that someone can help!
UPDATE 2
Screenshot of the debugger:
The value of arg is 24, which no normal object would live at, so the casting has nothing whatsoever to do with this.
Unfortunately, this "answer" can only be a wild guess because you haven't shown the calling code.
If arg is always 24, I would check for something similar to this:
class Something
{
public:
void dostuff() { mapper.work(); }
private:
// 24 bytes of "stuff" before this member (including possibly a vtable)
Mapper mapper;
};
Something *thing = 0;
thing->dostuff(); // Thing->mapper will have address 24.
It's also possible that you have an uninitialised variable
Mapper* mapper;
mapper->work(); // Oops, uninitialised
which just happens to be 24.