I have a crash using protobuf, i need to use it with reflection.
enum ipVersionType{
ipv4 = 0;
ipv6 = 1;
}
message IpAddress {
required ipVersionType ipVersion = 1;
required uint32 IpPart1 = 2;
required uint32 IpPart2 = 3;
required uint32 IpPart3 = 4;
required uint32 IpPart4 = 5;
}
message TcpUdpCdr {
...
optional IpAddress DestinationIp = 8;
optional IpAddress UEIP = 11;
optional uint32 PacketUpLink = 12;
....
}
message cdr {
optional TcpUdpCdr tcpCdr = 1;
}
When i use TcpUdpCdr without cdr, i got no crash.
If I use TcpUdpCdr within cdr, i got a crash.
here is the code that i am using to set Ipaddress
//Fill proto ip address sruct
ProtoCdr::IpAddress * ipAddressMsg = new ProtoCdr::IpAddress();
ipAddressMsg->set_ipversion(ProtoCdr::ipVersionType::ipv4);
ipAddressMsg->set_ippart1(pi_ipAddress.GetAddresPointer()[0]);
ipAddressMsg->set_ippart2(pi_ipAddress.GetAddresPointer()[1]);
ipAddressMsg->set_ippart3(pi_ipAddress.GetAddresPointer()[2]);
ipAddressMsg->set_ippart4(pi_ipAddress.GetAddresPointer()[3]);
google::protobuf::Message& find_msg = cdrMsg.GetReflection()->GetMessage... with local recursive function
find_msg is of type TcpUdpCdr
find_msg.GetReflection()->SetAllocatedMessage(
&find_msg,
ipAddressMsg,
this->m_fdArray[m_iNestingSize-1]);
Up to here ther is no crash...
If i try to get the pointer with GetReflection GetMessage, i receive the same pointer address that have been Set with SetAllocatedMessage but when i try to use it it crash. It crash for the second IpAddress UEIP but not for the first one....
The problem is that you're using SetAllocatedMessage to assign ownership of ipAddressMsg to find_msg, but ipAddressMsg is already owned by cdrMsg. Because ipAddressMsg is now owned by two different protos, you will quickly run into problems because both owners will eventually try to delete it. I think your code is trying to use that submessage after it has already been deleted.
Note that in general with protobuf, methods that contain the word release or allocated are a bit of an advanced feature--they can improve performance but they increase complexity and require you to do more manual memory management. With those methods it's easy to accidentally run into memory leaks, double deletes, and use-after-delete. Unless you really need the performance, it's best to avoid them. In your case, you could just assign to the result MutableMessage() instead of calling SetAllocatedMessage().
Related
I'm trying to write a program whose job it is to go into shared memory, retrieve a piece of information (a struct 56 bytes in size), then parse that struct lightly and write it to a database.
The catch is that it needs to do this several dozens of thousands of times per second. I'm running this on a dedicated Ubuntu 14.04 server with dual Xeon X5677's and 32GB RAM. Also, Mongo is running PerconaFT as its storage engine. I am making an uneducated guess here, but say worst case load scenario would be 100,000 writes per second.
Shared memory is populated by another program who's reading information from a real time data stream, so I can't necessarily reproduce scenarios.
First... is Mongo the right choice for this task?
Next, this is the code that I've got right now. It starts with creating a list of collections (the list of items I want to record data points on is fixed) and then retrieving data from shared memory until it catches a signal.
int main()
{
//these deal with navigating shared memory
uint latestNotice=0, latestTurn=0, latestPQ=0, latestPQturn=0;
symbol_data *notice = nullptr;
bool done = false;
//this is our 56 byte struct
pq item;
uint64_t today_at_midnight; //since epoch, in milliseconds
{
time_t seconds = time(NULL);
today_at_midnight = seconds/(60*60*24);
today_at_midnight *= (60*60*24*1000);
}
//connect to shared memory
infob=info_block_init();
uint32_t used_symbols = infob->used_symbols;
getPosition(latestNotice, latestTurn);
//fire up mongo
mongoc_client_t *client = nullptr;
mongoc_collection_t *collections[used_symbols];
mongoc_collection_t *collection = nullptr;
bson_error_t error;
bson_t *doc = nullptr;
mongoc_init();
client = mongoc_client_new("mongodb://localhost:27017/");
for(uint32_t symbol = 0; symbol < used_symbols; symbol++)
{
collections[symbol] = mongoc_client_get_collection(client, "scribe",
(infob->sd+symbol)->getSymbol());
}
//this will be used later to sleep one millisecond
struct timespec ts;
ts.tv_sec=0;
ts.tv_nsec=1000000;
while(continue_running) //becomes false if a signal is caught
{
//check that new info is available in shared memory
//sleep 1ms if it isn't
while(!getNextNotice(¬ice,latestNotice,latestTurn)) nanosleep(&ts, NULL);
//get the new info
done=notice->getNextItem(item, latestPQ, latestPQturn);
if(done) continue;
//just some simple array math to make sure we're on the right collection
collection = collections[notice - infob->sd];
//switch on the item type and parse it accordingly
switch(item.tp)
{
case pq::pq_event:
doc = BCON_NEW(
//decided to use this instead of std::chrono
"ts", BCON_DATE_TIME(today_at_midnight + item.ts),
//item.pr is a uint64_t, and the guidance I've read on mongo
//advises using strings for those values
"pr", BCON_UTF8(std::to_string(item.pr).c_str()),
"sz", BCON_INT32(item.sz),
"vn", BCON_UTF8(venue_labels[item.vn]),
"tp", BCON_UTF8("e")
);
if(!mongoc_collection_insert(collection, MONGOC_INSERT_NONE, doc, NULL, &error))
{
LOG(1,"Mongo Error: "<<error.message<<endl);
}
break;
//obviously, several other cases go here, but they all look the
//same, using BCON macros for their data.
default:
LOG(1,"got unknown type = "<<item.tp<<endl);
break;
}
}
//clean up once we break from the while()
if(doc != nullptr) bson_destroy(doc);
for(uint32_t symbol = 0; symbol < used_symbols; symbol++)
{
collection = collections[symbol];
mongoc_collection_destroy(collection);
}
if(client != nullptr) mongoc_client_destroy(client);
mongoc_cleanup();
return 0;
}
My second question is: is this the fastest way to do this? The retrieval from shared memory isn't perfect, but this program is getting way behind its supply of data, far moreso than I need it to be. So I'm looking for obvious mistakes with regards to efficiency or technique when speed is the goal.
Thanks in advance. =)
I wonder why this code doesn't work:
#include <iostream>
using namespace std;
int main()
{
int *pointer = (int*)0x02F70BCC;
cout<<*pointer;
return 0;
}
In my opinion it should write on the screen value of 0x02F70BCC,
instead of this my programm crashes.
I know that memory with adress 0x02F70BCC stores value of 20.
But like I said no matter what it just doesn't want to show correct number.
Please help me guys, detailed explanation would be very nice of you.
It doesn't work, because you won't get access to every location in memory you want. Not every location in memory is valid, you may want to read about Virtual Address Space.
Some addresses are reserved for device drivers and kernel mode operations. Another range of addresses (for example 0xCCCCCCCC and higher) may be reserved for uninitialized pointers.
Even if some location is valid, operating system may still deny access to write to/read from certain location, if that would cause undefined behaviour or violate system safety.
EDIT
I think you might be interested in creating some kind of "GameHack", that allows you to modify amount of resources, number of units, experience level, attributes or anything.
Memory access is not a simple topic. Different OSes use different strategies to prevent security violations. But many thing can be done here, after all there is a lot software for doing such things.
First of all, do you really need to write your own tool? If you just want some cheating, use ArtMoney - it is a great memory editor, that I have been using for years.
But if you really have to write it manually, you need to do some research first.
On Windows, for example, I would start from these:
ReadProcessMemory
WriteProcessMemory
Also, I am quite certain, that one of possible techniques is to pretend, that you are a debugger:
DebugActiveProcess.
EDIT 2
I have done some research and it looks, that on Windows (I assume this is your platform, since you mentioned gaming; can't imagine playing anything on crappy Linux), steps required to write another process' memory are:
1. Enumerate processes: (EnumProcesses)
const size_t MAX_PROC_NUM = 512;
DWORD procIDs[MAX_PROC_NUM] = { 0 };
DWORD idsNum = 0;
if(!EnumProcesses(procIDs, sizeof(DWORD) * MAX_PROC_NUM, &idsNum))
//handle error here
idsNum /= sizeof(DWORD); //After EnumProcesses(), idsNum contains number of BYTES!
2. Open required process. (OpenProcess,GetModuleFileNameEx)
const char* game_exe_path = "E:\\Games\\Spellforce\\Spellforce.exe"; //Example
HANDLE game_proc_handle = nullptr;
DWORD proc_access = PROCESS_QUERY_INFORMATION | PROCESS_VM_READ | PROCESS_VM_WRITE; //read & write memory, query info needed to get .exe name
const DWORD MAX_EXE_PATH_LEN = 1024;
for(DWORD n = 0 ; n < idsNum ; ++idsNum)
{
DWORD current_id = procIDs[n];
HANDLE current_handle = OpenProcess(proc_access, false, current_id);
if(!current_handle)
{
//handle error here
continue;
}
char current_path[MAX_EXE_PATH_LEN];
DWORD length = GetModuleFileNameEx(current_handle, nullptr, current_path, MAX_EXE_PATH_LEN);
if(length > 0)
{
if(strcmp(current_path, game_exe_path) == 0) //that's our game!
{
game_proc_handle = current_handle;
break;
}
}
CloseHandle(current_handle); //don't forget this!
}
if(!game_proc_handle)
//sorry, game not found
3. Write memory (WriteProcessMemory)
void* pointer = reinterpret_cast<void*>(0x02F70BCC);
int new_value = 5000; //value to be written
BOOL success = WriteProcessMemory(game_proc_handle, pointer, &new_value, sizeof(int), nullptr);
if(success)
//data successfully written!
else
//well, that's... em...
This code is written just 'as is', but I see no errors, so you can use it as your starting point. I also provided links for all functions I used, so with some additional research (if necessary), you can achieve what you are trying to.
Cheers.
When you use,
cout<<*pointer;
the program tries to dereference the value of the pointer and writes the value at the address.
If you want to print just the pointer, use:
cout << pointer;
Example:
int main()
{
int i = 20;
int* p = &i;
std::cout << *p << std::endl; // print the value stored at the address
// pointed to by p. In this case, it will
// print the value of i, which is 20
std::cout << p << std::endl; // print the address that p points to
// It will print the address of i.
}
So I used this example of the HeapWalk function to implement it into my app. I played around with it a bit and saw that when I added
HANDLE d = HeapAlloc(hHeap, 0, sizeof(int));
int* f = new(d) int;
after creating the heap then some new output would be logged:
Allocated block Data portion begins at: 0X037307E0
Size: 4 bytes
Overhead: 28 bytes
Region index: 0
So seeing this I thought I could check Entry.wFlags to see if it was set as PROCESS_HEAP_ENTRY_BUSY to keep a track of how much allocated memory I'm using on the heap. So I have:
HeapLock(heap);
int totalUsedSpace = 0, totalSize = 0, largestFreeSpace = 0, largestCounter = 0;
PROCESS_HEAP_ENTRY entry;
entry.lpData = NULL;
while (HeapWalk(heap, &entry) != FALSE)
{
int entrySize = entry.cbData + entry.cbOverhead;
if ((entry.wFlags & PROCESS_HEAP_ENTRY_BUSY) != 0)
{
// We have allocated memory in this block
totalUsedSpace += entrySize;
largestCounter = 0;
}
else
{
// We do not have allocated memory in this block
largestCounter += entrySize;
if (largestCounter > largestFreeSpace)
{
// Save this value as we've found a bigger space
largestFreeSpace = largestCounter;
}
}
// Keep a track of the total size of this heap
totalSize += entrySize;
}
HeapUnlock(heap);
And this appears to work when built in debug mode (totalSize and totalUsedSpace are different values). However, when I run it in Release mode totalUsedSpace is always 0.
I stepped through it with the debugger while in Release mode and for each heap it loops three times and I get the following flags in entry.wFlags from calling HeapWalk:
1 (PROCESS_HEAP_REGION)
0
2 (PROCESS_HEAP_UNCOMMITTED_RANGE)
It then exits the while loop and GetLastError() returns ERROR_NO_MORE_ITEMS as expected.
From here I found that a flag value of 0 is "the committed block which is free, i.e. not being allocated or not being used as control structure."
Does anyone know why it does not work as intended when built in Release mode? I don't have much experience of how memory is handled by the computer, so I'm not sure where the error might be coming from. Searching on Google didn't come up with anything so hopefully someone here knows.
UPDATE: I'm still looking into this myself and if I monitor the app using vmmap I can see that the process has 9 heaps, but when calling GetProcessHeaps it returns that there are 22 heaps. Also, none of the heap handles it returns matches to the return value of GetProcessHeap() or _get_heap_handle(). It seems like GetProcessHeaps is not behaving as expected. Here is the code to get the list of heaps:
// Count how many heaps there are and allocate enough space for them
DWORD numHeaps = GetProcessHeaps(0, NULL);
HANDLE* handles = new HANDLE[numHeaps];
// Get a handle to known heaps for us to compare against
HANDLE defaultHeap = GetProcessHeap();
HANDLE crtHeap = (HANDLE)_get_heap_handle();
// Get a list of handles to all the heaps
DWORD retVal = GetProcessHeaps(numHeaps, handles);
And retVal is the same value as numHeaps, which indicates that there was no error.
Application Verifier had been set up previously to do a full page heap verifying of my executable and was interfering with the heaps returned by GetProcessHeaps. I'd forgotten about it being set up as it was done for a different issue several days ago and then closed without clearing the tests. It wasn't happening in debug build because the application builds to a different file name for debug builds.
We managed to detect this by adding a breakpoint and looking at the callstack of the thread. We could see the AV DLL had been injected in and that let us know where to look.
I have a BYTE array as follows:
BYTE* m_pImage;
m_pImage = new BYTE[m_someLength];
And at various stages of my program data is copied to this array like so:
BYTE* pDestinationBuffer = m_pImage + m_imageOffset;
memcpy( pDestinationBuffer, (BYTE*)data, dataLength );
But when I go to delete my buffer like so:
delete[] m_pImage;
I am getting the
HEAP CORRUPTION DETECTED - CRT detected that the application wrote to memory after the end of heap buffer
Now I have experimented with a simple program to try and replicate the error in order to help me investigate whats going on. I see from that following that if I create an array of size 5 but write over the end of it and try to delete it I get the exact same error.
int* myArray = new int[5];
myArray[0] = 0;
myArray[1] = 1;
myArray[2] = 2;
myArray[3] = 3;
myArray[4] = 4;
myArray[5] = 5; // writing beyond array bounds
delete[] myArray;
Now my question is how can I possibly debug or find out what is overwriting my original buffer. I know that something is overwriting the end of the buffer, so is there a way for visual studio to help me debug this easily.
The code above that is copying to the data buffer is called several times before the delete soits hard to keep a track of the m_pImage contents and the data copied to it. (Its about 2M worth of data)
Now my question is how can I possibly debug or find out what is overwriting my original buffer.
I would recommend to use assert() statement as much as possible. In this case it should be:
BYTE* pDestinationBuffer = m_pImage + m_imageOffset;
assert( dataLength + m_imageOffset <= m_someLength );
memcpy( pDestinationBuffer, (BYTE*)data, dataLength );
then compile into debug mode and run. Benefit of this method - you will not have any overhead in release mode, where asserts are not evaluated.
On Windows you can use the Application Verifier to find this kind of overwrite
Heap corruption is a tough bug to find. Most times, when the error is reported, the memory has already been corrupted by some up stream code that executed previously. If you decide to use Application Verifier (and you should), I'd also encourage you to try GFLags and PageHeap. They are some additional tools that allow you to set registry flags for debugging these types of problems.
I have the following problem.
I use the following function to receive a string from a buffer until a newline occurs.
string get_all_buf(int sock) {
int n = 1, total = 0, found = 0;
char c;
char temp[1024*1024];
string antw = "";
while (!found) {
n = recv(sock, &temp[total], sizeof(temp) - total - 1, 0);
if (n == -1) {
break;
}
total += n;
temp[total] = '\0';
found = (strchr(temp, '\n') != 0);
if (found == 0){
found = (strchr(temp, '\r\n') != 0);
}
}
antw = temp;
size_t foundIndex = antw.find("\r\n");
if (foundIndex != antw.npos)
antw.erase ( antw.find ("\r\n"), 2 );
foundIndex = antw.find("\n");
if (foundIndex != antw.npos)
antw.erase ( antw.find ("\n"), 2 );
return answ;
}
So use it like this:
string an = get_all_buf(sClient);
If I create an exe file everything works perfectly.
But if I create a dll and run it using rundll32 the application closes at "string an = get_all_buf(sClient);" without any error message...
I tried to fix this for hours now, and I am currently a bit desperate...
P.S. sorry for obvious errors or bad coding style, I just started learning C++.
char temp[1024*1024];
declares a 1Mb structure on the stack. This may be too large and overflow available stack memory. You could instead give it static scope
static char temp[1024*1024];
or allocate it dynamically
char* temp = (char*)malloc(1024*1024);
// function body
free(temp);
Alternatively, assuming the mention of run32.dll means you're working on Windows, you could investigate keeping it on the stack by using the /STACK linker option. This probably isn't the best approach - you've already found it causes problems when you change build settings or try to target other platforms.
Instead of creating temp variable on the stack, I'd create it dynamically (on the heap), but not using raw malloc and free as showed in a previous answer, but using modern C++ and std::vector:
#include <vector>
std::vector<char> temp(1024*1024);
This is exception safe, and you don't have to pay attention to release the allocated memory: std::vector's destructor will do that automatically (also in case of exceptions thrown).
Instead of sizeof(temp), in your code you can use temp.size() (which will return the count of elements in the vector, and since this is a vector of chars, it will return just the total vector size in chars i.e. in bytes).
You can still use operator[] for std::vector, as you do for raw C arrays.
Note also that if you are building a DLL and the above function is exposed at the DLL interface, since this function has a C++ interface with a STL class (std::string) at the boundary, you must pay attention that both your DLL and your clients are built with dynamic linking to the same CRT, and with the same compiler and the same compiler settings (e.g. you can't mix a DLL built with VS2008/VC9 with a .EXE built with VS2010/VC10, or a release-build DLL with a debug-build EXE built with the same compiler).