I am copying the contents of one STL vector to another.
The program is something like this
std::vector<uint_8> l_destVector(100); //Just for illustration let us take the size as 100.
std::vector<uint_8> l_sourceVector; //Let us assume that source vector is already populated.
memcpy( l_destVector.data(), l_sourceVector.data(), l_sourceVector.size() );
The above example is pretty simplistic but in my actual code the size
of destination vector is dynamically calculated.
Also the source vector is getting populated dynamically making it possible to have different length of data.
Hence it increases the chance of buffer overrun.
The problem I faced is my program is not crashing at the point of memcpy when there is a buffer overrun but sometime later making it hard to debug.
How do we explain this behavior?
/******************************************************************************************************/
Based on the responses I am editing the question to make my concern more understandable.
So, this is a legacy code and there are lot of places where vector has been copied using memcpy, and we do not intend to change the existing code. My main concern here is "Should memcpy not guarantee immediate crash, if not why ?", I would honestly admit that this is not very well written code.
A brief illustration of actual use is as follows.
In the below method, i_DDRSPDBuffer and i_dataDQBuffer where generated based on some logic in the calling method.
o_dataBuffer was assigned a memory space that would have been sufficient to take the data from two input buffers, but some recent changes in method that calls updateSPDDataToRecordPerDimm, is causing overrun in one of the flows.
typedef std::vector<uint8_t> DataBufferHndl;
errHdl_t updateSPDDataToRecordPerDimm(
dimmContainerIterator_t i_spdMmap,
const DataBufferHndl & i_DDRSPDBuffer,
const DataBufferHndl & i_dataDQBuffer,
DataBufferHndl & o_dataBuffer)
{
uint16_t l_dimmSPDBytes = (*i_spdMmap).second.dimmSpdBytes;
// Get the Data Buffer Handle for the input and output vectors
uint8_t * l_pOutLDimmSPDData = o_dataBuffer.data();
const uint8_t * l_pInDDRSPDData = i_DDRSPDBuffer.data();
const uint8_t * l_pInDQData = i_dataDQBuffer.data();
memcpy(l_pOutLDimmSPDData, l_pInDDRSPDData, l_dimmSPDBytes);
memcpy(l_pOutLDimmSPDData + l_dimmSPDBytes,
l_pInDQData, LDIMM_DQ_DATA_BYTES);
memcpy(l_pOutLDimmSPDData ,
l_pInDQData, LDIMM_DQ_DATA_BYTES); ====> Expecting the crash here but the crash happens some where after the method updateSPDDataToRecordPerDimm returns.
}
It doesn't have to crash, it's undefined behaviour.
If you had used std::copy instead with std::vector<uint_8>::iterators in debug mode, you probably would've hit an assertion which would've caught it.
Do not do that! It will eventually bite you.
Use either std::copy and and output_iterator, or since you know the size of the destination, resize the vector to the correct size or create a vector of the correct size and pipe the contents straight in, or simply the assignment operator.
It doesn't crash right at the moment of memcpy because you 'only' overwrite the memory behind the allocated vector. As long as your program does not read from that corrupt memory and use the data, your program will continue to run.
As already mentioned before, using memcpy is not the recommended way to copy the contents of stl containers. You'd be on the safe side with
std::copy
std::vector::assign
And in both cases you'd also get the aforementioned iterator debugging which will trigger close to the point where the error actually is.
Related
I'm writing a fairly straightforward function that sends an array over to a file descriptor. However, in order to send the data, I need to append a one byte header.
Here is a simplified version of what I'm doing and it seems to work:
void SendData(uint8_t* buffer, size_t length) {
uint8_t buffer_to_send[length + 1];
buffer_to_send[0] = MY_SPECIAL_BYTE;
memcpy(buffer_to_send + 1, buffer, length);
// more code to send the buffer_to_send goes here...
}
Like I said, the code seems to work fine, however, I've recently gotten into the habit of using the Google C++ style guide since my current project has no set style guide for it (I'm actually the only software engineer on my project and I wanted to use something that's used in industry). I ran Google's cpplint.py and it caught the line where I am creating buffer_to_send and threw some comment about not using variable length arrays. Specifically, here's what Google's C++ style guide has to say about variable length arrays...
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Variable-Length_Arrays_and_alloca__
Based on their comments, it appears I may have found the root cause of seemingly random crashes in my code (which occur very infrequently, but are nonetheless annoying). However, I'm a bit torn as to how to fix it.
Here are my proposed solutions:
Make buffer_to_send essentially a fixed length array of a constant length. The problem that I can think of here is that I have to make the buffer as big as the theoretically largest buffer I'd want to send. In the average case, the buffers are much smaller, and I'd be wasting about 0.5KB doing so each time the function is called. Note that the program must run on an embedded system, and while I'm not necessarily counting each byte, I'd like to use as little memory as possible.
Use new and delete or malloc/free to dynamically allocate the buffer. The issue here is that the function is called frequently and there would be some overhead in terms of constantly asking the OS for memory and then releasing it.
Use two successive calls to write() in order to pass the data to the file descriptor. That is, the first write would pass only the one byte, and the next would send the rest of the buffer. While seemingly straightforward, I would need to research the code a bit more (note that I got this code handed down from a previous engineer who has since left the company I work for) in order to guarantee that the two successive writes occur atomically. Also, if this requires locking, then it essentially becomes more complex and has more performance impact than case #2.
Note that I cannot make the buffer_to_send a member variable or scope it outside the function since there are (potentially) multiple calls to the function at any given time from various threads.
Please let me know your opinion and what my preferred approach should be. Thanks for your time.
You can fold the two successive calls to write() in your option 3 into a single call using writev().
http://pubs.opengroup.org/onlinepubs/009696799/functions/writev.html
I would choose option 1. If you know the maximum length of your data, then allocate that much space (plus one byte) on the stack using a fixed size array. This is no worse than the variable length array you have shown because you must always have enough space left on the stack otherwise you simply won't be able to handle your maximum length (at worst, your code would randomly crash on larger buffer sizes). At the time this function is called, nothing else will be using the further space on your stack so it will be safe to allocate a fixed size array.
I'm writing a small program that reads the bytes from a file in binary file in groups of 16 bytes (please don't ask why), modifies them, and then writes them to another file.
The fstream::read function reads into a char * buffer, which I was initially passing to a function that looks like this:
char* modify (char block[16], std::string key)
The modification was done on block which was then returned. On roaming the posts of SO, I realized that it might be a better idea to use std::vector<char>. My immediate next worry was how to convert a char * to a std::vector<char>. Once again, SO gave me an answer.
But now what I'm wondering is: If its such a good idea to use std::vector<char> instead of char*, why do the fstream functions use char* at all?
Also, is it a good idea to convert the char* from fstream to std::vector<char> in the first place?
EDIT: I now realize that since fstream::read is used to write data into objects directly, char * is necessary. I must now modify my question. Firstly, why are there no overloaded functions for fstream::read? And secondly, in the program that I've written about, which is a better option?
To use it with a vector, do not pass a pointer to the vector. Instead, pass a pointer to the vector content:
vector<char> v(size);
stream.read(&v[0], size);
fstream() functions let you use char*s so you can point them at arbitrary pre-allocated buffers. std::vector<char> can be sized to provide an appropriate buffer, but it will be on the heap and there's allocation costs involved with that. Sometimes too you may want to read or write data to a specific location in memory - even in shared memory - rather than accepting whatever heap memory the vector happens to have allocated. Further, you may want to use fstream without having included the vector header... it's nice to be able to avoid unnecessary includes as it reduces compilation time.
As your buffers are always 16 bytes in size, it's probably best to allocate them as char [16] data members in an appropriate owning object (if any exists), or on the stack (i.e. some function's local variable).
vector<> is more useful when the alternative is heap allocation - whether because the size is unknown at compile time, or is particularly large, or you want more flexible control of the memory lifetime. It's also useful when you specifically want some of the other vector functionality, such as ability to change the number of elements afterwards, to sort the bytes etc. - it seems very unlikely you'll want to do any of that so a vector raises questions in the mind of the person reading your code about what you'll do for no good purpose. Still, the choice of char[16] vs. vector appears (based on your stated requirements) more a matter of taste than objective benefit.
I'm wrapping up user space linux socket functionality in some C++ for an embedded system (yes, this is probably reinventing the wheel again).
I want to offer a read and write implementation using a vector.
Doing the write is pretty easy, I can just pass &myvec[0] and avoid unnecessary copying. I'd like to do the same and read directly into a vector, rather than reading into a char buffer then copying all that into a newly created vector.
Now, I know how much data I want to read, and I can allocate appropriately (vec.reserve()). I can also read into &myvec[0], though this is probably a VERY BAD IDEA. Obviously doing this doesn't allow myvec.size to return anything sensible. Is there any way of doing this that:
Doesn't completely feel yucky from a safety/C++ perspective
Doesn't involve two copies of the data block - once from kernel to user space and once from a C char * style buffer into a C++ vector.
Use resize() instead of reserve(). This will set the vector's size correctly -- and after that, &myvec[0] is, as usual, guaranteed to point to a continguous block of memory.
Edit: Using &myvec[0] as a pointer to the underlying array for both reading and writing is safe and guaranteed to work by the C++ standard. Here's what Herb Sutter has to say:
So why do people continually ask whether the elements of a std::vector (or std::array) are stored contiguously? The most likely reason is that they want to know if they can cough up pointers to the internals to share the data, either to read or to write, with other code that deals in C arrays. That’s a valid use, and one important enough to guarantee in the standard.
I'll just add a short clarification, because the answer was already given. resize() with argument greater than current size will add elements to the collection and default - initialize them. If You create
std::vector<unsigned char> v;
and then resize
v.resize(someSize);
All unsigned chars will get initialized to 0. Btw You can do the same with a constructor
std::vector<unsigned char> v(someSize);
So theoretically it may be a little bit slower than a raw array, but if the alternative is to copy the array anyway, it's better.
Reserve only prepares the memory, so that there is no reallocation needed, if new elements are added to the collection, but You can't access that memory.
You have to get an information about the number of element written to Your vector. The vector won't know anything about it.
Assuming it's a POD struct, call resize rather than reserve. You can define an empty default constructor if you really don't want the data zeroed out before you fill the vector.
It's somewhat low level, but the semantics of construction of POD structs is purposely murky. If memmove is allowed to copy-construct them, I don't see why a socket-read shouldn't.
EDIT: ah, bytes, not a struct. Well, you can use the same trick, and define a struct with just a char and a default constructor which neglects to initialize it… if I'm guessing correctly that you care, and that's why you wanted to call reserve instead of resize in the first place.
If you want the vector to reflect the amount of data read, call resize() twice. Once before the read, to give yourself space to read into. Once again after the read, to set the size of the vector to the number of bytes actually read. reserve() is no good, since calling reserve doesn't give you permission to access the memory allocated for the capacity.
The first resize() will zero the elements of the vector, but this is unlikely to create much of a performance overhead. If it does then you could try Potatoswatter's suggestion, or you could give up on the size of the vector reflecting the size of the data read, and instead just resize() it once, then re-use it exactly as you would an allocated buffer in C.
Performance-wise, if you're reading from a socket in user mode, most likely you can easily handle data as fast as it comes in. Maybe not if you're connecting to another machine on a gigabit LAN, or if your machine is frequently running 100% CPU or 100% memory bandwidth. A bit of extra copying or memsetting is no big deal if you are eventually going to block on a read call anyway.
Like you, I'd want to avoid the extra copy in user-space, but not for performance reasons, just because if I don't do it, I don't have to write the code for it...
I'm trying to debug a legacy code written for Linux. Sometimes the application gets a segfault when it reaches the memcpy call in the following method:
std::vector<uint8> _storage;
size_t _wpos;
void append(const uint8 *src, size_t cnt)
{
if (!cnt)
return;
if (_storage.size() < _wpos + cnt)
_storage.resize(_wpos + cnt);
memcpy(&_storage[_wpos], src, cnt);
_wpos += cnt;
}
The values are as follows:
_storage.size() is 1000
_wpos is 0
*src points to an array of uint8 with 3 values: { 3, 110, 20 }
cnt is 3
I have no idea why this happens since this method gets called thousands of times during the application's runtime but it sometimes gets a segfault.
Any one has any idea how to solve this?
Your code looks good in terms of the data that is written. Are you absolutely sure that you're passing in the right src pointer? What happens when you run the code with a debugger such as gdb? It should halt on the segfault, and then you can print out the values of _storage.size(), src, and cnt.
I'm sure you'll find that (at least) one of those is not at all what you're expecting. You might have passed an invalid src; you might have passed an absurdly large cnt.
I'd suggest to run valgrind on your program.
It's very valuable to spot early memory corruption as it may be the case with your program (since it's not a systematic crash you got).
For the values you give, I can't see why that would segfault. It's possible that your segfault is a delayed failure due to an earlier memory management mistake. Writing past the end of the vector in some earlier function could cause some of the vector's internal members to be corrupted, or you may have accidentally freed part of the memory used by the vector earlier. I'd check the other functions that manipulate the vector to see if any of them are doing any suspicious casting.
I see the size of the vector increasing. I never see it decreasing.
Next to that, vector has expquisit memory management support builtin. You can insert your values right to the end:
vector.insert( src, src+cnt );
This will both expand the vector to the right size, and copy the values.
The only thing I can think of is that _storage.resize() fails (which should throw a bad_alloc exception).
Another alternative would be to append each value separately with a call to push_back() (probably far slower though).
I see one problem here.
The memcpy() function copies n bytes from memory, so if cnt is the number of elements, you need a *sizeof(uint8) in the call to memcpy.
In a comment to my other answer, you said that "The vector gets cleaned up in another method since it is a class member variable. I'll test insert and see what happen".
What about thread-safety? Are you absolutely sure that the clearing method does not clear 'while' the resize is happening, or immediately after it? Since it's a 'sometimes' problem, it may be induced by concurrent access to the memory management in the vector.
I'm currently using vectors as c-style arrays to send and recieve data through Winsock.
I have a std::vector and I'm using that as my 'byte array'.
The problem is, I'm using two vectors, one for each send, and one for each recv, but what I'm doing seems to be fairly inefficient.
Example:
std::string EndBody("\r\n.\r\n");
std::fill(m_SendBuffer.begin(),m_SendBuffer.end(),0);
std::copy(EndBody.begin(),EndBody.end(),m_SendBuffer.begin());
SendData();
SendData just calls send the appropriate amount of times and ensures everything works as it should.
Anyway. Unless I zero out the vector before each use I get errors with stuff overlapping. Is there a more efficient way for me to do what I'm doing? Because it seems that zeroing out the entire buffer on each call is horribly inefficient.
Thanks.
you can use m_SendBuffer.clear()
otherwise the end() method would not know what is the real size of the buffer.
clear() is not a very expensive method to call. Unless you're working on some 486 or something it shouldn't affect your performances
Seems like the other posters are focusing on the cost of clearing the buffer, or the size of the buffer. Yet you don't really need to clear or zero out the whole buffer, or know its size, for what you're doing. The 'errors with stuff overlapping' is a problem with SendData, that you've not posted the code for. Presumably SendData doesn't know how much of the buffer it needs to send unless the data within it is zero-terminated. if that assumption is correct, all you have to do is zero-terminate the data correctly.
std::copy(EndBody.begin(),EndBody.end(),m_SendBuffer.begin());
m_SendBuffer[EndBody.size()] = 0;
SendData();
Wouldn't calling clear mean the vector gets a new size of 0? If the OP is using the vector as a large chunk of memory then they'd have to then call resize after clear to ensure the appropriate space is available for calls to send and recv.
Calling clear then resize on the vector would be around the same as just filling it with zeros would it not?
vector::clear
vector::resize
fill
As far as I understand the STL docs, calling clear simply sets the .end() value to be the same as .begin() and sets size to zero,which is instant.
It doesn't change the amount of memory allocated or where the memory is (any iterator will obviously be invalid, but the data tends to linger!). The .capacity() doesn't change and neither does the data stored there, as you have already discovered. If you are always using .begin() .end() and STL iterators to access the area this won't matter.
Don't forget, method variables of a class aren't initialised unless you include them in your initialisation list. Adding m_SendBuffer(BUFSIZE,0) there might do the trick.