I'm pretty new to this so any clarification would be appreciated. When using the function ReadFile, how does the type of the lpBuffer interact with the parameter of "number of bytes to read"?
For instance what if you had an unsigned short MyShort[5] as lpBuffer, and then you set bytes to read as 2. Will all data be stored in MyShort[0]? Or would the first byte go into MyShort[0] and the second byte go into MyShort[1]? What happens when you set bytes to read is increase say to 9? Will 16bits go into MyShort[0] and then 16 more into MyShort[1] etc...?
Thanks
lpBuffer is always treated as a pointer to an array of specified amount of bytes (nNumberOfBytesToRead). The amount of bytes actually read will be stored in the variable pointed to by lpNumberOfBytesRead parameter or as async (overlapped) result later. So in your case if you request to read 2 bytes it may either read two bytes storing both of them in MyShort[0], or just a single byte stored in lower half of MyShort[0] or nothing at all. If you request to read 9 bytes then it will ready up to 9 bytes storing 2 + 2 + 2 + 2 + 1 bytes sequentially.
Related
I use memcpy() to write data to a device, with a logic analyzer/PCIe analyzer, I can see the actual stores.
My device gets more stores than expected.
For example,
auto *data = new uint8_t[1024]();
for (int i=0; i<50; i++){
memcpy((void *)(addr), data, i);
}
For i=9, I see these stores:
4B from byte 0 to 3
4B from byte 4 to 7
3B from byte 5 to 7
1B-aligned only, re-writing the same data -> inefficient and useless store
1B the byte 8
In the end, all the 9 Bytes are written but memcpy creates an extra store of 3B re-writing what it has already written and nothing more.
Is it the expected behavior? The question is for C and C++, I'm interested in knowing why this happens, it seems very inefficient.
Is it the expected behavior?
The expected behavior is that it can do anything it feels like (including writing past the end, especially in a "read 8 bytes into a register, modify the first byte in the register, then write 8 bytes" way) as long as the result works as if the rules for the C abstract machine were followed.
Using a logic analyzer/PCIe analyzer to see the actual stores is so far beyond the scope of "works as if the rules for the abstraction machine were followed" that it's unreasonable to have any expectations.
Specifically; you can't assume the writes will happen in any specific order, can't assume anything about the size of any individual write, can't assume writes won't overlap, can't assume there won't be writes past the end of the area, can't assume writes will actually occur at all (without volatile), and can't even assume that CHAR_BIT isn't larger than 8 (or that memcpy(dest, source, 10); isn't asking to write 20 octets/"8 bit bytes").
If you need guarantees about writes, then you need to enforce those guarantees yourself (e.g. maybe create a structure of volatile fields to force the compiler to ensure writes happen in a specific order, maybe use inline assembly with explicit fences/barriers, etc).
The following illustrates why memcpy may be implemented this way.
To copy 9 bytes, starting at a 4-byte aligned address, memcpy issues these instructions (described as pseudo code):
Load four bytes from source+0 and store four bytes to destination+0.
Load four bytes from source+4 and store four bytes to destination+4.
Load four bytes from source+5 and store four bytes to destination+5.
The processor implements the store instructions with these data transfer in the hardware:
Since destination+0 is aligned, store 4 bytes to destination+0.
Since destination+4 is aligned, store 4 bytes to destination+4.
Since destination+5 is not aligned, store 3 bytes to destination+3 and store 1 byte to destination+8.
This is an easy and efficient way to write memcpy:
If length is less than four bytes, jump to separate code for that.
Loop copying four bytes until fewer than four bytes are left.
if length is not a multiple of four, copy four bytes from source+length−4 to destination+length−4.
That single step to copy the last few bytes may be more efficient than branching to three different cases with various cases.
Imagine you had a uint64_t bytes and you know that you only need 7 bytes because the integers you store will not exceed the limit of 7 bytes.
When writing a file you could do something like
std::ofstream fout(fileName);
fout.write((char *)&bytes, 7);
to only write 7 bytes.
The question I'm trying to figure out is whether endianess of a system affects the bytes that are written to the file. I know that endianess affects the order in which the bytes are written, but does it also affect which bytes are written? (Only for the case when you write less bytes than the integer usually has.)
For example, on a little endian system the first 7 bytes are written to the file, starting with the LSB. On a big endian system what is written to the file?
Or to put it differently, on a little endian system the MSB(the 8th byte) is not written to the file. Can we expect the same behavior on a big endian system?
Endianess affects only the way (16, 32, 64) int are written. If you are writing bytes, (as it is your case) they will be written in the exact same order you are doing it.
For example, this kind of writing will be affected by endianess:
std::ofstream fout(fileName);
int i = 67;
fout.write((char *)&i, sizeof(int));
uint64_t bytes = ...;
fout.write((char *)&bytes, 7);
This will write exactly 7 bytes starting from the address of &bytes. There is a difference between LE and BE systems how the eight bytes in memory are laid out, though (let's assume the variable is located at address 0xff00):
0xff00 0xff01 0xff02 0xff03 0xff04 0xff05 0xff06 0xff07
LE: [byte 0 (LSB!)][byte 1][byte 2][byte 3][byte 4][byte 5][byte 6][byte 7 (MSB)]
BE: [byte 7 (MSB!)][byte 6][byte 5][byte 4][byte 3][byte 2][byte 1][byte 0 (LSB)]
Starting address (0xff00) won't change if casting to char*, and you'll print out the byte at exactly this address plus the next six following ones – in both cases (LE and BE), address 0xff07 won't be printed. Now if you look at my memory table above, it should be obvious that on BE system, you lose the LSB while storing the MSB, which does not carry information...
On a BE-System, you could instead write fout.write((char *)&bytes + 1, 7);. Be aware, though, that this yet leaves a portability issue:
fout.write((char *)&bytes + isBE(), 7);
// ^ giving true/false, i. e. 1 or 0
// (such function/test existing is an assumption!)
This way, data written by a BE-System would be misinterpreted by a LE-system, when read back, and vice versa. Safe version would be decomposing each single byte as geza did in his answer. To avoid multiple system calls, you might decompose the values into an array instead and print out that one.
If on linux/BSD, there's a nice alternative, too:
bytes = htole64(bytes); // will likely result in a no-op on LE system...
fout.write((char *)&bytes, 7);
The question I'm trying to figure out is whether endianess of a system affects the bytes that are written to the file.
Yes, it affects the bytes are written to the file.
For example, on a little endian system the first 7 bytes are written to the file, starting with the LSB. On a big endian system what is written to the file?
The first 7 bytes are written to the file. But this time, starting with the MSB. So, in the end, the lowest byte is not written in the file, because on big endian systems, the last byte is the lowest byte.
So, this is not what you've wanted, because you lose information.
A simple solution is to convert uint64_t to little endian, and write the converted value. Or just write the value byte-by-byte in a way that a little endian system would write it:
uint64_t x = ...;
write_byte(uint8_t(x));
write_byte(uint8_t(x>>8));
write_byte(uint8_t(x>>16));
// you get the idea how to write the remaining bytes
I need help identifying the following technique. Is a lengthy read so please try to follow. My question is if this is a known standard, does it have a name, can anyone relate or seen this before. What is the benefit. Also in case you wonder, this is related to a packet captured on a long forgotten online PS2 game and I am part of a team that is trying to bring it back.
Note that this is not the size as described by the ip protocol this size representation is withing the actual payload and it is for client and server consumption.
The following read describes the how the size of the message is being represented.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The true packet length is 94 bytes long.
These are bytes 5-6 [CF E0] on the payload data after all of the ip protocol stuff.
Also, note that we must interpret these two bytes as being in little endian format. Thus, we should think of these two bytes as being
[E0 CF]
We determine the Packet Class from these two bytes by taking the first nibble(4 bits) of the first byte. In this particular case, this is just 0xE. We would then identify this packet as having a packet class of 0xE. This was identified as a Session Initiator Packet Class.
Now, to determine the packet length from the remaining nibble and second byte. First we convert the second byte to decimal, we get 0xCF = 207. The different between this value and the actual length is 207-94=113 bytes. Originally I knew this byte was proportional to the packet length, but just had some offset. I wasn't sure where this offset came from. Additionally, this offset seemed to change for different packets. More study was required.
Eventually, I found out that each packet class had a different offset. So I needed to examine only packets in the same packet class to figure out the offset for that packet class. In doing this, I made a table of all the reported lengths (in byte 5) and compared that to the actual packet length. What I discovered is that
almost all of reported packet lengths in byte 5 were greater than 0x80=128.
the second nibble in the other byte worked as a type of multiplier for the packet length
that each packet class had an associated minimum packet length and maximum packet length that could be represented. For the 0xC packet class I was examining, the minimum packet size was 18 bytes and the maximum packet size was approximately 10*128 +17 = 1297 bytes.
This led to the following approach to extract the packet length from the fifth and sixth byte packet header. First note that we have previously determined the packet class to be 0xE and that the minimum packet size associated with this packet class is 15 bytes. Now, take the second nibble of the first byte [0xE0] = 0 in this case and multiply it by 128 bytes 0*128 = 0 bytes. Now add this to the second byte [0xCF] = 207 in this case and subtract out 128. So 0 + 207 - 128 = 79. Now we need to add in the minimum packet size for this packet class 0xE = 15 byte minimum packet size. So (0*128)+(207-128) -15 = 94. This is the reported true packet size.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This formula was tested on 20,000 subsequent packets and it works. But why go trough all that trouble just to indicate the size of the message that follows? I thought it was a form of encryption but the rest of the message is not encrypted at all. The formula is understood but I don't see a benefit. I am thinking that maybe is a way to optimize the size of the packet by passing a number greater than 255 using only one byte but that only saves me exactly one byte, throwing another byte yields a Max value of 65,535 so why not throw another byte into the byte stream. I am sure one extra byte is not going to have a great impact on the network so what could be the purpose. I thought that maybe someone else would see what's missing or connected to some kind of documented standard, protocol, pattern, technique or something that is documented somewhere.
Also, I do not take credit for figuring out the formula above, that was done by another team member.
My best guess is that the receiver uses some form of variable-length base128 encoding, like LEB128.
But in this case, the sender, knowing the actual max size fits in 11 bits, forces the encoding to to use 2 bytes, and overloads the high nibble for "class". This makes the header size and construction time constant. And the receiver side can just mask out the class and run it through a standard decoder.
Send:
len -= minlen[class]
byte[5]=(len&0x7F)|0x80;
byte[6]=(len>>7)|(class<<4);
Receive:
class = byte[6]>>4;
byte[6]&=0xF;
len = decode(&byte[5]) + minlen[class];
where:
int decode(byte* data) {
int v=*data&0x7F;
while (*data & 0x80) {
data++;
v+=*data&0x7F;
}
return v;
}
One other possibility is that byte[5] is signed, and length is reconstructed by
(int8_t)byte[5] + 128*((byte[6]&0xF)+1) + minlen[byte[6]>>4];
But I can't think of any reason to construct it this way.
I wanna prepend the size of the vector in the buffer. But I don't know exactly what the type of the size is. After all, std::size_t can't be a fixed size. In my mind, I intend to use uint64_t instead. Then the buffer would like this:
8 bytes length | 4 bytes element1 | 4 bytes element2 | ... |
Now the question is uint64_t doesn't mean std::size_t. Any better ideas would be appreciated.
You can use any type you want so long as it can hold the value you are using. Since it's already a size_t, just keep it that way. Decide how many bytes you want to use to represent the value and what value you need each byte to be and write code to encode/decode each byte correctly.
You are almost there. No current platform uses size_t greater than 64 bits (and it would take several days to transfer 64bits worth of int32 over experimental 100TBit fibre). The steps are:
const uint64_t len = vec.length();
Write the eight bytes into your tcp buffer in a defined order.
Write the four bytes of each int into the tcp buffer in a defined order.
Send.
Note that the last two steps will have to be in a loop for more than a few thousand elements..
per the following code, I get the size of a character pointer is 8 bytes. Yet this site has a size of 1 byte for the char pointer.
#include <stdio.h>
int main(void ){
char *a = "saher asd asd asldasdas;daksd ahwal";
printf(" nSize = %d \n", sizeof(a));
return 0;
}
Is this always the case? I am writing a connector for a simple database I am implementing and want to read TEXT field of mysql into my database. Since TEXT has variable size, I was wondering if my column Type/metadata can have a fixed size of 8 bytes where I store the pointer in memory to the string (char *)?
per the following code, I get the size of a character pointer is 8 bytes. Yet this site has a size of 1 byte for the char pointer.
It's implementation-defined. It's usually 8 on a 64-bit Intel system and 4 on a 32-bit Intel system. Don't rely on it being any particular size.
I am writing a connector for a simple database I am implementing and want to read TEXT field of mysql into my database. Since TEXT has variable size, I was wondering if my column can have a fixed size of 8 bytes where I store the pointer in memory to the string (char *)?
It makes no sense at all to store pointers into memory in a database. A database is for persistent data. On the other hand, data stored in memory is liable to disappear whenever a process exits (or the system is restarted).
No, it is not. Size of a pointer depends on CPU architecture. Some architecture even have different sizes depending on "type" of the pointer. On x86_64, pointers are 48 bits wide. 64 bits are used because individual bits are not addressable. One could, however, use pointer packing to serialize/deserialize pointers into 48-bit chunks.
A variable can be different sizes based on the computer that you are using. This is causing the discrepancy between your results and the results you see online.
However, the variable will always be the same size on the same machine.
The size of any pointer in one platform is the same.. regardless of the data type char, string, object, etc.
In PC with 64 operating system (and also the compiler support 64 bit), the size of pointer is 8 byte (64 bit address space)..
Another platform may have 4 byte, 2 byte, or 1 byte (like an 8 bit micro controller)..