I'm using the qextserialport classes in Qt to implement serial transmissions between devices. Now, I'm required to send a file between devices connected via USB using the serialport. I have used the serialport for various functions in the past weeks but I have no idea where to start implementing this. I thought about reading data event-driven until there is no more data to read, determine the size (number of bytes) of the file before hand and send it together with the data so that it would be clear if data went missing. I also have a correct function that calculates the CCIITT 16 bit checksum so I can use that for checking as well. My question therefore is:
Can someone please send me a link to a site that could help solve my problem and explain to me what would be the most simple and effective way to send a file and receive a file via the qextserialport class in Qt. ANY help would be awsum!
You need a protocol. Simple one could be:
send length of file name as raw binary number, for example 2 bytes in network byte order
(max name length 65535 bytes)
send that many bytes of file name, encoded with UTF-8
send file size as raw binary number, for example 4 bytes in network byte order (max file size 4 gigs)
send that many bytes of file contents
You might want to add info like file date, and checksum. More advanced would be to split file to chunks, so if there is transmission error, you don't have to re-send everything. Etc.
Also, study protocols like Kermit, xmodem, zmodem to see how it's been done in the modem and BBS era. Maybe use an existing protocol like that instead of creating your own.
Note: while you could use QDataStream, it requires reliable channel, so it's not as easy as it may seem at first (requires extra buffering).
Related
I have been messing with Wireshark for a while, and I wonder if anyone could help me. I have recorded a random browsing with it, and I saved it to a pcap file. I would like to create a C/C++ program (I know many exist, but I want to practise) that extracts every info from the packets, like source and target IP, port used, data, etc. My finish goal in learning is to extract an image or a Youtube video or anything from the stream (I know, I'll have to group the packets and sort them, etc.), but that's a later project I guess. :)
I am using libpcap (on Linux), and my code so far can read the offline file packet by packet, and - since I know they are PPP packets in my case - if I load a self defined structure with the info from the 20th byte of the packet, I can view the mac addresses and the ip addresses.
My problems:
1) How do I know/determine without Wireshark that what kind of data link type, is used? (Ethernet, WiFi, PPP, etc)
2) How do I read further data of packets? If I just read one byte, my program doesn't do anything, every variable gets empty.
I have a ppphdr struct, which contains:
u_int16_t htype;
u_int16_t ptype;
u_char hplen;
u_char plen;
u_int16_t oper;
u_char sha[6];
u_char spa[4];
u_char tha[6];
u_char tpa[4];
And I call this for every packet:
pppheader = (struct ppphdr*)(packet+20);
Because the ppp frame starts from the 20th byte. It gives back sender and target mac and IP address.
After I continue reading the next few bytes, with the same call just different struct, it comes back empty, and the program stops after 1 packet. I'm trying to use this guide: http://www.tcpipguide.com/free/t_PPPGeneralFrameFormat.htm
How do I know/determine without Wireshark that what kind of data link type, is used? (Ethernet, WiFi, PPP, etc)
Wireshark itself works with different file formats. Two of them that you are probably interested in are "pcap" and "pcap-ng".
If you have recorded data in "pcap" format, the link type is stored in the "Link-layer header type" field in the pcap file header; see the pcap-savefile man page.
If you have recorded data using "pcap-ng" format, then link type is stored in Interface Description Block.
You can read more about these two formats here and there.
If you are reading a pcap or pcap-ng file with libpcap, the pcap_datalink() routine will return a DLT_ value specifying the link-layer header type. See the list of link-layer header types for a description of the DLT_ values and the headers that correspond to them. DLT_EN10MB is for Ethernet (the "10MB" is historical - it's used for all Ethernet speeds); DLT_PPP is the most likely type for PPP. If you have Wi-Fi packets with Wi-Fi headers (if you don't capture in monitor mode, you'll probably get Ethernet headers, and DLT_EN10MB on Wi-Fi adapters), you'll get DLT_IEEE802_11; if you also have "radio metadata" headers before the 802.11 headers, you'll get something such as DLT_IEEE802_11_RADIO or DLT_IEEE802_11_RADIO_AVS or DLT_PRISM_HEADER.
Do NOT assume what the link-layer header type is for the packets you will get from libpcap. ALWAYS call pcap_datalink() to determine the link-layer header type, and use that to parse the packets; if your code doesn't know how to parse packets for a particular DLT_ value, it should report an error and exit.
How do I read further data of packets? If I just read one byte, my program doesn't do anything, every variable gets empty.
Assuming that you record Ethernet data, you need to parse/process data in accordance with standard specifications. For example, first parse Ethernet frame. Even at that point, Ethernet frame can be of variable length. For example, given that tcpdump/wireshark does not record Preamble field, you need to read 15 octets to determine how much more you can/should read.
After you are done with Ethernet frame, you need to parse IP, then possibly UDP and/or TCP. Some other data can be in other formats, but in each and every case you have to carefully study the format specification and parse the data accordingly. Reading one byte will not get you anywhere. So I'd recommend you to start from learning basic network layers - Ethernet, IP, UDP, first, and then get back to the problem of parsing them.
At the end of the day, Wireshark is an open source program that does most of what you want to do as an exercise. Meaning that you can always download the source code, see what it does and learn from it.
Hope it helps. Good Luck!
I am trying to receive a file (audio, .CAF) from a socket in C (C++ solution ok as well). I have the socket communication working, having tested it with strings. My problem is I don't know what to supply to the 2nd arg in recv(socket, buffer, buffer_size, 0). What type should I make "buffer"? I basically want to receive an audio file, and will then play it. But don't know how to receive the audio file itself.
Any thoughts?
Thanks,
Robin
Typically, you'll have the audio encoded in some format. For simplicity, let's assume it's Wave format.
One way of doing things would be to encapsulate chunks of the audio (Say, 50 ms chunks) for sending over the network.
However, you can't blindly send data over the network and expect it to work. On your computer, data may be organized one way (little or big endian), and it could be organized in the opposite way on the other computer.
In that case, the client will get data that he interprets as being completely different than what you intended. So, you'll need to properly serialize it somehow.
Once you properly serialize the data though (or not, if both computers use the same endianess!), you can just send() it and rcev() it, then just pass it off to a decoder to deal with.
I'd love to offer more information, but that's about the extent of my knowledge on the subject. It really depends on what exactly you're doing, so it's hard to give any more information without some more specifics as to what you're doing (with regards to audio format, for one).
Some more information:
http://en.wikipedia.org/wiki/Serialization
http://en.wikipedia.org/wiki/Endianness
Edit: As pointed out in the comments of this answer, you should probably not worry about serialization at all if you're using a standard format. Just pass it over the network in chunks that are usable by your decoder (Send a frame at a time, for example) then decode those individual frames (or possibly multiple frames) on the client side.
buffer is going to be a pointer to an array you've allocated to store the data the comes across the wire.
It depends on the socket library you're using, but usually it expects void* (which is just a generic pointer type).
You might do something like this:
uint8[1000] myBuffer;
recv(sock,myBuffer,1000,0);
It gets tricky because this only gives you enough room for 8,000bytes, which might not be enough to hold your audio file, so you'll have to handle multiple recv() calls until you get the entire audio file.
I've heard that we can somehow send an image file with binary over a socket...
But I have no idea on how to convert an image file into binary or how to even think of sending it over a socket...
Was hoping if someone could post a simple example? or point me in the right direction :) I am also using QT for just my gui, but not using QT socket programming.
Thanks so much :D I really appreciate it
Question # djc:
How would you get the directory path for an image, and somehow use the send command on that image? I'm basically using C++. But this is also a question I've had for awhile.
Any image files you have are already binary. You can just send them over the socket.
You will need to know, or have the user tell you, a path that will find the image file.
Once you have that, then you logically open the file, read it into a buffer, and then write that buffer over the socket, and finally close the file (always close what you open and free what you allocate). However, there are details to be sorted - like how does the receiving end know that the data that follows is an image and how big it is (so it knows when you've sent it all). Your protocol will, presumably, define a bit pattern (one or two bytes) that identifies the message as an image, and then probably use four bytes to specify the size of the image, followed by the correct number of bytes. You can find the size of a file using the POSIX-based stat() system call. Alternatively, you can send a series of packets containing parts of the image (again with a type - this time, of type 'image packet' instead of 'image') plus the length of the packet (which might only be a 16-bit unsigned integer, for a maximum size of 65535 bytes), plus an 'end image packet' for the last segment. This is perhaps easier for the sender; it is easy for the receiver if the data goes direct to file, but messy if the receiver needs the image in memory.
I have to send mesh data via TCP from one computer to another... These meshes can be rather large. I'm having a tough time thinking about what the best way to send them over TCP will be as I don't know much about network programming.
Here is my basic class structure that I need to fit into buffers to be sent via TCP:
class PrimitiveCollection
{
std::vector<Primitive*> primitives;
};
class Primitive
{
PRIMTYPES primType; // PRIMTYPES is just an enum with values for fan, strip, etc...
unsigned int numVertices;
std::vector<Vertex*> vertices;
};
class Vertex
{
float X;
float Y;
float Z;
float XNormal;
float ZNormal;
};
I'm using the Boost library and their TCP stuff... it is fairly easy to use. You can just fill a buffer and send it off via TCP.
However, of course this buffer can only be so big and I could have up to 2 megabytes of data to send.
So what would be the best way to get the above class structure into the buffers needed and sent over the network? I would need to deserialize on the recieving end also.
Any guidance in this would be much appreciated.
EDIT: I realize after reading this again that this really is a more general problem that is not specific to Boost... Its more of a problem of chunking the data and sending it. However I'm still interested to see if Boost has anything that can abstract this away somewhat.
Have you tried it with Boost's TCP? I don't see why 2MB would be an issue to transfer. I'm assuming we're talking about a LAN running at 100mbps or 1gbps, a computer with plenty of RAM, and don't have to have > 20ms response times? If your goal is to just get all 2MB from one computer to another, just send it, TCP will handle chunking it up for you.
I have a TCP latency checking tool that I wrote with Boost, that tries to send buffers of various sizes, I routinely check up to 20MB and those seem to get through without problems.
I guess what I'm trying to say is don't spend your time developing a solution unless you know you have a problem :-)
--------- Solution Implementation --------
Now that I've had a few minutes on my hands, I went through and made a quick implementation of what you were talking about: https://github.com/teeks99/data-chunker There are three big parts:
The serializer/deserializer, boost has its own, but its not much better than rolling your own, so I did.
Sender - Connects to the receiver over TCP and sends the data
Receiver - Waits for connections from the sender and unpacks the data it receives.
I've included the .exe(s) in the zip, run Sender.exe/Receiver.exe --help to see the options, or just look at main.
More detailed explanation:
Open two command prompts, and go to DataChunker\Debug in both of them.
Run Receiver.exe in one of the
Run Sender.exe in the other one (possible on a different computer, in which case add --remote-host=IP.ADD.RE.SS after the executable name, if you want to try sending more than once and --num-sends=10 to send ten times).
Looking at the code, you can see what's going on, creating the receiver and sender ends of the TCP socket in the respecitve main() functions. The sender creates a new PrimitiveCollection and fills it in with some example data, then serializes and sends it...the receiver deserializes the data into a new PrimitiveCollection, at which point the primitive collection could be used by someone else, but I just wrote to the console that it was done.
Edit: Moved the example to github.
Without anything fancy, from what I remember in my network class:
Send a message to the receiver asking what size data chunks it can handle
Take a minimum of that and your own sending capabilities, then reply saying:
What size you'll be sending, how many you'll be sending
After you get that, just send each chunk. You'll want to wait for an "Ok" reply, so you know you're not wasting time sending to a client that's not there. This is also a good time for the client to send a "I'm canceling" message instead of "Ok".
Send until all packets have been replied with an "Ok"
The data is transfered.
This works because TCP guarantees in-order delivery. UDP would require packet numbers (for ordering).
Compression is the same, except you're sending compressed data. (Data is data, it all depends on how you interpret it). Just make sure you communicate how the data is compressed :)
As for examples, all I could dig up was this page and this old question. I think what you're doing would work well in tandem with Boost.Serialization.
I would like to add one more point to consider - setting TCP socket buffer size in order to increase socket performance to some extent.
There is an utility Iperf that let test speed of exchange over the TCP socket. I ran on Windows a few tests in a 100 Mbs LAN. With the 8Kb default TCP window size the speed is 89 Mbits/sec and with 64Kb TCP window size the speed is 94 Mbits/sec.
In addition to how to chunk and deliver the data, another issue you should consider is platform differences. If the two computers are the same architecture, and the code running on both sides is the same version of the same compiler, then you should, probably, be able to just dump the raw memory structure across the network and have it work on the other side. If everything isn't the same, though, you can run into problems with endianness, structure padding, field alignment, etc.
In general, it's good to define a network format for the data separately from your in-memory representation. That format can be binary, in which case numeric values should be converted to standard forms (mainly, changing endianness to "network order", which is big-endian), or it can be textual. Many network protocols opt for text because it eliminates a lot of formatting issues and because it makes debugging easier. Personally, I really like JSON. It's not too verbose, there are good libraries available for every programming language, and it's really easy for humans to read and understand.
One of the key issues to consider when defining your network protocol is how the receiver knows when it has received all of the data. There are two basic approaches. First, you can send an explicit size at the beginning of the message, then the receiver knows to keep reading until it's gotten that many bytes. The other is to use some sort of an end-of-message delimiter. The latter has the advantage that you don't have to know in advance how many bytes you're sending, but the disadvantage that you have to figure out how to make sure the the end-of-message delimiter can't appear in the message.
Once you decide how the data should be structured as it's flowing across the network, then you should figure out a way to convert the internal representation to that format, ideally in a "streaming" way, so you can loop through your data structure, converting each piece of it to network format and writing it to the network socket.
On the receiving side, you just reverse the process, decoding the network format to the appropriate in-memory format.
My recommendation for your case is to use JSON. 2 MB is not a lot of data, so the overhead of generating and parsing won't be large, and you can easily represent your data structure directly in JSON. The resulting text will be self-delimiting, human-readable, easy to stream, and easy to parse back into memory on the destination side.
Is there a good method on how to transfer a file from say... a client to a server?
Probably just images, but my professor was asking for any type of files.
I've looked around and am a little confused as to the general idea.
So if we have a large file, we can split that file into segments...? Then send each segment off to the server.
Should I also use a while loop to receive all the files / segments on the server side? Also, how will my server know if all the segments were received without previously knowing how many segments there are?
I was looking on the Cplusplus website and found that there is like a binary transfer of files...
Thanks for all the help =)
If you are using TCP:
You are right, there is no way to "know" how much data you will be receiving. This gives you a few options:
1) Before transmitting the image data, first send the number of bytes to be expected. So your first 4 bytes might be the 4-byte integer "4096". Then your client can read the first 4 bytes, "know" that it is expecting 4096 bytes, and then malloc(4096) so it can expect the rest. Then, your server can send() 4096 bytes worth of image data.
When you do this, be aware that you might have to recv() multiple times - for one reason or another, you might not have received all 4096 bytes. So you will need to check the return value of recv() to make sure you have gotten everything.
2) If you are just sending one file, you could just have your receiver read it. And it can keep recv()ing from the socket until the server closes the connection. This is a bit harder - you will have to keep track of how much you have received, and then if your buffer is full, you will have to reallocate it. I don't recommend this method, but it would technically accomplish the task.
If you are using UDP:
This means that you don't have reliable transfer. So packets might be dropped. They might also arrive out of order. So if you are going to use UDP, you must fragment your data into little segments. Both the sender and receiver must have agreement on how large a segment is (100 bytes? 1000 bytes?)
Not only that, but you must also transmit a sequence number with each packet - that is, label each packet #1, #2, etc. Because your client must be able to tell: if any packets are missing (you receive packets 1, 2 and 4 - and are thus missing #3) and to make sure they are in order (you receive 3, 2, then 1 - but when you save them to the file, you must make sure the packets are saved in the correct order, 1, 2, then 3).
So for your assignment, well, it will depend on what protocol you have to/are allowed to use.
If you use a UDP-based transfer protocol, you will have to break the file up into chunks for network transmission. You'll also have to reassemble them in the correct order on the receiving end and verify the results. If you use a TCP-based transfer protocol, all of this will be taken care of under the hood.
You should consult Beej's Guide to Network Programming for how best to send and receive data and use sockets in general. It explains most of the things about which you are asking.
There are many ways of transferring files. If your transferring files in a lossless manor, then your basically going to divide the file into chunks. Tag each chunk with a sequence number. Send the chunks to the other side and reconstitute the file. Stream oriented protocols are simpler since packets will be retransmitted if lost. If your using an unreliable protocol, then you will need to retransmit missing packets and resequenced chunks which are not in the correct order.
If lossy transfer is acceptable (like transferring video or on-line game data), then use an unreliable protocol. Lossy transfer is simpler because you don't have to retransmit missing chunks. All you need to do is make sure the chunks are processed in the proper sequence.
Many protocols send a terminator packet to indicate the end of transmission. You could use this strategy if you don't want to send the number of chunks to the other side before transmission.