I have been messing with Wireshark for a while, and I wonder if anyone could help me. I have recorded a random browsing with it, and I saved it to a pcap file. I would like to create a C/C++ program (I know many exist, but I want to practise) that extracts every info from the packets, like source and target IP, port used, data, etc. My finish goal in learning is to extract an image or a Youtube video or anything from the stream (I know, I'll have to group the packets and sort them, etc.), but that's a later project I guess. :)
I am using libpcap (on Linux), and my code so far can read the offline file packet by packet, and - since I know they are PPP packets in my case - if I load a self defined structure with the info from the 20th byte of the packet, I can view the mac addresses and the ip addresses.
My problems:
1) How do I know/determine without Wireshark that what kind of data link type, is used? (Ethernet, WiFi, PPP, etc)
2) How do I read further data of packets? If I just read one byte, my program doesn't do anything, every variable gets empty.
I have a ppphdr struct, which contains:
u_int16_t htype;
u_int16_t ptype;
u_char hplen;
u_char plen;
u_int16_t oper;
u_char sha[6];
u_char spa[4];
u_char tha[6];
u_char tpa[4];
And I call this for every packet:
pppheader = (struct ppphdr*)(packet+20);
Because the ppp frame starts from the 20th byte. It gives back sender and target mac and IP address.
After I continue reading the next few bytes, with the same call just different struct, it comes back empty, and the program stops after 1 packet. I'm trying to use this guide: http://www.tcpipguide.com/free/t_PPPGeneralFrameFormat.htm
How do I know/determine without Wireshark that what kind of data link type, is used? (Ethernet, WiFi, PPP, etc)
Wireshark itself works with different file formats. Two of them that you are probably interested in are "pcap" and "pcap-ng".
If you have recorded data in "pcap" format, the link type is stored in the "Link-layer header type" field in the pcap file header; see the pcap-savefile man page.
If you have recorded data using "pcap-ng" format, then link type is stored in Interface Description Block.
You can read more about these two formats here and there.
If you are reading a pcap or pcap-ng file with libpcap, the pcap_datalink() routine will return a DLT_ value specifying the link-layer header type. See the list of link-layer header types for a description of the DLT_ values and the headers that correspond to them. DLT_EN10MB is for Ethernet (the "10MB" is historical - it's used for all Ethernet speeds); DLT_PPP is the most likely type for PPP. If you have Wi-Fi packets with Wi-Fi headers (if you don't capture in monitor mode, you'll probably get Ethernet headers, and DLT_EN10MB on Wi-Fi adapters), you'll get DLT_IEEE802_11; if you also have "radio metadata" headers before the 802.11 headers, you'll get something such as DLT_IEEE802_11_RADIO or DLT_IEEE802_11_RADIO_AVS or DLT_PRISM_HEADER.
Do NOT assume what the link-layer header type is for the packets you will get from libpcap. ALWAYS call pcap_datalink() to determine the link-layer header type, and use that to parse the packets; if your code doesn't know how to parse packets for a particular DLT_ value, it should report an error and exit.
How do I read further data of packets? If I just read one byte, my program doesn't do anything, every variable gets empty.
Assuming that you record Ethernet data, you need to parse/process data in accordance with standard specifications. For example, first parse Ethernet frame. Even at that point, Ethernet frame can be of variable length. For example, given that tcpdump/wireshark does not record Preamble field, you need to read 15 octets to determine how much more you can/should read.
After you are done with Ethernet frame, you need to parse IP, then possibly UDP and/or TCP. Some other data can be in other formats, but in each and every case you have to carefully study the format specification and parse the data accordingly. Reading one byte will not get you anywhere. So I'd recommend you to start from learning basic network layers - Ethernet, IP, UDP, first, and then get back to the problem of parsing them.
At the end of the day, Wireshark is an open source program that does most of what you want to do as an exercise. Meaning that you can always download the source code, see what it does and learn from it.
Hope it helps. Good Luck!
Related
I'm parsing a file with lots of tcp packets which i need to parse. The problem is that they get segmented and i can't find any indication when and where they do so. No flags or anything else indicates, that the middle of current packet may contain the beginning of the next one. The protocol above tcp is FIX(used in online trading) but i'd like for my code to be able to work with any protocols(or at least understand which is protocol is it).
I'm writing code in C++ and can't use any additional libraries.
So, how do i figure out what is the protocol above tcp and where it gets segmented ?
You can't. TCP/IP is conceptually a stream, not a sequence of messages (the fact that it is ultimately implemented as a sequence of packets is irrelevant). When you write a sequence of bytes to a TCP/IP stream, that sequence is added to the stream; it is not treated as a message which should maintain its own identity. No notion of message begin/end is transmitted along with the stream, unless you do so yourself in your own protocol.
If you find this hard to believe, consider how it works for files: if you write a sequence of bytes to a file, that sequence does not somehow become a record that you can later identify and retrieve. If you want that kind of structure you have to add it yourself. The same is true for TCP/IP.
The transport packets used to implement TCP/IP have no relation to the data blocks you specify with your API calls; they are merely a way to implement the TCP/IP stream. For some use cases there may appear to be a mapping, but this is accidental.
The only way to split a TCP/IP stream back into separate messages is by using knowledge of the protocol running on top of TCP/IP. In your case this is FIX. I assume you know how that works; you can use that knowledge to correctly split the FIX data back into its original messages. A generic TCP/IP message splitter cannot be made.
As I can see your problem is to separate TCP packets. To solve it you can relay on length of payload (this answer) and checksum. If checksum is correct for data with specified length, than your packet is correct, if no - you need seek in thee previous part for start of the packet or drop this part of data. At least this approach will help you to find point where dada was segmented.
For more precise answer it will be better to see little part of data.
But main your problem is segmentation of packets. For better performance you should try to exclude this problem (maybe change network card to Intel).
I'm using the qextserialport classes in Qt to implement serial transmissions between devices. Now, I'm required to send a file between devices connected via USB using the serialport. I have used the serialport for various functions in the past weeks but I have no idea where to start implementing this. I thought about reading data event-driven until there is no more data to read, determine the size (number of bytes) of the file before hand and send it together with the data so that it would be clear if data went missing. I also have a correct function that calculates the CCIITT 16 bit checksum so I can use that for checking as well. My question therefore is:
Can someone please send me a link to a site that could help solve my problem and explain to me what would be the most simple and effective way to send a file and receive a file via the qextserialport class in Qt. ANY help would be awsum!
You need a protocol. Simple one could be:
send length of file name as raw binary number, for example 2 bytes in network byte order
(max name length 65535 bytes)
send that many bytes of file name, encoded with UTF-8
send file size as raw binary number, for example 4 bytes in network byte order (max file size 4 gigs)
send that many bytes of file contents
You might want to add info like file date, and checksum. More advanced would be to split file to chunks, so if there is transmission error, you don't have to re-send everything. Etc.
Also, study protocols like Kermit, xmodem, zmodem to see how it's been done in the modem and BBS era. Maybe use an existing protocol like that instead of creating your own.
Note: while you could use QDataStream, it requires reliable channel, so it's not as easy as it may seem at first (requires extra buffering).
I need to write (c++, linux) rtp analyzer like:http://mmlab.disi.unitn.it/wiki/index.php/Analyzer. But I can't understand some technical moments, such as how server, wich receives udp-datagrams, determines that one udp-datagram had finished and another udp-datagram started and how it identifies udp-datagrams, containing rtp. It would be great to see the code of analyzer, but I haven't find it.
Could you help me to find the answers to my questions (I wonder to know how it can be implemented via sockets). Thank you very much, I will really appreciate it!
If I do understand your question correctly, you can follow the steps stated below:
Write a server program in c++ using one of the socket libraries, and create a UDP socket.
Listen to the port to which your client is sending the RTP packets.
While you listen to the specified UDP port, you will get UDP packets one by one.
When you get a UDP packet parse the length field which is a 2 byte field, and starts at 4th byte.
Length - 8 (UDP header size in bytes) is what you looking for, and actually the size of the RTP packet.
Get the RTP packet by getting the bits between 8th byte of the UDP packet to Length'the byte.
Then you have your RTP packet, you can also parse it in the same way by looking at how it looks like in RFC3550 by IETF.
(Also good to specify that UDP packets might arrive out of order, and you can use the sequence number field in the RTP header, to reorder them)
I am just starting out with network programming and I am having trouble keeping up with some of the stuff. Namely because there is a lot of auto-magic stuff going on under the hood that is hidden away from me, and I cant get my head around it all. In my current app, I take user input like so:
char buff[1024];
cout << "Enter a message:";
fgets(buff, 1024, stdin);
And i can easily send that off by essentially doing this:
sendto(m_socket, buff, size, flags, (SOCKADDR*)&addr, sizeof(addr));
The other variables (size,flags,etc) are obviously initialized and configured somewhere between the input and sendto call.
This all works fine, and the other end receives the data fine. the server does
recvfrom(m_socket, (char*)data, max, flags, (SOCKADDR*)&fromAddr, &fromLength)
and I can then just print out 'data' into the console, and it'd be my message.
So where exactly is my UDP packets header? Did i have to specify it? when i pass in some data, is the 'data' part of a standard UDP packet filled with the data i specify, and the header automatically populated for me (source IP, destination IP)? If so how do i even access it?
A lot of websites I have looked at talk about headers and what not. It seems to be this very specific thing. But I dont understand how this illusive header works.. I'd like to define my own header with things like segment number, total segments, etc. but i dont know how to go about this at all. My googleing isnt sending me in the right direction either..!
The UDP headers are not available to the application when you use the standard interfaces from the operating system (socket() with SOCK_DGRAM, sendto(), recvfrom() and such). They are automatically handled by the operating system's network stack. They are automatically added when you send the message and automatically stripped when you receive.
Depending on the operating system, there are some means you can write and manage the UDP headers directly over the IP routing layer, but that is certainly unusual, and would probably require administrative privileges.
If you want to define your own headers for your own purposes, you must do so inside the body of the message, i.e. parse and interpret what you send and receive sendto() and recvfrom(), thus creating what is called an application protocol. Networking is a layered architecture, where applications sits upon UDP or TCP, that sits upon IP that (usually) sits upon Ethernet or Wi-Fi. Each one has its own headers that is stripped when the data is handled to the above layer, and you only gets what you send on the application layer (maybe you can miss some packets or get them out of order, because UDP doesn't give you those guarantees, as TCP does).
UDP does not contain such things as headers, segment numbers and total segments. In fact, a UDP datagram contains nothing except the buffer that you sendto.
You can add extra information by including them with your message. Instead of simply putting your message in your buffer, make a larger buffer into which you put whatever "header" information you would like and then put your message afterwards.
From UDP's perspective, all it sees is data. It is up to the program on the receiving end to parse this data and realize that a certain part it is metadata and the other part of it is regular data.
You will have to define your message format such that it is easy for the receiving program to extract the "header" portion of the buffer. There are two common schemes: fixed headers and variable length headers.
If you are going for a fixed header, you assume that the first X bytes of every UDP message are metadata. You pick the X and it never changes per message. The recipient then knows to read X bytes as the header and all the rest as a message.
If you are going for a variable length header, you need to be able to tell the receiving end how long the header is for each particular message. A common scheme is that the first 4 bytes of the header contain an integer which says how long the header part of the message is. Then the receiving end reads that many bytes and interprets that as the header. Then it reads the rest of the data and interprets that as the message.
Lastly, the source IP and the destination IP are not properties of your UDP messages but rather properties of your socket. You can retrieve this information by inspecting the socket itself.
You can receive the UDP headers based the sockets you open up. You can receive the complete packet using socket(AF_PACKET,SOCK_RAW,htons(ETH_P_ALL));
If this socket is enabled for recvfrom then you receive the complete packet including your headers.
Is there a good method on how to transfer a file from say... a client to a server?
Probably just images, but my professor was asking for any type of files.
I've looked around and am a little confused as to the general idea.
So if we have a large file, we can split that file into segments...? Then send each segment off to the server.
Should I also use a while loop to receive all the files / segments on the server side? Also, how will my server know if all the segments were received without previously knowing how many segments there are?
I was looking on the Cplusplus website and found that there is like a binary transfer of files...
Thanks for all the help =)
If you are using TCP:
You are right, there is no way to "know" how much data you will be receiving. This gives you a few options:
1) Before transmitting the image data, first send the number of bytes to be expected. So your first 4 bytes might be the 4-byte integer "4096". Then your client can read the first 4 bytes, "know" that it is expecting 4096 bytes, and then malloc(4096) so it can expect the rest. Then, your server can send() 4096 bytes worth of image data.
When you do this, be aware that you might have to recv() multiple times - for one reason or another, you might not have received all 4096 bytes. So you will need to check the return value of recv() to make sure you have gotten everything.
2) If you are just sending one file, you could just have your receiver read it. And it can keep recv()ing from the socket until the server closes the connection. This is a bit harder - you will have to keep track of how much you have received, and then if your buffer is full, you will have to reallocate it. I don't recommend this method, but it would technically accomplish the task.
If you are using UDP:
This means that you don't have reliable transfer. So packets might be dropped. They might also arrive out of order. So if you are going to use UDP, you must fragment your data into little segments. Both the sender and receiver must have agreement on how large a segment is (100 bytes? 1000 bytes?)
Not only that, but you must also transmit a sequence number with each packet - that is, label each packet #1, #2, etc. Because your client must be able to tell: if any packets are missing (you receive packets 1, 2 and 4 - and are thus missing #3) and to make sure they are in order (you receive 3, 2, then 1 - but when you save them to the file, you must make sure the packets are saved in the correct order, 1, 2, then 3).
So for your assignment, well, it will depend on what protocol you have to/are allowed to use.
If you use a UDP-based transfer protocol, you will have to break the file up into chunks for network transmission. You'll also have to reassemble them in the correct order on the receiving end and verify the results. If you use a TCP-based transfer protocol, all of this will be taken care of under the hood.
You should consult Beej's Guide to Network Programming for how best to send and receive data and use sockets in general. It explains most of the things about which you are asking.
There are many ways of transferring files. If your transferring files in a lossless manor, then your basically going to divide the file into chunks. Tag each chunk with a sequence number. Send the chunks to the other side and reconstitute the file. Stream oriented protocols are simpler since packets will be retransmitted if lost. If your using an unreliable protocol, then you will need to retransmit missing packets and resequenced chunks which are not in the correct order.
If lossy transfer is acceptable (like transferring video or on-line game data), then use an unreliable protocol. Lossy transfer is simpler because you don't have to retransmit missing chunks. All you need to do is make sure the chunks are processed in the proper sequence.
Many protocols send a terminator packet to indicate the end of transmission. You could use this strategy if you don't want to send the number of chunks to the other side before transmission.