I'm trying to write a C++ program to scan a pcap file and filter out certain packets. I tried using winpcap to scan and filter but I can't find a way to separate retransmissions. Does anyone know how this filtering can be done?
You will have to implement several TCP variables to do this.
Read section 3.2 of RFC 793.
Related
I am a newbie on pcap parsing and I would like to ask you for some help about this task (I am using Debian 9):
A.pcap is a pcap file that contains the network packet to parse, along with other packets to discard
B.so is a library file that contains the binary to parse the packets the task cares about. There are no header files, so I need to use a binary inspection
Both files can be downloaded at this link:
https://www.dropbox.com/s/ustehbd8lmejddv/task.zip?dl=0
First thing I try to check and dig inside both files using:
nm -gC B.so
tcpick -C -yP -r A.pcap
tcpdump -qns 0 -A -r A.pcap
Now I should parse only the inbound OrderField packets and retrieve the following fields:
1. OrderStatus
2. OrderLocalID
3. LimitPrice
4. Direction
5. InstrumentID
I believe I am having troubles with tcpick and tcpdump because I can't access to any of those information, I can only see a long list of MAC/IP addresses plus some "random" chars.
Do you have any suggestions?
Thank you in advance.
I'm not interviewing the company and have nothing to do with them, but just having fun to reverse engineering this stuff. I just give some idea and hints found from the problem and internet.
You can use libpcap to retrieve every TCP packet from the file, and reconstruct the related inbound transaction to fulfill the requirement of the problem.
The OrderField packet might be related to this https://github.com/fakechris/femas_api/blob/master/traderapidemo/TraderApi4LNX64/USTPFtdcUserApiStruct.h
The compression method is ZeroCompress. Therefore you can directly call the shared library CompressUtil::Zerodecompress to unpack the TCP packet payload
Use the data structure typedef found in the github to extract these 4 necessary fields.
Disclaimer: I might be wrong because I am not able to verify the result.
I have a Qt TCP Server and Client program which can interact with each other. The Server can send some function generated data to the socket using Qtextstream. And the Client reads the data from the socket using simple readAll() and displays to a QtextEdit.
Now my data from Server side is huge (around 7000+ samples ) and I need the data to appear on the Client side instantaneously. I have learned that using XML will help in my case. So, I made an Qt XML Server and it generates the whole xml data into a .xml file. I read the .xml file in Client side and I can get to display its contents. I used the DOM method for parsing. But I get the data to display only when all the 7000+ samples have been generated on the Server side.
I need clarifications on these questions:
How do I write each element of the XML Server side in to a String and send them through socket? I learnt tagName() can help me, but I have not been able to figure out how.
Is there any other way other than the String method to get a single element generated in the Server side to appear in the Client side.
PS: I am a newbie, forgive my ignorance. Thank you.
Most DOM XML parsers require a complete, well-formed XML document before they'll do anything with it. That's precisely what you see: your data is processed only after all of the samples have been received.
You need to use an incremental parser that doesn't care about the XML document not being complete yet.
On the other hand: if you're not requiring XML for interoperability with 3rd party systems, you're probably wasting a lot of resources by using it. I don't know where you've "learned" that XML will "help in your case". To me it's not learning, it's just following the crowd without understanding what's going on. Is your requirement to use XML or to move the data around? Moving data around has been a well understood problem for decades. Computers "speak" binary. No need to work around it, you know. If all you need is to move around some numbers, use QDataStream and be done with it. It'll be two orders of magnitude faster than the fastest XML parsers, you'll transmit an order of magnitude less data, and everyone will live happily ever after*.
*living happily ever after not guaranteed, individual results may vary.
I have some large pcap (packet capture) files collected with tcpdump. I would like to filter out packets that contain a specific string. I want my output to still be in pcap format. I've found several ways of only displaying packets matching a regex from a pcap file, but what I need is to filter such files rather than filter out the display (e.g. stdout) of the packets. The output needs to be pcap files with the matched packets removed.
I suspect dpkt (a python module) might help, but I'd prefer to do this using an existing (C/C++) tool, if possible. I'll accept code as an answer (maybe a good dpkt example with benchmarks will convince me to just go that way as well ;-)).
Thanks in advance!
Answer:
Per Nim's answer, it's possible to do this via Wireshark/tshark. For others' reference, here's an example commandline, where I check for the string match within udp packets (this example can be built on to do tcp or specific protocol field searching):
tshark -r infile -R 'not udp matches "my_search_string"' -w outfile
Thanks again!
This website has a very nice example of how you can read the pcap file in C, a quick google search will reveal how you can re-write the file.
Alternatively, AFAIK Wireshark may allow you to do this already - i.e. open the file, apply a filter and save the file (and a quick run through Wireshark - reveals that it does indeed offer this).
When you write a dissector for Wireshark, how do you test it? Looking for the visual output in the UI is not sufficient for a none-trivial protocol.
Is there a good way for unit testing of the dissector?
EDIT:
The structure of protocol frames is dynamic. The dissector must somehow interpret the content.
For example if the fifth field is one a byte array follows as sixth field. If it's two you have a double array and if it's three you have to add a zero terminated string.
This usually never happens in a daily work capture. That's why you need a synthetic capture data even with "impossible" content.
To test a Wireshark dissector I found this useful:
Define a set of packets that the dissector should analyse including malformed packets
Implement the packets as a hex dump
Define the expected output
For each packet dump
Generate pcap files with text2pcap
Run the dissector with tshark
Extract the payload from the PDML output of tshark
Compare the XML output with the expected XML output
This can be improved by filtering the XML output since the PDML also includes the packet bytes, what can be annoying if the payload is large or/and complex.
The suggested arguments to the wireshark executables are
text2pcap -T 1024,9876 foo.txt foo.pcap
tshark -T pdml -r "foo.pcap"
To extract the dissector output it's useful to use an XPATH expression with the .NET CLR class XmlNode. This can be done e.g. this way:
XmlNode output = tsharkOutput.SelectSingleNode("packet/proto[#name='foo']");
XmlNodeList refList = referenceDocument.SelectNodes("proto[#name='foo']");
You can use something like Scapy or PacketSender to generate test packets.
I guess I'm old fashioned. A dissector's primary purpose is transforming data to a human readable form, so I tested mine by having humans read it.
I suppose you could do more automated testing by exporting to txt or pdml from file->export, or implementing some sort of test wrapper around your plugin DLL.
You could parse the output of tshark.
Just for updating the post.
Tshark uses the same plugins as Wireshark, and loads them in the same
manner. tshark is also used in this way in the Wireshark CI build
tests, see the test directory of the Wireshark sources for some test
script examples.
https://code.wireshark.org/review/gitweb?p=wireshark.git;a=tree;f=test
- grahamb.
source: https://ask.wireshark.org/questions/36721/tshark-for-plugin-testing
im now currently developing a standalone c++ program that would list all the access URL in a browser and its corresponding response time....
at this point of time, i can already sniff all out and in going packets. i am using winpcap for this...
retrieved packets were filtered to by only those 'tcp port 80(http) or 443(https)'...
and know i want to read some http headers. the problem i have is that usually ip are fragmented.
I want to know how to reassemble this and how to have some details about the http..
Note: i want to implement that of WIRESHARK.. in every packet/frame, it has a
'REASSEMBLED TCP SEGMENT'
any idea or tutorials how i can easily attain this?!..
thanks alot!
You'll have to do the same thing TCP does to reassemble packets, which means parsing the header of the packets and sequencing them into another buffer. The worst program logic is probably dealing with missing information; you'll then have to see if it was flagged and retransmitted.
There are a number of RFCs which cover this: 675, 793, 1122 and others. If looking through those seems overwhelming, maybe back off and look at the Roadmap RFC, rfc 4614.