I am using gRPC for streaming telemetry data. Testing team wants to check if protobuf encoding is used by gRPC? Can anyone help me how I can verify that?
Is there a way to capture the send packets using Wireshark & verify if protobuf encoding is used?
Tried searching online but didn't get any useful results.
There are several possible ways to proceed.
Boring - Look At the Documentation
First is to consult the docs. A glance here is pretty good confirmation.
Though it's important to understand that there's several layers of the network stack here - gRPC uses Protobuf to encode application messages that are transferred across networks using http/2, which in turn may be configured to use TLS encryption, all of which ends up being sent over tcp sessions.
So, yes, gRPC does use Protobuf, but that's not the end of it (there's http/2 and possibly TLS too). The Testing Team may need to fully understand all of that.
Look At the Source Code
The source code all appears to be open source, one could double check that.
The More Interesting Way
Thirdly, and I recommend doing this regardless of whether or not the documentation was adequate to answer your question, the more fun way using Wireshark!
Wireshark can be easily extended. Basically, WireShark can unpick gRPC all the way to the proto messages themselves (unless you've turned on any of the security features in gRPC, in which case Wireshark will get stymied by not knowing the encryption keys). Wireshark can do this, because it looks like gRPC uses http/2, which is a defined protocol, but won't know how to decode the messages being transported using http/2.
What you can do is to use your .proto file to help generate a dissector plug-in for Wireshark, so that individual messages can be decoded within Wireshark. See here, and here
Okay, so that's a bit of work to implement, but it can be worthwhile, especially if "testing" is significant. What it means is that Wireshark, and a PCAP file in particular, can be an authoritive record of the telemetry stream that is independent of the software at either end of the telemetry link. Plus, it's a properly timestamped record, which might also be important.
You've also got a very powerful diagnostics tool for if anything is going wrong. E.g. if the telemetry data isn't looking right, where is the problem creeping in? Having a Wireshark plugin that can be used to examine the gPRC data all the way in to the message content is a useful tool.
The Wireshark PCAP file and dissection would also be a good way of seeing whatever else might be going on at the same time.
I've done this before, though I was using ASN.1 and created a system-specific ASN.1 dissector for Wireshark. Wireshark used in this way is very cool.
Related
First of all, a big thank you to anyone willing to help!
You can skip the following wall of text if you want.
Background
I am trying to make a proxy program for my own minecraft server network. I wish to use multiple servers to allow people to play creative and survival on separate servers (to reduce serverload and increase my capacity). I am planning on using the transporter plugin for the server side of things, but I'd like to prevent having to install a clientpatch.
I discovered bungeecord, but it didn't do what I needed it to (namely, forge support for a modded server I plan on adding.) for mc 1.4.7 (which I need to use due to the server I plan on adding).
At the moment the system is running using a bungeecord proxy. However, md_5 told me that I'd need to write my own proxy (or mod his) to make it work the way I want. However, I really can't wrap my head around java (I just don't get it). So I decided to write my own.
The problem
After some research I discovered this to be doable. The hardest part would be to parse the packets. So I dug around for a library to do it for me. I came across libmcnet, which seemed to be what I wanted, but all the data it produced was garbled.
Looking over some packet dumps, and referencing it with wiki.vg, I discovered that libmcnet was giving me big-endian. However, my code and computers assume little-endian.
What I have done about it
After some thinking and looking over the code I thought it easier to write my own parser using some defines from libmcnet (namely include/mcnet/packets.h). So I copied the packets.h and read.h/read.c (both of which were heavily modified) and started recoding it. I did well enough, I think, as my system can successfully parse the server's 0xff (kick) packet during the server ping.
New problems
The next hurdle proved to be decrypting the client's data. It doesn't seem to use the same data standards as the server. For instance, the packet id the server transmits is 16 bit while the client's seems to be 8 bit.
So what I want to know:
Why does my code work fine on data from the server, but fail miserably when the client sends data? I'm pretty sure that libmcnet's code (on which I based it) is wrong as well.
Can anybody help me make this code work?
Code: http://pastebin.com/jg26yity (I figured this'd be cleaner)
Thank you all for your help!
*I'll add information if anybody requests more...
PS. First question asked here, so my post is a bit messy...
I figured it out, apparently I can't assume a library written to parse the minecraft protocol actually has proper code to read the protocol...
I was digging around and verifying what was read versus manually reading it... Appearantly a byte isn't a byte with minecraft and I thought the library had it covered.
is there any windows api that can send mail using smtp along with attachment.
I have heard its not possible,and i have to use other socket methods,if its true how can I do that??
please suggest c++ or c solution only,no c# or java(like system.net.mail etc)
This is one of those places that (at least in my experience) you're better off without libraries. Microsoft provides (at least) MAPI, Simple MAPI, and CDO as ways of sending email. Unfortunately, at least in my experience all of them (especially MAPI) are considerably more complex than doing the job on your own.
At least as long as you're dealing with an unsecured email connection, doing the job without a library is pretty simple: connect to the server on port 25. Send it a few strings and the email. When you get down to it, the "Simple" is "Simple Mail Transfer Protocol" is completely warranted -- it really is pretty simple to do.
If you want to do a secure connection you'll almost certainly want to use a library for that though -- creating a secure connection is a decidedly non-trivial task. Since you're using Windows, the obvious choice would be InternetOpen and InternetConnect. Contrary to the documentation, InternetConnect will let you create a connection on almost any port. It only lists a half dozen possibilities or so, but if you just pass an actual port number, such as 465 or 587, that should work -- though there are a few ports WinInet blocks as a (probably ineffective) security measure.
Dealing with an attachment is a (mostly) separate question. An attachment simply ends up as text in the body of the email, with enough of a header to tell the receiving email program to interpret it as an attachment instead of text. There are three reasonably popular possibilities: BinHex, UUencode, and MIME. If you just want to do simple attachments, UUencode is probably the way to go. MIME is more capable, but considerably more complex. The main reason to use it would be if you wanted to do things other than simple attachments (e.g., email with files embedded rather than attached). BinHex is marginally simpler to implement, but wastes more space -- the primary reason to use it is for a few ancient email clients that can't handle anything else.
Here's a link to a wxWidget smtp class. If you take a look at it, you should get an idea of how to send an email message.
I'm looking to write a small program which will intercept network packets (on the local machine) and modify them before they go out on the network. I need to be able to modify the headers as well, not just the data.
I've already looked through several possibilities but am unsure which one is best to pursue. There are open source packet filters out there, but filtering only seems to be able to either allow or reject packets, not much else.
The other solution would be to write an NDIS intermediate driver, but writing drivers is a beyond me. Even the simple pass-thru example in the WinDDK is thousands of lines. I'm also not looking forward to having to constantly reinstall a driver and reboot to test my code.
I'd ideally like the program to be self contained, and not rely on the installation of 3rd party drivers/software/whatever.
So if you people could point me in the right direction, throw some helpful links my way, whatever, I'd appreciate it.
Depends what kind of packets do you want to filter/modify.
If you're after application-level filtering, and want to get your hands on HTTP or similar packets, your best bet would probably be an LSP. Note however, following this path has certain disadvantages. First MS seems to be trying to get rid of this technology, and IIRC a part of Windows 7 logo requirements is "no LSP in your product", they seem to be promoting the Windows Filtering Platform. Second, you'd be very surprised with how much trouble you're getting into in terms of 3rd party LSP compatibility. Third, a very dummy LSP is still around 2 KLOC :)
If you're after an IP level packet filtering you'd need to go for a driver.
Windows Filtering Platform provides you with functionality needed in either case. However, it's only available on Windows Vista and later products, so no XP there. Another thing to take into consideration, WFP was only capable of allow/reject packets in user-land, and if you need to modify them, you'd need to go kernel-mode. (At least that what the situation was at the time it appeared, maybe they've improved something by now).
IMHO, If you want to modify packets you'll need something to talk to the hardware, a driver of some kind. If you do not want to use your own, you should get a 3rd party driver to inter-operate with.
For filtering there's libraries like: winpcap or libpcap.
Also have a look here: http://www.ntkernel.com/w&p.php?id=7
Another link: http://bittwist.sourceforge.net/
Hope this helps!
winpcap is only able to filter packets with precompiled conditions. What you need is to write LSP-level network driver. You won't need to reboot every time you reinstall it, but it can really modify packets before they go out to the network.
More info here: http://blogs.msdn.com/wndp/archive/2006/02/09/529031.aspx or here: http://www.microsoft.com/msj/0599/LayeredService/LayeredService.aspx
I'm no expert but I'm looking to do something similar on my LAN. I want to intercept packets form one single fixed IP and modify them before they go to my router then out onto the internet. I also want to capture and modify the returning packets prior to allowing them through to my host. The method I had envisaged was something like this...
ARP poison the host and router so my sniffing machine was having all packets passed through it.
Analyse the packets that I will want to modify in future and look for unique characteristics to those packets so I can catch just them.
Write a macro/script that looked for said characteristic in real-time and then modified it on the fly before sending it on its' way.
I know Cain&Abel for Windows is able (haha) to ARP poison but I'm not sure if it can provide raw dump of packet contents. Wireshark is able to dump all but not sure if it can ARP poison so as just to get what I'm after, if not then I can easily connect the host I want to intercept to my sniffer machine via ethernet and then share the internet via the sniffer so that all packets will go through the sniffer machine anyway.
So step 1 can be accomplished, I don't know if said programs have the ability to filter based on specifics yet but I'm guessing they do.
That's as far as I am with it. Hope this is of help to someone and maybe someone else can take this further?
Basically my question is the exact same one as this:
Simple client/server, TCP/IP encrypting the message stream, SSL
The difference is that I need this for pure C++, not .NET. I cannot use 3rd party libraries, so unless it's a Windows system component (like the above) I need something with source so I can get the generel idea and build it myself.
Thanks :)
Quoting the other question for reference:
"Writing a little TCP/IP client server
app. Basically it creates a server,
and then you can create several
different clients and set up a bit of
a chat session. What I am wondering is
there is any way to incorporate, using
standard .net libraries some form of
encryption?
m_mainSocket = new
Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
Is there any way of speficying tcp
using rsa?
Or would you (me that is) have to
write some custom libaries to do key
exchange and then encrypt the
subsequent chat messages? I have done
that before for uni but that was in
java but I know it would'nt be hard to
convert them. Just trying not to have
to reinvent the wheel...
Or what about utilising a ssl?
Thanks, Ron."
Have you considered using the ASIO library? think-async dot com/Asio/
There is an example specifically for an SSL based client/server. http://think-async.com/Asio/asio-1.4.1/doc/asio/examples.html#asio.examples.ssl
Its as "pure c++" as you can get.
You can always look at OpenSSL which is open source, but that would be like implement SSL yourself. I would suggest wrapping OpenSSL and use it. Or use the SSL tunnel application available in OpenSSL.
Writing your own encryption code is "not recommended". It's easy enough to make a simple mistake when using one of these libraries, let alone when you try to write one yourself.
What you really want to use is OpenSSL with Boost.ASIO on top of it. If you can't do that then your next best alternative is to use the Internet Explorer COM object. This isn't quite as flexible, but might work out fine depending on what your exact needs are. You can also explore the Win32 API. Last I looked there weren't enough crypto APIs widely available to do this. The final way of dealing with this is to wrap the .NET APIs so that you can make use of them from native C++.
Only if none of that works out for you should you even consider writing this yourself. You will make mistakes and your application will be less secure as a result. So, before you start trying to write your own crypto code you could also try to look at tunnelling SOCKS over SSH and use somebody else's SSH implementation. The next thing I would look at is to buy in the code rather than write it yourself. The code won't be as good as open source offerings as it will be less used so will have more security problems, but it will still be better than anything you would write on your first outing doing this.
Only if you've exhausted all of these options should you think about writing this yourself. Once you think about it you should try all of the other options again to make sure that you didn't miss getting one of them to work for you the first time around.
If you do still write your own implementation then throw it away and use one of the other options before putting it into production use as there will be mistakes that compromise the security to the extent where you probably may as well not have bothered.
Sorry to sound down on all of this, but getting these things right is really hard and not something you can do by just taking a quick look at somebody else's implementation.
I'm developing an application in which distributed components talk to one another over a network, in an asynchronous, pub/sub kind of way.
For this, I like the idea of sending XML over sockets - it's asynchronous, I don't need a server of any kind, and it can work locally or over a network. I would have to roll my own pub/sub mechanism, which probably won't be too onerous - I could use the IP address as the id of the subscriber. The implementation language will be C++. I've used this method before with good results. But, I wonder if there is a better way? Any advice welcome.
Edit: I haven't made a decision yet, but I'm interested in cparcode's comment on protobuf:
"I never understood the popularity of using XML for network comms. Any way, I realize that the protocol is only part of your problem but on that subject, there's Google's protobuf too: http://code.google.com/p/protobuf/ – caparcode (Mar 11 at 1:01)"
Instead of using XML it might be best to use an existing message passing framework. Have a look at libt2n or d-bus
But it you have used your own socket implementation in the past with good results i don't see a reason why you should change. If it ain't broke don't fix it :)
Instead of just straight sockets, you might consider a solution like AMQP.
When you're talking about using socket connections for a pub/sub interface, that usually means point-to-point communication, which isn't always a scalable solution. AMQP really addresses this sort of pub/sub problem. It's free, it's open-source, and it works.
Given that they've already solved the pub/sub problem, you might want to leverage off their work instead of doing it yourself.
I recommend OpenAMQ.
That being said, it really depends on what sort of environment you're working in. AMQP requires an AMQP broker (an application responsible for routing messages) to be running somewhere in the system.
Take a look at ActiveMQ. It's a JMS provider and also has bindings in lots of other languages, works on several transports, and has a selection of protocols that it uses for its messages.
RabbitMQ is also an AMQP broker, that supports other protocols like XMPP, HTTP, STOMP, SMTP as well, using adaptors.
Especially if you have done it before, and it still meets your needs, stick with it.
How about embedding a web server in your app? EasyHTTPD is open source, C++, and pretty simple to use. The asynchronous part is easy enough. You either spin up a new thread on the sender or the receiver.
If you're looking for XML, and cross-platform to boot, you may consider an XML-RPC mechanism like SOAP (I've used gSoap for communication between a C server on linux with a C++ client on win32, and it worked fine once you figure it all out).
If you are not heavily inclined towards XML and you want something simple and efficient then consider looking at YAMI library. It is rather simple to use and the given link contains enough documentation, rationale and examples to get started.