As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
What's the best tool that you use to monitor Web Service, SOAP, WCF, etc. traffic that's coming and going on the wire? I have seen some tools that made with Java but they seem to be a little crappy. What I want is a tool that sits in the middle as a proxy and does port redirection (which should have configurable listen/redirect ports). Are there any tools work on Windows to do this?
For Windows HTTP, you can't beat Fiddler. You can use it as a reverse proxy for port-forwarding on a web server. It doesn't necessarily need IE, either. It can use other clients.
Wireshark does not do port redirection, but sniffs and interprets a lot of protocols.
You might find Microsoft Network Monitor helpful if you're on Windows.
Wireshark (or Tshark) is probably the defacto standard traffic inspection tool. It is unobtrusive and works without fiddling with port redirecting and proxying. It is very generic, though, as does not (AFAIK) provide any tooling specifically to monitor web service traffic - it's all tcp/ip and http.
You have probably already looked at tcpmon but I don't know of any other tool that does the sit-in-between thing.
I tried Fiddler with its reverse proxy ability which is mentioned by #marxidad and it seems to be working fine, since Fiddler is a familiar UI for me and has the ability to show request/responses in various formats (i.e. Raw, XML, Hex), I accept it as an answer to this question. One thing though. I use WCF and I got the following exception with reverse proxy thing:
The message with To 'http://localhost:8000/path/to/service' cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher. Check that the sender and receiver's EndpointAddresses agree
I have figured out (thanks Google, erm.. I mean Live Search :p) that this is because my endpoint addresses on server and client differs by port number. If you get the same exception consult to the following MSDN forum message:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2302537&SiteID=1
which recommends to use clientVia Endpoint Behavior explained in following MSDN article:
http://msdn.microsoft.com/en-us/magazine/cc163412.aspx
I've been using Charles for the last couple of years. Very pleased with it.
I second Wireshark. It is very powerful and versatile.
And since this tool will work not only on Windows but also on Linux or Mac OSX, investing your time to learn it (quite easy actually) makes sense. Whatever the platform or the language you use, it makes sense.
Regards,
Richard
Just Programmer
http://sili.co.nz/blog
I find WebScarab very powerful
Check out Paros Proxy.
JMeter's built-in proxy may be used to record all HTTP request/response information.
Firefox "Live HTTP headers" plugin may be used to see what is happening on the browser side when sending/receiving request.
Firefox "Tamper data" plugin may be useful when you need to intercept and modify request.
I use LogParser to generate graphs and look for elements in IIS logs.
Related
There is an old OC4J server where 200+ web services are deployed. Now all have been ported to weblogic. The question is whether can I check if anyone is still accessing the webservices deployed in oc4j. Can't rely on app logs as this is quite old (not much standard followed)
If anyone is accessing is it possible to understand who is it?
PS: Am a newbie please help me in understanding this (if possible) in detail.
Much appreciated.
You can use a tool like TCPDump to locate in real-time traffic on the LAN where the destination and port number match the host/port of the running OC4J
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I had to "sporadically" work with Heimdal / MIT Gssapi for kerberos authentication over past couple of years. I had to build an application that was to be used as a web-service running on a Linux box, and serve client applications like browsers, running on Windows and/or Linux Desktops and Workstations. Surely not the easiest of beasts to tame. Eventually when summarizing my work, I could record that the difficulties emanated due to challenges in multiple dimensions. Getting started with gssapi programming is truly a challenge just because of poor documentation, and practically non-existant tutorials. Googling mostly results in either some theoretical discussion on what's kerberos, or leads to content written with presumption that you already know everything besides some particular semantic issue.
Some really good hacks around here contributed to help me, I therefore suppose it would be a good idea to summarize the stuff, from a developer's perspective, and share it here as some sort of a wiki, to give something back to this fantastic place, and fellow programmers.
Haven't really done a wiki like this before, and I am surely no authority on GSSAPI nor Kerberos, so please be kind, but more than that please contribute and correct my mistakes. Site Editors, I am counting on you to do your magic ;)
Getting your project completed successfully will require 3 specific things to be done correctly:
Setup of your test environment
Setup of your libraries
Your code
As I said already, such projects are beasts, just because all the three haven't been put together on the same page anywhere.
Ok So let's begin at the beginning.
Unavoidable theory for a newbie
GSSAPI helps a client application to provide credentials for a server to authoritatively identify the user. Extremely useful because the server applications can modulate their served responses if they wish to, as per the user. Very naturally therefore both - the client and the server applications must be kerberized, or as some would state kerberos-aware.
The kerberos based authentication, requires both the client and server applications, to be members of a Kerberos Realm. KDC (Kerberos Domain Controller) is the designated authority that rules the realm. Microsoft's AD servers are one of the most popularly experienced examples of a KDC, though you can of course be using a *NIX based KDC. But surely without a KDC there can be no Kerberos business at all. Desktops, Servers & workstations joined into the domain identify each other as long as all of them remain joined into the domain.
For your initial experiments, setup the client & server applications in the same realm.
Though Kerberos Authentication can surely be also used across realms by creating trusts between KDCs of these realms, or even merging keytabs from different KDCs that do not trust each other. Your code will not really need any change to accommodate such different and complex-sounding scenarios.
Kerberos Authentication basically works via "tickets (or tokens)". When a member joins the realm, the KDC "grants tokens" to each of them. These tokens are unique; time and FQDN are essential factors for these tickets.
Before you even think of the very first line of your code make sure you have got these two right:
Pitfall #1 When you setup your development and test environment, make sure everything is tested and addressed as FQDN. For example if you want to check connectivity, ping using FQDN, not IP. Needless to say therefore, they must necessarily have the same DNS service configuration.
Pitfall #2 Make sure all the host systems - that are running your KDC, client software, server software have the same time server. Time synchronization is something that one forgets, and realizes to be amiss after a lot of hair-splitting, and head-banging!
Both, the client and server applications NEED kerberos keytabs. So if your application is going to run in a *NIX host, and be a part of a Microsoft Domain, you have to get a kerberos keytab generated, before we start to look at the remaining preparatory steps for gss programming.
Step-by-Step Guide to Kerberos 5 (krb5 1.0) Interoperability at is an absolute must-read.
GSS-API Programming Guide is an excellent bookmark.
Depending upon your *NIX distribution you can install the headers & libraries for building your code. My suggestion however is to download the source and build it yourself. Yes, you might not get it right at one go, but it surely is worth the trouble.
Pitfall #3 Make sure that your application is running in an Kerberos aware environment.
I really learnt this the hard way, but maybe because I am not so smart. In my earliest stages of gssapi programming struggle, I had discovered that kerberos keytabs were absolutely necessary for making my application kerberos-aware. But I simply couldn't find anything about how to load these keytabs in my application. You know why?!! Because no such api exists!!!
Because: The application is to be run in an environment which is aware of the keytabs.
Ok, let me make this simple: Your application that is supposed to do the GSSAPI / Kerberos things has to run after you have set environment variable KRB5_KTNAME to the path where you have stored the keytabs. So either you do something like:
export KRB5_KTNAME=<path/to/your/keytab>
or make use of setenv to set KRB5_KTNAME in your application sufficiently before the very first line of your code that uses gssapi is run.
We are now ready to do the necessary things in the application's code.
I understand there are quite a few other aspects that must be reviewed by the application developer, to write and test an application. I know of a few environment variables, that can be important.
Can anybody please shed some more light upon that?
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
I'm looking to develope a c/c++ application for linux that converts ipv4 packet received to ipv6 and viceversa ( losing some ipv6 only features )
step 1: how to receive all necessary info from an incoming packet? Should I use raw packet library to read all TCP/UDP flags and info about the packet?
Any documentation about that? ( I'm already looking at beej.us guide)
step2: i'm looking to use this program into a linux machine (ie Ubuntu) as a router to forward all packets received from an ipv6 machine to a net card, to an ipv4 machine connected to the ipv4 card on the router.
How to receive and parse all packets in this application ( except the packets directed to the router machine IP ) ? Is it possible in the 'application level' or should I touch the kernel? If yes, where could i get some documentation about this?
Goal: have an http or other common protocol works between the 2 machines connected via the router
Greatly appreciate any hints
Since converting between IPv4 and IPv6 necessarily implies changing IP addresses in the packet, NAT is required by definition. Your project comes down to implenting a NAT router.
Read up on NAT64 to find out more about the particular flavour of NAT you are looking for.
In the course of trying to implement a router in userspace, I think that tun devices are probably the best design choice for sending and receiving packets. This is in fact the approach chosen by TAYGA (the first NAT64 implementation listed on the above-cited Wikipedia page).
Implementing a router (of any type, let alone a NAT) in userspace is a fairly complex and ambitious project, so the best two pieces of advice I can give are:
Do not implement this yourself. Instead, contribute your efforts to improving one of the existing open source implementations.
Failing that, study one of the existing open source implementations for inspiration.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I am experimenting and attempting to make a minimal web crawler. I understand the whole process at a very high level. So getting into the next layer of details, how does a program 'connect' to different websites to extract the HTML?
Am I using Sockets to connect to servers and sending http requests? Am I giving commands to the terminal to run telnet or ssh?
Also, is C++ a good language of choice for a web crawler?
Thanks!
Also, is C++ a good language of choice for a web crawler?
Depends. How good are you at C++.
C++ is a good language to write an advanced high speed crawler in, because of its speed (and you need that to processes the HTML pages). But it is not the easiest language to write a crawler with so probably not a good choice if you are experimenting.
Based on your question you don't have the experience to write an advanced crawler so are probably looking to build a simple serial crawler. For this speed is not a priority as the bottleneck is the download of the page across the web (not the processing of the page). So I would pick another language (maybe python).
Short answer, no. I prefer coding in C++ but this instance calls for a Java application. The API has many html parsers plus built in socket protocols. This project will be a pain in C++. I coded one once in java and it was somewhat blissful.
BTW, there are many web crawlers out there but I assume you have custom needs :-)
If you plan to stick with C++, then you should consider using the libcurl library, instead of implementing the HTTP protocol from scratch using sockets. There are C++ bindings available for that library.
From curl's webpage:
libcurl is a free and easy-to-use client-side URL transfer library,
supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS,
LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet
and TFTP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP
uploading, HTTP form based upload, proxies, cookies, user+password
authentication (Basic, Digest, NTLM, Negotiate, Kerberos), file
transfer resume, http proxy tunneling and more!
libcurl is highly portable, it builds and works identically on
numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD,
Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga,
OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare,
DOS and more...
libcurl is free, thread-safe, IPv6 compatible, feature rich, well
supported, fast, thoroughly documented and is already used by many
known, big and successful companies and numerous applications.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I want to create a Mobile messaging service, but I don't know which is better to use a socket programming or a web service ?.
What are the concerns that I need to take in consideration when creating such service ? such as cost of connection .. etc.
If you need more details just tell me before voting the question down or to be closed !
Webservices are generally speaking "easier" to do, thanks to the tremendous interest in them and the support for them in developer tools and through libraries and frameworks.
However, especially if your payload is small (think messages the size of a typical SMS or tweet), the overhead you create with webservices is prohibitive: bytes sent over a wireless network like GPRS or UMTS are still very expensive, compared to bytes carried over cable or ADSL. And web services carry several layers of invisible info around that the end customer will also have to pay.
So, if your use case is based on short messages, I'd at least advise to do some bandwidth simulation calculations, and base your decision on bandwidth savings vs increased complexity of your app.
While looking at sockets, also have a look at UDP: if you can live with the fact that basically you throw someone a packet, and without designing some ack mechanism into your protocol you'll never be sure the message arrived, it's very efficient because there is no traffic to create and maintain a connection, and even pretty long messages can very well be transported inside 1 UDP packet.
EDIT based on comment:
stream socket: not sure how you define streams, but streams and messages are two very distinct concepts for me, a stream is a typically longer sequence of data being sent, whereas a message is an entity that's sent, and optionally acknowledged or answered by the receiver.
bandwidth simulation: the easiest way to understand what I'm talking about is to get Wireshark and add up everything that gets transported across the net to do a simple request of a very short string - you'll see several layers of administrative info (ie invisible, just there to make the different protocol layers work) that are all traffic paid for by the end user. Then, write a small mock service using UDP to transport the same message, or use a tool like netcat, good tutorial here, and add up the bytes that get transported. You'll see pretty huge differences in the number of bytes that are exchanged.
EDIT2, something I forgot to mention: mobile networks used to be open, transparent networks with devices identified by public IP addresses. There's a rapid evolution towards NATed mobile networks which has its impact on how devices inside and outside this "walled garden" can communicate (NAT traversal). You'll need to take this into account when designing your communication channel.
As for the use of streams for a chat application: it offers some conceptual advantages, but you can very well layer a chat app on top of UDP, look here or here