As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
I'm looking to develope a c/c++ application for linux that converts ipv4 packet received to ipv6 and viceversa ( losing some ipv6 only features )
step 1: how to receive all necessary info from an incoming packet? Should I use raw packet library to read all TCP/UDP flags and info about the packet?
Any documentation about that? ( I'm already looking at beej.us guide)
step2: i'm looking to use this program into a linux machine (ie Ubuntu) as a router to forward all packets received from an ipv6 machine to a net card, to an ipv4 machine connected to the ipv4 card on the router.
How to receive and parse all packets in this application ( except the packets directed to the router machine IP ) ? Is it possible in the 'application level' or should I touch the kernel? If yes, where could i get some documentation about this?
Goal: have an http or other common protocol works between the 2 machines connected via the router
Greatly appreciate any hints
Since converting between IPv4 and IPv6 necessarily implies changing IP addresses in the packet, NAT is required by definition. Your project comes down to implenting a NAT router.
Read up on NAT64 to find out more about the particular flavour of NAT you are looking for.
In the course of trying to implement a router in userspace, I think that tun devices are probably the best design choice for sending and receiving packets. This is in fact the approach chosen by TAYGA (the first NAT64 implementation listed on the above-cited Wikipedia page).
Implementing a router (of any type, let alone a NAT) in userspace is a fairly complex and ambitious project, so the best two pieces of advice I can give are:
Do not implement this yourself. Instead, contribute your efforts to improving one of the existing open source implementations.
Failing that, study one of the existing open source implementations for inspiration.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Today I am thinking about connect two computer without tcp/ip. Actually i am searching: connection without ip; if i manage to connect without ip, these network is untraceable.
My full question is :
It is possible to connect two computer without tcp/ip over internet.
May these scenario impossible for the ISP. I don't know.
If possible, It can be competitor of Internet.
From the first line of Wikipedia on Internet:
The Internet is a global system of interconnected computer networks that use the standard Internet protocol suite (TCP/IP) to link several billion devices worldwide.
The internet is built upon the IP framework. You can't "not use" IP through the internet. That's like to say I want to use the post system without addresses. Without the IP framework, there is no way to identify devices from each other or have any standard format to route packets anywhere at all. This is not to say that it is the only way to establish networked communications, it's just the most popular and most used way.
Regarding the first part of your question: It is possible to connect two computer without tcp/ip? There are plenty of ways this is done e.g. Bluetooth, RS-232, proprietary RF communications and so forth.
Also, towards competitor of Internet is that really such a good idea? For once we have one system that is universally compatible with all devices around the globe (almost!). I don't think the rest of the world would be keen on a brand new system unless it is much much much better (in which it'll probably be implemented into the Internet Protocol Suite anyway).
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I'd like to hear your guys opinion on architecture decision.
I'm building high-performance upload server. The purpose of server(s), in simple terms, is to receive huge amount of files from clients and save them on NAS.
Requirements are:
Files will be uploaded via HTTP POSTs in "multipart/form-data" encoding.
Each file will be at least few megabytes long.
Clients are browsers on computers and mobile clients.
Because at least 50% of clients will be mobile clients I expect these TCP/IP sessions to be pretty long.
It should not be affected by Slowloris attacks.
Target platform is Linux CentOS 6.2.
As server receives files it saves some info to database, performs authentication and etc.
My current approach is to implement everything (http parsing and etc) myself using Boost Asio. I like it because I have greater control on what happens with data. I'm thinking about writing it to file directly from socket using memory mapped files.
Would you say it might be easier to implement it as extension to Nginx ? Is Nginx a viable option given the requirements listed above?
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I am experimenting and attempting to make a minimal web crawler. I understand the whole process at a very high level. So getting into the next layer of details, how does a program 'connect' to different websites to extract the HTML?
Am I using Sockets to connect to servers and sending http requests? Am I giving commands to the terminal to run telnet or ssh?
Also, is C++ a good language of choice for a web crawler?
Thanks!
Also, is C++ a good language of choice for a web crawler?
Depends. How good are you at C++.
C++ is a good language to write an advanced high speed crawler in, because of its speed (and you need that to processes the HTML pages). But it is not the easiest language to write a crawler with so probably not a good choice if you are experimenting.
Based on your question you don't have the experience to write an advanced crawler so are probably looking to build a simple serial crawler. For this speed is not a priority as the bottleneck is the download of the page across the web (not the processing of the page). So I would pick another language (maybe python).
Short answer, no. I prefer coding in C++ but this instance calls for a Java application. The API has many html parsers plus built in socket protocols. This project will be a pain in C++. I coded one once in java and it was somewhat blissful.
BTW, there are many web crawlers out there but I assume you have custom needs :-)
If you plan to stick with C++, then you should consider using the libcurl library, instead of implementing the HTTP protocol from scratch using sockets. There are C++ bindings available for that library.
From curl's webpage:
libcurl is a free and easy-to-use client-side URL transfer library,
supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS,
LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet
and TFTP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP
uploading, HTTP form based upload, proxies, cookies, user+password
authentication (Basic, Digest, NTLM, Negotiate, Kerberos), file
transfer resume, http proxy tunneling and more!
libcurl is highly portable, it builds and works identically on
numerous platforms, including Solaris, NetBSD, FreeBSD, OpenBSD,
Darwin, HPUX, IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga,
OS/2, BeOs, Mac OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare,
DOS and more...
libcurl is free, thread-safe, IPv6 compatible, feature rich, well
supported, fast, thoroughly documented and is already used by many
known, big and successful companies and numerous applications.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I want to create a Mobile messaging service, but I don't know which is better to use a socket programming or a web service ?.
What are the concerns that I need to take in consideration when creating such service ? such as cost of connection .. etc.
If you need more details just tell me before voting the question down or to be closed !
Webservices are generally speaking "easier" to do, thanks to the tremendous interest in them and the support for them in developer tools and through libraries and frameworks.
However, especially if your payload is small (think messages the size of a typical SMS or tweet), the overhead you create with webservices is prohibitive: bytes sent over a wireless network like GPRS or UMTS are still very expensive, compared to bytes carried over cable or ADSL. And web services carry several layers of invisible info around that the end customer will also have to pay.
So, if your use case is based on short messages, I'd at least advise to do some bandwidth simulation calculations, and base your decision on bandwidth savings vs increased complexity of your app.
While looking at sockets, also have a look at UDP: if you can live with the fact that basically you throw someone a packet, and without designing some ack mechanism into your protocol you'll never be sure the message arrived, it's very efficient because there is no traffic to create and maintain a connection, and even pretty long messages can very well be transported inside 1 UDP packet.
EDIT based on comment:
stream socket: not sure how you define streams, but streams and messages are two very distinct concepts for me, a stream is a typically longer sequence of data being sent, whereas a message is an entity that's sent, and optionally acknowledged or answered by the receiver.
bandwidth simulation: the easiest way to understand what I'm talking about is to get Wireshark and add up everything that gets transported across the net to do a simple request of a very short string - you'll see several layers of administrative info (ie invisible, just there to make the different protocol layers work) that are all traffic paid for by the end user. Then, write a small mock service using UDP to transport the same message, or use a tool like netcat, good tutorial here, and add up the bytes that get transported. You'll see pretty huge differences in the number of bytes that are exchanged.
EDIT2, something I forgot to mention: mobile networks used to be open, transparent networks with devices identified by public IP addresses. There's a rapid evolution towards NATed mobile networks which has its impact on how devices inside and outside this "walled garden" can communicate (NAT traversal). You'll need to take this into account when designing your communication channel.
As for the use of streams for a chat application: it offers some conceptual advantages, but you can very well layer a chat app on top of UDP, look here or here
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
What's the best tool that you use to monitor Web Service, SOAP, WCF, etc. traffic that's coming and going on the wire? I have seen some tools that made with Java but they seem to be a little crappy. What I want is a tool that sits in the middle as a proxy and does port redirection (which should have configurable listen/redirect ports). Are there any tools work on Windows to do this?
For Windows HTTP, you can't beat Fiddler. You can use it as a reverse proxy for port-forwarding on a web server. It doesn't necessarily need IE, either. It can use other clients.
Wireshark does not do port redirection, but sniffs and interprets a lot of protocols.
You might find Microsoft Network Monitor helpful if you're on Windows.
Wireshark (or Tshark) is probably the defacto standard traffic inspection tool. It is unobtrusive and works without fiddling with port redirecting and proxying. It is very generic, though, as does not (AFAIK) provide any tooling specifically to monitor web service traffic - it's all tcp/ip and http.
You have probably already looked at tcpmon but I don't know of any other tool that does the sit-in-between thing.
I tried Fiddler with its reverse proxy ability which is mentioned by #marxidad and it seems to be working fine, since Fiddler is a familiar UI for me and has the ability to show request/responses in various formats (i.e. Raw, XML, Hex), I accept it as an answer to this question. One thing though. I use WCF and I got the following exception with reverse proxy thing:
The message with To 'http://localhost:8000/path/to/service' cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher. Check that the sender and receiver's EndpointAddresses agree
I have figured out (thanks Google, erm.. I mean Live Search :p) that this is because my endpoint addresses on server and client differs by port number. If you get the same exception consult to the following MSDN forum message:
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2302537&SiteID=1
which recommends to use clientVia Endpoint Behavior explained in following MSDN article:
http://msdn.microsoft.com/en-us/magazine/cc163412.aspx
I've been using Charles for the last couple of years. Very pleased with it.
I second Wireshark. It is very powerful and versatile.
And since this tool will work not only on Windows but also on Linux or Mac OSX, investing your time to learn it (quite easy actually) makes sense. Whatever the platform or the language you use, it makes sense.
Regards,
Richard
Just Programmer
http://sili.co.nz/blog
I find WebScarab very powerful
Check out Paros Proxy.
JMeter's built-in proxy may be used to record all HTTP request/response information.
Firefox "Live HTTP headers" plugin may be used to see what is happening on the browser side when sending/receiving request.
Firefox "Tamper data" plugin may be useful when you need to intercept and modify request.
I use LogParser to generate graphs and look for elements in IIS logs.