I'm trying to build an application which allows a user to transfer files/directories from its computer to another computer when he wants in LAN ~ TCP/IP without any intervention of the receiver computer's user.
To deal this, I think Sockets would be the best alternative. Because if I use FTP and let the receiver's computer's FTP port open continually that would be a vulnerability.
Is the use of Sockets the best choise?
If yes, how to send directories and non-text files throw sockets?
When it comes to security, it's really a matter of "whatever you do, it will only be as safe as the keeping of the password/credentials needed to log in". Using FTPS or SSH protocols will encrypt the traffic between the machines, ensuring that nobody outside can "see" what the files are (or passwords, etc). SSH also has features to identify if the remote machine suddenly changes, so you can identify if somebody has introduced a "man in the middle" attack (that is, pretending to be your actual machine you are sending to)
As for sending non-text files, it shouldn't really be any different than sending text-files in all cases I'm aware of. Of coruse, if you use FTP, you need to set the protocol to "binary mode" before sending binary files, as some systems will otherwise "modify" the content (e.g. translating CR, LF and CRLF sequences to match the target - and a JPG image will certainly look quite weird when all the bytes with value 0x0A has been replaced with 0x0D 0x0A in the file...).
Of course, you could also set up a web-server with suitable software on the receiving machine and use HTTP/HTTPS protocol to upload files - with or without password protection (and in HTTPS, the password is safe as long as nobody "outside group of trust" has access to the actual sending/receiving machine, as the traffic is encrypted).
There are literally several hundred other solutions. Without understanding MUCH more about exactly what problem you are trying to solve, it's hard to make very concrete solutions.
You are going to need some sort of server on the receiving machine as there is not normally any process listening and writing what it receives into the filesystem. Have a practice with netcat (also known as nc) before you write too much code. See here.
Related
How can I upload a file in FTP in segmented way ? Is there any open source tool/library so that I can use it?.
Is there any server side change needed to combine the uploads? Currently I am using vsftpd.
The first thing to consider is that segmented transfers are not considered to be good net citizen behaviour. (i.e. you are gaming the system by setting up multi downloads on a shared link, gaining more than your fair share of bandwidth) As such, the protocol definitions do not support specifically segmented upload. (Or download for that matter) Resume yes.
Segmented DOWNLOAD is a hack by some tools that use the RESUME function of the protocol to transfer different parts of the same file at the same time.. this behaviour has a "NON-STANDARD" and not the intention of the protocol specifications.
Segmented UPLOAD is possible but the client AND ftpd server (or whatever protocol server your using) would need to support this NO-STANDARD and frowned upon implementation.
Again, this is not supported specifically in any standards as such poor behaviour is not encouraged by an open standard.
HOWEVER, you will find tools like lftp that support segmented ftp downloads. But currently, I have not seen any implementation of segmented upload that uses common open protocols like ftp.
I did find a java (Custom open source) based udp tool that did this, but udp needs tcp fallback if you want reliability in the internet. (udp is dropped by some internet gateways)
In FTP protocol, you can implement a transfer by parts using REST command.
The REST command defines offset in a file, where transfer starts. You then transfer as many bytes as you want. And then you can restart the transfer again from a further offset.
vsftpd server supports REST command.
I am no expert in network programming although I do have some knowledge of Winsock, for any experts out there I am wanting to know if there is a way I can capture data at the socket coming from an application on my machine and do something with it. ie: I sent a message via MSN but I want to capture it from a custom application before it actually gets sent.
Thanks.
You can certainly capture the packets. Tools like Wireshark are proof of that (have a look at the WinPCap library). Just keep in mind that you are capturing what an application sends, so if the application sends encrypted data using SSL/TLS or similar, that is what you are going to get. You won't be able to decrypt and view the original data without the security keys used.
Altering and/or discarding packets, on the other hand, is much harder, requiring much lower level access to the system, but it is possible (see WinDivert, for example).
Please tell me is it possile to know when a program is trying to download a file ( like in Internet Download Manager ). I want to catch that event (hook it), get the download url, and then destroy the event.
Thanks in advance..
#Jerry Coffin:Sr, I forgot to tell you that this feature of IDM is not active by default. It is only turned on when you enable the "Use advance browser integration" option at "Download/Options" of IDM menu.
Like here :
http://files.myopera.com/UenX/files/Detect.jpg
+ Check the (1) options, OK, then reboot.
+ After reboot, the (2) option will appear, check it, OK, and now run your software. You should see some thing likes (3)
( this appear when I run the msgr9us.exe ( Yahoo! Messenger setup file) )
Give it a try..
For a specific program such as Internet Explorer, doing this is quite reasonable (IE includes hooks to invoke your code under the right circumstances). For most programs it's not possible though -- they simply don't generate any "event" for you to hook and "destroy".
To make a long story short, to get anywhere with this, you'll almost certainly need to handle the situation on a case-by-base basis, writing code specific to each application you want to deal with -- and know that any other application and even newer versions of the applications you've dealt with will probably break what you're trying to do.
Not really. Consider how a browser typically downloads a file: it opens a TCP socket connection to a remote server, either on port 23 or 80, and using the FTP protocol or HTTP protocol on that connection. These things you can detect, intercept and modify with high reliability. But there are other programs that use other mthods. for instance, P2P filesharing programs such as BitTorrent do not use HTTP or FTP, nor do they download a file from a single server.
So, while you don't need to understand every program, you must be able to detect and understand every file download protocol instead.
you could hook the network stream and filter for http download requests.
you'll need some library to capture network traffic (e.g. http://en.wikipedia.org/wiki/Pcap).
Then you'll have to parse the network packets for the appropriate HTTP messages (sorry, I can't give them to you, I don't know them). I don't know if you can actually prevent packets from being sent though.
Another (easier) way would be to implement a proxy server (or modify an existing one) to do what you want. Then you just have to connect the IE to your proxy using the proxy server settings. Check for example Privoxy, which already does some kind of filtering.
I need to have as part of a desktop application a file server which should respond as fast as possible to file transfer requests (from remote clients, usually located on the same LAN). There will be many file requests for small sized files. The server should be able to provide both upload and download services.
I am not tight to any particual technology so I am open to any programming language, toolkits, libraries as long as they can run on Windows.
My initial take is to go with a C/C++ implementation using Windows Sockets or use the services provided by libraries such as Boost (asio or such). I have also thought of Erlang but that I'll have to learn and so the performance benefits should justify the increased development time due to having to learn the language.
LATER EDIT: I appreciate the answers that say use FTP or HTTP or basically anything that has been already created but considering you still want to write one from scratch, what would you do?
Why not just go with FTP? You should be able to find an adequate server implementation in any language, and client access libraries too.
It sounds like a lot of wheel-reinvention. Granted, FTP is not ideal, and has a few odd spots, but ... it's there, it's standard, well-known, and already very widely implemented.
For frequent uploads of small files, the fastest way would be to implement your own proprietary protocol, but that would require a considerable amount of work - and also it would be non-standard, meaning future integration would be difficult unless you are able to implement your protocol in any client you'll support. If you choose to do it anyway, this is my suggestion for a simple protocol:
Command: 1 byte to identify what'll be done: (0x01 for upload request, 0x02 for download request, 0x11 for upload response, 0x12 for download response, etc).
File name: can be fixed-size or prefixed with a byte for the length (assuming the name is less than 255 bytes)
Checksum, MD5 for instance (if upload request or download response)
File size (if upload request or download response)
payload (if upload request or download response)
This could be implemented on top of a simple TCP socket. You can also use UDP, avoiding the cost of establishing a connection but in this case you have to deal with retransmission control.
Before deciding to implement your own protocol, take a look at HTTP libraries like libcurl, you could make your server use standard HTTP commands like GET for download and POST for upload. This would save a lot of work and you'll be able to test the download with any web browser.
Another suggestion to improve performance is to use as the file repository not the filesystem, but something like SQLite. You can create a single table containing one char column for the file name and one blob column for the file contents. Since SQLite is lightweight and does an efficient caching, you'll most of the time avoid the disk access overhead.
I'm assuming you don't need client authentication.
Finally: although C++ is your preference to give you raw native code speed, rarely this is the major bottleneck in this kind of application. Most probably will be disk access and network bandwidth. I'm mentioning this because in Java you'll probably be able to make a servlet to do exactly the same thing (using HTTP GET for download and POST for upload) with less than 100 lines of code. Use Derby instead of SQLite in this case, put that servlet in any container (Tomcat, Glassfish, etc) and it's done.
If all the machines are running on Windows on the same LAN, why do you need a server at all? Why not simply use Windows file sharing?
I would suggest not to use FTP, or SFTP, or any other connection oriented technique. Instead, go for a connectionless protocol or technique.
The reason is that, if you require lots of small files to be uploaded or downloaded, and the response should be as fast as possible, you want to avoid the cost of setting up and destroying connections.
I would suggest that you look at either using an existing implementation or implementing your own HTTP or HTTPS server/service.
Your bottlenecks are likely to come from one of the following sources:
Harddisk I/O - The WD velociraptor is supposed to have a random access speed of about 100MB/s. Also, it is important whether you set it up as RAID0,1,5 or what nots. Some read fast but write slow. Trade-offs.
Network I/O - Assuming that you have the fastest harddisks in a fast RAID setup, unless you use Gbit I/O, your network will be slow. If your pipes are big, you still need to supply it with data.
Memory cache - The in-memory file-system cache will need to be big enough to buffer all the network I/O so that it does not slow you down. That will require large amounts of memory for the kind of work you're looking at.
File-system structure - Assuming that you have gigabytes worth of memory, then the bottleneck will most likely be the data-structure that you use for the file-system. If the file-system structure is cumbersome it will slow you down.
Assuming that all the other problems are solved, then do you worry about your application itself. Notice, that most of the bottlenecks are outside your software control. Therefore, whether you code it in C/C++ or use specific libraries, you will still be at the mercy of the OS and hardware.
Sounds like you should use an SFTP (SSH) server, it's firewall/NAT safe, secure, and already does what you want and more. You could also use SAMBA or windows file sharing for an even more simple implementation.
Why not use something existing, for example a normal Web server handles a lot of small files (images) very well and fast.
And lots of people already spent time in optimizing the code.
And the second benefit is that the transfer is done with HTTP which is an established protocol. And is easily switched to SSL if you need more security.
For the uploads, they are also no problem with a script or custom module - with the same method you can also add authorization.
As long as you don't need to dynamically seek the files i guess this would be one of the best solutions.
It's a new part to an existing desktop application? What's the goal of the server? Is it protecting the files that are uploaded/downloaded and providing authentication and/or authorisation? Does it provide some kind of structure for the uploads to be stored in?
One option may be to install Apache HTTP Server on the machine and serve the file via that. Use POST to upload and GET to download.
If the clients are within a LAN could you not just share a drive?
I want to setup a statistics monitoring platform to watch a specific service, but I'm not quiet sure how to go about it. Processing the intercepted data isn't my concern, just how to go about it. One idea was to setup a proxy between the client application and the service so that all TCP traffic went first to my proxy, the proxy would then delegate the intercepted messages to an awaiting thread/fork to pass the message on and recieve the results. The other was to try and sniff the traffic between client & service.
My primary goal is to avoid any serious loss in transmission speed between client & application but get 100% complete communications between client & service.
Environment: UBuntu 8.04
Language: c/c++
In the background I was thinking of using a sqlite DB running completely in memory or a 20-25MB memcache dameon slaved to my process.
Update:
Specifically I am trying to track the usage of keys for a memcache daemon, storing the # of sets/gets success/fails on the key. The idea is that most keys have some sort of separating character [`|_-#] to create a sort of namespace. The idea is to step in between the daemon and the client, split the keys apart by a configured separator and record statistics on them.
Exactly what are you trying to track? If you want a simple count of packets or bytes, or basic header information, then iptables will record that for you:
iptables -I INPUT -p tcp -d $HOST_IP --dport $HOST_PORT -j LOG $LOG_OPTIONS
If you need more detailed information, look into the iptables ULOG target, which sends each packet to userspace for analysis.
See http://www.netfilter.org for very thorough docs.
If you want to go the sniffer way, it might be easier to use tcpflow instead of tcpdump or libpcap. tcpflow will only output TCP payload so you don't need to care about reassembling the data stream yourself. If you prefer using a library instead of gluing a bunch of programs together you might be interested in libnids.
libnids and tcpflow are also available on other Unix flavours and do not restrict you to just Linux (contrarily to iptables).
http://www.circlemud.org/~jelson/software/tcpflow/
http://libnids.sourceforge.net/
You didn't mention one approach: you could modify memcached or your client to record the statistics you need. This is probably the easiest and cleanest approach.
Between the proxy and the libpcap approach, there are a couple of tradeoffs:
- If you do the packet capture approach, you have to reassemble the TCP
streams into something usable yourself. OTOH, if your monitor program
gets bogged down, it'll just lose some packets, it won't break the cache.
Same if it crashes. You also don't have to reconfigure anything; packet
capture is transparent.
- If you do the proxy approach, the kernel handles all the TCP work for
you. You'll never lose requests. But if your monitor bogs down, it'll bog
down the app. And if your monitor crashes, it'll break caching. You
probably will have to reconfigure your app and/or memcached servers so
that the connections go through the proxy.
In short, the proxy will probably be easier to code, but implementing it may be a royal pain, and it had better be perfect or its taking down your caching. Changing the app or memcached seems like the sanest approach to me.
BTW: You have looked at memcached's built-in statistics reporting? I don't think its granular enough for what you want, but if you haven't seen it, take a look before doing actual work :-D
iptables provides libipq, a userspace packet queuing library. From the manpage:
Netfilter provides a mechanism for
passing packets out of the stack for
queueing to userspace, then receiving
these packets back into the kernel
with a verdict specifying what to do
with the packets (such as ACCEPT or
DROP). These packets may also be
modified in userspace prior to
reinjection back into the kernel.
By setting up tailored iptables rules that forward packets to libipq, in addition to specifying the verdict for them, it's possible to do packet inspection for statistics analysis.
Another viable option is manually sniff packets by means of libpcap or PF_PACKET socket with the socket-filter support.