I have an application that transfers files via socket, desire that this also make transfers directories.
How can I download a full DIRECTORY via socket?
The program works like this, it asks the user to enter the remote directory where the file is to transfer Example: C:\users\server\file.dat
After it makes a validation to confirm the existence of the file,
and finally it transfers byte by byte.
The problem is that when the user type a directory validation fail, an example is if I type C:\users\SERVER\DIRECTORY
the program then returns an error FAILED IN READ BYTES
The solution would be to zip the directory, but the server only works with command line "shell", and Windows has nothing to zip by native command line.
Any suggestions?
You cannot read a directory like an ordinary file. You should search for all files yourself and send them one by one. This means you should also send paths to all files and reproduce their hierarchy yourself.
Related
I have an access to some server where there is a lot of data. I can't copy the whole of data on my computer.
I can't compile on the server the program I want because the server doesn't have all libs I need.
I don't think that the server admin would be very happy to see me coming and asking to him to install some libs just for me...
So, I try to figure if there is a way to open a file like with,
FILE *fopen(const char *filename, const char *mode);
or
void std::ifstream::open(const char* filename, ios_base::openmode mode = ios_base::in);
but over a SSH connection. Then reading the file like I do for usual program.
both computer and server are running linux
I assume you are working on your Linux laptop and the remote machine is some supercomputer.
First non-technical advice: ask permission first to access the data remotely. In some workplaces you are not allowed to do that, even if it technically possible.
You could sort-of use libssh for that purpose, but you'll need some coding and read its documentation.
You could consider using some FUSE file system (on your laptop), e.g. some sshfs; you would then be able to access some supercomputer files as /sshfilesystem/foo.bar). It is probably the slowest solution, and probably not a very reliable one. I don't really recommend it.
You could ask permission to use NFS mounts.
Maybe you might consider some HTTPS access (if the remote computer has it for your files) using some HTTP/HTTPS client library like libcurl (or the other way round, some HTTP/HTTPS server library like libonion)
And you might (but ask permission first!) use some TLS connection (e.g. start manually a server like program on the remote supercomputer) perhaps thru OpenSSL or libgnutls
At last, you should consider installing (i.e. asking politely the installation on the remote supercomputer) or using some database software (e.g. a PostgreSQL or MariaDB or Redis or MongoDB server) on the remote computer and make your program become a database client application ...
BTW, things might be different if you access a few dozen of terabyte sized files in a random access (each run reading a few kilobytes inside them), or a million files, of which a given run access only a dozen of them with sequential reads, each file of a reasonable size (a few megabytes). In other words, DNA data, video films, HTML documents, source code, ... are all different cases!
Well, the answer to your question is no, as already stated several times (unless you think about implementing ssh yourself which is out of scope of sanity).
But as you also describe your real problem, it's probably just asking the wrong question, so -- looking for alternatives:
Alternative 1
Link the library you want to use statically to your binary. Say you want to link libfoo statically:
Make sure you have libfoo.a (the object archive of your library) in your library search path. Often, development packages for a library provided by your distribution already contain it, if not, compile the library yourself with options to enable the creation of the static library
Assuming the GNU toolchain, build your program with the following flags: -Wl,-Bstatic -lfoo -Wl,-Bdynamic (instead of just -lfoo)
Alternative 2
Create your binary as usual (linked against the dynamic library) and put that library (libfoo.so) e.g. in ~/lib on the server. Then run your binary there with LD_LIBRARY_PATH=~/lib ./a.out.
You can copy parts of file to your computer over SSH connection:
copy part of source file using dd command to temporary file
copy temporary file to your local box using scp or rsync
You can create a shell script to automate this if you need to do that multiple times.
Instead of fopen on a path, you can use popen on an ssh command. (Don't forget that FILE * streams obtained from popen are closed with pclose and not fclose).
You can simplify the interface by writing a function which wraps popen. The function accepts just the remote file name, and then generates the ssh command to fetch that file, properly escaping everything, like spaces in the file name, shell meta-characters and whatnot.
FILE *stream = popen("ssh user#host cat /path/to/remote/file", "r");
if (stream != 0) {
/* ... */
pclose(stream);
}
popen has some drawbacks because it processes a shell command. Because the argument to ssh is also a shell command that is processed on the remote end, it raises issues of double escaping: passing a command through as a shell command.
To do something more robust, you can create a pipe using pipe, then fork and exec* the ssh process, installing the write end of the pipe as its stdout, and use fdopen to create a FILE * stream on the reading end of the pipe in the parent process. This way, there is accurate control over the arguments which are handed to the process: at least locally, you're not running a shell command.
You can't directly(1) open a file over ssh with fopen() or ifstream::open. But you can leverage the existing ssh binary. Simply have your program read from stdin, and pipe the file to it via ssh:
ssh that_server cat /path/to/largefile | ./yourprogram
(1) Well, if you mount the remote system using sshfs you can access the files over ssh as if they were local.
I have a c++ project that I would like to send to someone in executable form. The issue is the program must read from a .txt that I created (specific deliminators). Currently my program reads from a file path that is specific to my computer,
parseFile("/Users/David/Desktop/FinalProject/Store.txt");
How could I package the .txt file and the executable file together, where the exec. reads specifically from the that .txt on anyone's machine?
Note: I am using Xcode
Change your programs to receive 'file path' as a parameter. Write a note(ReadMe) with the program to specify the file format and added a sample data file with the package
tl;dr: if you just put the text file in the same folder with your executable, you can open it with parseFile("Store.txt");
In most runtime implementations, there is a notion of a "working directory." When you open up an executable via the graphical shell (by double clicking it or something to that effect) the working directory is the same as the directory the executable is in.
Now, if you try to open a file in your program via a path that isn't fully qualified, then the path that gets used will be relative to the working directory.
A fully qualified path is a discrete path that points to a single entity in your filesystem. "/Users/David/Desktop/FinalProject/Store.txt" is one such example, as it starts at root (/ on *nix, DriveLetter:\ on Windows) and says exactly which directories you need to traverse to get to your file.
A path that is not fully qualified (which basically means that it doesn't start at the root of your filesystem) can be used to perform relative file addressing. Most runtimes will assume that any path that is not fully qualified is meant to be relative to the working directory, which basically means that the path that actually gets opened is the result of concatenating your provided path to the end of the working directory.
As an example, if you opened your binary, which is stored as /Users/David/Desktop/FinalProject/a.exe, then the working directory would be set to /Users/David/Desktop/FinalProject/. If your program then tried to open "Store.txt", the runtime would see that you're trying to open a path that isn't fully qualified, so it would assume you meant to open a file relative to the working directory, which would then be /Users/David/Desktop/FinalProject/ + Store.txt, which would be /Users/David/Desktop/FinalProject/Store.txt.
The nice thing about this is that if you move your binary, the working directory moves too. if you move a.exe along with Store.txt to /Users/David/Desktop/FinalProject(copy)/, then when you open /Users/David/Desktop/FinalProject(copy)/a.exe, the working directory will be /Users/David/Desktop/FinalProject(copy)/ now, and now when you call parseFile("Store.txt"), it will instead open up /Users/David/Desktop/FinalProject(copy)/Store.txt. This holds true when moving to other computers, too.
It's worth noting that if your binary is run from a command line utility, the working directory will often be the directory the command line shell is in, rather than the executable's directory. It is, however, a part of the C standard that the first command line parameter to main() should be the name of the executable, and most implementations supply you with the fully qualified path. With some minimal parsing, you can use that to determine what path to use as a base for addressing files.
Is it possible to use regex for matching file names in FTP to get files from server ?
I need to do FTP to server and need to download the files whose file names are ending with the same value. In my case, it is 14_04_25_144238.
I am not sure if it is doable. But, just out of curiosity, asking this.
Can we use regex like .*14_04_25_144238 in the ftp get command ?
Thanks in advance.
Dinesh S
You want the mget command.
From the Unix man page
mget remote-files
Expand the remote-files on the remote machine and
do a get for each file name thus produced. See glob for details on
the filename expansion. Resulting file names will then be
processed according to case, ntrans, and nmap settings. Files are
transferred into the local working directory, which can be changed
with 'lcd directory'; new local directories can be created with '!
mkdir directory'.
If you want to turn of the prompting of each file, then you also need this:
prompt
Toggle interactive prompting. Interactive prompting occurs
during multiple file transfers to allow the user to selectively retrieve or store files. If prompting is turned off (default is on), any mget or mput will transfer all files, and any
mdelete will delete all files
.
I am writing a c++ program that interfaces with an Apache FtpServer using libcurl. I was originally using the LIST command to get the contents of a directory but it was giving me a lot of information I didn't but had to parse any ways which lead to a lot unneeded overhead (especially when I was working with hundreds of thousands of files). In addition I needed a valid time stamp and it was giving me a shorthand that didn't include the year (so on January 1 all of the files on my computer looked outdated compared to the FTP server's). My solution was to use the NLST command to get only the names, then download the timestamps of each using MDTM. This worked awesome but then I ran into the major problem of not being able to tell if a file was a directory or not.
I am thinking the easiest way to do this is using the permissions to see if the first flag is set to d. FTP doesn't appear to have this functionality. Is there an easy way to tell if a filename is a directory or file?
You can try to use the CWD command on the file to test if it is a directory or not. But failure may mean lack of permission, so you need to check the error code. For example:
ftp> cd atom.xml
550 Can't change directory to atom.xml: Not a directory
Alternatively, you can use the NLST command again on the file you want to test. If it is a plain file, you will just get the filename back. Otherwise, you will get a list of contents of the directory.
ftp> nlist atom.xml
200 PORT command successful
150 Connecting to port 53912
atom.xml
226 1 matches total
ftp> mkdir foo
257 "foo" : The directory was successfully created
ftp> nlist foo
200 PORT command successful
150 Connecting to port 53928
226 0 matches total
pConnect->SetCurrentDirectory( "C:\\FilesToSendToServer" ); //Need this set on client, i believe currently setting on server.
CFtpFileFind finder(pConnect);
finder.FindFile( "*", INTERNET_FLAG_DONT_CACHE | INTERNET_FLAG_EXISTING_CONNECT );
finder.FindNextFile();
CString filename = finder.GetFileName(); //finds the only file in that directory - test.txt
pConnect->PutFile( filename, "C:\\FilesReceived\\FILE_SENT_FROM_CLIENT.txt", FTP_TRANSFER_TYPE_BINARY, 1 )) //filename set to test.txt correctly
On the client I have the directory FilesToSendToServer with one test file - "test.txt"
I also have the working directory of this application on the client - C\uploadApp\
The code above sets the current directory correctly and "finds" the test file.
However, when PutFile executes with the filename parameter set correctly (test.txt) the functin returns the error file not found.
So as a test, I put a copy of test.txt in the applications working directory uploadApp and it DID send the file to the server.
Why does the file need to be in the working directory to be sent if the active directory is set in the CFtpConnection object?
(Does this mean for any file I want to send from the client I have to copy it over to the directory of the application?)
Thank You.
EDIT
Looks like SetCurrentDirectory calls FtpSetCurrentDirectory which
determine the remote site's current working directory
So for a GET this would make sense...is there an alternate function for a PUT - to set the active directory of the local machine? (FYI - Unable to make a distinction between the remote directory on the server testing on same machine...if testing on target the SetCurrentDirectory should fail, as it is looking on the server...I assume)
It looks like you were meaning to call the win32 API SetCurrentDirectory. This will change the local directory instead of the remote directory that pConnect->SetCurrentDirectory changes. Try ::SetCurrentDirectory if you're in a namespace. Of course your working directory will no longer be C:\uploadApp...
To change the working directory on the client side you should be able to call Win32 ::SetCurrentDirectory.
You will find it easier to get this code working if you get in the habit of checking for errors on any Win32 call, including those encapsulated within MFC.