FTP folders synchronization c / obj c - c++

I have folder (with a lot of sublfolders and files) on remote server and only FTP protocol supported.
I should have synchronized copy on my device every time when user press "SYNC" button.
Content has size more than 1gb - download every time is not good idea :)
I've trying get folders structure (recursively via LIST and CWD) and get files modification date to analyze - should I download this file or no.
Is it one correct way or I missed better solution ? For now it can take around 4 minutes only for get files structure :(
Is it possible get all folders structure with filecustom fields (like modification date)?
Im on objC (iOS) and using CFNetwork sample from SimpleFTPSample (from Apple).
update:
I've found dir -R command - is this command standard (like LIST) or no ?
thanks,

I've replace library to libCURL and now it works in 1 sec... I'm shock!

Related

How can I create a link between mnt/sdcard and sdcard on android x86 4.3?

I am trying to give the virtual machine android x86 ultimately have a fake sdcard. It is getting tricky for me and I am running through the multiple scenarios. The problem is stemming from applications that point towards mnt/sdcard for references, but the directory is empty because it is not automatically synching with the sdcard folder.
On the website they say to make 2 partitions, one for the os and the other for the sdcard. I am not making any progress with that method.
My next choice is to copy all files from the sdcard folder into the mnt/sdcard folder. This works unfortunately I must execute a chmod command on mnt/sdcard under su every time the phone starts up. I am trying to make this a one time operation instead.
My last resort is to execute a script that does the remedial commands on start up, copying the contents of the sdcard folder to mnt/sdcard. Still this does not account for anything added to the sdcard folder afterwards. The manual creation of a symlink between the two folders is the best option for longevity.
That doesn't include the partition example. For anyone familiar with android x86 on VM's The process is as follows:
Create two partitions
a. Bootable primary sda1 linux/ntfs
b. primary sda2 fat36
Install android on sda1
reboot device/vm
Set partition 2 in kernel / GRUB, 2 options
a. scroll to vesa and press tab and enter
b. Press A and enter "DATA=/dev/sda1 SDCARD=/dev/sda2"
after setup complete navigate to settings and click storage
a. Mount and format sdcard shazam the sdcard works and is mounted

Monitoring a directory for subdirectory complete creation and then launching another process, c++

So I have an idea that I would like to implement and it's as follows:
Monitor a specific directory.
once a sub-directory is not only created but completed (i.e. a folder that's being downloaded or copied has just completed) the code calls a procedure or a scheme to compress the folder.
I have a sort of an idea of implementing this using ReadDirectoryChangesW. However my question is how to wait for changes, but when a change happens, it waits for its completeness. The second question would be how to identify the subfolder that's completed so I can call the compression scheme and supply it as an argument.
Thank you.
Since it's labelled "winapi", just set the NTFS compression attribute on the subdirectory as soon as you see it. Any new files in that directory will be automatically compressed as they're created.

C++ / Qt: track filesystem changes

I'm currently trying to implement a c++ program which monitors a folder on the filesystem. On initalizing the application, it scans the Directory and saves some meta information about it.
when something is chenged while the program is active, i can read changes to the folder (for examlpe changing the name of a folder or a file). But i can't track changes to the Directory while the program isn't running. Upon startup i would get
Removed folder X
Added folder Y
instead of
Renamed folder X to Y
is it possible to identify a directory in another way than it's path/name?
if yes, how would i gather that information in C++ / Qt ?
Rather than reinventing the wheel, you could just use the class QFileSystemWatcher which the Qt docs states: -
The QFileSystemWatcher class provides an interface for monitoring
files and directories for modifications
If you want the program to run all the time, then you may want to look at creating a service (in Windows) or daemon (Linux / OSX).
This is filesystem-specific, but generally yes this is possible. FAT is the main exception, I think. But you won't find code for this in the C++ Standard Library or Qt. It's just too unusual, so you'll need OS-specific code if not filesystem-specific.
Like TheDarkNight said, you need to use QFileSystemWatcher to avoid portability and so other problem.
But if you want to continue your approach:
In GNU/Linux land, you can check this with inode struct of directory (take care of symbolic link issue).
inode struct have an index for example you can get it on shell with:
ls -id /path/to/your/folder
There is an API to access inode. You can google inode struct linux for it.
In Windows garden, you can get file id when accessing handle in the struct BY_HANDLE_FILE_INFORMATION:
nFileIndexHigh
The high-order part of a unique identifier that is associated with a file.
For more information, see nFileIndexLow.
nFileIndexLow
The low-order part of a unique identifier that is associated with a file.

inotify : How to be notified of new files in the directory after they are complete in transfer?

A file is copied from machine1/dir1 to machine2/dir2. I have added a inotify watch on the dir2 for any new files created. Now if the file is large, it might take a few seconds to be fully on the new machine. If I'm not wrong, inotify will notify the application as soon as it detects an event. So if the file status has to be checked, How should it be done ?
Save downloaded file with temporary filename (or to other directory) and rename it to expected filename when file moved successfully.
Nginx for example use this method to store cached data
Caching data is first written to the temporary file which is then moved to the final location in a cache directory. Cheap and atomic rename syscall is performed instead of a full file copy So it's better to use the same file system in both locations
There's no way to answer this because it depends on the application environment and requirements. It might do to see that the file hasn't been modified for 60 seconds. It might require checking every few seconds. It depends.
Using IN_CLOSE_WRITE works if its only a scp from one machine to another. Otherwise, it depends on the way the file is uploaded from one machine to another. If its a one time open and close , IN_CLOSE_WRITE is the way to do it.
Both the answers above make sense depending on how we do it.

Automatically checking for a new version of my application

Trying to honor a feature request from our customers, I'd like that my application, when Internet is available, check on our website if a new version is available.
The problem is that I have no idea about what have to be done on the server side.
I can imagine that my application (developped in C++ using Qt) has to send a request (HTTP ?) to the server, but what is going to respond to this request ? In order to go through firewalls, I guess I'll have to use port 80 ? Is this correct ?
Or, for such a feature, do I have to ask our network admin to open a specific port number through which I'll communicate ?
#pilif : thanks for your detailed answer. There is still something which is unclear for me :
like
http://www.example.com/update?version=1.2.4
Then you can return what ever you want, probably also the download-URL of the installer of the new version.
How do I return something ? Will it be a php or asp page (I know nothing about PHP nor ASP, I have to confess) ? How can I decode the ?version=1.2.4 part in order to return something accordingly ?
I would absolutely recommend to just do a plain HTTP request to your website. Everything else is bound to fail.
I'd make a HTTP GET request to a certain page on your site containing the version of the local application.
like
http://www.example.com/update?version=1.2.4
Then you can return what ever you want, probably also the download-URL of the installer of the new version.
Why not just put a static file with the latest version to the server and let the client decide? Because you may want (or need) to have control over the process. Maybe 1.2 won't be compatible with the server in the future, so you want the server to force the update to 1.3, but the update from 1.2.4 to 1.2.6 could be uncritical, so you might want to present the client with an optional update.
Or you want to have a breakdown over the installed base.
Or whatever. Usually, I've learned it's best to keep as much intelligence on the server, because the server is what you have ultimate control over.
Speaking here with a bit of experience in the field, here's a small preview of what can (and will - trust me) go wrong:
Your Application will be prevented from making HTTP-Requests by the various Personal Firewall applications out there.
A considerable percentage of users won't have the needed permissions to actually get the update process going.
Even if your users have allowed the old version past their personal firewall, said tool will complain because the .EXE has changed and will recommend the user not to allow the new exe to connect (users usually comply with the wishes of their security tool here).
In managed environments, you'll be shot and hanged (not necessarily in that order) for loading executable content from the web and then actually executing it.
So to keep the damage as low as possible,
fail silently when you can't connect to the update server
before updating, make sure that you have write-permission to the install directory and warn the user if you do not, or just don't update at all.
Provide a way for administrators to turn the auto-update off.
It's no fun to do what you are about to do - especially when you deal with non technically inclined users as I had to numerous times.
Pilif answer was good, and I have lots of experience with this too, but I'd like to add something more:
Remember that if you start yourapp.exe, then the "updater" will try to overwrite yourapp.exe with the newest version. Depending upon your operating system and programming environment (you've mentioned C++/QT, I have no experience with those), you will not be able to overwrite yourapp.exe because it will be in use.
What I have done is create a launcher. I have a MyAppLauncher.exe that uses a config file (xml, very simple) to launch the "real exe". Should a new version exist, the Launcher can update the "real exe" because it's not in use, and then relaunch the new version.
Just keep that in mind and you'll be safe.
Martin,
you are absolutely right of course. But I would deliver the launcher with the installer. Or just download the installer, launch it and quit myself as soon as possible. The reason is bugs in the launcher. You would never, ever, want to be dependent on a component you cannot update (or forget to include in the initial drop).
So the payload I distribute with the updating process of my application is just the standard installer, but devoid of any significant UI. Once the client has checked that the installer has a chance of running successfully and once it has downloaded the updater, it runs that and quits itself.
The updater than runs, installs its payload into the original installation directory and restarts the (hopefully updated) application.
Still: The process is hairy and you better think twice before implementing an Auto Update functionality on the Windows Platform when your application has a wide focus of usage.
in php, the thing is easy:
<?php
if (version_compare($_GET['version'], "1.4.0") < 0){
echo "http://www.example.com/update.exe";
}else{
echo "no update";
}
?>
if course you could extend this so the currently available version isn't hard-coded inside the script, but this is just about illustrating the point.
In your application you would have this pseudo code:
result = makeHTTPRequest("http://www.example.com/update?version=" + getExeVersion());
if result != "no update" then
updater = downloadUpdater(result);
ShellExecute(updater);
ExitApplication;
end;
Feel free to extend the "protocol" by specifying something the PHP script could return to tell the client whether it's an important, mandatory update or not.
Or you can add some text to display to the user - maybe containing some information about what's changed.
Your possibilities are quite limitless.
My Qt app just uses QHttp to read tiny XML file off my website that contains the latest version number. If this is greater than the current version number it gives the option to go to the download page. Very simple. Works fine.
I would agree with #Martin and #Pilif's answer, but add;
Consider allowing your end-users to decide if they want to actually install the update there and then, or delay the installation of the update until they've finished using the program.
I don't know the purpose/function of your app but many applications are launched when the user needs to do something specific there and then - nothing more annoying than launching an app and then being told it's found a new version, and you having to wait for it to download, shut down the app and relaunch itself. If your program has other resources that might be updated (reference files, databases etc) the problem gets worse.
We had an EPOS system running in about 400 shops, and initially we thought it would be great to have the program spot updates and download them (using a file containing a version number very similar to the suggestions you have above)... great idea. Until all of the shops started up their systems at around the same time (8:45-8:50am), and our server was hit serving a 20+Mb download to 400 remote servers, which would then update the local software and cause a restart. Chaos - with nobody able to trade for about 10 minutes.
Needless to say that this caused us to subsequently turn off the 'check for updates' feature and redesign it to allow the shops to 'delay' the update until later in the day. :-)
EDIT: And if anyone from ADOBE is reading - for god's sake why does the damn acrobat reader insist on trying to download updates and crap when I just want to fire-it-up to read a document? Isn't it slow enough at starting, and bloated enough, as it is, without wasting a further 20-30 seconds of my life looking for updates every time I want to read a PDF?
DONT THEY USE THEIR OWN SOFTWARE??!!! :-)
On the server you could just have a simple file "latestversion.txt" which contains the version number (and maybe download URL) of the latest version. The client then just needs to read this file using a simple HTTP request (yes, to port 80) to retrieve http://your.web.site/latestversion.txt, which you can then parse to get the version number. This way you don't need any fancy server code --- you just need to add a simple file to your existing website.
if you keep your files in the update directory on example.com, this PHP script should download them for you given the request previously mentioned. (your update would be yourprogram.1.2.4.exe
$version = $_GET['version'];
$filename = "yourprogram" . $version . ".exe";
$filesize = filesize($filename);
header("Pragma: public");
header("Expires: 0");
header("Cache-Control: post-check=0, pre-check=0");
header("Content-type: application-download");
header('Content-Length: ' . $filesize);
header('Content-Disposition: attachment; filename="' . basename($filename).'"');
header("Content-Transfer-Encoding: binary");
This makes your web browser think it's downloading an application.
The simplest way to make this happen is to fire an HTTP request using a library like libcurl and make it download an ini or xml file which contains the online version and where a new version would be available online.
After parsing the xml file you can determine if a new version is needed and download the new version with libcurl and install it.
Just put an (XML) file on your server with the version number of the latest version, and a URL to the download the new version from. Your application can then request the XML file, look if the version differs from its own, and take action accordingly.
I think that simple XML file on the server would be sufficient for version checking only purposes.
You would need then only an ftp account on your server and build system that is able to send a file via ftp after it has built a new version. That build system could even put installation files/zip on your website directly!
If you want to keep it really basic, simply upload a version.txt to a webserver, that contains an integer version number. Download that check against the latest version.txt you downloaded and then just download the msi or setup package and run it.
More advanced versions would be to use rss, xml or similar. It would be best to use a third-party library to parse the rss and you could include information that is displayed to your user about changes if you wish to do so.
Basically you just need simple download functionality.
Both these solutions will only require you to access port 80 outgoing from the client side. This should normally not require any changes to firewalls or networking (on the client side) and you simply need to have a internet facing web server (web hosting, colocation or your own server - all would work here).
There are a couple of commercial auto-update solutions available. I'll leave the recommendations for those to others answerers, because I only have experience on the .net side with Click-Once and Updater Application Block (the latter is not continued any more).