I've written a small program which is able to upload files to a server via ftp. Because of the large size of some files I want to create a progress bar for the user. So during the upload I need to know at certain intervals how many bytes have been sent to the server in order to derive the percentage of the file that has been uploaded. What I have tried so far:
While I call the function FtpPutFile() to upload the file, I spawn a thread with the following code:
hInternet = InternetOpen(NULL,INTERNET_OPEN_TYPE_DIRECT,NULL,NULL,0);
hFtpSession = InternetConnect(hInternet, ftpserver, port, user, pass, INTERNET_SERVICE_FTP, 0, 0);
int filesize = 0; // 2GB max
hFile = FtpOpenFile(hFtpSession,szFileTitle,GENERIC_READ,FTP_TRANSFER_TYPE_BINARY,0);
filesize = FtpGetFileSize(hFile,0);
cout << "Size: " << filesize << endl;
However this doesn't seem to work as filesize keeps returning a value of -1. I think this is due to the fact that I'm writing to a file (uploading part) and at the same time I'm trying to read it to get the file size. And I think this is not possible (please correct me if I'm wrong).
My main question: is there another way to create a progress bar for ftp uploading? Perhaps counting the bytes before they are uploaded using the function readBytesCount() (not sure if this is possible at all).
You want to:
Call InternetSetStatusCallback to set a function that will be called periodically during the transfer.
Pass a (non-zero) value the last parameter to FtpOpenFile. This will be passed back to your status callback function during the transfer.
Then, during the FTP operation, your callback function will be invoked periodically with information about the progress of the transfer, which it can then display to the user.
I don't believe this will let you show actual bytes as they're being transferred though -- if memory serves it mostly shows the discrete steps in a transfer, like opening the handle, resolving names, sending/receiving cookies, and finally closing the handle.
To deal with the actual bytes being written during the transfer of the file itself, you'd typically read a buffer-full of data from the local file, then write that buffer with InternetWriteFile. With that, you can compute the percentage transferred as the number of bytes written so far divided by the total size of the file (and multiply by 100).
Well, I resolved such a problem by sending the file by chunks and updating the progress in place of sending the whole file by one call to FtpPutFile
I mean:
FtpOpenFile(...)
for( ... )
{
InternetWriteFile(... dwChunkSize...)
UpdateProgressBar(dwChunkSize)
}
InternetCloseHandle(...)
Related
Through the same class object which includes 2 zeromq subscriber and 1 zeromq request socket, I create objects in different threads. I use inproc zeromq sockets and that belong to same ZContext.
Each time I create the object the number of open files (lsof | wc -l) in the server (operating Centos 7) system increases incrementally. After creating the first object the open file # increases by amount of 300 and the second one increases the open file number by 304 and continuously growing.
As my programme can use many of these objects during runtime this can result in too many open files error for zeromq even though I set the limit to 524288 (ulimit -n). As the # of objects getting higher each object consumes the open file limit much more as some of them around 1500.
During runtime my programme crashes with the too many open files error at the times of many objects created and threads doing their work (sending messages to another server or clients) on the objects.
How can I overcome this through?
example code:
void Agent::run(void *ctx) {
zmq::context_t *_context = (zmq::context_t *) ctx;
zmq::socket_t dataSocket(*(_context),ZMQ_SUB);
zmq::socket_t orderRequestSocket(*(_context),ZMQ_REQ);//REQ
std::string bbpFilter = "obprice.1;
std::string bapFilter = "obprice.2"
std::string orderFilter = "order";
dataSocket.connect("inproc://ordertrade_publisher");
dataSocket.connect("inproc://orderbook_prices_pub");
orderRequestSocket.connect("inproc://frontend_oman_agent");
int rc;
try {
zmq::message_t filterMessage;
zmq::message_t orderMessage;
rc = dataSocket.recv(&filterMessage);
dataSocket.recv(&orderMessage);
//CALCULATION AND SEND ORDER
// end:
return;
}
catch(std::exception& e) {
std::cerr<< "Exception:" << e.what() << std::endl;
Order.cancel_order(orderRequestSocket);
return;
}
}
I'm running into this as well. I'm not sure I have a solution, but I see that a context (zmq::context_t) has a maximum number of sockets. See zmq_ctx_set for more detail. This limit defaults to ZMQ_MAX_SOCKETS_DFLT which appears to be 1024.
You might just need to increase the number of sockets your context can have, although I suspect there might be some leaking going on (at least in my case).
UPDATE:
I was able to fix my leak through a combination of socket options:
ZMQ_RCVTIMEO - I was already using this to avoid waiting forever if the other end wasn't there. My system handles this by only making one request on a socket, then closing it.
ZMQ_LINGER - set to 0 so the socket doesn't wait around trying to send the failed message. The default behavior is infinite linger. This is probably the key to your problem
ZMQ_IMMEDIATE - this option restricts the queueing of messages to only completed connections. Without a queue, there's no need for the socket to linger.
I can't say for sure if I need both linger and immediate, but they both seemed appropriate to my use case; they might help yours. With these options set, my number of open files does not grow infinitely.
first of all a little background on my situation:
- Qt/C++ UI desktop application
- embedded device (Stm32l4xx family) +ATWINC1500 wifi module
I'm developing the gui application in order to send commands and files to the emdedded device via sockets.
For simple commands I've done all successfully, but for sending files (text files in GCODE format) I am stuck with some issues.
The embedded device has already a socket management(not written by me, so I have not the possibility to modify the way sockets are managed, coming from third party company), and the reception of that type of files is managed in a way that the API waits for every single line of the file being sent, and then wrotes it into a reserved portion of the flash.
My problem is that when I send file from qt Application(by reading each line and and calling write() on the line, in reality my socket sends an entire chunk of the file, like 50 lines, resulting in my device not managing the file reception.
My sending code is this:
void sendGCODE(const QString fileName)
{
QFile *file = new QFile(fileName,this);
bool result = true;
if (file->open(QIODevice::ReadOnly))
{
while (!file->atEnd())
{
QByteArray bytes(file->readLine());
result = communicationSocket->write(bytes);
communicationSocket->flush();
if(result)
{
console->append("-> GCODE line sent:"+ QString(bytes));
}
else
{
console->append("-> Error sending GCODE line!");
}
}
file->close();
}
}
Have anyone of you guys any hints on what I am doing wrong?
I've already searched and someone suggests on other topic that for this purpose it should be better to use UDP instead of TCP sockets, but unfortunately I cannot touch the embedded-device-side code.
thank you all!
EDIT
After suggestions from comments, I've sniffed tcp packets and the packets are sent correctly(i.e. each packet contains a single line). BUT... at the receiver(device), I understood that there is something regarding memory which is not well managed. an example:
sender sends the line "G1 X470.492 Y599.623 F1000" ; receiver receives correctly the string "G1 X470.492 Y599.623 F1000"
next, if the line length is less than the previous sent, i.e. sending "G1 Z5", the receiver receives: "G1 Z5\n\n.492 Y599.623 F1000", so it is clear that the buffer used to store the data packet is not re-initialized from previous packet content, and the new part overwrites the previous values where the remaining part is from the previous packet
I'm trying to figure out how I could reset that part of memory.
This is all wrong. TCP is not a message-oriented protocol. There is no way to ensure that the TCP packets contain any particular amount of data. The receiver code on the device mustn't expect that either - you perhaps misunderstood the receiver's code, or are otherwise doing something wrong (or the vendor is). What the receiver must do is wait for a packet, add the packet's data to a buffer, then extract and process as many complete lines as it can, then move the remaining data to the beginning of the buffer. And repeat that on every packet.
Thus you're looking for the wrong problem at the wrong place, unless your device never ever had a chance of working. If that device works OK with other software, then your "packetized" TCP assumption doesn't hold any water.
Here's how to proceed:
If the device is commercially available and has been tested to work, then you're looking in the wrong place.
If the device is a new product and still in development, then someone somewhere did something particularly stupid and you either need to fix that stupidity, or have the vendor fix it, or hire a consultant to fix it. But just to be completely clear: that's not how TCP works, and you cannot just accept that "it's how it is".
According to this solution to send out an image through TCP. Since the code is very elegant compared to other ways and both image and file are data, I believe that we can use almost the same code to send out a file.
So if I want to send a file from a client to a server.
On the client side
get file size
send file size
// Above steps will always work, so I will only show code after here
read file content into a buffer
char buf[size];
read(fs,buf,size);
send the buffer
int bytes = 0;
for (uint i = 0;i<size;i+=bytes)
{
if ((bytes = send(sock,buf+i,size-i,0))<0)
{
fprintf(stderr,"Can not send file\n");
close(fd);
return false;
}
fprintf(stderr,"bytes write = %d\n",bytes);
}
And on the server side
recv file size
recv stuff into a buffer with size from step 1
char buf[size];
int bytes=0;
for (uint i = 0;i<size;i+=bytes)
{
if ((bytes = recv(sock,buf+i,size-i,0))<0)
{
fprintf(stderr,"Can not receive file\n");
return false;
}
fprintf(stderr,"bytes read = %d\n",bytes);
}
write buffer to a file
fwrite(buf,sizeof(char),size,fs);
This code will compile and run.
When I send out an cpp binary file(24k) from client to server, since both client and server are on the same machine (OS X), this binary file will be received and can be executed.
But if the server forward the file back to the client, and client forward this file back to the server multiple times, this binary file will be corrupted. But the number of bytes sent and number of bytes received are the same, and the file size is still 24k.
I am wondering what is going wrong here.
Is this an OS bug?
Thanks,
Neither send(), nor recv(), guarantees that the number of bytes requested will actually be sent or received. In that case, the return value will still be positive, but less than the number of bytes that was requested in the system call.
This is extensively documented in the manual page documentation for send() and recv(). Please reread your operating system's documentation for these system call.
It is the application's responsibility to try again, to send or receive the remaining bytes.
This code assumes that the number of bytes that was sent is the number of bytes it requested to be sent. It does appear to handle recv()'s return status properly, but not send()'s. After a fewer number of bytes was sent, this code still assumes that the entire contents were sent or received, and the fwrite() system call will end up writing junk instead of the latter part of the file.
If both client and server are in the same folder, then in this case it is just like copying and pasting a file.
So when client send out a file, it will
open file
get file name/size + send name/size + send data
close file
On the server side,
get file name/size
open the same file again
get file content
close file
So the problem will occur on step 2 by causing a race condition.
I'm writing an SDL application for Linux, that works from the console (no X server). One function I have is a file copy mechanism, that copies specific files from HDD to USB Flash device, and showing progress of this copy in the UI. To do this, I'm using simple while loop and copying file by 8kB chunks to get copy progress. The problem is, that it's slow. I get to copy a 100 MB file in nearly 10 minutes, which is unacceptable.
How can I implement faster file copy? I was thinking about some asynchronous API that would read file from HDD to a buffer and store the data to USB in separate thread, but I don't know if I should implement this myself, because it doesn't look like an easy task. Maybe you know some C++ API/library that can that for me? Or maybe some other, better method?
Don't synchronously update your UI with the copy progress, that will slow things down considerably. You should run the file copy on a separate thread from the main UI thread so that the file copy can proceed as fast as possible without impeding the responsiveness of your application. Then, the UI can update itself at the natural rate (e.g. at the refresh rate of your monitor).
You should also use a larger buffer size than 8 KB. Experiment around, but I think you'll get faster results with larger buffer sizes (e.g. in the 64-128 KB range).
So, it might look something like this:
#define BUFSIZE (64*1024)
volatile off_t progress, max_progress;
void *thread_proc(void *arg)
{
// Error checking omitted for expository purposes
char buffer[BUFSIZE];
int in = open("source_file", O_RDONLY);
int out = open("destination_file", O_WRONLY | O_CREAT | O_TRUNC);
// Get the input file size
struct stat st;
fstat(in, &st);
progress = 0;
max_progress = st.st_size;
ssize_t bytes_read;
while((bytes_read = read(in, buffer, BUFSIZE)) > 0)
{
write(out, buffer, BUFSIZE);
progress += bytes_read;
}
// copy is done, or an error occurred
close(in);
close(out);
return 0;
}
void start_file_copy()
{
pthread_t t;
pthread_create(&t, NULL, &thread_proc, 0);
}
// In your UI thread's repaint handler, use the values of progress and
// max_progress
Note that if you are sending a file to a socket instead of another file, you should instead use the sendfile(2) system call, which copies the file directly in kernel space without round tripping into user space. Of course, if you do that, you can't get any progress information, so that may not always be ideal.
For Windows systems, you should use CopyFileEx, which is both efficient and provides you a progress callback routine.
Let the OS do all the work:
Map the file to memory: mmap, will drastically speed up the reading process.
Save it to a file using msync.
I want to implement an progress bar in my C++ windows application when downloading a file using WinHTTP. Any idea how to do this? It looks as though the WinHttpSetStatusCallback is what I want to use, but I don't see what notification to look for... or how to get the "percent downloaded"...
Help!
Thanks!
Per the docs:
WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE
Data is available to be retrieved with
WinHttpReadData. The
lpvStatusInformation parameter points
to a DWORD that contains the number of
bytes of data available. The
dwStatusInformationLength parameter
itself is 4 (the size of a DWORD).
and
WINHTTP_CALLBACK_STATUS_READ_COMPLETE
Data was successfully read from the
server. The lpvStatusInformation
parameter contains a pointer to the
buffer specified in the call to
WinHttpReadData. The
dwStatusInformationLength parameter
contains the number of bytes read.
There may be other relevant notifications, but these two seem to be the key ones. Getting "percent" is not necessarily trivial because you may not know how much data you're getting (not all downloads have content-length set...); you can get the headers with:
WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE
The response header has been received
and is available with
WinHttpQueryHeaders. The
lpvStatusInformation parameter is
NULL.
and if Content-Length IS available then the percentage can be computed by keeping track of the total number of bytes at each "data available" notification, otherwise your guess is as good as mine;-).