When connecting to a server and sending a specific QUOTE command libcurl(7.29.0) appears to be requesting a directory listing even though I do not want this.
This is the response I get back which indicates after the MDTM command was sent libcurl then sends a LIST command which I do not want. How do I prevent it from sending LIST?
* Request has same path as previous transfer
> MDTM fileName
< 213 20130324223404
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||x|)
* Trying ip...
* Connecting to ip (ip) port x
> LIST
< 150 Here comes the directory listing.
Portion of curl session code (I am reusing a curl handle here that previously changed the remote directory to where fileName was and get a directory listing for other purposes):
headers = NULL;
headers = curl_slist_append(headers, "MDTM fileName");
curl_easy_setopt(curlSession, CURLOPT_QUOTE, headers);
curl_easy_setopt(curlSession, CURLOPT_WRITEHEADER, &fileModResponse);
res = curl_easy_perform(curlSession);
I downloaded the libcurl source and found the answer. Specifying the CURLOPT_NOBODY option prevents it from sending the FTP LIST command.
curl_easy_setopt(curlSession, CURLOPT_NOBODY, 1);
Related
I searched the Internet about the problem with file being uploaded on the FTP, no locking mechanism what results in not knowing if file that FTP client want to get is being uploaded.
Main articles that I found concerned about changing configuration of FTP server or getting files only when some time passed after last modification date:
How to detect that a file is being uploaded over FTP (only guettli suggested using STAT command)
https://community.boomi.com/s/article/ftpfilelockinghowtoavoidpickinguppartialorincompletefilesfromftpsftpsite
I want to use poller mechanism on FTP server that I do not have permission to administrate so I want to know if file which is being uploaded or appended. In FTP there is STAT [file] command that returns the status of the file, which exactly executes LIST [file] to get status code from FTP server. Can I rely on this command? And why using this command was not suggested in any article?
I have a while loop, and inside that loop I send a PUT request into google firebase REST api. It works very well, but if I want to fasten things up (the while loop waits for the curl response every round of the loop which is very slow sometimes, over 200ms), I'm trying to add the CURLOPT_TIMEOUT_MS and set it to a low 1 millisecond.
TLDR;
after adding line
curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, 1L);
My curl does not send anything to the server anymore. Or does the server somehow force the client to receive the returning value from the request?
You tell curl to fail the operation if it isn't completed within 1 millisecond. Not many requests are completed that quickly, especially not if you're using DNS or just use connections over the Internet.
So yes, most transfers will then just return CURLE_OPERATION_TIMEDOUT (28) with no content.
This is a bug of CURL.
If your timeout setting is less than 1s, it will directly return an error.
Solution is:
curl_easy_setopt(conn, CURLOPT_NOSIGNAL, 1);
conn is the pointer of CURL, e.g.:
CURL *conn = NULL;
curl_easy_setopt(conn, CURLOPT_NOSIGNAL, 1);
Is it possible to do? I know about custom request; so I send custom request with text "DELE", and set message ID that I want to delete. As a result, curl_easy_perform hangs until timeout appears. On web forums people write advice to send also "QUIT" command after "DELE"; but how can I send "QUIT" command if libcurl hangs?
libcurl debug output follows:
* Connected to pop-mail.outlook.com (157.55.1.215) port 995 (#2)
* SSL connection using DES-CBC3-SHA
* Server certificate:
* subject: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; CN=*.
hotmail.com
* start date: 2013-04-24 20:35:09 GMT
* expire date: 2016-04-24 20:35:09 GMT
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation
CA - G2
* SSL certificate verify result: unable to get local issuer certificate (
20), continuing anyway.
< +OK DUB0-POP132 POP3 server ready
> CAPA
< -ERR unrecognized command
> USER ************#hotmail.com
< +OK password required
> PASS ******************
< +OK mailbox has 1 messages
> DELE 1
< +OK message deleted
* Operation too slow. Less than 1000 bytes/sec transferred the last 10 seconds
> QUIT
* Operation too slow. Less than 1000 bytes/sec transferred the last 10 seconds
* Closing connection 2
So, the message is removed, but libcurl hangs until speed limit forces it to disconnect, which is bad idea. How to force it to stop after deleting of message and don't wait until timeout comes?
If you look at the libcurl documentation, CURLOPT_CUSTOMREQUEST says:
POP3
When you tell libcurl to use a custom request it will behave like a LIST or RETR command was sent where it expects data to be returned by the server. As such CURLOPT_NOBODY should be used when specifying commands such as DELE and NOOP for example.
That is why libcurl is hanging - it is waiting for more data that the server is not actually sending. So add CURLOPT_NOBODY to stop that waiting.
There's a recently added example code on the libcurl site showing exactly how to do this:
pop3-dele.c
I'm working on a python 2.7 script that must check in a Fedora Commons repository for the existence of some data in 20'000 objects.
Basically this means sending 20'000 HTTP requests to 20'000 different urls on the repository (that runs on a Tomcat server).
I wrote a script that does the job, but I've been warned by the server system administrator that it opens too many network connections, which causes some troubles.
My script uses so far urllib2 to make the HTTP requests.
response = urllib2.urlopen(url)
response_content = response.read()
And actually this code opens one new network connection per request.
I have tried to use other libraries to make the requests, but could not find any way to reuse the same connection for all requests. Both solutions below still open many network connections, even if their number is really lower (actually both solutions seem to open one connection for 100 HTTP requests, which is still around 200 connections in my case).
httplib:
url = "http://localhost:8080/fedora/objects/test:1234?test="
url_infos = urlparse(url)
conn = httplib.HTTPConnection(url_infos.hostname + ":" + str(url_infos.port))
for x in range(0, 20000):
myurl = url + str(x)
conn.request("GET", myurl)
r = conn.getresponse()
response_content = r.read()
print x, "\t", myurl, "\t", r.status
requests:
url = "http://localhost:8080/fedora/objects/test:1234?test="
s = requests.Session()
for x in range(0, 20000):
myurl = url + str(x)
r = s.get(myurl)
response_content = r.content
print x, "\t", myurl, "\t", r.status_code
Even if the number of connections is much better, ideally I'd like to use one or very few connections for all requests. Is that even possible ? Is this number of 100 requests per connection related to the system or to the server ? By the way I also tried to make the requests pointing to an Apache server and the result was the same.
The fact that both solutions shared some code like Lukasa said, and the fact that both results were equivalent whenever querying Apache or Tomcat
made me first think it was related to the Python code. But in fact it was related to the servers configurations.
The trick is that both Apache and Tomcat share a setting which indicates how many HTTP requests can be made within the same TCP connection. And both have a default value of 100.
Tomcat:
maxKeepAliveRequests:
The maximum number of HTTP requests which can be pipelined until the connection is closed by the server.
If not specified, this attribute is set to 100.
See http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Implementation
Apache:
MaxKeepAliveRequests:
The MaxKeepAliveRequests directive limits the number of requests allowed per connection when KeepAlive is on
Default: MaxKeepAliveRequests 100
See http://httpd.apache.org/docs/2.2/en/mod/core.html#maxkeepaliverequests
By modifying these values only a very few connections can be created indeed
I send mail from vmime using the following code:
vmime::string urlString;
urlString="smtp://outgoing.verizon.net";
vmime::utility::url url(urlString);
vmime::ref <vmime::net::transport> tr =
g_session->getTransport(url,vmime::create <interactiveAuthenticator>());
// You can also set some properties (see example7 to know the properties
// available for each service). For example, for SMTP:
tr->setProperty("options.need-authentication", true);
tr->setProperty("auth.username", userName);
tr->setProperty("auth.password", password);
fromString=userName+"#verizon.net";
vmime::mailbox from(fromString);
vmime::mailboxList to;
toString = toUserName+"#verizon.net";
to.appendMailbox(vmime::create <vmime::mailbox>(toString));
std::ostringstream data;
data<<subjectId;
// Connect to server
tr->connect();
// Send the message
vmime::string msgData = data.str();
vmime::utility::inputStreamStringAdapter vis(msgData);
tr->send(from, to, vis, msgData.length());
logMsg(DEBUG,2,"Thread Id: %ld,Sent the data in the transaction",pthread_self());
I see that the sent data is succesful from the log.
But when i connect to the box[to which the mail was sent] and check the inbox, i see 0 mails to that inbox.
There is no excepton or error from vmime.
when i connect to the web version of the mail box.Iam unable to see any transactions is sent box, even for the succesful mails.Can anyone help how i can see the mails sent in the sent box?
Thanks in advance.
use any sniffer to catch SMTP traffic from your host. if everything looks good (i.e. SMTP session correct), then it's not a problem of your host, but remote MTA (make sure your email wasn't detected as SPAM & etc)