How to fix issue with post requests from arduino, only being received 2-3 times before failing - c++

EDIT5:
I eventually fixed this issue by more or less throwing away half my code. Rather than sending data to a ruby server using HTTP, I'm now using MQTT to a broker to a NodeJS server. The NodeJS part here isn't important but to anyone else having this issue I STRONGLY recommend sending all IoT data using MQTT, and that's what solved my issue.
I'm currently trying to send data collected from sensors on an Arduino WiFi rev2, to my rails server hosted on Heroku. I do this by sending my data in a JSON format. My problem is that while my methods seem to work initially, with the first few POST requests being received and processed fine, after 2-3 requests the arduino hangs, and I receive status code: -2. I'm using the ArduinoHttpClient library.
I've tried using a local server, which has the same problem, aswell as sending the POST request via both curl and postman. Both curl and postman seem to work as expected, so I imagine the issue is with the arduino code although I can't be sure.
client.beginRequest();
client.post("/input");
client.sendHeader("Content-Type", "application/json");
client.sendHeader("Content-Length", postData.length());
client.beginBody();
client.println(postData);
client.endRequest();
LED(0,128,0);
Serial.println("Gone");
int statusCode = client.responseStatusCode();
String response = client.responseBody();
Serial.print("Status code: ");
Serial.println(statusCode);
Serial.print("Response: ");
Serial.println(response);
When this code fails, the arduino will hang for about 20-40 seconds and I will receive 'status code -3' from the Serial. However I have also received status code -2 and -4 in the past.
When it does succeed I receive the following: "Status code: 204" which is what I would expect.
EDIT:
I've since tried posting to requestcatcher.com, and the problem persisted. I'm therefore fairly confident this is an arduino problem, I also received the following output:
POST /input HTTP/1.1
Host: arduino.requestcatcher.com
Connection: close
Connection: close
Content-Length: 88
Content-Type: application/json
User-Agent: Arduino/2.2.0
{"inputs":[{"input_id":"1","value":1.778002}{"input_id":"2","value":18.037}],"id":"13"}
EDIT 2:
I accidentally discovered that the POST requests go through fine if the "Content-Length:" Header is omitted. Obviously no JSON actually gets sent so this does not fix my issue but it is likely that this header or the JSON itself is the issue.
EDIT 3:
Regardless of server I receive either status code -4, or -3, even on request catcher.
EDIT 4:
After various adjustments, code now looks as below. This seems to have helped a little and it fails less often but still does fail. I'm beginning to wonder if this is a problem with ArduinoHttpClient.
String postData = "";
serializeJson(doc, postData);
serializeJson(doc, Serial);
Serial.println(postData)
client.post("/input", "application/json", postData.c_str());
LED(0,128,0);
Serial.println("Gone");
int statusCode = client.responseStatusCode();
Serial.print("Status code: ");
Serial.println(statusCode);
client.stop();
doc.clear();
lastCycle = millis();

Try replacing
client.beginRequest();
client.post("/input");
client.sendHeader("Content-Type", "application/json");
client.sendHeader("Content-Length", postData.length());
client.beginBody();
client.println(postData);
client.endRequest();
with just
String contentType = "application/json";
client.post("/input", contentType, postData);
or
client.post("/input", "application/json", postData.c_str());
You don't need to explicitly specify the request headers - or call beginRequest(), etc. - when using the post() method(s) in that library.

Related

Lag when sending an http request using POCO C++ libraries

I have written a program making an http POST request to google-analytics using the following code:
Poco::Net::HTTPClientSession session("www.google-analytics.com");
Poco::Net::HTTPRequest req(Poco::Net::HTTPRequest::HTTP_POST, "/collect",Poco::Net::HTTPMessage::HTTP_1_1);
Poco::URI uri("");
uri.addQueryParameter("v", "1");
////// more parameters of the uri //////
req.setContentLength(uri.getRawQuery().length());
session.sendRequest(req) << uri.getRawQuery();
A user of the program has reported a 20 seconds freeze of the program every time a request is made. I have not been able to replicate the problem on any other computer, nor understand where it could come from.
He uses an ethernet connection to a modem TP-Link 300Mbps Wireless N Gigabit ADSL2+Modem Router TD-W8970. When the modem is turned off, the lag does not occur (my code throws an exception when sendRequest fails).
Any help/suggestion on the problem would be much appreciated!
Thanks.

FTP client request failure

I'm trying to develop an FTP protocol in c++. It takes the type of the request from the client as it is (get or put) however, the server always receives a put request in both cases and does none of the functionality.
Here the code that compare the type of the request:
if(strcmp(argv[3],"get")==0)
smsg.type=REQ_GET; //REQ_TIME;
else if (strcmp(argv[3],"put")==0)
{
smsg.type=REQ_PUT; //REQ_SIZE;
}
else err_sys("Wrong request type\n");
and this is the part that execute when a get request is sent to the server:
if(smsg.type=REQ_GET)
{
cout<<"Iam inside get"<<endl;
cout<<smsg.type<<endl;
//send out GET message
memcpy(smsg.buffer,&req,sizeof(req)); //copy the request to the msg's buffer
smsg.length=sizeof(req);
fprintf(stdout,"Send a GET request to %s\n",argv[1]);
if (msg_send(sock,&smsg) != sizeof(req))
err_sys("Sending req packet error.,exit");
//receive the response
if(msg_recv(sock,&rmsg)!=rmsg.length)
err_sys("recv response error,exit");
//cast it to the response structure
respp=(Resp *)rmsg.buffer;
printf("Response:%s\n\n\n",respp->response);}
when displaying "respp->response" it doesn't return anything, and always the size of the file is zero.
Does anyone have an idea about how to solve this problem?
Any help will be highly appreciated.
- if(smsg.type=REQ_GET)
+ if(smsg.type==REQ_GET)
next time try to use -Wall when compiling, it might catch this kind of error

Sending HTML tag to browser via socket connection with C++ Socket API

I am trying to make a simple http server with C++. I've followed the beej's guide of network programming in C++.
When I ran the server in some port (8080, 2127, etc.) it successfully send response to browser (Firefox) when it accessed via address bar with: localhost:PORT_NUMBER except in port 80.
This is the code i wrote:
printf("Server: Got connection from %s\n", this->client_ip);
if(!fork()) // This is the child process, fork() -> Copy and run process
{
close(this->server_socket); // Child doesn't need listener socket
// Try to send message to client
char message[] = "\r\nHTTP/1.1 \r\nContent-Type: text/html; charset=ISO-8859-4 \r\n<h1>Hello, client! Welcome to the Virtual Machine Web..</h1>";
int length = strlen(message); // Plus 1 for null terminator
int send_res = send(this->connection, message, length, 0); // Flag = 0
if(send_res == -1)
{
perror("send");
}
close(this->connection);
exit(0);
}
close(this->connection); // Parent doesn't need this;
The problem is, even I have added the header on very early of the response string, why does the browser not showing the HTML properly instead shows only plain text? It shows something like this:
Content-Type: text/html; charset=ISO-8859-4
<h1>Hello, client! Welcome to the Virtual Machine Web..</h1>
Not a big "Hello, client!.." string like a normally h1 tagged string. What is the problem? Am I missing something in the header?
Another question is, why is the server won't running in port 80? The error log in server says:
server: bind: Permission denied
server: bind: Permission denied
Server failed to bind
libc++abi.dylib: terminate called throwing an exception
Please help. Thank you. Edit: I'dont have any process on Port 80.
You need to terminate the HTTP response header with \r\n\r\n, rather than just \r\n. It should also start with something more like HTTP/1.1 200 OK\r\n, without the leading \r\n.
For your port problem, if you have nothing else running on the port in question, you may find that the socket created by the last run of your program is still sticking around. To work around this, you can use setsockopt to set the SO_REUSEADDR flag on the socket. (This is not recommended for general use, I believe because you may receive data not intended for your program, but for development it's extremely handy.)
Your request starts with \r\n when it shouldn't also it did not specify a status code and you need a blank line after all the headers.
char message[] = "HTTP/1.1 200 Okay\r\nContent-Type: text/html; charset=ISO-8859-4 \r\n\r\n<h1>Hello, client! Welcome to the Virtual Machine Web..</h1>";
As for your port 80 issue, some other application maybe bound to it.
you need to add "Content-length: ", and the length is your HTML code, just like this:
char msg[] = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-length: 20\r\n\r\n<h1>Hello World</h1>";

Broken HTML - browsers don't downloads whole HTTP response from my webserver, CURL does

Symptom
I think, I messed up something, because both Mozilla Firefox and Google Chrome produce the same error: they don't receive the whole response the webserver sends them. CURL never misses, the last line of the quick-scrolling response is always "</html>".
Reason
The reason is, that I send response in more part:
sendHeaders(); // is calls sendResponse with a fix header
sendResponse(html_opening_part);
for ( ...scan some data... ) {
sendResponse(the_data);
} // for
sendResponse(html_closing_part)
The browsers stop receiving data between sendResponse() calls. Also, the webserver does not close() the socket, just at the end.
(Why I'm doing this way: the program I write is designed for non-linux system, it will run on an embedded computer. It has not too much memory, which is mostly occupied by lwIP stack. So, avoid collecting the - relativelly - huge webpage, I send it in parts. Browsers like it, no broken HTML occurred as under Linux.)
Environment
The platform is GNU/Linux (Ubuntu 32-bit with 3.0 kernel). My small webserver sends the stuff back to the client standard way:
int sendResponse(char* data,int length) {
int x = send(fd,data,length,MSG_NOSIGNAL);
if (x == -1) {
perror("this message never printed, so there's no error \n");
if (errno == EPIPE) return 0;
if (errno == ECONNRESET) return 0;
... panic() ... (never happened) ...
} // if send()
} // sendResponse()
And here's the fixed header I am using:
sendResponse(
"HTTP/1.0 200 OK\n"
"Server: MyTinyWebServer\n"
"Content-Type: text/html; charset=UTF-8\n"
"Cache-Control: no-store, no-cache\n"
"Pragma: no-cache\n"
"Connection: close\n"
"\n"
);
Question
Is this normal? Do I have to send the whole response with a single send()? (Which I'm working on now, until a quick solution arrives.)
If you read RFC 2616, you'll see that you should be using CR+LF for the ends of lines.
Aside from that, open the browser developer tools to see the exact requests they are making. Use a tool like Netcat to duplicate the requests, then eliminate each header in turn until it starts working.
Gotcha!
As #Jim adviced, I've tried sending same headers with CURL, as Mozilla does: fail, broken pipe, etc. I've deleted half of headers: okay. I've added back one by one: fail. Deleted another half of headers: okay... So, there is error, only if header is too long. Bingo.
As I've said, there're very small amount of memory in the embedded device. So, I don't read the whole request header, only 256 bytes of them. I need only the GET params and "Host" header (even I don't need it really, just to perform redirects with the same "Host" instead of IP address).
So, if I don't recv() the whole request header, I can not send() back the whole response.
Thanks for your advices, dudes!

HTTP GET request problem

I am writing a simple downloader. I am trying to download jpg picture.
void accel::download(int threads){
char msg[] = "HEAD /logos/2011/cezanne11-hp.jpg HTTP/1.0\r\nConnection: close\r\n\r\n";
int back = send(socketC, (const char *)&msg, strlen(msg), 0);
char *buff = new char[500];
back = recv(socketC, buff, 500, 0);
cout << buff;
char *buff2 = new char[700];
char msg2[] = "GET /logos/2011/cezanne11-hp.jpg HTTP/1.0\r\nRange: bytes=0-400\r\nConnection: close\r\n\r\n";
back = send(socketC, (const char *)&msg2, strlen(msg2), 0);
back = recv(socketC, buff2, 700, 0);
cout << back;
}
TCP connection is already initialized and the first part of my code is working. It succesfuly sends HEAD message and receaves response. But when it tries to download the picture, the recv gets 0. What might be wrong?
Btw this is school project so I am not allowed to use some fancy libraries to perform this operation. This is full pictures address - http://www.google.com/logos/2011/cezanne11-hp.jpg
You don't recieve anything because you told the server that you didn't want to make a second request when you specified
Connection: close
In your HEAD request.
This tells the server that you're only going to make ONE request and not to bother waiting for a second.
Try changing your first request to a persistant 'keep-alive' connection.
"HEAD /logos/2011/cezanne11-hp.jpg HTTP/1.0\r\nConnection: keep-alive\r\n\r\n";
NOTE: If you don't want to server to go away you might want to change your second request to keep-alive too.
Generally speaking, HTTP closes the socket. (HTTP 1.1 has persistent (keep-alive) connections, though you seem to have asked the server to close the connection on you in your first command.)
So make sure that your socket is still open after your first receive; I'm willing to bet that it isn't.
Have you verified that the image is actually being sent to your client? Maybe you're not getting any response from the server.
Try using wireshark to inspect the actual network activity. That will let you see exactly what's being sent and received. It's possible you're not getting anything back from the server, or that there's an issue with your request that you might be able to spot in the actual network traffic.
After you follow through with what chrisaycock says you may want to add Host: to your requests. A lot of shared hosting around and IP only access is likely to start failing.