ALL,
I tried to search the suggested threads here but to no availability.
I need to send the file to the web server using libCURL POST request.
I am creating the file I'm sending so I know it is exist.
I also have a WireShark installed here so I know what is sent over.
During the debug session I can see that the file has been read and the buffer it is read to is sending out. However, in the WireShark it is all displayed as FF FF FF FF sequence.
And in the end the operation fail with "Error writing the body" error message returning to CURL.
Any idea what might be the problem?
Here is what I tried so far:
1.
result = curl_formadd( &first, &last, CURLFORM_COPYNAME, "MyFile",
CURLFORM_FILE, (const char *) fileName.c_str(),
CURLFORM_FILENAME, (const char *) fileName.c_str(),
CURLFORM_CONTENTSLENGTH, (long) file.Length(),
CURLFORM_CONTENTTYPE, "image/bitmap", CURLFORM_END );
2.
result = curl_formadd( &first, &last, CURLFORM_COPYNAME, "MyFile",
CURLFORM_FILENAME, (const char *) fileName.c_str(),
CURLFORM_STREAM, &file, CURLFORM_CONTENTSLENGTH,
(long) file.Length(), CURLFORM_CONTENTTYPE,
"image/bitmap", CURLFORM_END );
Then in both cases:
curl_easy_setopt( handle, CURLOPT_HTTPPOST, first );
curl_easy_setopt( handle, CURLOPT_POSTFIELDSIZE, file.Length() );
The header in both cases are formed correctly, but the data itself is not sent.
Does anybody have an idea? I feel like I'm missing something very simple...
ALL,
Turns out that the service I tried to connect to was down.
Thank you for listening.
Related
To provide a little background. I am not very experienced with Ethernet communications so I apologize in advance for that. I'm working on a project where I need to figure out how to upload a binary file.
I'm trying to upload a large binary file (~34MB) to an embedded device. I have a python code snippet that works but I'm trying to implement the same capability in a different application using C++. Using WireShark, I've captured the header from the python program that works as well as the header that I end up with in my C++ code which doesn't work as needed.
Success Uploading with Python
Here is the python code that works:
response = session.post('http://10.42.42.1:81/__FileUpload',
files={"upfile": open(filename, 'rb')},
stream=False)
Here is the header information extracted from the message which initiates the successful file upload:
POST /__FileUpload HTTP/1.1
Host: 10.42.42.1:81
Content-Length: 34112690
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.7.0 CPython/2.7.13 Windows/10
Connection: keep-alive
Content-Type: multipart/form-data; boundary=6a659e345a35419e99b66546c1bd9d4e
--6a659e345a35419e99b66546c1bd9d4e
Content-Disposition: form-data; name="upfile"; filename="TestFile.bin"
No Success with C++ Code
Here is the essence of the code that I'm using to upload the file in C++:
curl_mime *multipart;
curl_mimepart *part;
// Specify the target URL
std::string str(comms.BaseURL() + kFileUploadEndpoint);
curl_easy_setopt(pCurl, CURLOPT_URL, str.c_str());
multipart = curl_mime_init(pCurl);
part = curl_mime_addpart(multipart);
curl_mime_name(part, "upfile");
curl_mime_data(part, ("filename=\"" + FileName + "\"").c_str(), CURL_ZERO_TERMINATED);
part = curl_mime_addpart(multipart);
curl_mime_data_cb(part, fileSize, ReadCallback, SeekCallback, NULL, pFile);
curl_easy_setopt(pCurl, CURLOPT_MIMEPOST, multipart);
curl_easy_setopt(pCurl, CURLOPT_TIMEOUT, 90L);
res = curl_easy_perform(pCurl);
...
Here is the header information from running the C++ code:
--------------------------4977715f070a13da
Content-Disposition: form-data; name="upfile"
filename="TestFile.bin"
--------------------------4977715f070a13da
Content-Disposition: form-data
I realized that the header above does not contain the URL endpoint and such and noticed that apparently the message was split into two pieces. Here is the header content from the message sent before the message above.
POST /__FileUpload HTTP/1.1
Host: 10.42.42.1:81
Accept: */*
Content-Length: 31546130
Content-Type: multipart/form-data; boundary=------------------------4977715f070a13da
Expect: 100-continue
I can check the status from the embedded device during the upload and the one thing that I notice in particular is that when the upload is successful with python then the embedded device reports the filename being uploaded in the status content reply. However, when I run the C++ code the filename is blank when I check the status. Therefore the embedded device is obviously not able to extract the filename from the C++ message.
What the Embedded Device is Looking For
While I don't have access to the source code of the device I did get the following information from someone who does. He indicated that this is what the embedded device is looking for. It didn't help me to figure out how to get things working but it might help someone else more knowledgeable in this area.
<FORM METHOD=POST name="install" enctype="multipart/form-data" target="HiddenFrame" action="/__FileUpload" onsubmit="InstallAction(); return true;">
File to upload: <INPUT TYPE=FILE NAME="upfile" size=50><p>
<INPUT TYPE=SUBMIT VALUE="Submit" >
</FORM>
I would prefer using the libcurl 'curl_mime_...' methods to setup the file upload only because that approach is recommended over using the older HTTP post methods. However, I'm perfectly okay with using the older HTTP post methods if that is easier to do. I just want to get it working.
Thanks in advance for your time.
I ended up finding the solution.
It is posted here for anyone who could use it:
curl_mime *multipart;
curl_mimepart *part;
multipart = curl_mime_init(pCurl);
part = curl_mime_addpart(multipart);
curl_mime_name(part, "upfile");
curl_mime_filename(part, FileName.c_str());
curl_mime_data_cb(part, lSize, ReadCallback, SeekCallback, NULL, pFile);
part = curl_mime_addpart(multipart);
curl_easy_setopt(pCurl, CURLOPT_MIMEPOST, multipart);
// Now send the message
res = curl_easy_perform(pCurl);
// Free the post data
curl_mime_free(multipart);
...
curl_mime_data(part, ("filename=\"" + FileName + "\"").c_str(), CURL_ZERO_TERMINATED);
I presume this wants to set the mime part's file name, and then you should rather use curl_mime_filename, because the file name is not data. Each part has a name and data, but also meta-data such as file name. Setting the file name only of course then requires that you set the data separately.
If you rather want to set the data as well as the file name, then instead do it with curl_mime_filedata.
Also, take a look at the official libcurl example postit2.c.
What i want to achieve is to be able to upload a file of any size and also send post data in the same request. The server is guaranteed to be able to handle chunked encoding and also support unlimited size of size.
To achieve that I used CURLOPT_HTTPOST and set the header Transfer-Encoding: chunked, The problem is that I'm am not sure if libcurl is actually using chunked encoding at all. In my attempts to try and figure out I set CURLOPT_VERBOSE to 1 and implemented a custom debug callback to monitor the http headers. From curl I can only see 1 HTTP Request and only 1 HTTP Response (code 200). I didn't see any http code 201 responses.
My question is how do I enable chunk encoding for file uploads in multipart POST requests? If what I'm doing is correct why does libcurl not show me all the requests, how do I make sure it works like it is required?
Bellow is a short example of what I am doing in code.
//Assume `curl` has been initialised
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME,
"foo",
CURLFORM_COPYCONTENTS,
"bar",
CURLFORM_END);
curl_formadd(&formpost,
&lastptr,
CURLFORM_COPYNAME,
"file",
CURLFORM_FILE,
"filepath",
CURLFORM_FILENAME,
"filename",
CURLFORM_END);
headerlist = curl_slist_append(headerlist, "Transfer-Encoding: chunked");
curl_easy_setopt(curl, CURLOPT_URL, "https://foo/uploadfile");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write_to_string);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, curl_progress_cb);
curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &progress_id);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
CURLcode res = curl_easy_perform(curl);
Thanks in advance
First of, I'm very new to HTTP commands and the libcurl library, so there is a good chance that I'm not understanding something fundamental. That said, I'm attempting to replicate an HTTP POST command sent to a device via an internal server on a windows based MFC app. Essentially I'm sending a small bitmap image along with a command. I captured the command using Fiddler, and it looks something like:
POST /Service/MyCommand HTTP/1.1
Authorization: MyAuth
Content-Type: image/bmp
User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.7.0_51
Host: MyHost:MyPort
Accept: MyAccept
Connection: MyConnection
Content-Length: 15606
/* BMP Data */
I'm having two problems replicating this (using libcurl). First, my post command of 'Service/MyCommand' appears at the very end of the header, rather than after the 'POST /'. I have attempted to move it around, but it will cease to appear on my WireShark filter window. Second, when I attempt to set my content length to 15606, as with the original, the protocol on WireShark switches from "POST" to "TCP". I've attached the code below.
int CHttpPost::fnSendContent()
{
using namespace std;
int Error = 0;
CString str;
CURL* curl;
CURLcode res;
struct curl_httppost *formpost=NULL;
struct curl_httppost *lastptr=NULL;
struct curl_slist *headerlist=NULL;
static const char buf[] = "a"; // not sure what to do with this
curl_global_init(CURL_GLOBAL_ALL);
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "importfile", CURLFORM_FILE, "MyImage.bmp", CURLFORM_CONTENTTYPE, "image/bmp", CURLFORM_END);
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "action", CURLFORM_COPYCONTENTS, "upload", CURLFORM_END);
curl = curl_easy_init();
headerlist = curl_slist_append(headerlist, buf);
headerlist = curl_slist_append(headerlist, "Authorization: MyAuth");
headerlist = curl_slist_append(headerlist, "Content-Type: image/bmp");
headerlist = curl_slist_append(headerlist, "User-Agent: Mozilla/4.0 (Windows 7 6.1) Java/1.7.0_51");
headerlist = curl_slist_append(headerlist, "Accept: MyAccept");
headerlist = curl_slist_append(headerlist, "Connection: MyConnection");
headerlist = curl_slist_append(headerlist, "Content-Length: 15606");
//Set URL to recevie POST
curl_easy_setopt(curl, CURLOPT_VERBOSE, true);
curl_easy_setopt(curl,CURLOPT_POST, true);
curl_easy_setopt(curl, CURLOPT_HEADER, true);
curl_easy_setopt(curl, CURLOPT_URL, "MyHost:MyPort");
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "Service/MyCommand");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_global_cleanup();
return Error;
}
Any other suggestions or corrections you have are much appreciated as well.
EDIT: I was being an idiot by placing the "Service/MyCommand" in the postfield rather than the URL. I apparently misunderstood one of the tutorials. My content-length question still stands however.
You can do this much simpler. You, just use CURLOPT_POSTFIELDS to point to the BMP data, and make CURLOPT_POSTFIELDSIZE_LARGE be the full size of that data (15606 I would guess in this particular case).
You do not want curl_formadd() and the formpost stuff you use. That's for the CURLOPT_HTTPPOST option, which is for multipart formposts but you don't want that.
char *bmp_array = [your data];
long bmp_size = 15606;
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, bmp_array);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, bmp_size);
I am writing a IRC client in C++ (with the help of the SFML library), but it is behaving strangely. I send the NICK and USER commands and I can connect to the server, but the JOIN command has many strange thing happening that I have to write "Random code that magically works" to solve. I am pretty sure that the commands adhere to the IRC RFC as well.
I know that the sockets are sending what they are supposed to send and I have verified it with Wireshark, so what I post here is what the message of the packet is. In the following examples the socket is already connected to the IRC server (which in this case is irc.freenode.net)
This works:
char mess[] ="NICK lmno \n\rUSER lmno 0 * :lmno\n\rJOIN #mytest\n\r";
Socket.Send(mess, sizeof(mess));
This does not:
char msg[] = "NICK lmno \r\nUSER lmno 0 * :lmno \r\n";
char msga[] = "JOIN #mytest \r\n";
Socket.Send(msg, sizeof(msg));
Socket.Send(msga, sizeof(msga));
But curiously this works:
char msg[] = "NICK lmno \r\nUSER lmno 0 * :lmno \r\n";
char msga[] = "JOIN #mytest \r\n";
Socket.Send(msg, sizeof(msg));
Socket.Send(msga, sizeof(msga));
Socket.Send(msga, sizeof(msga));
I did do some research on this topic and no one seems to have the same problem. Stranger is that when I tried this in telnet, I only have to send JOIN once.
Is anyone able to give me some advice?
Thanks,
SFI
It might have to do with the terminating '\0' character at the end of a c-string. Try
Socket.Send(msg, sizeof(msg) - 1);
Socket.Send(msga, sizeof(msga) - 1);
And exact the media type then save the captured image with correct extension?
The browser usually sends a http GET request to the server (that holds the video stream) when it wants to play the video and that is what starts the file transfer, but this handshake may change according to the server you are negotiating with.
Below is a short code snippet that shows how to setup curl and download a file when you have the complete url of the file:
#include <curl/curl.h>
#include <stdio.h>
void get_page(const char* url, const char* file_name)
{
CURL* easyhandle = curl_easy_init();
curl_easy_setopt( easyhandle, CURLOPT_URL, url ) ;
FILE* file = fopen( file_name, "w");
curl_easy_setopt( easyhandle, CURLOPT_WRITEDATA, file) ;
curl_easy_perform( easyhandle );
curl_easy_cleanup( easyhandle );
}
int main()
{
get_page( "http://blog.stackoverflow.com/wp-content/themes/zimpleza/style.css", "style.css" ) ;
return 0;
}
Websites like youtube don't give the URL of the video so easily, and may even redirect you to another html page that you could parse to find the magic information needed to assembly the full URL of the video. I wrote a small bash script a long time ago to automate the process for finding a youtube's video URL and downloading the video. I know it doesn't work anymore so I'll paste it for education purposes only:
if [ -z "${1}" ]
then
echo 'Error !!! Missing URL or video_id !!!'
exit 1
fi
URL="http://www.youtube.com"
# Retrieve video_id from url passed by the user
VAR_VIDEO_ID="${1/*=}"
# Retrive t variable located in var swfHTML (javascript)
VAR_T=$(wget -qO - $URL/watch?v=$VAR_VIDEO_ID 2>&1 | perl -e 'undef $/; <STDIN> =~ m/&t=([^&]*)&/g; print "$1\n"';)
# Assemble magical string
FLV_URL="$URL/get_video?video_id="$VAR_VIDEO_ID"&t="$VAR_T"=&eurl=&el=detailpage&ps=default&gl=US&hl=en"
# Download flv from Youtube.com. Add 2>&1 before wget cmd to suppress logs
WGET_OUTPUT=$(wget $FLV_URL -O $VAR_VIDEO_ID.flv)
# Making sure the download went okay
if [ $? -ne 0 ]
then
# wget failed
echo -e 1>&2 $0: "!!! ERROR: wget failed !!!\n"
rm $VAR_VIDEO_ID.flv
exit 1
fi
And to answer your 2nd question, I believe that to identify the file/media type you will have to download the first bytes of the video stream since it contains the file header, and then check it for known file signatures. For instance, the first bytes of a FLV file should be:
46 4C 56 01
EDIT:
Downloading a video stream is not so different as one may believe. You will need to tell curl that you have your own method to save the stream data, and this can be specified by:
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, cb_writer);
curl_easy_setopt(curl_hndl, CURLOPT_WRITEDATA, url_data);
Where *cb_writter* is your callback that will be called by curl when new data arrives. Check the documentation on Callback Options for more info about these functions.
If you need a full example you can check this thread.
One more thing, if you are working with M-JPEG streams you should take a look at the cambozola implementation.