I've been trying to download some file through a web service and its perfectly downloaded, but now I'm trying to add a feature if the file wasn't downloaded completely, the app should download only the remaining byte array and append to the existing one, using
connection = (HttpConnection) cf.getConnection(_url).getConnection();
int alreadyDownloaded = 0;
if(connection.getResponseCode() == HttpConnection.HTTP_OK) {
inputStream = new DataInputStream(connection.openInputStream());
final int len = (int) connection.getLength();
if (len > 0) {
String filename = _url.substring(_url.lastIndexOf('/') + 1);
FileConnection outputFile = (FileConnection) Connector.open(path + filename, Connector.READ_WRITE, true);
if (!outputFile.exists()) {
outputFile.create();
} else {
alreadyDownloaded = (int) outputFile.fileSize();
connection.setRequestProperty("Range", "bytes=" + alreadyDownloaded + "-");
}
And at this line
connection.setRequestProperty("Range", "bytes=" + alreadyDownloaded + "-");
I get an exception that says
Stream not in setup state
How can I get rid of this error?
The problem is that you are calling this line
connection.setRequestProperty("Range", "bytes=" + alreadyDownloaded + "-");
after you have already opened the connection, and sent the request parameters. So, it's too late to now change the Range property.
From BlackBerry API docs:
After an output stream has been opened by the openOutputStream or
openDataOutputStream methods, attempts to change the request
parameters via setRequestMethod or the setRequestProperty are ignored.
Once the request parameters have been sent, these methods will throw
an IOException.
If you look further down in that document, they show an example, which explains a little more:
// Getting the response code will open the connection,
// send the request, and read the HTTP response headers.
// The headers are stored until requested.
rc = c.getResponseCode();
So, you just need to call setRequestProperty() before this line:
if(connection.getResponseCode() == HttpConnection.HTTP_OK) {
Related
I want to get the size of an http:/.../file before I download it. The file can be a webpage, image, or a media file. Can this be done with HTTP headers? How do I download just the file HTTP header?
Yes, assuming the HTTP server you're talking to supports/allows this:
public long GetFileSize(string url)
{
long result = -1;
System.Net.WebRequest req = System.Net.WebRequest.Create(url);
req.Method = "HEAD";
using (System.Net.WebResponse resp = req.GetResponse())
{
if (long.TryParse(resp.Headers.Get("Content-Length"), out long ContentLength))
{
result = ContentLength;
}
}
return result;
}
If using the HEAD method is not allowed, or the Content-Length header is not present in the server reply, the only way to determine the size of the content on the server is to download it. Since this is not particularly reliable, most servers will include this information.
Can this be done with HTTP headers?
Yes, this is the way to go. If the information is provided, it's in the header as the Content-Length. Note, however, that this is not necessarily the case.
Downloading only the header can be done using a HEAD request instead of GET. Maybe the following code helps:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://example.com/");
req.Method = "HEAD";
long len;
using(HttpWebResponse resp = (HttpWebResponse)(req.GetResponse()))
{
len = resp.ContentLength;
}
Notice the property for the content length on the HttpWebResponse object – no need to parse the Content-Length header manually.
Note that not every server accepts HTTP HEAD requests. One alternative approach to get the file size is to make an HTTP GET call to the server requesting only a portion of the file to keep the response small and retrieve the file size from the metadata that is returned as part of the response content header.
The standard System.Net.Http.HttpClient can be used to accomplish this. The partial content is requested by setting a byte range on the request message header as:
request.Headers.Range = new RangeHeaderValue(startByte, endByte)
The server responds with a message containing the requested range as well as the entire file size. This information is returned in the response content header (response.Content.Header) with the key "Content-Range".
Here's an example of the content range in the response message content header:
{
"Key": "Content-Range",
"Value": [
"bytes 0-15/2328372"
]
}
In this example the header value implies the response contains bytes 0 to 15 (i.e., 16 bytes total) and the file is 2,328,372 bytes in its entirety.
Here's a sample implementation of this method:
public static class HttpClientExtensions
{
public static async Task<long> GetContentSizeAsync(this System.Net.Http.HttpClient client, string url)
{
using (var request = new System.Net.Http.HttpRequestMessage(System.Net.Http.HttpMethod.Get, url))
{
// In order to keep the response as small as possible, set the requested byte range to [0,0] (i.e., only the first byte)
request.Headers.Range = new System.Net.Http.Headers.RangeHeaderValue(from: 0, to: 0);
using (var response = await client.SendAsync(request))
{
response.EnsureSuccessStatusCode();
if (response.StatusCode != System.Net.HttpStatusCode.PartialContent)
throw new System.Net.WebException($"expected partial content response ({System.Net.HttpStatusCode.PartialContent}), instead received: {response.StatusCode}");
var contentRange = response.Content.Headers.GetValues(#"Content-Range").Single();
var lengthString = System.Text.RegularExpressions.Regex.Match(contentRange, #"(?<=^bytes\s[0-9]+\-[0-9]+/)[0-9]+$").Value;
return long.Parse(lengthString);
}
}
}
}
WebClient webClient = new WebClient();
webClient.OpenRead("http://stackoverflow.com/robots.txt");
long totalSizeBytes= Convert.ToInt64(webClient.ResponseHeaders["Content-Length"]);
Console.WriteLine((totalSizeBytes));
HttpClient client = new HttpClient(
new HttpClientHandler() {
Proxy = null, UseProxy = false
} // removes the delay getting a response from the server, if you not use Proxy
);
public async Task<long?> GetContentSizeAsync(string url) {
using (HttpResponseMessage responce = await client.GetAsync(url))
return responce.Content.Headers.ContentLength;
}
I'm trying to send invoices to amazon mws through _UPLOAD_VAT_INVOICE_ following the java example in this guide:
Link
pdf file is a simple invoice of 85 kb
The error is status code 414 that is "Uri too long"
Debugging original amazon class MarketplaceWebServiceClient I see this:
if( request instanceof SubmitFeedRequest ) {
// For SubmitFeed, HTTP body is reserved for the Feed Content and the function parameters
// are contained within the HTTP header
SubmitFeedRequest sfr = (SubmitFeedRequest)request;
method = new HttpPost( config.getServiceURL() + "?" + getSubmitFeedUrlParameters( parameters ) );
getSubmitFeedUrlParameters method takes every parameter and add it to querystring. One of these parameters is contentMD5 from:
String contentMD5 = Base64.encodeBase64String(pdfDocument);
So there is a very large string representing pdf file passed as parameter. This causes error 414
But that class is the original one taken from MaWSJavaClientLibrary-1.1.jar
Can anybody help me please?
Thanks
For the last 2 days I was working on the same problem,
I changed like this and it works now
InputStream contentStream = new ByteArrayInputStream(pdfDocument);
String contentMD5 =computeContentMD5Header(new ByteArrayInputStream(pdfDocument));
public static String computeContentMD5Header(InputStream inputStream) {
// Consume the stream to compute the MD5 as a side effect.
DigestInputStream s;
try {
s = new DigestInputStream(inputStream,
MessageDigest.getInstance("MD5"));
// drain the buffer, as the digest is computed as a side-effect
byte[] buffer = new byte[8192];
while (s.read(buffer) > 0)
;
return new String(
org.apache.commons.codec.binary.Base64.encodeBase64(s
.getMessageDigest().digest()), "UTF-8");
} catch (NoSuchAlgorithmException e) {
throw new RuntimeException(e);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
In regard to Indy 10 of IdHTTP, many things have been running perfectly, but there are a few things that don't work so well here. That is why, once again, I need your help.
Download button has been running perfectly. I'm using the following code :
void __fastcall TForm1::DownloadClick(TObject *Sender)
{
MyFile = SaveDialog->FileName;
TFileStream* Fist = new TFileStream(MyFile, fmCreate | fmShareDenyNone);
Download->Enabled = false;
Urlz = Edit1->Text;
Url->Caption = Urlz;
try
{
IdHTTP->Get(Edit1->Text, Fist);
IdHTTP->Connected();
IdHTTP->Response->ResponseCode = 200;
IdHTTP->ReadTimeout = 70000;
IdHTTP->ConnectTimeout = 70000;
IdHTTP->ReuseSocket;
Fist->Position = 0;
}
__finally
{
delete Fist;
Form1->Updated();
}
}
However, a "Cancel Resume" button is still can't resume interrupted downloads. Meant, it is always sending back the entire file every time I call Get() though I've used IdHTTP->Request->Ranges property.
I use the following code:
void __fastcall TForm1::CancelResumeClick(TObject *Sender)
{
MyFile = SaveDialog->FileName;;
TFileStream* TFist = new TFileStream(MyFile, fmCreate | fmShareDenyNone);
if (IdHTTP->Connected() == true)
{
IdHTTP->Disconnect();
CancelResume->Caption = "RESUME";
IdHTTP->Response->AcceptRanges = "Bytes";
}
else
{
try {
CancelResume->Caption = "CANCEL";
// IdHTTP->Request->Ranges == "0-100";
// IdHTTP->Request->Range = Format("bytes=%d-",ARRAYOFCONST((TFist->Position)));
IdHTTP->Request->Ranges->Add()->StartPos = TFist->Position;
IdHTTP->Get(Edit1->Text, TFist);
IdHTTP->Request->Referer = Edit1->Text;
IdHTTP->ConnectTimeout = 70000;
IdHTTP->ReadTimeout = 70000;
}
__finally {
delete TFist;
}
}
Meanwhile, by using the FormatBytes function, found here, has been able to shows only the size of download files. But still unable to determine the speed of download or transfer speed.
I'm using the following code:
void __fastcall TForm1::IdHTTPWork(TObject *ASender, TWorkMode AWorkMode, __int64 AWorkCount)
{
__int64 Romeo = 0;
Romeo = IdHTTP->Response->ContentStream->Position;
// Romeo = AWorkCount;
Download->Caption = FormatBytes(Romeo) + " (" + IntToStr(Romeo) + " Bytes)";
ForSpeed->Caption = FormatBytes(Romeo);
ProgressBar->Position = AWorkCount;
ProgressBar->Update();
Form1->Updated();
}
Please advise and give an example. Any help would sure be appreciated!
In your DownloadClick() method:
Calling Connected() is useless, since you don't do anything with the result. Nor is there any guarantee that the connection will remain connected, as the server could send a Connection: close response header. I don't see anything in your code that is asking for HTTP keep-alives. Let TIdHTTP manage the connection for you.
You are forcing the Response->ResponseCode to 200. Don't do that. Respect the response code that the server actually sent. The fact that no exception was raised means the response was successful whether it is 200 or 206.
You are reading the ReuseSocket property value and ignoring it.
There is no need to reset the Fist->Position property to 0 before closing the file.
Now, with that said, your CancelResumeClick() method has many issues.
You are using the fmCreate flag when opening the file. If the file already exists, you will overwrite it from scratch, thus TFist->Position will ALWAYS be 0. Use fmOpenReadWrite instead so an existing file will open as-is. And then you have to seek to the end of the file to provide the correct Position to the Ranges header.
You are relying on the socket's Connected() state to make decisions. DO NOT do that. The connection may be gone after the previous response, or may have timed out and been closed before the new request is made. The file can still be resumed either way. HTTP is stateless. It does not matter if the socket remains open between requests, or is closed in between. Every request is self-contained. Use information provided in the previous response to govern the next request. Not the socket state.
You are modifying the value of the Response->AcceptRanges property, instead of using the value provided by the previous response. The server tells you if the file supports resuming, so you have to remember that value, or query it before then attempting to resumed download.
When you actually call Get(), the server may or may not respect the requested Range, depending on whether the requested file supports byte ranges or not. If the server responds with a response code of 206, the requested range is accepted, and the server sends ONLY the requested bytes, so you need to APPEND them to your existing file. However, if the server response with a response code of 200, the server is sending the entire file from scratch, so you need to REPLACE your existing file with the new bytes. You are not taking that into account.
In your IdHTTPWork() method, in order to calculate the download/transfer speed, you have to keep track of how many bytes are actually being transferred in between each event firing. When the event is fired, save the current AWorkCount and tick count, and then the next time the event is fired, you can compare the new AWorkCount and current ticks to know how much time has elapsed and how many bytes were transferred. From those value, you can calculate the speed, and even the estimated time remaining.
As for your progress bar, you can't use AWorkCount alone to calculate a new position. That only works if you set the progress bar's Max to AWorkCountMax in the OnWorkBegin event, and that value is not always know before a download begins. You need to take into account the size of the file being downloaded, whether it is being downloaded fresh or being resumed, how many bytes are being requested during a resume, etc. So there is lot more work involved in displaying a progress bar for a HTTP download.
Now, to answer your two questions:
How to retrieve and save the download file to a disk by using its original name?
It is provided by the server in the filename parameter of the Content-Disposition header, and/or in the name parameter of the Content-Type header. If neither value is provided by the server, you can use the filename that is in the URL you are requesting. TIdHTTP has a URL property that provides the parsed version of the last requested URL.
However, since you are creating the file locally before sending your download request, you will have to create a local file using a temp filename, and then rename the local file after the download is complete. Otherwise, use TIdHTTP.Head() to determine the real filename (you can also use it to determine if resuming is supported) before creating the local file with that filename, then use TIdHTTP.Get() to download to that local file. Otherwise, download the file to memory using TMemoryStream instead of TFileStream, and then save with the desired filename when complete.
when I click http://get.videolan.org/vlc/2.2.1/win32/vlc-2.2.1-win32.exe then the server will process requests to its actual url. http://mirror.vodien.com/videolan/vlc/2.2.1/win32/vlc-2.2.1-win32.exe. The problem is that IdHTTP will not automatically grab through it.
That is because VideoLan is not using an HTTP redirect to send clients to the real URL (TIdHTTP supports HTTP redirects). VideoLan is using an HTML redirect instead (TIdHTTP does not support HTML redirects). When a webbrowser downloads the first URL, a 5 second countdown timer is displayed before the real download then begins. As such, you will have to manually detect that the server is sending you an HTML page instead of the real file (look at the TIdHTTP.Response.ContentType property for that), parse the HTML to determine the real URL, and then download it. This also means that you cannot download the first URL directly into your target local file, otherwise you will corrupt it, especially during a resume. You have to cache the server's response first, either to a temp file or to memory, so you can analyze it before deciding how to act on it. It also means you have to remember the real URL for resuming, you cannot resume the download using the original countdown URL.
Try something more like the following instead. It does not take into account for everything mentioned above (particularly speed/progress tracking, HTML redirects, etc), but should get you a little closer:
void __fastcall TForm1::DownloadClick(TObject *Sender)
{
Urlz = Edit1->Text;
Url->Caption = Urlz;
IdHTTP->Head(Urlz);
String FileName = IdHTTP->Response->RawHeaders->Params["Content-Disposition"]["filename"];
if (FileName.IsEmpty())
{
FileName = IdHTTP->Response->RawHeaders->Params["Content-Type"]["name"];
if (FileName.IsEmpty())
FileName = IdHTTP->URL->Document;
}
SaveDialog->FileName = FileName;
if (!SaveDialog->Execute()) return;
MyFile = SaveDialog->FileName;
TFileStream* Fist = new TFileStream(MyFile, fmCreate | fmShareDenyWrite);
try
{
try
{
Download->Enabled = false;
Resume->Enabled = false;
IdHTTP->Request->Clear();
//...
IdHTTP->ReadTimeout = 70000;
IdHTTP->ConnectTimeout = 70000;
IdHTTP->Get(Urlz, Fist);
}
__finally
{
delete Fist;
Download->Enabled = true;
Updated();
}
}
catch (const EIdHTTPProtocolException &)
{
DeleteFile(MyFile);
throw;
}
}
void __fastcall TForm1::ResumeClick(TObject *Sender)
{
TFileStream* Fist = new TFileStream(MyFile, fmOpenReadWrite | fmShareDenyWrite);
try
{
Download->Enabled = false;
Resume->Enabled = false;
IdHTTP->Request->Clear();
//...
Fist->Seek(0, soEnd);
IdHTTP->Request->Ranges->Add()->StartPos = Fist->Position;
IdHTTP->Request->Referer = Edit1->Text;
IdHTTP->ConnectTimeout = 70000;
IdHTTP->ReadTimeout = 70000;
IdHTTP->Get(Urlz, Fist);
}
__finally
{
delete Fist;
Download->Enabled = true;
Updated();
}
}
void __fastcall TForm1::IdHTTPHeadersAvailable(TObject*Sender, TIdHeaderList *AHeaders, bool &VContinue)
{
Resume->Enabled = ( ((IdHTTP->Response->ResponseCode == 200) || (IdHTTP->Response->ResponseCode == 206)) && TextIsSame(AHeaders->Values["Accept-Ranges"], "bytes") );
if ((IdHTTP->Response->ContentStream) && (IdHTTP->Request->Ranges->Count > 0) && (IdHTTP->Response->ResponseCode == 200))
IdHTTP->Response->ContentStream->Size = 0;
}
#Romeo:
Also, you can try a following function to determine the real download filename.
I've translated this to C++ based on the RRUZ'function. So far so good, I'm using it on my simple IdHTTP download program, too.
But, this translation result is of course still need value improvement input from Remy Lebeau, RRUZ, or any other master here.
String __fastcall GetRemoteFileName(const String URI)
{
String result;
try
{
TIdHTTP* HTTP = new TIdHTTP(NULL);
try
{
HTTP->Head(URI);
result = HTTP->Response->RawHeaders->Params["Content-Disposition"]["filename"];
if (result.IsEmpty())
{
result = HTTP->Response->RawHeaders->Params["Content-Type"]["name"];
if (result.IsEmpty())
result = HTTP->URL->Document;
}
}
__finally
{
delete HTTP;
}
}
catch(const Exception &ex)
{
ShowMessage(const_cast<Exception&>(ex).ToString());
}
return result;
}
Can we Cache Dynamically Created Lists or View till the webservices are called in background. I want to achieve something like the FaceBook App does. I know its possible in Android Core but wanted to try it in Titanium (Android and IOS).
I would further explain it,
Consider I have a app which has a list. Now When I open for first time, it will obviously hit the webservice and create a dynamic list.
Now I close the app and again open the app. The old list should be visible till the webservice provides any data.
Yes Titanium can do this. You should use a global variable like Ti.App.myList if it is just an array / a list / a variable. If you need to store more complex data like images or databases you should use the built-in file system. There is a really good Documentation on the Appcelerator website.
The procedure for you would be as follows:
Load your data for the first time
Store your data in your preferred way (Global variable, file system)
During future app starts read out your local list / data and display it until your sync is successfull.
You should consider to implement some variable to check wether any update is needed to minimize the network use (it saves energy and provides a better user experience if the users internet connection is slow).
if (response.state == "SUCCESS") {
Ti.API.info("Themes successfully checked");
Ti.API.info("RESPONSE TEST: " + response.value);
//Create a map of the layout names(as keys) and the corresponding url (as value).
var newImageMap = {};
for (var key in response.value) {
var url = response.value[key];
var filename = key + ".jpg"; //EDIT your type of the image
newImageMap[filename] = url;
}
if (Ti.App.ImageMap.length > 0) {
//Check for removed layouts
for (var image in Ti.App.imageMap) {
if (image in newImageMap) {
Ti.API.info("The image " + image + " is already in the local map");
//Do nothing
} else {
//Delete the removed layout
Ti.API.info("The image " + image + " is deleted from the local map");
delete Ti.App.imageMap[image];
}
}
//Check for new images
for (var image in newImageMap) {
if (image in Ti.App.imageMap) {
Ti.API.info("The image " + image + " is already in the local map");
//Do nothing
} else {
Ti.API.info("The image " + image + " is put into the local map");
//Put new image in local map
Ti.App.imageMap[image] = newImageMap[image];
}
}
} else {
Ti.App.imageMap = newImageMap;
}
//Check wether the file already exists
for (var key in response.value) {
var url = response.value[key];
var filename = key + ".png"; //EDIT YOUR FILE TYPE
Ti.API.info("URL: " + url);
Ti.API.info("FILENAME: " + filename);
imagesOrder[imagesOrder.length] = filename.match(/\d+/)[0]; //THIS SAVES THE FIRST NUMBER IN YOUR FILENAME AS ID
//Case1: download a new image
var file = Ti.Filesystem.getFile(Ti.Filesystem.resourcesDirectory, "/media/" + filename);
if (file.exists()) {
// Do nothing
Titanium.API.info("File " + filename + " exists");
} else {
// Create the HTTP client to download the asset.
var xhr = Ti.Network.createHTTPClient();
xhr.onload = function() {
if (xhr.status == 200) {
// On successful load, take that image file we tried to grab before and
// save the remote image data to it.
Titanium.API.info("Successfully loaded");
file.write(xhr.responseData);
Titanium.API.info(file);
Titanium.API.info(file.getName());
};
};
// Issuing a GET request to the remote URL
xhr.open('GET', url);
// Finally, sending the request out.
xhr.send();
}
}
In addition to this code which should be placed in a success method of an API call, you need a global variable Ti.App.imageMap to store the map of keys and the corresponding urls. I guess you have to change the code a bit to fit your needs and your project but it should give you a good starting point.
I have a website where I am using selenium for integration testing.
I have link there that is generated using multiple variables from page.
I would like to verify the download pop up box is displaying if at all possible, when i am simulating click on link to download the file.
I know i can have JsUnit that will do that for me.
Any ideas?
Selenium is lousy when working with downloads. Clicking the link will get you into trouble.
You could, however, make a request using HttpURLConnection, Apache HttpComponents (or maybe just a file get through URL) for the link specified and assert a 200 OK response. Or try to get the file - this is my favourite tool for this with Selenium.
Thnx to Slanec i have took up your examples.
Ok after investigation I have decided that best solution will be
something along this line.
public int GetFileLenghtFromUrlLocation(string location)
{
int len = 0;
int timeoutInSeconds = 5;
// paranoid check for null value
if (string.IsNullOrEmpty(location)) return 0;
// Create a web request to the URL
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(location);
myRequest.Timeout = timeoutInSeconds * 1000;
myRequest.AddRange(1024);
try
{
// Get the web response
HttpWebResponse myResponse = (HttpWebResponse)myRequest.GetResponse();
// Make sure the response is valid
if (HttpStatusCode.OK == myResponse.StatusCode)
{
// Open the response stream
using (Stream myResponseStream = myResponse.GetResponseStream())
{
if (myResponseStream == null) return 0;
using (StreamReader rdr = new StreamReader(myResponseStream))
{
len = rdr.ReadToEnd().Length;
}
}
}
}
catch (Exception err)
{
throw new Exception("Error saving file from URL:" + err.Message, err);
}
return len;
}