I am try to upload a file to Amazon S3 using the AWS C++ SDK.
The call to CreateMultipartUpload returns successfully but the following call to UploadPart times out with the following error:
(Aws::String) m_message = "Unable to parse ExceptionName: RequestTimeout Message: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed."
I don't understand why the initiate call works but not the part upload call. There clearly isn't any network issue.
This is my code:
bool FileUploader::uploadChunk() {
Aws::S3::Model::UploadPartRequest request;
request.SetBucket("video");
request.SetKey(_key);
request.SetUploadId(_file->uploadId);
request.SetPartNumber(_file->chunksUploaded + 1);
long file_pos = _file->chunksUploaded * CHUNK_SIZE;
_input_file.seekg(file_pos, std::ios::beg);
_input_file.read(_file_buf, CHUNK_SIZE);
long n_bytes = _input_file.gcount();
if(n_bytes > 0) {
request.SetContentLength(n_bytes);
char_array_buffer buf2(_file_buf, _file_buf + n_bytes);
std::iostream *chunk_stream = new std::iostream(&buf2);
request.SetBody(std::shared_ptr<std::iostream>(chunk_stream));
Aws::S3::Model::UploadPartOutcome response = _client->UploadPart(request);
if(response.IsSuccess()) {
_file->chunksUploaded++;
_uploader->updateUploadStatus(_file);
}
return response.IsSuccess();
}
else {
return false;
}
}
The problem was my method of obtaining a stream for SetBody. I switched to using the boost library instead of a homegrown approach.
typedef boost::iostreams::basic_array_source<char> Device;
boost::iostreams::stream_buffer<Device> stmbuf(_file_buf, n_bytes);
std::iostream *stm = new std::iostream(&stmbuf);
request.SetBody(std::shared_ptr<Aws::IOStream>(stm));
This works well.
I also needed to keep track of the parts I was uploading for the call to CompleteMultipartUpload as follows:
Aws::S3::Model::CompletedPart part;
part.SetPartNumber(request.GetPartNumber());
part.SetETag(response.GetResult().GetETag());
_uploadedParts.AddParts(part);
Alternatively, you can use the TransferManager interface which will do this for you. It has an IOStream interface. In addition we provide a preallocated buffer implementation for iostream:
https://github.com/aws/aws-sdk-cpp/blob/master/aws-cpp-sdk-core/include/aws/core/utils/stream/PreallocatedStreamBuf.h
Related
I developed a little application that read data from a sensor, store them in SPIFFS memory of my wemos D1 mini (esp8266) and then create a JSON Document and send it via MQTT to my topic. The problem is that as long as I send a JSON Doc with 10 object everything works great, but when I increase the size of the doc over 10 object nothing works. Eventually I need to send a JSON doc with 100 object inside.
What have I already done?
I'm using PubSubClient and I already set the MAX_PACKET_SIZE to the correct value
Using arduinojson assistant I found out the size of my JSON Document (8192 bytes)
I tried to use mqtt.fx to test if the problem was the esp8266 or the mqtt broker. Using mqtt.fx I'm able to send a JSON doc with 100 objects
As soon as I increase the size of the JSON doc I get a wdt error from the serial monitor of my arduino IDE.
I search the internet for wdt error but I don't get what they are and how to solve my problem
Last things I already tried to show on the serial monitor the file.txt in the SPIFFS where I store the data and I can store and then read the 100 object
So in the end I think it's an esp8266 problem and not PubSubClient or MQTT. Am I right?
Does anyone of you here ever encountered this problem before or have some other test I can run?
I search the internet for wdt error but I don't get what they are and how to solve my problem
WDT stands for a Watch Dog Timer. https://os.mbed.com/cookbook/WatchDog-Timer#:~:text=A%20watchdog%20timer%20(WDT)%20is,a%20software%20or%20hardware%20fault.
A watchdog timer (WDT) is a hardware timer that automatically generates a system reset if the main program neglects to periodically service it. It is often used to automatically reset an embedded device that hangs because of a software or hardware fault. Some systems may also refer to it as a computer operating properly (COP) timer. Many microcontrollers including the mbed processor have watchdog timer hardware.
Let's paint a better picture with an example. Let's say that you setup a WDT with a time of 10 seconds. Then the WDT starts counting down from 10 seconds. If it reaches 0 the processor will reset. "Feeding" the WDT will reset the countdown to the original value in this case 10 seconds. So if the WDT has counted down to 4 seconds remaining and you feed it, it resets the countdown back to 10 and starts counting down again.
Does anyone of you here ever encountered this problem before or have some other test I can run?
It looks to me like sending a larger JSON object takes a longer period of time than what the WDT is set for. One possibility would be to break up the JSON object into multiple pieces and send it in smaller chunks instead of one large one. This way the time between WDT "feedings" is reduced. I have no idea if this would be possible for you to change. But this should at least give you a better idea of what's happening.
OK in the end the problem was that sending a large JsonDocument triggered the WDT and the only way I found to overcome this problem was, as suggested by adamvz, to create a main file with all the 100 object, then call a function to split that file in 10 smaller one and send each of them over the internet through an HTTP request or Mosquitto.
Supposing you already created the main file in the spiffs memory, then:
This to split the main file:
void WritePacks() {
sourceFile = LittleFS.open("/file.txt", "r");
if (!sourceFile) {
Serial.println(F("Error: file.txt open failed"));
} else {
Serial.println("File open w/ success");
for (byte idx = 0; idx < outputCount; idx++) {
String aLine;
aLine.reserve(capacity);
if (sourceFile.available() == 0) break;
destinationFile = LittleFS.open(outputFileNames[idx], "w");
if (!destinationFile) {
Serial.print(F("can't open destination "));
Serial.println(outputFileNames[idx]);
break;
} else {
int lineCount = 0;
while (sourceFile.available() && (lineCount <= 10)) {
aLine = sourceFile.readStringUntil('\n');
destinationFile.println(aLine); // double check if the '\n' is in the String or not (--> print or println accordingly)
lineCount++;
}
outputIndex = idx;
Serial.println(outputIndex);
destinationFile.close();
}
} // end for
sourceFile.close();
}
}//end WritePacks
This to publish:
//------ HTTP Publish ------
void httpPublish(){
const char * outputFileNames[] = {"/out1.txt", "/out2.txt", "/out3.txt", "/out4.txt", "/out5.txt", "/out6.txt", "/out7.txt", "/out8.txt", "/out9.txt", "/out10.txt"};
const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
byte outputIndex = 0;
File sourceFile;
File destinationFile;
//Serial.println(capacity);
for (byte idx = 0; idx < outputCount; idx++) {
DynamicJsonDocument doc(capacity);
DynamicJsonDocument globalDoc(capacity);
StaticJsonDocument <1024> localDoc;
String aLine;
aLine.reserve(capacity);
destinationFile = LittleFS.open(outputFileNames[idx], "r");
if (!destinationFile) {
Serial.print(F("can't open destination "));
Serial.println(outputFileNames[idx]);
break;
} else {
Serial.print("Reading: ");
Serial.println(outputFileNames[idx]);
//int lineCount = 0;
while (destinationFile.available()) {
aLine = destinationFile.readStringUntil('\n');
DeserializationError error = deserializeJson(localDoc, aLine);
if (!error) globalDoc.add(localDoc);
else{ Serial.println("Error Writing All files");}
}//while
JsonObject Info = doc.createNestedObject("Info");
Info["Battery"] = battery;
Info["ID"] = id;
Info["Latitudine"] = latitudine;
Info["Longitudine"] = longitudine;
JsonArray Data = doc.createNestedArray("Data");
Data.add(globalDoc);
HTTPClient http;
//Send request
http.begin("yourURL");
char buffer[capacity];
size_t n = serializeJson(doc, buffer);
http.POST(buffer);
Serial.println(buffer);
http.end();
destinationFile.close();
}
}// end for
}//end httpPublish
I use google cloud cpp library to upload/download a file, using UploadFile and DownloadToFile methods accordingly.
How it possible to cancel a file transfer from another thread?
Thanks in advance!
There is no way (currently) to cancel a download in progress. But you could write something like this that is easy to cancel:
void MyDownload(
gcs::Client client, std::string bucket_name, std::string object_name,
std::string filename, bool& canceled) {
auto reader = client.ReadObject(bucket_name, object_name);
auto writer = std::ofstream(filename);
std::vector<char> buffer(4 * 1024 * 1024L);
do {
if (canceled) return; // TODO - not thread safe
reader.read(buffer.data(), buffer.size());
writer.write(buffer.data(), reader.gcount());
} while(not reader.eof() and reader.good() and writer.good());
}
I am now implementing the Raft algorithm, and I want to use gRPC stream to do this. My main idea is to create 3 streams for each node to every other peers, one stream will transmit one type of RPCs, there are AppendEntries, RequestVote and InstallSnapshot. I write some code with limited help from route_guide, because in its bidirectional stream demo RouteChat, the client send all its data before it starts to read.
Firstly, I want to write to a stream at any time, so I write the following codes
void RaftMessagesStreamClientSync::AsyncRequestVote(const RequestVoteRequest& request){
std::string peer_name = this->peer_name;
debug("GRPC: Send RequestVoteRequest from %s to %s\n", request.name().c_str(), peer_name.c_str());
request_vote_stream->Write(request);
}
Meanwhile, I want a thread keep reading from a stream, like the following codes, which is called immediately after RaftMessagesStreamClientSync is constructed.
void RaftMessagesStreamClientSync::handle_response(){
// strongThis is a must
auto strongThis = shared_from_this();
t1 = new std::thread([strongThis](){
RequestVoteResponse response;
while (strongThis->request_vote_stream->Read(&response)) {
debug("GRPC: Recv RequestVoteResponse from %s, me %s\n", response.name().c_str(), strongThis->raft_node->name.c_str());
...
}
});
...
In order to initialize 3 streams, I have to write the constructor like this, I use 3 ClientContext here because the document says one ClientContext for one RPC
struct RaftMessagesStreamClientSync : std::enable_shared_from_this<RaftMessagesStreamClientSync>{
typedef grpc::ClientReaderWriter<RequestVoteRequest, RequestVoteResponse> CR;
typedef grpc::ClientReaderWriter<AppendEntriesRequest, AppendEntriesResponse> CA;
typedef grpc::ClientReaderWriter<InstallSnapshotRequest, InstallSnapshotResponse> CI;
std::unique_ptr<CR> request_vote_stream;
std::unique_ptr<CA> append_entries_stream;
std::unique_ptr<CI> install_snapshot_stream;
ClientContext context_r;
ClientContext context_a;
ClientContext context_i;
std::thread * t1 = nullptr;
std::thread * t2 = nullptr;
std::thread * t3 = nullptr;
...
}
RaftMessagesStreamClientSync::RaftMessagesStreamClientSync(const char * addr, struct RaftNode * _raft_node) : raft_node(_raft_node), peer_name(addr) {
std::shared_ptr<Channel> channel = grpc::CreateChannel(addr, grpc::InsecureChannelCredentials());
stub = raft_messages::RaftStreamMessages::NewStub(channel);
// 1
request_vote_stream = stub->RequestVote(&context_r);
// 2
append_entries_stream = stub->AppendEntries(&context_a);
// 3
install_snapshot_stream = stub->InstallSnapshot(&context_i);
}
~RaftMessagesStreamClientSync() {
raft_node = nullptr;
t1->join();
t2->join();
t3->join();
delete t1;
delete t2;
delete t3;
}
Then I implement the server side
Status RaftMessagesStreamServiceImpl::RequestVote(ServerContext* context, ::grpc::ServerReaderWriter< ::raft_messages::RequestVoteResponse, RequestVoteRequest>* stream){
RequestVoteResponse response;
RequestVoteRequest request;
while (stream->Read(&request)) {
...
}
return Status::OK;
}
Then 2 problems happen:
When I test with 3 nodes, which actually creates 2 RaftMessagesStreamServiceImpl for each node, the statement from 1 to 3 cost a long time to execute.
There is no RPC received from server side.
There are similar problems when using Bidi Aysnc Server, However I can't figure out how this post can help me.
UPDATE
After some debugging, I found request_vote_stream->Write(request) returns 0, which, according to the document, means the stream is closed. However why is it closed?
After some debugging, I found that the two problem are all due to one problem that I create a client before I create a server.
Because I originally uses unary RPC calls, so a previous call from client only causes a gRPC error code 14. The program continues because every call sent after the server is created can be handled correctly.
However, when it comes to streaming calls, stub->RequestVote(&context_r) will end up calling a blocking function ClientReaderWriter::ClientReaderWriter, which will try to connect to the server, which is not created now.
/// Block to create a stream and write the initial metadata and \a request
/// out. Note that \a context will be used to fill in custom initial metadata
/// used to send to the server when starting the call.
ClientReaderWriter(::grpc::ChannelInterface* channel,
const ::grpc::internal::RpcMethod& method,
ClientContext* context)
: context_(context),
cq_(grpc_completion_queue_attributes{
GRPC_CQ_CURRENT_VERSION, GRPC_CQ_PLUCK,
GRPC_CQ_DEFAULT_POLLING}), // Pluckable cq
call_(channel->CreateCall(method, context, &cq_)) {
if (!context_->initial_metadata_corked_) {
::grpc::internal::CallOpSet<::grpc::internal::CallOpSendInitialMetadata>
ops;
ops.SendInitialMetadata(context->send_initial_metadata_,
context->initial_metadata_flags());
call_.PerformOps(&ops);
cq_.Pluck(&ops);
}
}
As a consequence, the connection has not yet been established.
I upload a file to S3. Directly after the request I get an exception from the MonitoringManager and I don't know what I am doing wrong. We are using multiple threads in our application.
Exception: Assertion failed. Program: ... Monitor...ger.cpp Line 55
Expresion: s_monitors
The cpp file: https://github.com/aws/aws-sdk-cpp/blob/master/aws-cpp-sdk-core/source/monitoring/MonitoringManager.cpp Line 55
uploadFileToS3(...);
method 'uploadFileToS3':
bool result = false;
const Aws::SDKOptions options;
Aws::InitAPI(options);
{
std::shared_ptr<Aws::Utils::Threading::Executor> m_executor = Aws::MakeShared<Aws::Utils::Threading::PooledThreadExecutor>("TransferTests", 4);
Aws::Transfer::TransferManagerConfiguration config(m_executor.get());
config.s3Client = client;
auto transmanager = Aws::Transfer::TransferManager::Create(config);
std::shared_ptr<Aws::Transfer::TransferHandle> handle = transmanager->UploadFile(fileDestination, Aws::String(S3_BUCKET_NAME),
Aws::String(s3key), Aws::String("multipart/form-data"), metadata);
handle->WaitUntilFinished();
result = isAwsActionSuccessful(handle) && boost::filesystem::remove(fileDestination);
}
Aws::ShutdownAPI(options);
return result;
The issue was that my application used multiple threads so the API was initialized and shutdown multiple times. The problem was solved when I executed the initialize / shutdown of the API just once in my application.
How may I download a file in C++ with wxWidgets?
Been googling and everything and nothing shows up! Help appreciated!
Use wxHTTP class for that.
wxHTTP Example Code:
#include <wx/sstream.h>
#include <wx/protocol/http.h>
wxHTTP get;
get.SetHeader(_T("Content-type"), _T("text/html; charset=utf-8"));
get.SetTimeout(10); // 10 seconds of timeout instead of 10 minutes ...
while (!get.Connect(_T("www.google.com")))
wxSleep(5);
wxApp::IsMainLoopRunning();
wxInputStream *httpStream = get.GetInputStream(_T("/intl/en/about.html"));
if (get.GetError() == wxPROTO_NOERR)
{
wxString res;
wxStringOutputStream out_stream(&res);
httpStream->Read(out_stream);
wxMessageBox(res);
}
else
{
wxMessageBox(_T("Unable to connect!"));
}
wxDELETE(httpStream);
get.Close();
If you want more flexible solution consider using libcurl.
Depends on where you want to 'download' it from, and how the file server allows files to be downloaded. The server might use FTP, or HTTP, or something more obscure. There is no way to tell from your question which has no useful information in it.
In general, I would not use wxWidgets for this task. wxWidgets is a GUI frmaework, with some extras for various things that may or may not be helpful in your case.
From HTTP as Andrejs suggest, from FTP using wxFTP
wxFTP ftp;
// if you don't use these lines anonymous login will be used
ftp.SetUser("user");
ftp.SetPassword("password");
if ( !ftp.Connect("ftp.wxwindows.org") )
{
wxLogError("Couldn't connect");
return;
}
ftp.ChDir("/pub");
wxInputStream *in = ftp.GetInputStream("wxWidgets-4.2.0.tar.gz");
if ( !in )
{
wxLogError("Coudln't get file");
}
else
{
size_t size = in->GetSize();
char *data = new char[size];
if ( !in->Read(data, size) )
{
wxLogError("Read error");
}
else
{
// file data is in the buffer
...
}
delete [] data;
delete in;
}
http://docs.wxwidgets.org/stable/wx_wxftp.html#wxftp
You did not define what "downloading a file" means to you.
If you want to use HTTP to retrieve some content, you should use an HTTP client library like libcurl and issue the appropriate HTTP GET request.