Upload an image file takes too long on AWS S3 c++ SDK - c++

Question

Using AWS S3 C++ SDK for uploading .jpg images to a certain IAM user introduce huge time delays that in any case are caused due to network traffic and latency issues. I am using free-tier S3 version and MSVC 2017 64bit for my application (on Windows 10 PC). Here is a sample code:
Aws::SDKOptions options;
Aws::InitAPI(options);
Aws::Client::ClientConfiguration config;
config.region = Aws::Region::US_EAST_2;
Aws::S3::S3Client s3_client(Aws::Auth::AWSCredentials(KEY,ACCESS_KEY), config);
const Aws::String bucket_name = BUCKET;
const Aws::String object_name = "image.jpg";
Aws::S3::Model::PutObjectRequest put_object_request;
put_object_request.SetBucket(bucket_name);
put_object_request.SetKey(object_name);
std::shared_ptr<Aws::IOStream> input_data =
Aws::MakeShared<Aws::FStream>("PutObjectInputStream",
"../image.jpg",
std::ios_base::in | std::ios::binary);
put_object_request.SetBody(input_data);
put_object_request.SetContentType("image/jpeg");
input_data->seekg(0LL, input_data->end);
put_object_request.SetContentLength(static_cast<long>(input_data->tellg()));
auto put_object_outcome = s3_client.PutObject(put_object_request);
When I upload images bigger than 100KB the total
PutObject(put_object_request);
time of execution exceeds 2min for a 520KB image.
I have tried the same example using Python boto3 and the total upload time for the same image is around 25s.
Have anyone faced the same issue?

Answer 1

After a better look to AWS github repo I figure out the issue.
The problem was that WinHttpSyncHttpClient was making timouts and reset upload activity internally thus not exiting the upload thread and finally aborting the transaction. By adding a custom timeout value the problem solved.
I used multipart Upload to re-implement the example as it seems more robust and manageable. Although I thought that it is unavailable for C++ SDK, its not the case as the TransferManager does the same job for C++ (not by using S3 headers as Java, .NET and PHP does).
Thanks KaibaLopez and SCalwas from AWS github repo who help me solve the issues (issue1, issue2). I am pasting an example code is case anyone face the same issue:
#include "pch.h"
#include <iostream>
#include <fstream>
#include <filesystem>
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentials.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/Bucket.h>
#include <aws/transfer/TransferManager.h>
#include <aws/transfer/TransferHandle.h>
static const char* KEY = "KEY";
static const char* BUCKET = "BUCKET_NAME";
static const char* ACCESS_KEY = "AKEY";
static const char* OBJ_NAME = "img.jpg";
static const char* const ALLOCATION_TAG = "S3_SINGLE_OBJ_TEST";
int main()
{
Aws::SDKOptions options;
Aws::InitAPI(options);
{
Aws::Client::ClientConfiguration config;
config.region = Aws::Region::US_EAST_2;
config.requestTimeoutMs = 20000;
auto s3_client = std::make_shared<Aws::S3::S3Client>(Aws::Auth::AWSCredentials(KEY, ACCESS_KEY), config);
const Aws::String bucket_name = BUCKET;
const Aws::String object_name = OBJ_NAME;
const Aws::String key_name = OBJ_NAME;
auto s3_client_executor = Aws::MakeShared<Aws::Utils::Threading::DefaultExecutor>(ALLOCATION_TAG);
Aws::Transfer::TransferManagerConfiguration trConfig(s3_client_executor.get());
trConfig.s3Client = s3_client;
trConfig.uploadProgressCallback =
[](const Aws::Transfer::TransferManager*, const std::shared_ptr<const Aws::Transfer::TransferHandle>&transferHandle)
{ std::cout << "Upload Progress: " << transferHandle->GetBytesTransferred() <<
" of " << transferHandle->GetBytesTotalSize() << " bytes" << std::endl;};
std::cout << "File start upload" << std::endl;
auto tranfer_manager = Aws::Transfer::TransferManager::Create(trConfig);
auto transferHandle = tranfer_manager->UploadFile(object_name.c_str(),
bucket_name.c_str(), key_name.c_str(), "multipart/form-data", Aws::Map<Aws::String, Aws::String>());
transferHandle->WaitUntilFinished();
if(transferHandle->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED)
std::cout << "File up" << std::endl;
else
std::cout << "Error uploading: " << transferHandle->GetLastError() << std::endl;
}
Aws::ShutdownAPI(options);
return 0;
}