AWS CPP TransferManager vs GetObjectRequest Stream to File fstream OOM - c++

Question

I am using the AWS CPP SDK ( https://github.com/aws/aws-iot-device-sdk-cpp ) to download a file from S3 on a small Linux system ( only 32 MB RAM ). I was using the GetObjectRequest class as shown below. It worked great and downloaded the file to the FStream on my system so that it did not consume too much RAM.
Now, I want to convert the download code to the TransferManager method in order to get progress callbacks. I've rewritten that part of the code and it is shown below as well. It starts out fine, prints the percentage downloaded, but when it reached ~14 MB RAM ( roughly the amount available in Linux at the time of download ) it is killed by the kernel for using too much RAM.
I created a file stream just like I did for the GetObjectRequest. What am I doing wrong? How can I fix this? Thanks.
Old way that did not use all the RAM:
// Old way
GetObjectRequest getObjectRequest;
getObjectRequest.SetBucket(bucket.c_str());
getObjectRequest.SetKey(keyName.c_str());
getObjectRequest.SetResponseStreamFactory([&destination](){
return Aws::New<Aws::FStream>(
"s3file", destination, std::ios_base::out); });
GetObjectOutcome getObjectOutcome = SessionClient->GetObject(getObjectRequest);
if(getObjectOutcome.IsSuccess())
{
std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
}
else
{
std::cout << "<AWS DOWNLOAD> Get FW failed: " << getObjectOutcome.GetError().GetMessage() << std::endl;
exit(1);
}
New way that eventually uses too much RAM and is killed by the kernel:
// New way
Aws::Transfer::TransferManagerConfiguration transferConfig;
transferConfig.s3Client = SessionClient;
std::shared_ptr<Aws::Transfer::TransferHandle> requestPtr(nullptr);
transferConfig.downloadProgressCallback =
[](const Aws::Transfer::TransferManager*, const Aws::Transfer::TransferHandle& handle)
{
std::cout << "\r" << "<AWS DOWNLOAD> Download Progress: " << static_cast<int>(handle.GetBytesTransferred() * 100.0 / handle.GetBytesTotalSize()) << " Percent " << handle.GetBytesTransferred() << " bytes\n";
};
Aws::Transfer::TransferManager transferManager(transferConfig);
requestPtr = transferManager.DownloadFile(bucket.c_str(), keyName.c_str(), [&destination](){
Aws::FStream *stream = Aws::New<Aws::FStream>("s3file", destination, std::ios_base::out);
stream->rdbuf()->pubsetbuf(NULL, 0);
return stream; });
requestPtr->WaitUntilFinished();
size_t retries = 0;
//just make sure we don't fail because a download part failed. (e.g. network problems or interuptions)
while (requestPtr->GetStatus() == Aws::Transfer::TransferStatus::FAILED && retries++ < 5)
{
std::cout << "<AWS DOWNLOAD> FW Download trying download again!" << std::endl;
transferManager.RetryDownload(requestPtr);
requestPtr->WaitUntilFinished();
}
// Check status
if ( requestPtr->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED ) {
if ( requestPtr->GetBytesTotalSize() == requestPtr->GetBytesTransferred() ) {
std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl;
exit(0);
}
else {
std::cout << "<AWS DOWNLOAD> Get FW failed - Bytes downloaded did not equal requested number of bytes: " << requestPtr->GetBytesTotalSize() << requestPtr->GetBytesTransferred() << std::endl;
exit(1);
}
}
else {
std::cout << "<AWS DOWNLOAD> Get FW failed - download was never completed even after retries" << std::endl;
exit(1);
}

Answer 1

TransferManager only really makes things easier once you are in the land of 10mb or larger and you want to take advantage of parallelization. It will allocate the max heap size up front and not grow the heap larger than that. Given your RAM constraints, I wouldnt use TransferManager. You can still receive the progress notifications. Check the callback mechanisms in the AmazonWebServiceRequest class.