Does AWS CLI upload files larger than 4GB to Amazon Glacier? - amazon-web-services

Question

The AWS Command Line Interface (CLI) allows to upload a file to AWS Glacier. But there is also a limit of 4GB for file uploads in the AWS Rest API. If I need to upload a file larger than 4GB through the Rest API, I need to use the multi-part upload.
My question is: does the AWS CLI handle internally file uploads larger than 4GB, or do I need to handle myself the multipart upload when handling files larger than 4GB? Can I just pass a 20Gb file to the upload-archive option of the AWS CLI and it will just work? If the CLI can't handle large file uploads directly, there is any command line tool that does it for me (freeing me from the trouble of implementing all of the checksum computing, error handling and retry logic when a part upload fails)?
I understand that the 4GB limit is on the AWS Rest API, but I could not find anything about how this limit is handled in the CLI. I could just make the test, but my upload speed is not so fast and I fear wasting a few hours before discovering that it does not work.

Answer 1

I'm using glacier-cmd (https://github.com/uskudnik/amazon-glacier-cmd-interface), works pretty well, but seems to be unsupported recently. Sometimes it has a timeout with big files (~50GB).

Answer 2

The below script will work fine. I had created chunks for treehash calculation and fileparts uploads saperately. It worked fine.
#!/bin/bash
date1=$(date +"%s")
byteSize=1073741824
CHUNK_SIZE=1073741824
hashsize=1048576
if [[ -z "${1}" ]]; then
echo "No file provided."
exit 1
fi
ARCHIVE="/mnt/dbfiles/mahipal/splitfiles/${1}"
ARCHIVE_SIZE=`cat "${ARCHIVE}" | wc --bytes`
cd /mnt/dbfiles/mahipal/splitfiles
rm -rf TEMP
rm -rf HASH
mkdir TEMP
mkdir HASH
cd /mnt/dbfiles/mahipal/splitfiles/TEMP
date3=$(date +"%s")
split -d --bytes=${CHUNK_SIZE} "${ARCHIVE}" chunk -a 4
date4=$(date +"%s")
diff2=$(($date4-$date3))
cd /mnt/dbfiles/mahipal/splitfiles/HASH
date5=$(date +"%s")
split -d --bytes=${hashsize} "${ARCHIVE}" chunk -a 5
date6=$(date +"%s")
diff3=$(($date6-$date5))
cd /mnt/dbfiles/mahipal/splitfiles/TEMP
lastpartsize=`expr $(ls -l | tail -1 | awk '{print$5}') + 0`
lastfile=$(ls -l | tail -1 | awk '{print$9}')
cont=$(ls -l | wc -l)
cnt=`expr $cont - 2`
fileCount=$(ls -1 | grep "^chunk" | wc -l)
echo "Total parts to upload: " $fileCount
files=$(ls | grep "^chunk")
init=$(/bin/aws glacier initiate-multipart-upload --account-id - --part-size $byteSize --vault-name final_vault --archive-description "${1}_${ARCHIVE_SIZE}_${byteSize}")
echo "---------------------------------------"
uploadId=$(echo $init | jq '.uploadId' | xargs)
touch commands.txt
i=0
for f in $files
do
byteStart=$((i*byteSize))
byteEnd=$((i*byteSize+byteSize-1))
echo /bin/aws glacier upload-multipart-part --body $f --range "'"'bytes '"$byteStart"'-'"$byteEnd"'/*'"'" --account-id - --vault-name final_vault --upload-id $uploadId >> commands.txt
i=$(($i+1))
if [ "$i" == "$cnt" ]
then
byteEnd=`expr $byteEnd + 1`
byteEnd2=$((i*byteSize+lastpartsize-1))
byteSize=$lastpartsize
echo /bin/aws glacier upload-multipart-part --body $lastfile --range "'"'bytes '"$byteEnd"'-'"$byteEnd2"'/*'"'" --account-id - --vault-name final_vault --upload-id $uploadId >> commands.txt
break
fi
done
parallel --load 100% -a commands.txt --no-notice --bar
cd /mnt/dbfiles/mahipal/splitfiles/HASH
files=$(ls | grep "^chunk")
for f in $files
do
openssl dgst -sha256 -binary ${f} > "hash${f:5}"
done
echo "List Active Multipart Uploads:"
echo "Verify that a connection is open:"
/bin/aws glacier list-multipart-uploads --account-id - --vault-name final_vault >> /mnt/dbfiles/mahipal/splitfiles/TEMP/commands.txt
echo "-------------"
echo "Contents of commands.txt"
cd /mnt/dbfiles/mahipal/splitfiles/TEMP
cat commands.txt
# Calculate tree hash.
cd /mnt/dbfiles/mahipal/splitfiles/HASH
echo "Calculating tree hash..."
while true; do
COUNT=`ls hash* | wc -l`
if [[ ${COUNT} -le 2 ]]; then
TREE_HASH=$(cat hash* | openssl dgst -sha256 | awk '{print $2}')
break
fi
ls hash* | xargs -n 2 | while read PAIR; do
PAIRARRAY=(${PAIR})
if [[ ${#PAIRARRAY[#]} -eq 1 ]]; then
break
fi
cat ${PAIR} | openssl dgst -sha256 -binary > temphash
rm ${PAIR}
mv temphash "${PAIRARRAY[0]}"
done
done
cd /mnt/dbfiles/mahipal/splitfiles/TEMP
echo "Finalizing..."
/bin/aws glacier complete-multipart-upload --account-id=- --vault-name="final_vault" --upload-id="$uploadId" --checksum="${TREE_HASH}" --archive-size=${ARCHIVE_SIZE} >>commands.txt
RETVAL=$?
if [[ ${RETVAL} -ne 0 ]]; then
echo "complete-multipart-upload failed with status code: ${RETVAL}" >>commands.txt
echo "Aborting upload ${uploadId}" >>commands.txt
/bin/aws glacier abort-multipart-upload --account-id=- --vault-name="final_vault" --upload-id="${uploadId}" >>commands.txt
exit 1
fi
echo "--------------"
echo "Deleting temporary commands.txt file"
#rm commands.txt
date2=$(date +"%s")
diff=$(($date2-$date1))
echo "Total Split Duration for Chunk Part Size: $(($diff2/ 3600 )) hours $((($diff2 % 3600) / 60)) minutes $(($diff2 % 60)) seconds" >>commands.txt
echo "Total Split Duration for hash Part Size: $(($diff3/ 3600 )) hours $((($diff3 % 3600) / 60)) minutes $(($diff3 % 60)) seconds" >>commands.txt
echo "Total upload Duration: $(($diff/ 3600 )) hours $((($diff % 3600) / 60)) minutes $(($diff % 60)) seconds" >>commands.txt
echo "Done."
exit 0