I am trying to upload an 18GB .vhd file to S3.
I have done the following process :
aws s3api create-multipart-upload --bucket amcaebucket --key 'multipart/01'
where my-bucket is the name of my bucket.
I got the following response :
{
"Bucket": "amcaebucket",
"UploadId": "xxxxxxxxxxx",
"Key": "multipart/01"
}
My next command :
aws s3api upload-part --bucket amcaebucket --key 'multipart/01' --part-number 1 --upload-id "xxxxxxxxxxxx"
I got the following error :
An error occurred (AllAccessDisabled) when calling the UploadPart operation: All access to this object has been disabled.
What do I do??
Related
I'm trying to delete an empty folder structure using the cli and noting seems to be working. The root folder is /+7000/ and I'm positive it's because of the "+"
My rm commands are working on other folders without special characters and the cli isn't returning an error. How would I build this script to recognize this folder and get rid of it?
Test Scripts
(%2B is '+' in hex)
>aws s3 ls
2022-07-13 10:29:36 0 %2B7000/
>
>
//Attempts
> aws s3 rm s3://namespace/ --exclude "*" --include "%2B7000/"
> aws s3 rm s3://namespace/ --exclude "*" --include "*7000/"
> aws s3 rm s3://namespace/ --exclude "*" --include "[+]7000/"
> aws s3 rm s3://namespace/ --exclude "*" --include "'+7000'/"
> aws s3 rm s3://namespace/"\+7000/"
> aws s3 rm s3://namespace/%2B7000/
delete: s3://namespace/%2B7000/
Most attempts return a successful deletion but the folder is still there.
Output from aws s3api list-objects --bucket namespace
{
"Key": "+7000/",
"LastModified": "2022-07-13T14:29:36.884Z",
"ETag": "\"100\"",
"Size": 0,
"StorageClass": "STANDARD",
"Owner": {
"DisplayName": "vmsstg",
"ID": "1-2-3-4"
}
}
If aws s3 rm isn't working, you can try the 'lower-level' API call:
aws s3api delete-object --bucket namespace --key %2B7000/
Given that you said that %2B is plus, and based on your comment, you can use:
aws s3api delete-object --bucket namespace --key "+7000/"
I guess the plus sign got translated into 'URL Encode' syntax somewhere along the way.
Another approach is to use an AWS SDK (eg boto3 for Python) to retrieve the Key and then delete the object by passing back the exact value. This would avoid any encoding in the process.
I am trying to bulk update all s3 buckets with default encryption to that i generate a json file using below command
aws s3api list-buckets --query "Buckets[].Name" >> s3.json
My results was names of all s3 buckets.
How do i pass in that json file into the command so i can enable default encryption.
I also tried below
aws s3api list-buckets --query 'Buckets[*].[Name]' --output text | xargs -I {} bash -c 'aws s3api put-bucket-encryption --bucket {} --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}''
But iam getting below error
Error parsing parameter '--server-side-encryption-configuration': Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
JSON received: {Rule
aws s3api put-bucket-encryption --bucket bucketnames --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'
I have tried below but it does not work.
aws s3api put-bucket-encryption \
--bucket value \
--server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}' \
--cli-input-json file://s3bucket.json
Pleas let me know how to update my command to enable default encryption.
Below is the code snippet to solve your problem:
# Check if bucket is SSE enabled and then encrypt using SSE AES256:
#!/bin/bash
#List all buckets and store names in a array.
arr=(`aws s3api list-buckets --query "Buckets[].Name" --output text`)
# Check the status before encryption:
for i in "${arr[#]}"
do
echo "Check if SSE is enabled for bucket -> ${i}"
aws s3api get-bucket-encryption --bucket ${i} | jq -r .ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault.SSEAlgorithm
done
# Encrypt all buckets in your account:
for i in "${arr[#]}"
do
echo "Encrypting bucket with SSE AES256 for -> ${i}"
aws s3api put-bucket-encryption --bucket ${i} --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'
done
aws s3api list-buckets --query "Buckets[].Name" \
| jq .[] \
| xargs -I '{}' aws s3api put-bucket-encryption --bucket {} --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]}'
Worked for me
If you wanted to do it in Python it would be something like this (not tested!):
import boto3
s3_client = boto3.client('s3')
response = s3_client.list_buckets()
for bucket in response['Buckets']
s3_client.put_bucket_encryption(
Bucket=bucket,
ServerSideEncryptionConfiguration={
'Rules': [
{
'ApplyServerSideEncryptionByDefault': {
'SSEAlgorithm': 'AES256'
}
},
]
}
)
I have a DynamoDB's Table called "ZombieSession" and the "SessionId" primary key with "S" type.
The local service is running in http://localhost:8181.
For local tests, I'm trying execute these commands:
(1)
aws dynamodb delete-item --table-name ZombieSession --key
'4ae40a08-007c-4785-babd-caff0ed12d1d' --endpoint-url
http://localhost:8181 --region us-east-1
That results in:
Error parsing parameter '--key': Invalid JSON:
'4ae40a08-007c-4785-babd-caff0ed12d1d'
and
(2)
aws dynamodb delete-item --table-name ZombieSession --key
'{"SessionId":{"S":"4ae40a08-007c-4785-babd-caff0ed12d1d"}}'
--endpoint-url http://localhost:8181 --region us-east-1
That results in:
Error parsing parameter '--key': Invalid JSON:
'{SessionId:{S:4ae40a08-007c-4785-babd-caff0ed12d1d}}'
I don't found any documentation example about this.
What's the appropriate command for this operation?
I discovered that the value of --key parameter need have the quotation mark with escape:
aws dynamodb delete-item --table-name ZombieSession --key
"{\"SessionId\":{\"S\":\"4ae40a08-007c-4785-babd-caff0ed12d1d\"}}"
--endpoint-url http://localhost:8181 --region us-east-1
Getting following error for put-bucket-lifecycle-configuration :
[root#ADM-PROD-OMNI noc-scripts]# aws s3api put-bucket-lifecycle-configuration --bucket noc-try --lifecycle-configuration lifecycle.json
usage: aws [options] <command> <subcommand> [parameters]
aws: error: argument operation: Invalid choice, valid choices are:
abort-multipart-upload | complete-multipart-upload
copy-object | create-bucket
create-multipart-upload | delete-bucket
delete-bucket-cors | delete-bucket-lifecycle
delete-bucket-policy | delete-bucket-replication
delete-bucket-tagging | delete-bucket-website
delete-object | delete-objects
get-bucket-acl | get-bucket-cors
get-bucket-lifecycle | get-bucket-location
get-bucket-logging | get-bucket-notification
get-bucket-notification-configuration | get-bucket-policy
get-bucket-replication | get-bucket-request-payment
get-bucket-tagging | get-bucket-versioning
get-bucket-website | get-object
get-object-acl | get-object-torrent
head-bucket | head-object
list-buckets | list-multipart-uploads
list-object-versions | list-objects
list-parts | put-bucket-acl
put-bucket-cors | put-bucket-lifecycle
put-bucket-logging | put-bucket-notification
put-bucket-notification-configuration | put-bucket-policy
put-bucket-replication | put-bucket-request-payment
put-bucket-tagging | put-bucket-versioning
put-bucket-website | put-object
put-object-acl | restore-object
upload-part | upload-part-copy
wait | help
But
get-bucket-lifecycle is working this means my aws is configured :
[root#ADM-PROD-OMNI noc-scripts]# aws s3api get-bucket-lifecycle --bucket 4sm-wrapup
RULES clear multipart failed files Enabled
**OR**
[root#ADM-PROD-OMNI noc-scripts]# aws s3api get-bucket-lifecycle --bucket noc-try
A client error (NoSuchLifecycleConfiguration) occurred when calling the GetBucketLifecycle operation: The lifecycle configuration does not exist
Also tried :
[root#ADM-PROD-OMNI noc-scripts]# aws s3api put-bucket-lifecycle --bucket noc-try --lifecycle-configuration lifecycle.json
Error parsing parameter '--lifecycle-configuration': Expected: '=', received: '.' for input:
lifecycle.json
^
Please let me know what is wrong here ?
In your first example it is pretty obvious that the method "put-bucket-lifecycle-configuration" does not exist, and you need to be using "put-bucket-lifecycle" instead, which you said you also tried and received a different error.
Different errors are good!
The new error suggests improper syntax when calling your .json configuration file, and/or improper structured JSON.
Here is documentation on the "put-bucket-lifecycle": put-bucket-lifecycle
Here is an example of calling a .json config file:
aws s3api put-bucket-lifecycle --bucket my-bucket --lifecycle-configuration file://lifecycle.json
Here is an example of JSON file:
{
"Rules": [
{
"ID": "Move to Glacier after sixty days (objects in logs/2015/)",
"Prefix": "logs/2015/",
"Status": "Enabled",
"Transition": {
"Days": 60,
"StorageClass": "GLACIER"
}
},
{
"Expiration": {
"Date": "2016-01-01T00:00:00.000Z"
},
"ID": "Delete 2014 logs in 2016.",
"Prefix": "logs/2014/",
"Status": "Enabled"
}
]
}
BELOW JSON FILE IS TESTED AND WORKS AS SHOWN IN THE SUBSEQUENT SCREENSHOTS:
{
"Rules": [
{
"ID": "multipart-upload-rule",
"Prefix": "noc-try",
"Status": "Enabled",
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 3 }
}
]
}
CLI COMMAND TO CREATE LIFECYCLE CONFIG USING ABOVE JSON FILE:
aws s3api put-bucket-lifecycle --bucket testbucket1478921 --lifecycle-configuration file://c:/tmp/test.json
If I'm uploading data to S3 using the aws-cli (i.e. using aws s3 cp), does aws-cli do any work to confirm that the resulting file in S3 matches the original file, or do I somehow need to manage that myself?
Based on this answer and the Java API documentation for putObject(), it looks like it's possible to verify the MD5 checksum after upload. However, I can't find a definitive answer on whether aws-cli actually does that.
It matters to me because I'm intending to upload GPG-encrypted files from a backup process, and I'd like some confidence that what's been stored in S3 actually matches the original.
According to the faq from the aws-cli github, the checksums are checked in most cases during upload and download.
Key points for uploads:
The AWS CLI calculates the Content-MD5 header for both standard and
multipart uploads.
If the checksum that S3 calculates does not match
the Content-MD5 provided, S3 will not store the object and instead
will return an error message back the AWS CLI.
The AWS CLI will retry this error up to 5 times before giving up and exiting with a nonzero exit code.
The AWS support page How do I ensure data integrity of objects uploaded to or downloaded from Amazon S3? describes how to achieve this.
Firstly determine the base64 encoded md5sum of the file you wish to upload:
$ md5_sum_base64="$( openssl md5 -binary my-file | base64 )"
Then use the s3api to upload the file:
$ aws s3api put-object --bucket my-bucket --key my-file-name --body my-file-path --content-md5 "$md5_sum_base64"
Note the use of the --content-md5 flag, the help for this flag states:
--content-md5 (string) The base64-encoded 128-bit MD5 digest of the part data.
This does not say much about why to use this flag, but we can find this information in the API documentation for put object:
To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error. Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value.
Using this flag causes S3 to verify that the file hash serverside matches the specified value. If the hashes match s3 will return the ETag:
{
"ETag": "\"599393a2c526c680119d84155d90f1e5\""
}
The ETag value will usually be the hexadecimal md5sum (see this question for some scenarios where this may not be the case).
If the hash does not match the one you specified you get an error.
A client error (InvalidDigest) occurred when calling the PutObject operation: The Content-MD5 you specified was invalid.
In addition to this you can also add the file md5sum to the file metadata as an additional check:
$ aws s3api put-object --bucket my-bucket --key my-file-name --body my-file-path --content-md5 "$md5_sum_base64" --metadata md5chksum="$md5_sum_base64"
After upload you can issue the head-object command to check the values.
$ aws s3api head-object --bucket my-bucket --key my-file-name
{
"AcceptRanges": "bytes",
"ContentType": "binary/octet-stream",
"LastModified": "Thu, 31 Mar 2016 16:37:18 GMT",
"ContentLength": 605,
"ETag": "\"599393a2c526c680119d84155d90f1e5\"",
"Metadata": {
"md5chksum": "WZOTosUmxoARnYQVXZDx5Q=="
}
}
Here is a bash script that uses content md5 and adds metadata and then verifies that the values returned by S3 match the local hashes:
#!/bin/bash
set -euf -o pipefail
# assumes you have aws cli, jq installed
# change these if required
tmp_dir="$HOME/tmp"
s3_dir="foo"
s3_bucket="stack-overflow-example"
aws_region="ap-southeast-2"
aws_profile="my-profile"
test_dir="$tmp_dir/s3-md5sum-test"
file_name="MailHog_linux_amd64"
test_file_url="https://github.com/mailhog/MailHog/releases/download/v1.0.0/MailHog_linux_amd64"
s3_key="$s3_dir/$file_name"
return_dir="$( pwd )"
cd "$tmp_dir" || exit
mkdir "$test_dir"
cd "$test_dir" || exit
wget "$test_file_url"
md5_sum_hex="$( md5sum $file_name | awk '{ print $1 }' )"
md5_sum_base64="$( openssl md5 -binary $file_name | base64 )"
echo "$file_name hex = $md5_sum_hex"
echo "$file_name base64 = $md5_sum_base64"
echo "Uploading $file_name to s3://$s3_bucket/$s3_dir/$file_name"
aws \
--profile "$aws_profile" \
--region "$aws_region" \
s3api put-object \
--bucket "$s3_bucket" \
--key "$s3_key" \
--body "$file_name" \
--metadata md5chksum="$md5_sum_base64" \
--content-md5 "$md5_sum_base64"
echo "Verifying sums match"
s3_md5_sum_hex=$( aws --profile "$aws_profile" --region "$aws_region" s3api head-object --bucket "$s3_bucket" --key "$s3_key" | jq -r '.ETag' | sed 's/"//'g )
s3_md5_sum_base64=$( aws --profile "$aws_profile" --region "$aws_region" s3api head-object --bucket "$s3_bucket" --key "$s3_key" | jq -r '.Metadata.md5chksum' )
if [ "$md5_sum_hex" == "$s3_md5_sum_hex" ] && [ "$md5_sum_base64" == "$s3_md5_sum_base64" ]; then
echo "checksums match"
else
echo "something is wrong checksums do not match:"
cat <<EOM | column -t -s ' '
$file_name file hex: $md5_sum_hex s3 hex: $s3_md5_sum_hex
$file_name file base64: $md5_sum_base64 s3 base64: $s3_md5_sum_base64
EOM
fi
echo "Cleaning up"
cd "$return_dir"
rm -rf "$test_dir"
aws \
--profile "$aws_profile" \
--region "$aws_region" \
s3api delete-object \
--bucket "$s3_bucket" \
--key "$s3_key"