I am trying to upload encrypted data to S3. This code successfully encrypts the data, however it uploads the original unencrypted file to S3. How do I tell it to upload the encrypted data instead?
note-the commented decryption lines were to test the data had been encrypted and decrypted properly
session = botocore.session.get_session()
client = session.create_client('kms',
region_name = 'us-east-1',
aws_access_key_id = '[YOUR ACCESS KEY]',
aws_secret_access_key = '[YOUR SECRET ACCESSKEY]')
key_id = '[KEY ID]'
plaintext='[FILEPATH\FILENAME.CSV]'
ciphertext = client.encrypt(KeyId=key_id, Plaintext=plaintext)
#decrypt_ciphertext = client.decrypt(CiphertextBlob =
ciphertext['CiphertextBlob'])
print('Ciphertext: ', ciphertext)
#print('Decrypted Ciphertext: ', decrypt_ciphertext)
s3 = boto3.client('s3',
aws_access_key_id='[YOUR ACCESS KEY]',
aws_secret_access_key='[YOUR SECRET ACCESS KEY]')
filename = '[FILEPATH\FILENAME.CSV]'
bucket_name = '[BUCKET NAME]'
# Uploads the given file using a managed uploader, which will split up large
# files automatically and upload parts in parallel.
s3.upload_file(filename, bucket_name, filename)
The KMS encrypt() command does not work on files. Rather, it accepts incoming text in Plaintext and outputs encrypted text in CiphertextBlob.
Your code is responsible for reading the source file and passing the contents to encrypt(), and it is then responsible for writing the contents out to disk.
See also:
AWS Encryption SDK for Python Example Code - AWS Encryption SDK
s3-client-side-encryption/put.py at master · tedder/s3-client-side-encryption · GitHub
Related
I have written a Lambda function in AWS to download a file from an S3 location to /tmp directory (local Lambda space).
I am able to download the file however, the file size is changing here, not sure why?
def data_processor(event, context):
print("EVENT:: ", event)
bucket_name = 'asr-collection'
fileKey = 'cc_continuous/testing/1645136763813.wav'
path = '/tmp'
output_path = os.path.join(path, 'mydir')
if not os.path.exists(output_path):
os.makedirs(output_path)
s3 = boto3.client("s3")
new_file_name = output_path + '/' + os.path.basename(fileKey)
s3.download_file(
Bucket=bucket_name, Key=fileKey, Filename=output_path + '/' + os.path.basename(fileKey)
)
print('File size is: ' + str(os.path.getsize(new_file_name)))
return None
Output:
File size is: 337964
Actual size: 230MB
downloaded file size is 330KB
I tried download_fileobj() as well
Any idea how can i download the file as it is, without any data loss?
The issue can be that the bucket you are downloading from was from a different region than the Lambda was hosted in. Apparently, this does not make a difference when running it locally.
Check your bucket locations relative to your Lambda region.
Make a note that setting the region on your client will allow you to use a lambda in a different region from your bucket. However if you intend to pull down larger files you will get network latency benefits from keeping your lambda in the same region as your bucket.
Working with S3 resource instance instead of client fixed it.
s3 = boto3.resource('s3')
keys = ['TestFolder1/testing/1651219413148.wav']
for KEY in keys:
local_file_name = '/tmp/'+KEY
s3.Bucket(bucket_name).download_file(KEY, local_file_name)
I have tried some luarocks library to like lua-resty-s3 of OPM.
I successfully upload text file using below codes. But can't figure out how to upload image file in S3 using Lua.
local awss3 = require('resty.s3')
local util = require("resty.s3_util")
local s3 = awss3:new(AWSAccessKeyId, AWSSecretAccessKey, aws_bucket, {timeout=1000*10, aws_region=aws_region})
local key = "key"
local content = "image_data"
local headers = util.new_headers()
headers['content-type'] = 'text/plain'
s3:put(key, content, headers)
I am new to AWS and boto. The data I want to download is on AWS, and I have the access key and the secret key. My problem is I do not understand the approaches I found. For instance, this code:
import boto
import boto.s3.connection
def download_data_connect_s3(access_key, secret_key, region, bucket_name, key, local_path):
conn = boto.connect_s3(aws_access_key_id = access_key,\
aws_secret_access_key = secret_key,\
host='s3-{}.amazonaws.com'.format(region),\
calling_format = boto.s3.connection.OrdinaryCallingFormat()\
)
bucket = conn.get_bucket(bucket_name)
key = bucket.get_key(key)
key.get_contents_to_filename(local_path)
print('Downloaded File {} to {}'.format(key, local_path))
region = 'us-west-1'
access_key = # the key here
secret_key = # the secret key here
bucket_name = 'temp_name'
key = '<folder…/filename>' unique identifer
local_path = # local path
download_data_connect_s3(access_key, secret_key, region, bucket_name, key, local_path)
What I don't understand is the 'key' 'bucket_name' and 'local path'. What is 'key' in comparison to access key and secret key? I was not given a 'key'. Also, is the 'bucket_name' the name of the bucket on AWS (I was not provided with the bucket name); and local path the directory where I want to save the data?
You are right.
bucket_name = name of your S3 bucket
key = is object key. It's full path of the file in side the bucket. (ex: you have a file named a.txt in folder x, so key = x/a.txt. Refer to this link
local_path = where you want to save the data in local machine
It sounds like the data is stored in Amazon S3.
You can use the AWS Command-Line Interface (CLI) to access Amazon S3.
To view the list of buckets in that account:
aws s3 ls
To view the contents of a bucket:
aws s3 ls bucket-name
To copy a file from a bucket to the current directory:
aws s3 cp s3://bucket-name/filename.txt .
Or sync a whole folder:
aws s3 sync s3://bucket-name/folder/ local-folder/
I'm trying to decrypt a kms encrypted file and running in to the following error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 3: invalid start byte
I'm using the sample decrypt code.
I'm able to decrypt the file using the command line.
The exception is being thrown from here:
cipher_text.decode('utf-8')
Code: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/kms/api-client/snippets.py
Please let me know if I'm missing something here.
When you use the Python library, all inputs must be base64-encoded, and the outputs will be base64-encoded as well. In the encrypt function in snippets.py, you can see that the code is base64-encoding the plaintext before passing it to the KMS encrypt API.
encoded_text = base64.b64encode(plaintext)
When you use the gcloud kms encrypt command, you do not have to base64 encode the plaintext yourself, and the ciphertext is not base64-encoded.
So, when you pass the ciphertext from gcloud kms encrypt to the Python library to decrypt, you must base64-encode it first. Change the decrypt function in snippets.py to base64-encode the file data before sending it on.
# Read cipher text from the input file.
with io.open(encrypted_file_name, 'rb') as encrypted_file:
ciphertext = encrypted_file.read()
encoded_text = base64.b64encode(ciphertext)
# Use the KMS API to decrypt the text.
cryptokeys = kms_client.projects().locations().keyRings().cryptoKeys()
request = cryptokeys.decrypt(
name=name, body={'ciphertext': encoded_text.decode('utf-8')})
response = request.execute()
You can think of the base64-encoding as being a transport-layer implementation detail: it's only necessary so that arbitrary binary data can be sent in JSON, which only accepts Unicode strings. So, the Cloud KMS API requires this data to be base64-encoded, and must base64-encode the output as well. But the gcloud command does this work for you, so you don't have to do it.
I think the Python sample code is misleading. It should always base64-encode inputs to the API and base64-decode outputs, instead of only doing it sometimes. I'll look at updating the Python sample code shortly, and double check the sample code for the other languages.
Given the date of the question, the accepted answer should be #Russ (also, thank you for updating the git).
Since the documentation changed a little, here is a function that deals with an already encrypted json file.
Encrypted using the GCloud Command Line:
gcloud kms encrypt \
--plaintext-file=[SECRETS.json] \
--ciphertext-file=[ENCRYPTED-SECRETS.json.enc] \
--location=[REGION] \
--keyring=[KEYRING-NAME] \
--key=[KEY-NAME]
Here is the function for decrypting said file (cipher_file being the path to [ENCRYPTED-SECRETS.json.enc]):
def decrypt(cipher_file):
project_id = "project"
location_id = "region"
key_ring_id = "key-ring"
crypto_key_id = "key"
# Creates an API client for the KMS API.
client = kms_v1.KeyManagementServiceClient()
# The resource name of the CryptoKey.
name = client.crypto_key_path_path(project_id, location_id, key_ring_id,
crypto_key_id)
# Use the KMS API to decrypt the data.
with io.open(cipher_file, "rb") as file:
c_text = file.read()
response = client.decrypt(name, c_text)
secret_dict = json.loads(response.plaintext.decode("utf-8"))
return secret_dict
I want to extract the attachment from email and save it into my new S3 bucket. So far, I have configured AWS Simple Email Service to intercept incoming emails. Now I have an AWS lambda python function, which gets triggered on S3 Put.
Until this it is working. But my lambda is giving error saying: "[Errno 2] No such file or directory: 'abc.docx': OSError". I see that the attachment with the name abc.docx is mentioned in the raw email in S3.
I assume the problem is in my upload_file. Could you please help me here.
Please find below the relevant parts of my code.
s3 = boto3.client('s3')
s3resource = boto3.resource('s3')
waiterFlg = s3.get_waiter('object_exists')
waiterFlg.wait(Bucket=bucket, Key=key)
response = s3resource.Bucket(bucket).Object(key)
message = email.message_from_string(response.get()["Body"].read())
if len(message.get_payload()) == 2:
attachment = msg.get_payload()[1]
s3resource.meta.client.upload_file(attachment.get_filename(), outputBucket, attachment.get_filename())
else:
print("Could not see file/attachment.")
You can download the attachment to /tmp directory in Lambda and then upload to S3.
The following code solved the issue:
open('/tmp/newFile.docx', 'wb') as f:
f.write(attachment.get_payload(decode=True))
s3r.meta.client.upload_file('/tmp/newFile.docx', outputBucket, attachment.get_filename())