How to get the Initialization Vector (IV) from the AWS Encryption CLI? - amazon-web-services

I'm encrypting a file using the AWS Encryption CLI using a command like so:
aws-encryption-cli --encrypt --input test.mp4 --master-keys key=arn:aws:kms:us-west-2:123456789012:key/exmaple-key-id --output . --metadata-output -
From the output of the command, I can clearly see that it's using an Initialization Vector (IV) of strength 12, which is great, but how do I actually view the IV? In order to pass the encrypted file to another service, like AWS Elastic Transcoder, where it'll do the decryption itself, I need to actually know what the IV was that was used for encrypting the file.
{
"header": {
"algorithm": "AES_256_GCM_IV12_TAG16_HKDF_SHA384_ECDSA_P384",
"content_type": 2,
"encrypted_data_keys": [{
"encrypted_data_key": "...............",
"key_provider": {
"key_info": "............",
"provider_id": "..........."
}
}],
"encryption_context": {
"aws-crypto-public-key": "..............."
},
"frame_length": 4096,
"header_iv_length": 12,
"message_id": "..........",
"type": 128,
"version": "1.0"
},
"input": "/home/test.mp4",
"mode": "encrypt",
"output": "/home/test.mp4.encrypted"
}

Unfortunately, you won't be able to use the AWS Encryption SDK CLI to encrypt data for Amazon Elastic Transcoder's consumption.
One of the primary benefits of the AWS Encryption SDK is the message format[1] which packages all necessary information about the encrypted message into a binary blob and provides a more scalable way of handling large messages. Extracting the data primitives from that blob is not recommended and even if you did, they may or may not be directly compatible with another system, depending on how you used the AWS Encryption SDK and what that other system expects.
In the case of Elastic Transcoder, they expect the raw ciphertext encrypted using the specified AES mode[2]. This is not compatible with the AWS Encryption SDK format.
[1] https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html
[2] https://docs.aws.amazon.com/elastictranscoder/latest/developerguide/create-job.html#create-job-request-inputs-encryption

Related

How to specify filepath via --cli-input-json in s3api

I'm trying to issue an aws s3api put-object command with all arguments specified via the --cli-input-json document, which in this case looks like so:
{
"Body": "this is the part giving me trouble",
"Bucket": "my-bucket",
"Key": "my-key"
}
For the Body property, I can't figure out how to specify a file (on the local system) to put to S3. I've tried both:
"Body": "the_filepath"
"Body": "file://the_filepath"
... but neither work (both result in an Invalid base64 error).
I know I can add the file to the command line call via --body file://the_filepath, but I'm trying to put all command args into the JSON document. I'm also trying to avoid reading in the contents of the object by the controlling script.
I'm stumped and I can't seem to find AWS CLI documentation on this use case.

S3 Lambda Trigger not triggering for EVERY file upload

In Python Django, I save multiple video files.
Save 1:
Long Video
Short Video
Save 2:
Long Video
Short Video
Save 3:
Long Video
Short Video
I have a lambda trigger that uses media converter to add HLS formats to these videos as well as generate thumbnails. These 3 saves are done in very short time periods between each other since they are assets to a Social Media Post object.
For some reason the S3 triggers for only some of the files.
Save 1 triggers S3 Lambda but not Save 2.
Save 3 also triggers S3 Lambda.
My assumption is that the S3 trigger has some sort of downtime in between identifying new file uploads (In which case, I think the period in between these file uploads are near instant).
Is this assumption correct and how can I circumvent it?
It should fire for all objects.
When Amazon S3 triggers an AWS Lambda function, information about the object that caused the trigger is passed in the events field:
{
"Records": [
{
"eventSource": "aws:s3",
"awsRegion": "us-west-2",
"eventTime": "1970-01-01T00:00:00.000Z",
"eventName": "ObjectCreated:Put",
"s3": {
"bucket": {
"name": "my-s3-bucket",
"arn": "arn:aws:s3:::example-bucket"
},
"object": {
"key": "HappyFace.jpg",
"size": 1024,
...
}
}
}
]
}
Note that this is an array, so it is possible that multiple objects could be passed to one Lambda function. I have never definitively seen this happen, but the sample code from AWS certainly assumes this can happen based on their sample code:
def lambda_handler(event, context):
for record in event['Records']: # <-- Looping here
bucket = record['s3']['bucket']['name']
key = unquote_plus(record['s3']['object']['key'])
...
Therefore, I would recommend:
Print the event at the start of the function to put it into the log for later examination
Use a loop to go through all records that might be passed
Let us know what you found!

Using CloudWatch Event : How to Pass JSON Object to CodeBuild as an Environment Variable

Summary: I can't specify a JSON object using CloudWatch target Input Transformer, in order to pass the object contents as an environment variable to a CodeBuild project.
Background:
I trigger an AWS CodeBuild job when an S3 bucket receives any new object. I have enabled CloudTrail for S3 operations so that I can use a CloudWatch rule that has my S3 bucket as an Event Source, with the CodeBuild project as a Target.
If I setup the 'Configure input' part of the Target, using Input Transformer, I can get single 'primitive' values from the event using the format below:
Input path textbox:
{"zip_file":"$.detail.requestParameters.key"}
Input template textbox:
{"environmentVariablesOverride": [ {"name":"ZIP_FILE", "value":<zip_file>}]}
And this works fine if I use 'simple' single strings.
However, for example, if I wish to obtain the entire 'resources' key, which is a JSON object, I need to have knowledge of each of the keys within, and the object structure, and manually recreate the structure for each key/value pair.
For example, the resources element in the Event is:
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::mybucket/myfile.zip"
},
{
"accountId": "1122334455667799",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::mybucket"
}
],
I want the code in the buildspec in CodeBuild to do the heavy lifting and parse the JSON data.
If I specify in the input path textbox:
{"zip_file":"$.detail.resources"}
Then CodeBuild project never gets triggered.
Is there a way to get the entire JSON object, identified by a specific key, as an environment variable?
Check this...CodeBuild targets support all the parameters allowed by StartBuild API. You need to use environmentVariablesOverride in your JSON string.
{"environmentVariablesOverride": [ {"name":"ZIPFILE", "value":<zip_file>}]}
Please,avoid using '_' in the environment name.

Is it possible to download the contents of a public Lambda layer from AWS given the ARN?

I want to download the public arn for a more compact version of spacy from this GitHub repository.
"arn:aws:lambda:us-west-2:113088814899:layer:Klayers-python37-spacy:27"
How can I achieve this?
You can get it from a Arn using the get-layer-version-by-arn function in the CLI.
You can run the below command to get the source of the Lambda layer you requested.
aws lambda get-layer-version-by-arn \
--arn "arn:aws:lambda:us-west-2:113088814899:layer:Klayers-python37-spacy:27"
An example of the response you will receive is below
{
"LayerVersionArn": "arn:aws:lambda:us-west-2:123456789012:layer:AWSLambda-Python37-SciPy1x:2",
"Description": "AWS Lambda SciPy layer for Python 3.7 (scipy-1.1.0, numpy-1.15.4) https://github.com/scipy/scipy/releases/tag/v1.1.0 https://github.com/numpy/numpy/releases/tag/v1.15.4",
"CreatedDate": "2018-11-12T10:09:38.398+0000",
"LayerArn": "arn:aws:lambda:us-west-2:123456789012:layer:AWSLambda-Python37-SciPy1x",
"Content": {
"CodeSize": 41784542,
"CodeSha256": "GGmv8ocUw4cly0T8HL0Vx/f5V4RmSCGNjDIslY4VskM=",
"Location": "https://awslambda-us-west-2-layers.s3.us-west-2.amazonaws.com/snapshots/123456789012/..."
},
"Version": 2,
"CompatibleRuntimes": [
"python3.7"
],
"LicenseInfo": "SciPy: https://github.com/scipy/scipy/blob/master/LICENSE.txt, NumPy: https://github.com/numpy/numpy/blob/master/LICENSE.txt"
}
Once you run this you will get a response returned with a key of "Content", containing a subkey of "Location" which references the S3 path to download the layer contents.
You can download from this path, you will then need to configure this as a Lambda layer again after removing any dependencies.
Please ensure in this process that you only remove unnecessary dependencies.

linking to a google cloud bucket file in a terminal command?

I'm trying to find my way with Google Cloud.
I have a Debian VM Instance that I am running a server on. It is installed and working via SSH Connection in a browser window. The command to start the server is "./ninjamsrv config-file-path.cfg"
I have the config file in my default google firebase storage bucket as I will need to update it regularly.
I want to start the server referencing the cfg file in the bucket, e.g:
"./ninjamsrv gs://my-bucket/ninjam-config.cfg"
But the file is not found:
error opening configfile 'gs://my-bucket/ninjam-config.cfg'
Error loading config file!
However if I run:
"gsutil acl get gs://my-bucket/"
I see:
[
{
"entity": "project-editors-XXXXX",
"projectTeam": {
"projectNumber": "XXXXX",
"team": "editors"
},
"role": "OWNER"
},
{
"entity": "project-owners-XXXXX",
"projectTeam": {
"projectNumber": "XXXXX",
"team": "owners"
},
"role": "OWNER"
},
{
"entity": "project-viewers-XXXXX",
"projectTeam": {
"projectNumber": "XXXXX",
"team": "viewers"
},
"role": "READER"
}
]
Can anyone advise what I am doing wrong here? Thanks
The first thing to verify is if indeed the error thrown is a permission one. Checking the logs related to the VM’s operations will certainly provide more details in that aspect, and a 403 error code would confirm if this is a permission issue. If the VM is a Compute Engine one, you can refer to this documentation about logging.
If the error is indeed a permission one, then you should verify if the permissions for this object are set as “fine-grained” access. This would mean that each object would have its own set of permissions, regardless of the bucket-level access set. You can read more about this here. You could either change the level of access to “uniform” which would grant access to all objects in the relevant bucket, or make the appropriate permissions change for this particular object.
If the issue is not a permission one, then I would recommend trying to start the server from the same .cfg file hosted on the local directory of the VM. This might point the error at the file itself, and not its hosting on Cloud Storage. In case the server starts successfully from there, you may want to re-upload the file to GCS in case the file got corrupted during the initial upload.