Related
I have a AWS CF script which creates the stack via the parent template according to the account (Development & Production).
2 templates are names as - DevLaunchTemplate and ProdLaunchTemplate
In the resource I am using the FindInMap function to get the Launch template name according to the account.
Now the ask is to make this CF as a single use for both the accounts and to avoid using the "DevLaunchTemplate" name in the version. Is there a way that I can do this?
"Mappings": {
"TimeZoneMapping": {
"ca-central-1": {
"TimeZone": "Eastern Standard Time",
"Culture": "en-CA"
},
"ap-southeast-2": {
"TimeZone": "AUS Eastern Standard Time",
"Culture": "en-AU"
},
"us-east-1": {
"TimeZone": "Eastern Standard Time",
"Culture": "en-US"
},
"us-east-2": {
"TimeZone": "Eastern Standard Time",
"Culture": "en-US"
},
"us-west-1": {
"TimeZone": "Pacific Standard Time",
"Culture": "en-US"
},
"us-west-2": {
"TimeZone": "Pacific Standard Time",
"Culture": "en-US"
}
},
"Account" : {
"123456789" : {
"LaunchConfig" : "DevLaunchTemplate"
},
"987654321" : {
"LaunchConfig" : "ProdLaunchTemplate"
}
}
},
"Resources": {
"AutoScalingGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"DependsOn": ["DevLaunchTemplate","ProdLaunchTemplate"],
"Properties": {
"AutoScalingGroupName": {
"Fn::Join" : [ "-", ["cld", { "Fn::Select" : [ "1", { "Fn::Split": ["-", {"Fn::Sub": "${AWS::StackName}"}]}]}, { "Fn::Select" : [ "2", { "Fn::Split": ["-", {"Fn::Sub": "${AWS::StackName}"}]}]}]]
},
"AvailabilityZones": [
{
"Fn::Select": [
0,
{
"Fn::GetAZs": ""
}
]
},
{
"Fn::Select": [
1,
{
"Fn::GetAZs": ""
}
]
}
],
"VPCZoneIdentifier": {
"Ref": "Subnets"
},
"LaunchTemplate": {
"LaunchTemplateName": {
"Fn::Sub": [
"${AWS::StackName}-${FindInMapLaunchConfig}",
{
"FindInMapLaunchConfig": {
"Fn::FindInMap": [
"Account",
{
"Ref": "AWS::AccountId"
},
"LaunchConfig"
]
}
}
]
},
"Version" : { "Fn::GetAtt" : [ "DevLaunchTemplate", "LatestVersionNumber" ] }
},
"MaxSize": 1,
"MinSize": 1,
"DesiredCapacity": "1",
"Cooldown": "900",
"NotificationConfiguration": {
"TopicARN": {
"Ref": "SNSTopic"
},
},
},
"CreationPolicy": {
"ResourceSignal": {
"Count": "1",
"Timeout": "PT30M"
}
}
},
"DevLaunchTemplate":{
"Type":"AWS::EC2::LaunchTemplate",
"Properties":{
"LaunchTemplateName":{
"Fn::Sub": "${AWS::StackName}-DevLaunchTemplate"
},
"LaunchTemplateData":{
"IamInstanceProfile":{
"Arn":{"Fn::GetAtt": ["ProvisioningToolProfile", "Arn"]}
},
"InstanceMarketOptions":{
"MarketType": "spot",
"SpotOptions": {
"MaxPrice" : "0.5"
}
},
"ImageId":{
"Ref": "AMIId"
},
"InstanceType":{
"Ref": "InstanceType"
},
"KeyName":{
"Ref": "KeyName"
},
"SecurityGroupIds":[{
"Ref": "SecurityGroup"
}],
"EbsOptimized": "false",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"VolumeSize": "40",
"VolumeType": "gp3",
"Encrypted": "true"
}
}
]
}
}
},
"ProdLaunchTemplate":{
"Type":"AWS::EC2::LaunchTemplate",
"Properties":{
"LaunchTemplateName":{
"Fn::Sub": "${AWS::StackName}-ProdLaunchTemplate"
},
"LaunchTemplateData":{
"IamInstanceProfile":{
"Arn":{"Fn::GetAtt": ["ProvisioningToolProfile", "Arn"]}
},
"ImageId":{
"Ref": "AMIId"
},
"InstanceType":{
"Ref": "InstanceType"
},
"KeyName":{
"Ref": "KeyName"
},
"SecurityGroupIds":[{
"Ref": "SecurityGroup"
}],
"EbsOptimized": "false",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"VolumeSize": "40",
"VolumeType": "gp3",
"Encrypted": "true"
}
}
]
}
},
How do I format the below code part to make it as a single use for both development and production account?
"Version" : { "Fn::GetAtt" : [ "DevLaunchTemplate", "LatestVersionNumber" ] }
I'm building event driven AWS stacks with Lambda+APIGateway+SQS+SNS+S3+DynamoDB.
One of my constant frustrations is that, if you bind any of the above to Lambda (either through event notifications or event source mappings), the formats of the event messages received by the Lambda are completely different - so a message sent by S3 is completely different to one sent by SQS which is completely different to one sent by DynamoDB etc.
Normally I have to set up a Cloudformation stack with an event source + event source mapping + Lambda, then push a message onto the event source to see what message actually results. What a giant pain.
Is there not a single combined resource out there which lists the different schema formats of different event messages ? Hoping someone can point me in the right direction.
Lambda console provides some example events in Configure test event. Here are the examples from the console for the services you mentioned.
APIGateway (aws proxy)
{
"body": "eyJ0ZXN0IjoiYm9keSJ9",
"resource": "/{proxy+}",
"path": "/path/to/resource",
"httpMethod": "POST",
"isBase64Encoded": true,
"queryStringParameters": {
"foo": "bar"
},
"multiValueQueryStringParameters": {
"foo": [
"bar"
]
},
"pathParameters": {
"proxy": "/path/to/resource"
},
"stageVariables": {
"baz": "qux"
},
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, sdch",
"Accept-Language": "en-US,en;q=0.8",
"Cache-Control": "max-age=0",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Host": "1234567890.execute-api.us-east-1.amazonaws.com",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Custom User Agent String",
"Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
"X-Forwarded-For": "127.0.0.1, 127.0.0.2",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"multiValueHeaders": {
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
],
"Accept-Encoding": [
"gzip, deflate, sdch"
],
"Accept-Language": [
"en-US,en;q=0.8"
],
"Cache-Control": [
"max-age=0"
],
"CloudFront-Forwarded-Proto": [
"https"
],
"CloudFront-Is-Desktop-Viewer": [
"true"
],
"CloudFront-Is-Mobile-Viewer": [
"false"
],
"CloudFront-Is-SmartTV-Viewer": [
"false"
],
"CloudFront-Is-Tablet-Viewer": [
"false"
],
"CloudFront-Viewer-Country": [
"US"
],
"Host": [
"0123456789.execute-api.us-east-1.amazonaws.com"
],
"Upgrade-Insecure-Requests": [
"1"
],
"User-Agent": [
"Custom User Agent String"
],
"Via": [
"1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)"
],
"X-Amz-Cf-Id": [
"cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA=="
],
"X-Forwarded-For": [
"127.0.0.1, 127.0.0.2"
],
"X-Forwarded-Port": [
"443"
],
"X-Forwarded-Proto": [
"https"
]
},
"requestContext": {
"accountId": "123456789012",
"resourceId": "123456",
"stage": "prod",
"requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
"requestTime": "09/Apr/2015:12:34:56 +0000",
"requestTimeEpoch": 1428582896000,
"identity": {
"cognitoIdentityPoolId": null,
"accountId": null,
"cognitoIdentityId": null,
"caller": null,
"accessKey": null,
"sourceIp": "127.0.0.1",
"cognitoAuthenticationType": null,
"cognitoAuthenticationProvider": null,
"userArn": null,
"userAgent": "Custom User Agent String",
"user": null
},
"path": "/prod/path/to/resource",
"resourcePath": "/{proxy+}",
"httpMethod": "POST",
"apiId": "1234567890",
"protocol": "HTTP/1.1"
}
}
SQS
{
"Records": [
{
"messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
"receiptHandle": "MessageReceiptHandle",
"body": "Hello from SQS!",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1523232000000",
"SenderId": "123456789012",
"ApproximateFirstReceiveTimestamp": "1523232000001"
},
"messageAttributes": {},
"md5OfBody": "7b270e59b47ff90a553787216d55d91d",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
"awsRegion": "us-east-1"
}
]
}
SNS
{
"Records": [
{
"EventSource": "aws:sns",
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:us-east-1:{{{accountId}}}:ExampleTopic",
"Sns": {
"Type": "Notification",
"MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
"TopicArn": "arn:aws:sns:us-east-1:123456789012:ExampleTopic",
"Subject": "example subject",
"Message": "example message",
"Timestamp": "1970-01-01T00:00:00.000Z",
"SignatureVersion": "1",
"Signature": "EXAMPLE",
"SigningCertUrl": "EXAMPLE",
"UnsubscribeUrl": "EXAMPLE",
"MessageAttributes": {
"Test": {
"Type": "String",
"Value": "TestString"
},
"TestBinary": {
"Type": "Binary",
"Value": "TestBinary"
}
}
}
}
]
}
S3 (put)
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "1970-01-01T00:00:00.000Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "EXAMPLE"
},
"requestParameters": {
"sourceIPAddress": "127.0.0.1"
},
"responseElements": {
"x-amz-request-id": "EXAMPLE123456789",
"x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "testConfigRule",
"bucket": {
"name": "example-bucket",
"ownerIdentity": {
"principalId": "EXAMPLE"
},
"arn": "arn:aws:s3:::example-bucket"
},
"object": {
"key": "test/key",
"size": 1024,
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901"
}
}
}
]
}
DynamoDB
{
"Records": [
{
"eventID": "c4ca4238a0b923820dcc509a6f75849b",
"eventName": "INSERT",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"NewImage": {
"Message": {
"S": "New item!"
},
"Id": {
"N": "101"
}
},
"ApproximateCreationDateTime": 1428537600,
"SequenceNumber": "4421584500000000017450439091",
"SizeBytes": 26,
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899"
},
{
"eventID": "c81e728d9d4c2f636f067f89cc14862c",
"eventName": "MODIFY",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"NewImage": {
"Message": {
"S": "This item has changed"
},
"Id": {
"N": "101"
}
},
"OldImage": {
"Message": {
"S": "New item!"
},
"Id": {
"N": "101"
}
},
"ApproximateCreationDateTime": 1428537600,
"SequenceNumber": "4421584500000000017450439092",
"SizeBytes": 59,
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899"
},
{
"eventID": "eccbc87e4b5ce2fe28308fd9f2a7baf3",
"eventName": "REMOVE",
"eventVersion": "1.1",
"eventSource": "aws:dynamodb",
"awsRegion": "us-east-1",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"OldImage": {
"Message": {
"S": "This item has changed"
},
"Id": {
"N": "101"
}
},
"ApproximateCreationDateTime": 1428537600,
"SequenceNumber": "4421584500000000017450439093",
"SizeBytes": 38,
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"eventSourceARN": "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTableWithStream/stream/2015-06-27T00:48:05.899"
}
]
}
I used to copy data from one DynamoDB to another DynamoDB using a pipeline.json. It works when the source table has provisioned capacity and doesn't matter if destination is set to provisioned/on demand. I want both of my tables set to On Demand capacity. But when i use the same template it doesn't work. Is there any way that we can do that, or is it still under development?
Here is my original functioning script:
{
"objects": [
{
"startAt": "FIRST_ACTIVATION_DATE_TIME",
"name": "DailySchedule",
"id": "DailySchedule",
"period": "1 day",
"type": "Schedule",
"occurrences": "1"
},
{
"id": "Default",
"name": "Default",
"scheduleType": "ONDEMAND",
"pipelineLogUri": "#{myS3LogsPath}",
"schedule": {
"ref": "DailySchedule"
},
"failureAndRerunMode": "CASCADE",
"role": "DataPipelineDefaultRole",
"resourceRole": "DataPipelineDefaultResourceRole"
},
{
"id": "DDBSourceTable",
"tableName": "#{myDDBSourceTableName}",
"name": "DDBSourceTable",
"type": "DynamoDBDataNode",
"readThroughputPercent": "#{myDDBReadThroughputRatio}"
},
{
"name": "S3TempLocation",
"id": "S3TempLocation",
"type": "S3DataNode",
"directoryPath": "#{myTempS3Folder}/#{format(#scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}"
},
{
"id": "DDBDestinationTable",
"tableName": "#{myDDBDestinationTableName}",
"name": "DDBDestinationTable",
"type": "DynamoDBDataNode",
"writeThroughputPercent": "#{myDDBWriteThroughputRatio}"
},
{
"id": "EmrClusterForBackup",
"name": "EmrClusterForBackup",
"amiVersion": "3.8.0",
"masterInstanceType": "m3.xlarge",
"coreInstanceType": "m3.xlarge",
"coreInstanceCount": "1",
"region": "#{myDDBSourceRegion}",
"terminateAfter": "10 Days",
"type": "EmrCluster"
},
{
"id": "EmrClusterForLoad",
"name": "EmrClusterForLoad",
"amiVersion": "3.8.0",
"masterInstanceType": "m3.xlarge",
"coreInstanceType": "m3.xlarge",
"coreInstanceCount": "1",
"region": "#{myDDBDestinationRegion}",
"terminateAfter": "10 Days",
"type": "EmrCluster"
},
{
"id": "TableLoadActivity",
"name": "TableLoadActivity",
"runsOn": {
"ref": "EmrClusterForLoad"
},
"input": {
"ref": "S3TempLocation"
},
"output": {
"ref": "DDBDestinationTable"
},
"type": "EmrActivity",
"maximumRetries": "2",
"dependsOn": {
"ref": "TableBackupActivity"
},
"resizeClusterBeforeRunning": "true",
"step": [
"s3://dynamodb-emr-#{myDDBDestinationRegion}/emr-ddb-storage-handler/2.1.0/emr-ddb-2.1.0.jar,org.apache.hadoop.dynamodb.tools.DynamoDbImport,#{input.directoryPath},#{output.tableName},#{output.writeThroughputPercent}"
]
},
{
"id": "TableBackupActivity",
"name": "TableBackupActivity",
"input": {
"ref": "DDBSourceTable"
},
"output": {
"ref": "S3TempLocation"
},
"runsOn": {
"ref": "EmrClusterForBackup"
},
"resizeClusterBeforeRunning": "true",
"type": "EmrActivity",
"maximumRetries": "2",
"step": [
"s3://dynamodb-emr-#{myDDBSourceRegion}/emr-ddb-storage-handler/2.1.0/emr-ddb-2.1.0.jar,org.apache.hadoop.dynamodb.tools.DynamoDbExport,#{output.directoryPath},#{input.tableName},#{input.readThroughputPercent}"
]
},
{
"dependsOn": {
"ref": "TableLoadActivity"
},
"name": "S3CleanupActivity",
"id": "S3CleanupActivity",
"input": {
"ref": "S3TempLocation"
},
"runsOn": {
"ref": "EmrClusterForBackup"
},
"type": "ShellCommandActivity",
"command": "(sudo yum -y update aws-cli) && (aws s3 rm #{input.directoryPath} --recursive)"
}
],
"parameters": [
{
"myComment": "This Parameter specifies the S3 logging path for the pipeline. It is used by the 'Default' object to set the 'pipelineLogUri' value.",
"id" : "myS3LogsPath",
"type" : "AWS::S3::ObjectKey",
"description" : "S3 path for pipeline logs."
},
{
"id": "myDDBSourceTableName",
"type": "String",
"description": "Source DynamoDB table name"
},
{
"id": "myDDBDestinationTableName",
"type": "String",
"description": "Target DynamoDB table name"
},
{
"id": "myDDBWriteThroughputRatio",
"type": "Double",
"description": "DynamoDB write throughput ratio",
"default": "1",
"watermark": "Enter value between 0.1-1.0"
},
{
"id": "myDDBSourceRegion",
"type": "String",
"description": "Region of the DynamoDB table",
"default": "us-west-2"
},
{
"id": "myDDBDestinationRegion",
"type": "String",
"description": "Region of the DynamoDB table",
"default": "us-west-2"
},
{
"id": "myDDBReadThroughputRatio",
"type": "Double",
"description": "DynamoDB read throughput ratio",
"default": "1",
"watermark": "Enter value between 0.1-1.0"
},
{
"myComment": "Temporary S3 path to store the dynamodb backup csv files, backup files will be deleted after the copy completes",
"id": "myTempS3Folder",
"type": "AWS::S3::ObjectKey",
"description": "Temporary S3 folder"
}
]
}
And here is the error message from Data Pipeline execution when source DynamoDB table is set to On Demand capacity:
at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:833)
at org.apache.hadoop.dynamodb.tools.DynamoDbExport.run(DynamoDbExport.java:79)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.dynamodb.tools.DynamoDbExport.main(DynamoDbExport.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
The following JSON file worked for upload (DynamoDB to S3) -
{
"objects": [
{
"id": "Default",
"name": "Default",
"scheduleType": "ONDEMAND",
"pipelineLogUri": "#{myS3LogsPath}",
"failureAndRerunMode": "CASCADE",
"role": "DataPipelineDefaultRole",
"resourceRole": "DataPipelineDefaultResourceRole"
},
{
"id": "DDBSourceTable",
"tableName": "#{myDDBSourceTableName}",
"name": "DDBSourceTable",
"type": "DynamoDBDataNode",
"readThroughputPercent": "#{myDDBReadThroughputRatio}"
},
{
"name": "S3TempLocation",
"id": "S3TempLocation",
"type": "S3DataNode",
"directoryPath": "#{myTempS3Folder}/data"
},
{
"subnetId": "subnet-id",
"id": "EmrClusterForBackup",
"name": "EmrClusterForBackup",
"masterInstanceType": "m5.xlarge",
"coreInstanceType": "m5.xlarge",
"coreInstanceCount": "1",
"releaseLabel": "emr-5.23.0",
"region": "#{myDDBSourceRegion}",
"terminateAfter": "10 Days",
"type": "EmrCluster"
},
{
"id": "TableBackupActivity",
"name": "TableBackupActivity",
"input": {
"ref": "DDBSourceTable"
},
"output": {
"ref": "S3TempLocation"
},
"runsOn": {
"ref": "EmrClusterForBackup"
},
"resizeClusterBeforeRunning": "true",
"type": "EmrActivity",
"maximumRetries": "2",
"step": [
"s3://dynamodb-dpl-#{myDDBSourceRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,org.apache.hadoop.dynamodb.tools.DynamoDBExport,#{output.directoryPath},#{input.tableName},#{input.readThroughputPercent}"
]
}
],
"parameters": [
{
"myComment": "This Parameter specifies the S3 logging path for the pipeline. It is used by the 'Default' object to set the 'pipelineLogUri' value.",
"id" : "myS3LogsPath",
"type" : "AWS::S3::ObjectKey",
"description" : "S3 path for pipeline logs."
},
{
"id": "myDDBSourceTableName",
"type": "String",
"description": "Source DynamoDB table name"
},
{
"id": "myDDBSourceRegion",
"type": "String",
"description": "Region of the DynamoDB table",
"default": "us-west-2"
},
{
"id": "myDDBReadThroughputRatio",
"type": "Double",
"description": "DynamoDB read throughput ratio",
"default": "1",
"watermark": "Enter value between 0.1-1.0"
},
{
"myComment": "Temporary S3 path to store the dynamodb backup csv files, backup files will be deleted after the copy completes",
"id": "myTempS3Folder",
"type": "AWS::S3::ObjectKey",
"description": "Temporary S3 folder"
}
]
}
And the following worked for download (S3 to DynamoDB) -
{
"objects": [
{
"id": "Default",
"name": "Default",
"scheduleType": "ONDEMAND",
"pipelineLogUri": "#{myS3LogsPath}",
"failureAndRerunMode": "CASCADE",
"role": "DataPipelineDefaultRole",
"resourceRole": "DataPipelineDefaultResourceRole"
},
{
"name": "S3TempLocation",
"id": "S3TempLocation",
"type": "S3DataNode",
"directoryPath": "#{myTempS3Folder}/data"
},
{
"id": "DDBDestinationTable",
"tableName": "#{myDDBDestinationTableName}",
"name": "DDBDestinationTable",
"type": "DynamoDBDataNode",
"writeThroughputPercent": "#{myDDBWriteThroughputRatio}"
},
{
"subnetId": "subnet-id",
"id": "EmrClusterForLoad",
"name": "EmrClusterForLoad",
"releaseLabel": "emr-5.23.0",
"masterInstanceType": "m5.xlarge",
"coreInstanceType": "m5.xlarge",
"coreInstanceCount": "1",
"region": "#{myDDBDestinationRegion}",
"terminateAfter": "10 Days",
"type": "EmrCluster"
},
{
"id": "TableLoadActivity",
"name": "TableLoadActivity",
"runsOn": {
"ref": "EmrClusterForLoad"
},
"input": {
"ref": "S3TempLocation"
},
"output": {
"ref": "DDBDestinationTable"
},
"type": "EmrActivity",
"maximumRetries": "2",
"resizeClusterBeforeRunning": "true",
"step": [
"s3://dynamodb-dpl-#{myDDBDestinationRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar,org.apache.hadoop.dynamodb.tools.DynamoDBImport,#{input.directoryPath},#{output.tableName},#{output.writeThroughputPercent}"
]
},
{
"dependsOn": {
"ref": "TableLoadActivity"
},
"name": "S3CleanupActivity",
"id": "S3CleanupActivity",
"input": {
"ref": "S3TempLocation"
},
"runsOn": {
"ref": "EmrClusterForLoad"
},
"type": "ShellCommandActivity",
"command": "(sudo yum -y update aws-cli) && (aws s3 rm #{input.directoryPath} --recursive)"
}
],
"parameters": [
{
"myComment": "This Parameter specifies the S3 logging path for the pipeline. It is used by the 'Default' object to set the 'pipelineLogUri' value.",
"id" : "myS3LogsPath",
"type" : "AWS::S3::ObjectKey",
"description" : "S3 path for pipeline logs."
},
{
"id": "myDDBDestinationTableName",
"type": "String",
"description": "Target DynamoDB table name"
},
{
"id": "myDDBWriteThroughputRatio",
"type": "Double",
"description": "DynamoDB write throughput ratio",
"default": "1",
"watermark": "Enter value between 0.1-1.0"
},
{
"id": "myDDBDestinationRegion",
"type": "String",
"description": "Region of the DynamoDB table",
"default": "us-west-2"
},
{
"myComment": "Temporary S3 path to store the dynamodb backup csv files, backup files will be deleted after the copy completes",
"id": "myTempS3Folder",
"type": "AWS::S3::ObjectKey",
"description": "Temporary S3 folder"
}
]
}
Also, the subnet ID fields in both the pipeline definitions are totally optional, but it is always good to set them.
I have a problem with parsing output/transforming it from aws autoscaling describe-auto-scaling-groups
The output looks like this:
{
"AutoScalingGroups": [
{
"AutoScalingGroupName": "eks-nodegroup-AZ1",
"AutoScalingGroupARN": "arn:aws:autoscaling:eu-central-1::autoScalingGroup:854a8f05-cd3c-421d-abf3-0f3730d0b068:autoScalingGroupName/eks-nodegroup-AZ1",
"LaunchTemplate": {
"LaunchTemplateId": "lt-XXXXXXXXXXXXX",
"LaunchTemplateName": "eks-nodegroup-AZ1",
"Version": "$Latest"
},
"MinSize": 1,
"MaxSize": 6,
"DesiredCapacity": 1,
"DefaultCooldown": 300,
"AvailabilityZones": [
"eu-central-1a"
],
"LoadBalancerNames": [],
"TargetGroupARNs": [],
"HealthCheckType": "EC2",
"HealthCheckGracePeriod": 300,
"Instances": [
{
"InstanceId": "i-XXXXXXXXXXXXXXX",
"AvailabilityZone": "eu-central-1a",
"LifecycleState": "InService",
"HealthStatus": "Healthy",
"LaunchTemplate": {
"LaunchTemplateId": "lt-XXXXXXXXXXXXX",
"LaunchTemplateName": "eks-nodegroup-AZ1",
"Version": "1"
},
"ProtectedFromScaleIn": false
}
],
"CreatedTime": "2019-09-24T17:24:57.805Z",
"SuspendedProcesses": [],
"VPCZoneIdentifier": "subnet-XXXXXXXXXXXX",
"EnabledMetrics": [],
"Tags": [
{
"ResourceId": "eks-nodegroup-AZ1",
"ResourceType": "auto-scaling-group",
"Key": "Name",
"Value": "eks-nodegroup-AZ1",
"PropagateAtLaunch": true
},
{
"ResourceId": "eks-nodegroup-AZ1",
"ResourceType": "auto-scaling-group",
"Key": "k8s.io/cluster-autoscaler/enabled",
"Value": "true",
"PropagateAtLaunch": true
},
{
"ResourceId": "eks-nodegroup-AZ1",
"ResourceType": "auto-scaling-group",
"Key": "k8s.io/cluster-autoscaler/k8s-team-sandbox",
"Value": "true",
"PropagateAtLaunch": true
},
{
"ResourceId": "eks-nodegroup-AZ1",
"ResourceType": "auto-scaling-group",
"Key": "kubernetes.io/cluster/k8s-team-sandbox",
"Value": "owned",
"PropagateAtLaunch": true
}
],
"TerminationPolicies": [
"Default"
],
"NewInstancesProtectedFromScaleIn": false,
"ServiceLinkedRoleARN": "arn:aws:iam::XXXXXXXXXXXXXXX:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
},
{
"AutoScalingGroupName": "eks-k8s-team-sandbox-AZ2",
"AutoScalingGroupARN": "arn:aws:autoscaling:eu-central-1::autoScalingGroup:25324f3a-b911-453c-b316-46657e850b19:autoScalingGroupName/eks-nodegroup-AZ2",
"LaunchTemplate": {
"LaunchTemplateId": "lt-XXXXXXXXXXXX",
"LaunchTemplateName": "eks-nodegroup-AZ2",
"Version": "$Latest"
},
"MinSize": 1,
"MaxSize": 6,
"DesiredCapacity": 1,
"DefaultCooldown": 300,
"AvailabilityZones": [
"eu-central-1b"
],
"LoadBalancerNames": [],
"TargetGroupARNs": [],
"HealthCheckType": "EC2",
"HealthCheckGracePeriod": 300,
"Instances": [
{
"InstanceId": "i-XXXXXXXXXXXX",
"AvailabilityZone": "eu-central-1b",
"LifecycleState": "InService",
"HealthStatus": "Healthy",
"LaunchTemplate": {
"LaunchTemplateId": "lt-XXXXXXXXXX",
"LaunchTemplateName": "eks-nodegroup-AZ2",
"Version": "1"
},
"ProtectedFromScaleIn": false
}
],
"CreatedTime": "2019-09-24T17:24:57.982Z",
"SuspendedProcesses": [],
"VPCZoneIdentifier": "subnet-XXXXXXXX",
"EnabledMetrics": [],
"Tags": [
{
"ResourceId": "eks-nodegroup-AZ2",
"ResourceType": "auto-scaling-group",
"Key": "Name",
"Value": "eks-nodegroup-AZ2",
"PropagateAtLaunch": true
},
{
"ResourceId": "eks-nodegroup-AZ2",
"ResourceType": "auto-scaling-group",
"Key": "k8s.io/cluster-autoscaler/enabled",
"Value": "true",
"PropagateAtLaunch": true
},
{
"ResourceId": "eks-nodegroup-AZ2",
"ResourceType": "auto-scaling-group",
"Key": "k8s.io/cluster-autoscaler/k8s-team-sandbox",
"Value": "true",
"PropagateAtLaunch": true
},
{
"ResourceId": "eks-nodegroup-AZ2",
"ResourceType": "auto-scaling-group",
"Key": "kubernetes.io/cluster/k8s-team-sandbox",
"Value": "owned",
"PropagateAtLaunch": true
}
],
"TerminationPolicies": [
"Default"
],
"NewInstancesProtectedFromScaleIn": false,
"ServiceLinkedRoleARN": "arn:aws:iam::ARN:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
}
]
}
I need to parse it to get :
{
"eks-nodegroup-AZ1" : "$DesiredCapacityForEks-nodegroup-AZ1",
"eks-nodegroup-AZ2" : "$DesiredCapacityForEks-nodegroup-AZ2",
"eks-nodegroup-AZ3" : "$DesiredCapacityForEks-nodegroup-AZ3",
"eks-nodegroup-AZX" : "$DesiredCapacityForEks-nodegroup-AZX",
}
The following expected output will be used for external data resource for terraform to be able to automate DesiredCapacity value during the ASG rolling-updates.
Thanks,
Dominik
Try with your response,
response | jq '.AutoScalingGroups[] | {(.AutoScalingGroupName): .DesiredCapacity}' | jq -s add
I was hoping someone could help with this, I'm trying to stream logs from a Windows Server 2012 with EC2config service installed.
I have followed the following documentation:
https://aws.amazon.com/blogs/devops/using-cloudwatch-logs-with-amazon-ec2-running-microsoft-windows-server/
Unfortunately nothing is streaming to cloudwatch logs.
Here is the Json I'm using:
{
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [
{
"Id": "ApplicationEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Application",
"Levels": "1"
}
},
{
"Id": "SystemEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "System",
"Levels": "7"
}
},
{
"Id": "SecurityEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Security",
"Levels": "7"
}
},
{
"Id": "ETW",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Microsoft-Windows-WinINet/Analytic",
"Levels": "7"
}
},
{
"Id": "IISLog",
"FullName": "AWS.EC2.Windows.CloudWatch.IISLogOutput,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\inetpub\\logs\\LogFiles\\W3SVC1"
"AccessKey": "",
"SecretKey": "",
"Region": "eu-west-1",
"LogGroup": "Web-Logs",
"LogStream": "IIStest"
}
},
{
"Id": "CustomLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\CustomLogs\\",
"TimestampFormat": "MM/dd/yyyy HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "Local"
}
},
{
"Id": "PerformanceCounter",
"FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"CategoryName": "Memory",
"CounterName": "Available MBytes",
"InstanceName": "",
"MetricName": "Memory",
"Unit": "Megabytes",
"DimensionName": "",
"DimensionValue": ""
}
},
{
"Id": "CloudWatchLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"AccessKey": "",
"SecretKey": "",
"Region": "eu-west-1",
"LogGroup": "Win2Test",
"LogStream": "logging-test"
}
},
{
"Id": "CloudWatch",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatch.CloudWatchOutputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters":
{
"AccessKey": "",
"SecretKey": "",
"Region": "eu-west-1",
"NameSpace": "Windows/Default"
}
}
],
"Flows": {
"Flows":
[
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs",
"IISLog"
]
}
}
}
At this moment in time i only want to stream the IIS logs, from my understanding the Cloudwatch Log group and stream should automatically create.
The issue with Flows section is the second component of Flow definition is missing :
instead of
"Flows": {
"Flows":
[
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs",
"IISLog"
]
}
It should be
[
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs",
"IISLog,CloudWatchLogs"
]
The Flows section defines the source and target of the components from Components section first being what/how to get and second is how to send.
e.g. consider following snippet here ApplicationEventLog and SystemEventLog will be sent to CloudWatch (refers to "Id" : "CloudWatch" defined in Components instead of AWS CloudWatch).
The second line defines second flow i.e. PerformanceCounter sent to CloudWatch1
"Flows": {
"Flows":
[
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs",
"PerformanceCounter,CloudWatch1"
]
}
hope this explains how it resolved the issue.
Looks like i made a few mistakes on the JSON file itself, specifically the FLOW area.
Got this working now :)