I have been trying to come up with a way to track any and all instance type changed that happen in our companies account. (ex: t2.micro to t2.nano)
I settled on creating a custom config rule that would alert us if the instance changed with a uncompliant warning, but I think this might be over complicating it and am suspecting that I should be using CloudWatch alarms or EventBridge.
I have used the following setup (from the CLI):
rdk create ec2_check_instance_type --runtime python3.7 --resource-types AWS::ED2::Instance --input-parameters '{"modify-instance-type":"*"}'
modify-instance-type seemed to be the only thing I could find which related to what I was looking for the lambda function to track and I used the wildcard to signify any changes.
I then added the following to the lambda function:
if configuration_item['resourceType'] != 'AWS::EC2::Instance':
return 'NOT_APPLICABLE'
if configuration_item['configuration']['instanceType'] == valid_rule_parameters['ModifyInstanceAttribute']:
return 'NON_COMPLIANT'
is there a different input-parameter that I should be using for this instead of "modify-instance-type"? so far this has returned nothing. I don't think it is evaluating properly.
or does anyone have a service that might be a better way to track configuration changes like this within aws that I'm just not thinking of?
Related
I have an existing CDK setup in which a CloudFormation distribution is configured using the deprecated CloudFrontWebDistribution API, now I need to configure a OriginRequestPolicy, so after some Googling, switched to the Distribution API (https://docs.aws.amazon.com/cdk/api/latest/docs/aws-cloudfront-readme.html) and reused the same "id" -
Distribution distribution = Distribution.Builder.create(this, "CFDistribution")
When I synth the stack I already see in the yaml that the ID - e.g. CloudFrontCFDistribution12345689 - is a different one than the one before.
When trying to deploy it will fail, since the HTTP Origin CNAMEs are already associated with the existing distribution. ("Invalid request provided: One or more of the CNAMEs you provided are already associated with a different resource. (Service: CloudFront, Status Code: 409, Request ID: 123457657, Extended Request ID: null)"
Is there a way to either add the OriginRequestPolicy (I just want to transfer an additional header) to the CloudFrontWebDistribution or a way to use the new Distribution API while maintaining the existing distribution instead of creating a new one?
(The same operation takes around 3 clicks in the AWS Console).
You could use the following trick to assign the logical ID yourself instead of relying on the autogenerated logical ID. The other option is to execute it in two steps, first update it without the additional CNAME and then do a second update with the additional CNAME.
const cfDistro = new Distribution(this, 'distro', {...});
cfDistro.node.defaultChild.overrideLogicalId('CloudfrontDistribution');
This will result in the following stack:
CloudfrontDistribution:
Type: AWS::CloudFront::Distribution
Properties:
...
Small edit to explain why this happens:
Since you're switching to a new construct, you're also getting a new logical ID. In order to ensure a rollback is possible, CloudFormation will first create all new resources and create the updated resources that need to be recreated. Only when creating and updating everything is done, it will clean up by removing the old resources. This is also the reason why a two-step approach would work when changing the logical IDs of the resources, or force a normal update by ensuring the same logical ID.
Thanks a lot #stijndepestel - simply assigning the existing logical ID worked on the first try.
Here's the Java variant of the code in the answer
import software.amazon.awscdk.services.cloudfront.CfnDistribution;
...
((CfnDistribution) distribution.getNode().getDefaultChild()).overrideLogicalId("CloudfrontDistribution");
I am currently using the boto3 SDK from a Lambda function in order to retrieve various information about the Sagemaker Notebook Instances deployed in my account (almost 70 so not that many...)
One of the operations I am trying to perform is listing the tags for each instance.
However, from time to time it takes ages to return the tags : my Lambda either gets stopped (I could increase the timeout but still...) or a ThrottlingException is raised from the sagemaker.list_tags function (which could be avoided by increasing the number of retry upon sagemaker boto3 client creation) :
sagemaker = boto3.client("sagemaker", config=Config(retries = dict(max_attempts = 10)))
instances_dict = sagemaker.list_notebook_instances()
if not instances_dict['NotebookInstances']:
return "No Notebook Instances"
while instances_dict:
for instance in instances_dict['NotebookInstances']:
print instance['NotebookInstanceArn']
start = time.time()
tags_notebook_instance = sagemaker.list_tags(ResourceArn=instance['NotebookInstanceArn'])['Tags']
print (time.time() - start)
instances_dict = sagemaker.list_notebook_instances(NextToken=instances_dict['NextToken']) if 'NextToken' in instances_dict else None
If you guys have any idea to avoid such delays :)
TY
As you've noted you're getting throttled. Rather than increasing the number of retries you might try to change the delay (i.e. increase the growth_factor). Seems to be configurable looking at https://github.com/boto/botocore/blob/develop/botocore/data/_retry.json#L83
Note that buckets (and refill rates) are usually at the second granularity. So with 70 ARNs you're looking at some number of seconds; double digits does not surprise me.
You might want to consider breaking up the work differently since adding retries/larger growth_factor will just increase the length of time the function will run.
I've had pretty good success at breaking things up so that the Lambda function only processes a single ARN per invocation. The Lambda is processing work (I'll typically use a SQS queue to manage what needs to be processed) and the rate of work is configurable via a combination of configuring the Lambda and the SQS message visibility.
Not know what you're trying to accomplish outside of your original Lambda I realize that breaking up the work this way might (or will) add challenges to what you're doing overall.
It's also worth noting that if you have CloudTrail enabled the tags will be part of the event data (request data) for the "EventName" (which matches the method called, i.e. CreateTrainingJob, AddTags, etc.).
A third option would be if you are trying to find all of the notebook instances with a specific tag then you can use Resource Groups to create a query and find the ARNs with those tags fairly quickly.
CloudTrail: https://docs.aws.amazon.com/awscloudtrail/latest/APIReference/Welcome.html
Resource Groups: https://docs.aws.amazon.com/ARG/latest/APIReference/Welcome.html
Lambda with SQS: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
Seems it's common practice to make use of count on a resource to conditionally create it in Terraform using a ternary statement.
I'd like to conditionally update an AWS Route 53 entry based on a push_to_prod variable. Meaning I don't want to delete the resource if I'm not pushing to production, I only want to update it, or leave the CNAME value as it is.
Has anyone done something like this before in Terraform?
Currently as it stands interpolation syntax isn't supported in lifecycle tags. You can read more here. Which will make this harder because you could use the "Prevent Destroy". However, without more specifics I am going to take my best guess on how to get your there.
I would use the allow_overwrite property on the Route53 record and set that based on your flag. That way if you are pushing to prod you can set it it false. Which should trigger creating a new one. I haven't tested that.
Also note that if you don't make any changes to the Route53 resource it should trigger any changes in Terraform to be applied. So updating any part of the record will trigger the deployment.
You may want to combine this with some lifecycle events, but I don't have enough time to dig into that specific resource and how it happens.
Two examples I can think of are:
type = "${var.push_to_prod == "true" ? "CNAME" : var.other_value}" - this will have a fixed other_value, there is no way to have terraform "ignore" the resource once it's being managed by terraform.
or
type = "${var.aws_route53_record_type}" and you can have dev.tfvars and prod.tfvars, with aws_route53_record_type defined as whatever you want for dev and CNAME for prod.
The thing is with what you're trying to do, "I only want to update it, or leave the CNAME value as it is.", that's not how terraform works. Terraform either manages the resource for you or it doesn't. If it's managing it, it'll update the resource based on the config you've defined in your .tf file. If it's not managing the resource it won't modify it. It sounds like what you're really after is the second solution where you pass in two different configs from your .tfvars file into your .tf file and based off the different configs, different resources are created. You can couple this with count to determine if a resource should be created or not.
Simple question, but I suspect it doesn't have a simple or easy answer. Still, worth asking.
We're creating an implementation for push notifications using AWS with our Web Server running on EC2, sending messages to a queue on SQS, which is dealt with using Lambda, which is sent finally to SNS to be delivered to the iOS/Android apps.
The question I have is this: is there a way to query SNS endpoints based on the custom user data that you can provide on creation? The only way I see to do this so far is to list all the endpoints in a given platform application, and then search through that list for the user data I'm looking for... however, a more direct approach would be far better.
Why I want to do this is simple: if I could attach a User Identifier to these Device Endpoints, and query based on that, I could avoid completely having to save the ARN to our DynamoDB database. It would save a lot of implementation time and complexity.
Let me know what you guys think, even if what you think is that this idea is impractical and stupid, or if searching through all of them is the best way to go about this!
Cheers!
There isn't the ability to have a "where" clause in ListTopics. I see two possibilities:
Create a new SNS topic per user that has some identifiable id in it. So, for example, the ARN would be something like "arn:aws:sns:us-east-1:123456789:know-prefix-user-id". The obvious downside is that you have the potential for a boat load of SNS topics.
Use a service designed for this type of usage like PubNub. Disclaimer - I don't work for PubNub or own stock but have successfully used it in multiple projects. You'll be able to target one or many users this way.
According the the [AWS documentation][1] if you try and create a new Platform Endpoint with the same User Data you should get a response with an exception including the ARN associated with the existing PlatformEndpoint.
It's definitely not ideal, but it would be a round about way of querying the User Data Endpoint attributes via exception.
//Query CustomUserData by exception
CreatePlatformEndpointRequest cpeReq = new CreatePlatformEndpointRequest().withPlatformApplicationArn(applicationArn).withToken("dummyToken").withCustomUserData("username");
CreatePlatformEndpointResult cpeRes = client.createPlatformEndpoint(cpeReq);
You should get an exception with the ARN if an endpoint with the same withCustomUserData exists.
Then you just use that ARN and away you go.
I have a Lambda function invoked by S3 put events, which in turn needs to process the objects and write to a database on RDS. I want to test things out in my staging stack, which means I have a separate bucket, different database endpoint on RDS, and separate IAM roles.
I know how to configure the lambda function's event source and IAM stuff manually (in the Console), and I've read about lambda aliases and versions, but I don't see any support for providing operational parameters (like the name of the destination database) on a per-alias basis. So when I make a change to the function, right now it looks like I need a separate copy of the function for staging and production, and I would have to keep them in sync manually. All of the logic in the code would be the same, and while I get the source bucket and key as a parameter to the function when it's invoked, I don't currently have a way to pass in the destination stuff.
For the destination DB information, I could have a switch statement in the function body that checks the originating S3 bucket and makes a decision, but I hate making every function have to keep that mapping internally. That wouldn't work for the DB credentials or IAM policies, though.
I suppose I could automate all or most of this with the SDK. Has anyone set something like this up for a continuous integration-style deployment with Lambda, or is there a simpler way to do it that I've missed?
I found a workaround using Lambda function aliases. Given the context object, I can get the invoked_function_arn property, which has the alias (if any) at the end.
arn_string = context.invoked_function_arn
alias = arn_string.split(':')[-1]
Then I just use the alias as an index into a dict in my config.py module, and I'm good to go.
config[alias].host
config[alias].database
One thing I'm not crazy about is that I have to invoke my function from an alias every time, and now I can't use aliases for any other purpose without affecting this scheme. It would be nice to have explicit support for user parameters in the context object.