Is it possible to run a managed AWS Config rule (for example https://docs.aws.amazon.com/config/latest/developerguide/root-account-mfa-enabled.html) on demand via the SDK?
Scenario:
As a consultant I want to easily assess a customer's environment without spending time applying all the AWS Config rules to my customer's environment. Instead I want to use the SDK to quickly execute many rules and get the results back.
Is this possible?
Cloud Custodian
For adhoc execution you might be best served by evaluating Cloud Custodian instead. When I tried this out previously, I was pretty impressed with the immediate value I could get with minimal deployment.
The adhoc nature of your execution can benefit as you can run a report only action, or actually have it create lambda functions to remediate in certain cases if you need that.
The tool is cross platform, dockerized as well, and most of the configuration for rules is yaml based, supporting AWS Config, Security Hub, AWS SSM, and more.
If you look at the Run Your First Policy section in AWS you'll see it can be as simple as:
AWS_ACCESS_KEY_ID="foo" AWS_SECRET_ACCESS_KEY="bar" custodian run --output-dir=. custodian.yml
There is a pretty extensive list of example-policies which include items like
AWS Config Integration
Can be deployed as config-rule for any resource type supported by config.
Can use config as resource database instead of querying service describe apis. Custodian supports server side querying resources with Config’s SQL expression language.
Can filter resources based on their compliance with one or more config rules.
Can be deployed as a config-poll-rule against any resource type supported by cloudformation.
source: AWS Config Integration
It supports custom config rules as well.
Note: I'm not involved in the project, just found it useful and promising for similar situations as you describe. Seems to reduce a lot of "DevOps plumbing" required to get value out of several AWS services with far less service specific knowledge and setup required.
Related
Is there some way to 'dehydrate' or extract an entire AWS setup? I have a small application that uses several AWS components, and I'd like to put the project on hiatus so I don't get charged every month.
I wrote / constructed the app directly through the various services' sites, such as VPN, RDS, etc. Is there some way I can extract my setup into files so I can save these files in Version Control, and 'rehydrate' them back into AWS when I want to re-setup my app?
I tried extracting pieces from Lambda and Event Bridge, but it seems like I can't just 'replay' these files using the CLI to re-create my application.
Specifically, I am looking to extract all code, settings, connections, etc. for:
Lambda. Code, Env Variables, layers, scheduling thru Event Bridge
IAM. Users, roles, permissions
VPC. Subnets, Route tables, Internet gateways, Elastic IPs, NAT Gateways
Event Bridge. Cron settings, connections to Lambda functions.
RDS. MySQL instances. Would like to get all DDL. Data in tables is not required.
Thanks in advance!
You could use Former2. It will scan your account and allow you to generate CloudFormation, Terraform, or Troposphere templates. It uses a browser plugin, but there is also a CLI for it.
What you describe is called Infrastructure as Code. The idea is to define your infrastructure as code and then deploy your infrastructure using that "code".
There are a lot of options in this space. To name a few:
Terraform
Cloudformation
CDK
Pulumi
All of those should allow you to import already existing resources. At least Terraform has a import command to import an already existing resource into your IaC project.
This way you could create a project that mirrors what you currently have in AWS.
Excluded are things that are strictly taken not AWS resources, like:
Code of your Lambdas
MySQL DDL
Depending on the Lambdas deployment "strategy" the code is either on S3 or was directly deployed to the Lambda service. If it is the first, you just need to find the S3 bucket etc and download the code from there. If it is the second you might need to copy and paste it by hand.
When it comes to your MySQL DDL you need to find tools to export that. But there are plenty tools out there to do this.
After you did that, you should be able to destroy all the AWS resources and then deploy them later on again from your new IaC.
I've been building serverless applications on AWS for the past few years, utilizing services such as Lambda, DynamoDB, SNS, SQS, Kinesis, etc., relying on the Serverless framework for local development and deployments. Due to professional reasons, I have now to switch to Google Cloud and I've been exploring the serverless space in that platform. Unfortunately, at first glance it doesn't seem to be as mature as AWS, which I don't know whether it's true or just caused by my lack of expertise. The reasons that make me claim that are basically the following:
There is no logical grouping of functions and resources: on AWS, Lambda functions are grouped in Applications, and can be deployed as a whole via SAM or the Serverless framework, which also allow creating any associated resource (databases, queues, event buses, etc.). It seems that on GCP functions are treated as individual entities, which makes managing them and orchestrating them harder.
Lack of tooling: both the SAM cli and the Serverless framework provide tools for local development and deployments. I haven't found anything on GCP like the former (the Functions Framework seems to cover it partially, but it doesn't handle deployments), and even though that the latter supports GCP, it's missing basic features, such as creating resources other than functions. What is more, GCP is not in the core framework and the plugin is looking for maintainers.
Less event sources: Lambda is directly integrated with a long list of services. On the other hand, Cloud Functions is integrated with just a few services, making HTTP triggers the only option on most cases. It seems they're trying to address this issue with Eventarc, but I don't think it's generally available yet.
Does anybody have any tips on how to setup a local environment for this kind of applications and how to manage them effectively?
Here some documentation that might be helpful for your case, even though required to take a deep look into it.
Configure Serverless VPC Access (which i think applies for 'setting up your local environment').
Cloud Run Quick start (which contains how to built and deploy serverless services with GCP Cloud Run using node.js, python, java, etc.
I am familiar with AWS SDKs (Python/Java) and the need for the same for a lot of services like S3, DDB, KMS etc.
Are there any valid use cases for using the AWS Java SDK e.g. to programmatically spin off an EC2 instance or an EMR cluster or Redshift cluster or RDS instance or for that matter any resource that requires setting up of an infrastructure/cluster?
If AWS console is not an option, and programmatic access is all we have then, don't we always end up using AWS CLI for corresponding services or CloudFormation or Terraform for that matter?
Generally its best practice to manage any infrastructure/service deployments by using an infrastructure as code solutions such as CloudFormation, CDK (which generates CloudFormation stacks under the hood) or Terraform.
Whilst you could use the SDK to create this services (and create a solution that is similar to the solutions above) you will be building a lot of functionality that other services have already created which would put more ownership on you to fix it if you want to support another service etc.
The SDKs simply provide every AWS API interaction in a programmatic way, even under the hood CloudFormation and Terraform will likely be using the SDKs to programmatically create the resources (although I am speculating) but would then add additional functionality such as state management and drift detection on top of this.
I only have seen services being created via the SDKs when that service is not available in the selected tool, and even then it would generally be wrapped within that tool (such as custom resources for CloudFormation).
In summary, yes you could use the SDK to generate these but unless there is a specific usecase to use the SDK I would advise using a tool that already manages this so you can focus more on your infrastructure/applications.
The AWS CLI is built using the AWS SDK for Python. Terraform is built using the AWS SDK for GoLang. You may want to stick with higher level infrastructure-as-code tools, but those tools wouldn't exist without the SDKs, and if you wanted to build a tool like that you would most likely build it on top of one of the SDKs.
There are also quite a few use-cases I've seen discussed here on StackOverflow for performing infrastructure automation through AWS Lambda, for example periodically stopping and starting RDS instances, where using one of the AWS SDKs in the Lambda code would likely be much easier than trying to get Terraform to run inside a Lambda function.
I am having a unique opportunity to suggest a workflow for IaC for a part of a big company which has number of technical agencies working for it.
I am trying to work out a solution that would be enterprise-level safe but have as much self-service as possible.
In scope:
Code management [repository per project/environment/agency/company]
Environment handling [build promotion/statefile per env, one statefile, terraform envs etc]
Governance model [Terraform Enterprise/PR system/custom model]
Testing and acceptance [manual acceptance/automated tests(how to test tf files?)/infra test environment]
I have read many articles, but most of them describe a situation of a development team in-house, which is much easier in terms of security and governance.
I would love to learn how what is the optimal solution for IaC management and govenance in enterprise. Is Terraform Enterprise a valid option?
I recommend using Terraform modules as Enterprise "libraries" for (infrastructure) code.
Then you can:
version, test, and accept your libraries at the Enterprise level
control what variables developers or clients can set (e.g. provide a module for AWS S3 buckets with configurable bucket name, but restricted ACL options)
provide abstractions over complex, repeated configurations to save time, prevent errors and encourage self-service (e.g. linking AWS API Gateway with AWS Lambda and Dynamodb)
For governance, it helps to have controlled cloud provider accounts or environments where every resource is deployed from scratch via Terraform (in addition to sandboxes where users can experiment manually).
For example, you could:
deploy account-level settings from Terraform (e.g. AWS password policy)
tag all Enterprise module resources automatically with
the person who last deployed changes (e.g. AWS caller ID)
the environment they used (with Terraform interpolation: "${terraform.workspace}")
So, there are lots of ways to use Terraform modules to empower your clients / developers without giving up Enterprise controls.
I am new to AWS (Amazon Web Services) as well as our own custom boto based python deployment scripts, but wanted to ask for advice or best practices for a simple configuration management task. We have a simple web application with configuration data for several different backend environments controlled by a command line -D defined java environment variable. Sometimes, the requirement comes up that we need to switch from one backend environment to another due to maintenance or deployment schedules of our backend services.
The current procedure requires python scripts to completely destroy and rebuild all the virtual infrastructure (load balancers, auto scale groups, etc.) to redeploy the application with a change to the command line parameter. On a traditional server infrastructure, we would log in to the management console of the container, change the variable, bounce the container, and we're done.
Is there a best practice for this operation on AWS environments, or is the complete destruction and rebuilding of all the pieces the only way to accomplish this task in an AWS environment?
It depends on what resources you have to change. AWS is evolving everyday in a fast paced manner. I would suggest you to take a look at the AWS API for the resources you need to deal with and check if you can change a resource without destroying it.
Ex: today you cannot change a Launch Group once it is created. you must delete it and create it again with the new configurations. but if you have one auto scaling group attached to that launch group you will have to delete the auto scaling group and so on.
IMHO a see no problems with your approach, but as I believe that there is always room for improvement, I think you can refactor it with the help of AWS API documentation.
HTH
I think I found the answer to my own question. I know the interface to AWS is constantly changing, and I don't think this functionality is available yet in the Python boto library, but the ability I was looking for is best described as "Modifying Attributes of a Stopped Instance" with --user-data as being the attribute in question. Documentation for performing this action using HTTP requests and the command line interface to AWS can be found here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_ChangingAttributesWhileInstanceStopped.html