Is there a way to unit test AWS Cloudformation template - unit-testing

Question

When we say that cloudformation is 'Infrastructure as Code', the next question that immediately comes to mind is how can this code be tested.
Can we do some sort of basic unit test of this code
And I am discounting the cloudformation validation because that just is a way of doing syntactic validation, and that I can do with any other free JSON/YAML validator.
I am more inclined towards some sort of functional validation, possibly testing that I have defined all the variables that are used as references.
Possibly testing that whatever properties I am using are actually supported ones for that component
Not expected that it should test if the permissions are correct or that I have not exhausted my limits. But atleast something beyond the basic JSON/YAML syntax validation

Answer 1

Here's a breakdown of how several methods of testing software can be applied to CloudFormation templates/stacks:
Linting
For linting (checking CloudFormation-template code for syntax/grammar correctness), you can use the ValidateTemplate API to check basic template structure, and the CreateChangeSet API to verify your Resource properties in more detail.
Note that ValidateTemplate performs a much more thorough check than a simple JSON/YAML syntax checker- it validates correct Template Anatomy, correct syntax/usage of Intrinsic Functions, and correct resolution of all Ref values.
ValidateTemplate checks basic CloudFormation syntax, but doesn't verify your template's Resources against specific property schemas. For checking the structure of your template's Parameters, Resources and Properties against AWS Resource types, CreateChangeSet should return an error if any parameters or resource properties are not well-formed.
Unit testing
Performing unit testing first requires an answer to the question: what is the smallest self-contained unit of functionality that can/should be tested? For CloudFormation, I believe that the smallest testable unit is the Resource.
The official AWS Resource Types are supported/maintained by AWS (and are proprietary implementations anyway) so don't require any additional unit tests written by end-user developers.
However, your own Custom Resources could and should be unit-tested. This can be done using a suitable testing framework in the implementation's own language (e.g., for Lambda-backed Custom Resources, perhaps a library like lambda-tester would be a good starting point).
Integration testing
This is the most important and relevant type of testing for CloudFormation stacks (which mostly serve to tie various Resources together into an integrated application), and also the type that could use more refinement and best-practice development. Here are some initial ideas on how to integration-test CloudFormation code by actually creating/updating full stacks containing real AWS resources:
Using a scripting language, perform a CloudFormation stack creation using the language's AWS SDK. Design the template to return Stack Outputs reflecting behavior that you want to test. After the stack is created by the scripting language, compare the stack outputs against expected values (and then optionally delete the stack afterwards in a cleanup process).
Use AWS::CloudFormation::WaitCondition resources to represent successful tests/assertions, so that a successful stack creation indicates a successful integration-test run, and a failed stack creation indicates a failed integration-test run.
Beyond CloudFormation, one interesting tool worth mentioning in the space of testing infrastructure-as-code is kitchen-terraform, a set of plugins for Test Kitchen which allow you to write fully-automated integration test suites for Terraform modules. A similar integration-testing harness could eventually be built for CloudFormation, but doesn't exist yet.

Answer 2

This tool “cfn-nag” parses a collection of CloudFormation templates and applies rules to find code patterns that could lead to insecure infrastructure.  The results of the tool include the logical resource identifiers for violating resources and an explanation of what rule has been violated.
Further Reading: https://stelligent.com/2016/04/07/finding-security-problems-early-in-the-development-process-of-a-cloudformation-template-with-cfn-nag/
While there are quite a number of particular rules the tool will attempt to match, the rough categories are:
IAM and resource policies (S3 Bucket, SQS, etc.)
Matches policies that are overly permissive in some way (e.g. wildcards in actions or principals)
Security Group ingress and egress rules
Matches rules that are overly liberal (e.g. an ingress rule open to 0.0.0.0/0, port range 1-65535 is open)
Access Logs
Looks for access logs that are not enabled for applicable resources (e.g. Elastic Load Balancers and CloudFront Distributions)
Encryption
(Server-side) encryption that is not enabled or enforced for applicable resources (e.g. EBS volumes or for PutObject calls on an S3 bucket)

Answer 3

New tool is on the market now. Test all the CloudFormation things! (with TaskCat)
What is taskcat?
taskcat is a tool that tests AWS CloudFormation templates. It deploys your AWS CloudFormation template in multiple AWS Regions and generates a report with a pass/fail grade for each region. You can specify the regions and number of Availability Zones you want to include in the test, and pass in parameter values from your AWS CloudFormation template. taskcat is implemented as a Python class that you import, instantiate, and run.
usage
follow this document : https://aws.amazon.com/blogs/infrastructure-and-automation/up-your-aws-cloudformation-testing-game-using-taskcat/
notes
taskcat can't read AWS_PROFILE environment variable. you need define the profile in part of general if it is not default profile.
general:
auth:
default: dev-account
Ref: https://github.com/aws-quickstart/taskcat/issues/434

Answer 4

The testing as you described (at least beyond JSON parsing) can be achieved partially by parsing CloudFormation templates by programmatic libraries that are used to generate/read templates. They do not test the template explicitly but can throw an exception or error upon conditions where you use a property that is not defined for a resource.
Check out go-cloudformation: https://github.com/crewjam/go-cloudformation
Other than that you need to run the stack to see errors. I believe that testing IaaC is one of the main challenges in infrastructure automation. Not only unit testing but also integration testing and continuous validation.

Answer 5

Speaking specifically of CloudFormation, AWS recommends using the taskcat, which is a tool that deploys the infrastructure / template within all AWS regions, in this process it already performs code validation.
TaskCat Github repository: https://github.com/aws-quickstart/taskcat
In addition, through the Visual Studio code you can use the extension Cloud conformity template scanner or use the feature that has been currently purchased by trend micro to make security validations whose name has changed from cloud conformity to Trend Micro Template scanner.
It will basically perform the validation of the template and architectural code linked to the model and use case of the Well Architected Framework from AWS.
About template scanner: https://aws.amazon.com/blogs/apn/using-shift-left-to-find-vulnerabilities-before-deployment-with-trend-micro-template-scanner/
The VS Code Extension Cloud Conformity: https://marketplace.visualstudio.com/items?itemName=raphaelbottino.cc-template-scanner
In addition, there is a VS Code extension Linter that you can use as a pre-commit for validation, the name is: CloudFormation Linter.
CloudFormation Linter: https://marketplace.visualstudio.com/items?itemName=kddejong.vscode-cfn-lint
You can also use more advanced features if you want to implement an infra as code pipeline using DevSecOps "SEC", this is Scout suite. It has its own validator for the cloud that can be run in a build container, it will audit the cloud to validate if there are resources that are outside a security standard.
Scout Suite Github repository: https://github.com/nccgroup/ScoutSuite
If you want to go deeper in the case of using validation and resource testing / compliance on AWS, I recommend you study about 'Compliance as code' using the config service.
Link to a presentation of this service: https://www.youtube.com/watch?v=fBewaclMo2s

Answer 6

I couldn't find a real unit testing solution for Cloudformation templates so I created one. https://github.com/DontShaveTheYak/cloud-radar
Cloud-Radar lets you take a template, pass in the parameters you want to set. Then render that template to its final form. Meaning all conditionals and intrinsic functions are solved.
This allows you to take a template like this and write the following tests:
from pathlib import Path
from unittest.mock import mock_open, patch
import pytest
from cloud_radar.cf.unit import Template
#pytest.fixture
def template():
template_path = Path(__file__).parent / "../../templates/log_bucket/log_bucket.yaml"
return Template.from_yaml(template_path.resolve(), {})
def test_log_defaults(template):
result = template.render({"BucketPrefix": "testing"})
assert "LogsBucket" in result["Resources"]
bucket_name = result["Resources"]["LogsBucket"]["Properties"]["BucketName"]
assert "us-east-1" in bucket_name
def test_log_retain(template):
result = template.render(
{"BucketPrefix": "testing", "KeepBucket": "TRUE"}, region="us-west-2"
)
assert "LogsBucket" not in result["Resources"]
bucket = result["Resources"]["RetainLogsBucket"]
assert "DeletionPolicy" in bucket
assert bucket["DeletionPolicy"] == "Retain"
bucket_name = bucket["Properties"]["BucketName"]
assert "us-west-2" in bucket_name
Edit: If you are interested in testing Cloudformation templates, then checkout my blog series Hypermodern Cloudformation.

Answer 7

There's a bash library xsh-lib/aws, a tool of it can deploy AWS CloudFormation templates from CLI.
The tool can be found at: https://github.com/xsh-lib/aws/blob/master/functions/cfn/deploy.sh
It handles template validation and uploading, stack naming, policing, updating, cleaning, time outing, rollbacking, and status checking.
Besides the above, it also handles nested templates and non-inline Lambdas. That saves the work for uploading nested templates and non-inline Lambda functions which could drive people nuts if doing tests manually.
It supports a customized config file which is making the deployment easier. A real-world config file is here.
A simple example call of the tool from your CLI looks like this:
xsh aws/cfn/deploy -t template.json -s mystack -o OPTIONS=Param1Key=Param1Value
xsh-lib/aws is a library of xsh, in order to use it you will have to install xsh first.