Required Cloudformation Script for Blue/Green deployment on ECS - amazon-web-services

I am trying to write a cloud-formation template for AWS ECS with blue green deployment support. This blue-green feature was added recently by AWS in ECS and couldn't find any reference for updating it in cloud-formation template. They have given documentation on, how to do it through UI but not through cloud-formation. I guess, AWS might not updated their cloud-formation documentation as it is a new feature. Any help to find the documentation would be appreciated. Thanking you in advance.

Support for blue/green deployment in CloudFormation has been added. It can be found here in the documentation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-service-deploymentcontroller.html
In the "Type" property you can choose "CODE_DEPLOY" as the deployment type. Hope this helps!

Currently cloudformation does not support the DeploymentController parameter in which you can specify CODE_DEPLOY.
Keep yourself update by visiting this page for documentation updates:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-service.html
For now - use custom cloudformation resource. Use Boto3 library to create the service with CODE_DEPLOY setting. Read more here:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.create_service
This is the python class which can create/delete/upadte ecs:
import boto3
from botocore.exceptions import ClientError
from typing import Any, Dict
client = boto3.client('ecs')
class Service:
#staticmethod
def create(**kwargs) -> Dict[str, Any]:
kwargs = dict(
cluster=kwargs.get('cluster'),
serviceName=kwargs.get('serviceName'),
taskDefinition=kwargs.get('taskDefinition'),
loadBalancers=kwargs.get('loadBalancers'),
serviceRegistries=kwargs.get('serviceRegistries'),
desiredCount=kwargs.get('desiredCount'),
clientToken=kwargs.get('clientToken'),
launchType=kwargs.get('launchType'),
platformVersion=kwargs.get('platformVersion'),
role=kwargs.get('role'),
deploymentConfiguration=kwargs.get('deploymentConfiguration'),
placementConstraints=kwargs.get('placementConstraints'),
placementStrategy=kwargs.get('placementStrategy'),
networkConfiguration=kwargs.get('networkConfiguration'),
healthCheckGracePeriodSeconds=kwargs.get('healthCheckGracePeriodSeconds'),
schedulingStrategy=kwargs.get('schedulingStrategy'),
deploymentController=kwargs.get('deploymentController'),
tags=kwargs.get('tags'),
enableECSManagedTags=kwargs.get('enableECSManagedTags'),
propagateTags=kwargs.get('propagateTags'),
)
kwargs = {key: value for key, value in kwargs.items() if key and value}
return client.create_service(**kwargs)
#staticmethod
def update(**kwargs: Dict[str, Any]) -> Dict[str, Any]:
filtered_kwargs = dict(
cluster=kwargs.get('cluster'),
service=kwargs.get('serviceName'),
desiredCount=kwargs.get('desiredCount'),
taskDefinition=kwargs.get('taskDefinition'),
deploymentConfiguration=kwargs.get('deploymentConfiguration'),
networkConfiguration=kwargs.get('networkConfiguration'),
platformVersion=kwargs.get('platformVersion'),
forceNewDeployment=kwargs.get('forceNewDeployment'),
healthCheckGracePeriodSeconds=kwargs.get('healthCheckGracePeriodSeconds')
)
try:
filtered_kwargs = {key: value for key, value in filtered_kwargs.items() if key and value}
return client.update_service(**filtered_kwargs)
except ClientError as ex:
if ex.response['Error']['Code'] == 'InvalidParameterException':
if 'use aws codedeploy' in ex.response['Error']['Message'].lower():
# For services using the blue/green (CODE_DEPLOY ) deployment controller,
# only the desired count, deployment configuration, and health check grace period
# can be updated using this API. If the network configuration, platform version, or task definition
# need to be updated, a new AWS CodeDeploy deployment should be created.
filtered_kwargs = dict(
cluster=kwargs.get('cluster'),
service=kwargs.get('serviceName'),
desiredCount=kwargs.get('desiredCount'),
deploymentConfiguration=kwargs.get('deploymentConfiguration'),
healthCheckGracePeriodSeconds=kwargs.get('healthCheckGracePeriodSeconds'),
)
filtered_kwargs = {key: value for key, value in filtered_kwargs.items() if key and value}
return client.update_service(**filtered_kwargs)
elif ex.response['Error']['Code'] == 'ServiceNotActiveException':
# We can not update ecs service if it is inactive.
return {'Code': 'ServiceNotActiveException'}
elif ex.response['Error']['Code'] == 'ServiceNotFoundException':
# If for some reason service was not found - don't update and return.
return {'Code': 'ServiceNotFoundException'}
raise
#staticmethod
def delete(**kwargs: Dict[str, Any]) -> Dict[str, Any]:
kwargs = dict(
cluster=kwargs.get('cluster'),
service=kwargs.get('serviceName'),
force=True
)
kwargs = {key: value for key, value in kwargs.items() if key and value}
return client.delete_service(**kwargs)

Related

Cloudformation "update your Lambda function code so that CloudFormation can attach the updated version"

I am deploying the CloudFormation template from this blog post. I had to update the Lambda functions from python 3.6 to 3.9 to get it to work. Now however I get the following error message:
> CloudFormation did not receive a response from your Custom Resource.
> Please check your logs for requestId
> [029f4ea5-cd25-4593-b1ee-d805dd30463f]. If you are using the Python
> cfn-response module, you may need to update your Lambda function code
> so that CloudFormation can attach the updated version.
Below is the lambda code in question - what does it mean to update the Lambda function "so that CloudFormation can attach the updated version"?
import util.cfnresponse
import boto3
import uuid
client = boto3.client('s3')
cfnresponse = util.cfnresponse
def lambda_handler(event, context):
response_data = {}
try:
if event["RequestType"] == "Create":
bucket_name = uuid.uuid4().hex+'-connect'
# response = client.create_bucket(
# Bucket=bucket_name,
# )
response_data["BucketName"] = bucket_name
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
From what I can tell the response format follows the current version of the response module API?
the cfnrespone lib has changed get updated. Old versions of the lib use the request lib. This CF is over 4 years old so it probably don't work due to this.
You can read about the update on the last rows in the README here:
https://github.com/gene1wood/cfnresponse

Triggering DAG from cloudfunction gen2,throws , .HTTPError: 400 Client Error: Bad Request for url: https://<composer-url>/api/v1/dags/test-dag/dagRuns

I am trying to trigger Composer2 DAG from cloud Function gen2 when a Bigquery table is inserted with some records.
Event I am listening to is - google.cloud.bigquery.v2.JobService.InsertJob
And source is - /projects/MYPROJECT/datasets/DATASET/tables/test_trigger.
I am able to get,the trigger to the cloud function,when a record is inserted, but when the cloud fnction is trying to trigger DAG,it throwing following error.
Triggering DAG from cloudfunction gen2,throws error, requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://<composer-url>/api/v1/dags/test-dag/dagRuns
Here is my main.py
from typing import Any
import composer2_airflow_rest_api
def trigger_dag_bq(data, context=None):
web_server_url = (
"https:<URL>composer.googleusercontent.com"
)
dag_id = 'test-dag'
composer2_airflow_rest_api.trigger_dag(web_server_url, dag_id,data)
And composer2_airflow_rest_api.py
from typing import Any
import google.auth
from google.auth.transport.requests import AuthorizedSession
import requests
# Following GCP best practices, these credentials should be
# constructed at start-up time and used throughout
# https://cloud.google.com/apis/docs/client-libraries-best-practices
AUTH_SCOPE = "https://www.googleapis.com/auth/cloud-platform"
CREDENTIALS, _ = google.auth.default(scopes=[AUTH_SCOPE])
def make_composer2_web_server_request(url: str, method: str = "GET", **kwargs: Any) -> google.auth.transport.Response:
"""
Make a request to Cloud Composer 2 environment's web server.
Args:
url: The URL to fetch.
method: The request method to use ('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT',
'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
"""
authed_session = AuthorizedSession(CREDENTIALS)
# Set the default timeout, if missing
if "timeout" not in kwargs:
kwargs["timeout"] = 90
return authed_session.request(method, url, **kwargs)
def trigger_dag(web_server_url: str, dag_id: str, data: dict) -> str:
"""
Make a request to trigger a dag using the stable Airflow 2 REST API.
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
Args:
web_server_url: The URL of the Airflow 2 web server.
dag_id: The DAG ID.
data: Additional configuration parameters for the DAG run (json).
"""
endpoint = f"api/v1/dags/{dag_id}/dagRuns"
request_url = f"{web_server_url}/{endpoint}"
json_data = {"conf": data.decode('utf-8')}
response = make_composer2_web_server_request(
request_url, method="POST", json=json_data
)
if response.status_code == 403:
raise requests.HTTPError(
"You do not have a permission to perform this operation. "
"Check Airflow RBAC roles for your account."
f"{response.headers} / {response.text}"
)
elif response.status_code != 200:
response.raise_for_status()
else:
return response.text
Does anyone know,whats going wrong?
All the resources are in same project.

AWS CDK Issue with Appsync

import this
from constructs import Construct
from aws_cdk import (
Duration,
Stack,
aws_iam as iam,
aws_appsync as appsync,
aws_dynamodb as dynamodb,
aws_lambda as lamb
)
class CdkStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
api = appsync.GraphqlApi(self, "Api",
name="demo",
schema=appsync.Schema.from_asset('graphql/schema.graphql'),
authorization_config=appsync.AuthorizationConfig(
default_authorization=appsync.AuthorizationMode(
authorization_type=appsync.AuthorizationType.IAM
)
),
xray_enabled=True
)
When I run 'cdk synth' I get the error --> AttributeError: module 'aws_cdk.aws_appsync' has no attribute 'GraphqlApi'. Did you mean: 'CfnGraphQLApi'?
However, GraphqlApi is an attribute in the pypi documentation.
You will find the GraphqlApi construct in the aws_cdk.aws_appsync_alpha package.
Not-yet-stable APIs are in separate "alpha" packages in CDK v2. The Appsync constructs are split between 2 packages, aws_cdk.aws_appsync_alpha for "experimental" APIs, and aws_cdk.aws_appsync for stable APIs.

How to get the list of Nitro system based EC2 instance type by CLI?

I know this page lists up the instance types which based on Nitro system but I would like to know the list in a dynamic way with CLI. (for example, using aws ec2 describe-instances). Is it possible to get Nitro based instance type other than parsing the static page? If so, could you tell me the how?
You'd have to write a bit of additional code to get that information. aws ec2 describe-instances will give you InstanceType property. You should use a programming language to parse the JSON, extract InstanceType and then call describe-instances like so: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-types.html?highlight=nitro
From the JSON you get back, extract hypervisor. That'll give you Nitro if the instance is Nitro.
Here's a Python code that might work. I have not tested it fully but you can tweak this to get the results you want.
"""List all EC2 instances"""
import boto3
def ec2_connection():
"""Connect to AWS using API"""
region = 'us-east-2'
aws_key = 'xxx'
aws_secret = 'xxx'
session = boto3.Session(
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret
)
ec2 = session.client('ec2', region_name = region)
return ec2
def get_reservations(ec2):
"""Get a list of instances as a dictionary"""
response = ec2.describe_instances()
return response['Reservations']
def process_instances(reservations, ec2):
"""Print a colorful list of IPs and instances"""
if len(reservations) == 0:
print('No instance found. Quitting')
return
for reservation in reservations:
for instance in reservation['Instances']:
# get friendly name of the server
# only try this for mysql1.local server
friendly_name = get_friendly_name(instance)
if friendly_name.lower() != 'mysql1.local':
continue
# get the hypervisor based on the instance type
instance_type = get_instance_info(instance['InstanceType'], ec2)
# print findings
print(f'{friendly_name} // {instance["InstanceType"]} is {instance_type}')
break
def get_instance_info(instance_type, ec2):
"""Get hypervisor from the instance type"""
response = ec2.describe_instance_types(
InstanceTypes=[instance_type]
)
return response['InstanceTypes'][0]['Hypervisor']
def get_friendly_name(instance):
"""Get friendly name of the instance"""
tags = instance['Tags']
for tag in tags:
if tag['Key'] == 'Name':
return tag['Value']
return 'Unknown'
def run():
"""Main method to call"""
ec2 = ec2_connection()
reservations = get_reservations(ec2)
process_instances(reservations, ec2)
if __name__ == '__main__':
run()
print('Done')
In the above answer , the statement "From the JSON you get back, extract hypervisor. That'll give you Nitro if the instance is Nitro " is not longer accurate.
As per the latest AWS documentation,
hypervisor - The hypervisor type of the instance (ovm | xen ). The value xen is used for both Xen and Nitro hypervisors.
Cleaned up, verified working code below:
# Get all instance types that run on Nitro hypervisor
import boto3
def get_nitro_instance_types():
"""Get all instance types that run on Nitro hypervisor"""
ec2 = boto3.client('ec2', region_name = 'us-east-1')
response = ec2.describe_instance_types(
Filters=[
{
'Name': 'hypervisor',
'Values': [
'nitro',
]
},
],
)
instance_types = []
for instance_type in response['InstanceTypes']:
instance_types.append(instance_type['InstanceType'])
return instance_types
get_nitro_instance_types()
Example output as of 12/06/2022 below:
['r5dn.8xlarge', 'x2iedn.xlarge', 'r6id.2xlarge', 'r6gd.medium',
'm5zn.2xlarge', 'r6idn.16xlarge', 'c6a.48xlarge', 'm5a.16xlarge',
'im4gn.2xlarge', 'c6gn.16xlarge', 'c6in.24xlarge', 'r5ad.24xlarge',
'r6i.xlarge', 'c6i.32xlarge', 'x2iedn.2xlarge', 'r6id.xlarge',
'i3en.24xlarge', 'i3en.12xlarge', 'm5d.8xlarge', 'c6i.8xlarge',
'r6g.large', 'm6gd.4xlarge', 'r6a.2xlarge', 'x2iezn.4xlarge',
'c6i.large', 'r6in.24xlarge', 'm6gd.xlarge', 'm5dn.2xlarge',
'd3en.2xlarge', 'c6id.8xlarge', 'm6a.large', 'is4gen.xlarge',
'r6g.8xlarge', 'm6idn.large', 'm6a.2xlarge', 'c6i.4xlarge',
'i4i.16xlarge', 'm5zn.6xlarge', 'm5.8xlarge', 'm6id.xlarge',
'm5n.16xlarge', 'c6g.16xlarge', 'r5n.12xlarge', 't4g.nano',
'm5ad.12xlarge', 'r6in.12xlarge', 'm6idn.12xlarge', 'g5.2xlarge',
'trn1.32xlarge', 'x2gd.8xlarge', 'is4gen.4xlarge', 'r6gd.xlarge',
'r5a.xlarge', 'r5a.2xlarge', 'c5ad.24xlarge', 'r6a.xlarge',
'r6g.medium', 'm6id.12xlarge', 'r6idn.2xlarge', 'c5n.2xlarge',
'g5.4xlarge', 'm5d.xlarge', 'i3en.3xlarge', 'r5.24xlarge',
'r6gd.2xlarge', 'c5d.large', 'm6gd.12xlarge', 'm6id.2xlarge',
'm6i.large', 'z1d.2xlarge', 'm5a.4xlarge', 'm5a.2xlarge',
'c6in.xlarge', 'r6id.16xlarge', 'c7g.8xlarge', 'm5dn.12xlarge',
'm6gd.medium', 'im4gn.8xlarge', 'm5dn.large', 'c5ad.4xlarge',
'r6g.16xlarge', 'c6a.24xlarge', 'c6a.16xlarge']
"""List all EC2 instances"""
import boto3
def ec2_connection():
"""Connect to AWS using API"""
region = 'us-east-2'
aws_key = 'xxx'
aws_secret = 'xxx'
session = boto3.Session(
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret
)
ec2 = session.client('ec2', region_name = region)
return ec2
def get_reservations(ec2):
"""Get a list of instances as a dictionary"""
response = ec2.describe_instances()
return response['Reservations']
def process_instances(reservations, ec2):
"""Print a colorful list of IPs and instances"""
if len(reservations) == 0:
print('No instance found. Quitting')
return
for reservation in reservations:
for instance in reservation['Instances']:
# get friendly name of the server
# only try this for mysql1.local server
friendly_name = get_friendly_name(instance)
if friendly_name.lower() != 'mysql1.local':
continue
# get the hypervisor based on the instance type
instance_type = get_instance_info(instance['InstanceType'], ec2)
# print findings
print(f'{friendly_name} // {instance["InstanceType"]} is {instance_type}')
break
def get_instance_info(instance_type, ec2):
"""Get hypervisor from the instance type"""
response = ec2.describe_instance_types(
InstanceTypes=[instance_type]
)
return response['InstanceTypes'][0]['Hypervisor']
def get_friendly_name(instance):
"""Get friendly name of the instance"""
tags = instance['Tags']
for tag in tags:
if tag['Key'] == 'Name':
return tag['Value']
return 'Unknown'
def run():
"""Main method to call"""
ec2 = ec2_connection()
reservations = get_reservations(ec2)
process_instances(reservations, ec2)
if name == 'main':
run()
print('Done')

how do you filter a list of firewall rules by sourceRanges 0.0.0.0/0 using firewalls.list Method

Im using the Method: firewalls.list but when I try filtering I get
"Invalid value for field 'filter': 'sourceranges=[0.0.0.0/0]'. Invalid list filter expression."
from pprint import pprint
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('compute', 'v1', credentials=credentials)
# Project ID for this request.
project = "helloworld-273420" # TODO: Update placeholder value.
sourceRanges = "0.0.0.0/0"
request = service.firewalls().list(project=project, filter=sourceRanges)
while request is not None:
response = request.execute()
for firewall in response['items']:
# TODO: Change code below to process each `firewall` resource:
pprint(firewall)
request = service.firewalls().list_next(previous_request=request, previous_response=response)
You can use instead a gcloud command to filter by sourceRanges:
gcloud compute firewall-rules list --filter="sourceRanges:(0.0.0.0/0)"
For more information check the usage of gcloud compute firewall-rules list:
https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/list
Cheers.
computeService = discovery.build('compute', 'v1', credentials=credentials)
fireruleRequest = computeService.firewalls().list(project=project)
while fireruleRequest is not None:
fireruleList = fireruleRequest.execute()
for firewall in fireruleList['items']:
if firewall['sourceRanges'] == ['0.0.0.0/0']:
pprint(firewall) #prints all fields related to the firewall. Use firewall['field'] for a specific field
# print('error: nothing valid')
fireruleRequest = computeService.firewalls().list_next(previous_request=fireruleRequest, previous_response=fireruleList)
The firewall.list method does not allow filtering of sourceRanges yet
refer to firewall().list docs for more info