I am deploying the CloudFormation template from this blog post. I had to update the Lambda functions from python 3.6 to 3.9 to get it to work. Now however I get the following error message:
> CloudFormation did not receive a response from your Custom Resource.
> Please check your logs for requestId
> [029f4ea5-cd25-4593-b1ee-d805dd30463f]. If you are using the Python
> cfn-response module, you may need to update your Lambda function code
> so that CloudFormation can attach the updated version.
Below is the lambda code in question - what does it mean to update the Lambda function "so that CloudFormation can attach the updated version"?
import util.cfnresponse
import boto3
import uuid
client = boto3.client('s3')
cfnresponse = util.cfnresponse
def lambda_handler(event, context):
response_data = {}
try:
if event["RequestType"] == "Create":
bucket_name = uuid.uuid4().hex+'-connect'
# response = client.create_bucket(
# Bucket=bucket_name,
# )
response_data["BucketName"] = bucket_name
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
From what I can tell the response format follows the current version of the response module API?
the cfnrespone lib has changed get updated. Old versions of the lib use the request lib. This CF is over 4 years old so it probably don't work due to this.
You can read about the update on the last rows in the README here:
https://github.com/gene1wood/cfnresponse
I am trying to trigger Composer2 DAG from cloud Function gen2 when a Bigquery table is inserted with some records.
Event I am listening to is - google.cloud.bigquery.v2.JobService.InsertJob
And source is - /projects/MYPROJECT/datasets/DATASET/tables/test_trigger.
I am able to get,the trigger to the cloud function,when a record is inserted, but when the cloud fnction is trying to trigger DAG,it throwing following error.
Triggering DAG from cloudfunction gen2,throws error, requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://<composer-url>/api/v1/dags/test-dag/dagRuns
Here is my main.py
from typing import Any
import composer2_airflow_rest_api
def trigger_dag_bq(data, context=None):
web_server_url = (
"https:<URL>composer.googleusercontent.com"
)
dag_id = 'test-dag'
composer2_airflow_rest_api.trigger_dag(web_server_url, dag_id,data)
And composer2_airflow_rest_api.py
from typing import Any
import google.auth
from google.auth.transport.requests import AuthorizedSession
import requests
# Following GCP best practices, these credentials should be
# constructed at start-up time and used throughout
# https://cloud.google.com/apis/docs/client-libraries-best-practices
AUTH_SCOPE = "https://www.googleapis.com/auth/cloud-platform"
CREDENTIALS, _ = google.auth.default(scopes=[AUTH_SCOPE])
def make_composer2_web_server_request(url: str, method: str = "GET", **kwargs: Any) -> google.auth.transport.Response:
"""
Make a request to Cloud Composer 2 environment's web server.
Args:
url: The URL to fetch.
method: The request method to use ('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT',
'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
"""
authed_session = AuthorizedSession(CREDENTIALS)
# Set the default timeout, if missing
if "timeout" not in kwargs:
kwargs["timeout"] = 90
return authed_session.request(method, url, **kwargs)
def trigger_dag(web_server_url: str, dag_id: str, data: dict) -> str:
"""
Make a request to trigger a dag using the stable Airflow 2 REST API.
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
Args:
web_server_url: The URL of the Airflow 2 web server.
dag_id: The DAG ID.
data: Additional configuration parameters for the DAG run (json).
"""
endpoint = f"api/v1/dags/{dag_id}/dagRuns"
request_url = f"{web_server_url}/{endpoint}"
json_data = {"conf": data.decode('utf-8')}
response = make_composer2_web_server_request(
request_url, method="POST", json=json_data
)
if response.status_code == 403:
raise requests.HTTPError(
"You do not have a permission to perform this operation. "
"Check Airflow RBAC roles for your account."
f"{response.headers} / {response.text}"
)
elif response.status_code != 200:
response.raise_for_status()
else:
return response.text
Does anyone know,whats going wrong?
All the resources are in same project.
import this
from constructs import Construct
from aws_cdk import (
Duration,
Stack,
aws_iam as iam,
aws_appsync as appsync,
aws_dynamodb as dynamodb,
aws_lambda as lamb
)
class CdkStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
api = appsync.GraphqlApi(self, "Api",
name="demo",
schema=appsync.Schema.from_asset('graphql/schema.graphql'),
authorization_config=appsync.AuthorizationConfig(
default_authorization=appsync.AuthorizationMode(
authorization_type=appsync.AuthorizationType.IAM
)
),
xray_enabled=True
)
When I run 'cdk synth' I get the error --> AttributeError: module 'aws_cdk.aws_appsync' has no attribute 'GraphqlApi'. Did you mean: 'CfnGraphQLApi'?
However, GraphqlApi is an attribute in the pypi documentation.
You will find the GraphqlApi construct in the aws_cdk.aws_appsync_alpha package.
Not-yet-stable APIs are in separate "alpha" packages in CDK v2. The Appsync constructs are split between 2 packages, aws_cdk.aws_appsync_alpha for "experimental" APIs, and aws_cdk.aws_appsync for stable APIs.
I know this page lists up the instance types which based on Nitro system but I would like to know the list in a dynamic way with CLI. (for example, using aws ec2 describe-instances). Is it possible to get Nitro based instance type other than parsing the static page? If so, could you tell me the how?
You'd have to write a bit of additional code to get that information. aws ec2 describe-instances will give you InstanceType property. You should use a programming language to parse the JSON, extract InstanceType and then call describe-instances like so: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-types.html?highlight=nitro
From the JSON you get back, extract hypervisor. That'll give you Nitro if the instance is Nitro.
Here's a Python code that might work. I have not tested it fully but you can tweak this to get the results you want.
"""List all EC2 instances"""
import boto3
def ec2_connection():
"""Connect to AWS using API"""
region = 'us-east-2'
aws_key = 'xxx'
aws_secret = 'xxx'
session = boto3.Session(
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret
)
ec2 = session.client('ec2', region_name = region)
return ec2
def get_reservations(ec2):
"""Get a list of instances as a dictionary"""
response = ec2.describe_instances()
return response['Reservations']
def process_instances(reservations, ec2):
"""Print a colorful list of IPs and instances"""
if len(reservations) == 0:
print('No instance found. Quitting')
return
for reservation in reservations:
for instance in reservation['Instances']:
# get friendly name of the server
# only try this for mysql1.local server
friendly_name = get_friendly_name(instance)
if friendly_name.lower() != 'mysql1.local':
continue
# get the hypervisor based on the instance type
instance_type = get_instance_info(instance['InstanceType'], ec2)
# print findings
print(f'{friendly_name} // {instance["InstanceType"]} is {instance_type}')
break
def get_instance_info(instance_type, ec2):
"""Get hypervisor from the instance type"""
response = ec2.describe_instance_types(
InstanceTypes=[instance_type]
)
return response['InstanceTypes'][0]['Hypervisor']
def get_friendly_name(instance):
"""Get friendly name of the instance"""
tags = instance['Tags']
for tag in tags:
if tag['Key'] == 'Name':
return tag['Value']
return 'Unknown'
def run():
"""Main method to call"""
ec2 = ec2_connection()
reservations = get_reservations(ec2)
process_instances(reservations, ec2)
if __name__ == '__main__':
run()
print('Done')
In the above answer , the statement "From the JSON you get back, extract hypervisor. That'll give you Nitro if the instance is Nitro " is not longer accurate.
As per the latest AWS documentation,
hypervisor - The hypervisor type of the instance (ovm | xen ). The value xen is used for both Xen and Nitro hypervisors.
Cleaned up, verified working code below:
# Get all instance types that run on Nitro hypervisor
import boto3
def get_nitro_instance_types():
"""Get all instance types that run on Nitro hypervisor"""
ec2 = boto3.client('ec2', region_name = 'us-east-1')
response = ec2.describe_instance_types(
Filters=[
{
'Name': 'hypervisor',
'Values': [
'nitro',
]
},
],
)
instance_types = []
for instance_type in response['InstanceTypes']:
instance_types.append(instance_type['InstanceType'])
return instance_types
get_nitro_instance_types()
Example output as of 12/06/2022 below:
['r5dn.8xlarge', 'x2iedn.xlarge', 'r6id.2xlarge', 'r6gd.medium',
'm5zn.2xlarge', 'r6idn.16xlarge', 'c6a.48xlarge', 'm5a.16xlarge',
'im4gn.2xlarge', 'c6gn.16xlarge', 'c6in.24xlarge', 'r5ad.24xlarge',
'r6i.xlarge', 'c6i.32xlarge', 'x2iedn.2xlarge', 'r6id.xlarge',
'i3en.24xlarge', 'i3en.12xlarge', 'm5d.8xlarge', 'c6i.8xlarge',
'r6g.large', 'm6gd.4xlarge', 'r6a.2xlarge', 'x2iezn.4xlarge',
'c6i.large', 'r6in.24xlarge', 'm6gd.xlarge', 'm5dn.2xlarge',
'd3en.2xlarge', 'c6id.8xlarge', 'm6a.large', 'is4gen.xlarge',
'r6g.8xlarge', 'm6idn.large', 'm6a.2xlarge', 'c6i.4xlarge',
'i4i.16xlarge', 'm5zn.6xlarge', 'm5.8xlarge', 'm6id.xlarge',
'm5n.16xlarge', 'c6g.16xlarge', 'r5n.12xlarge', 't4g.nano',
'm5ad.12xlarge', 'r6in.12xlarge', 'm6idn.12xlarge', 'g5.2xlarge',
'trn1.32xlarge', 'x2gd.8xlarge', 'is4gen.4xlarge', 'r6gd.xlarge',
'r5a.xlarge', 'r5a.2xlarge', 'c5ad.24xlarge', 'r6a.xlarge',
'r6g.medium', 'm6id.12xlarge', 'r6idn.2xlarge', 'c5n.2xlarge',
'g5.4xlarge', 'm5d.xlarge', 'i3en.3xlarge', 'r5.24xlarge',
'r6gd.2xlarge', 'c5d.large', 'm6gd.12xlarge', 'm6id.2xlarge',
'm6i.large', 'z1d.2xlarge', 'm5a.4xlarge', 'm5a.2xlarge',
'c6in.xlarge', 'r6id.16xlarge', 'c7g.8xlarge', 'm5dn.12xlarge',
'm6gd.medium', 'im4gn.8xlarge', 'm5dn.large', 'c5ad.4xlarge',
'r6g.16xlarge', 'c6a.24xlarge', 'c6a.16xlarge']
"""List all EC2 instances"""
import boto3
def ec2_connection():
"""Connect to AWS using API"""
region = 'us-east-2'
aws_key = 'xxx'
aws_secret = 'xxx'
session = boto3.Session(
aws_access_key_id = aws_key,
aws_secret_access_key = aws_secret
)
ec2 = session.client('ec2', region_name = region)
return ec2
def get_reservations(ec2):
"""Get a list of instances as a dictionary"""
response = ec2.describe_instances()
return response['Reservations']
def process_instances(reservations, ec2):
"""Print a colorful list of IPs and instances"""
if len(reservations) == 0:
print('No instance found. Quitting')
return
for reservation in reservations:
for instance in reservation['Instances']:
# get friendly name of the server
# only try this for mysql1.local server
friendly_name = get_friendly_name(instance)
if friendly_name.lower() != 'mysql1.local':
continue
# get the hypervisor based on the instance type
instance_type = get_instance_info(instance['InstanceType'], ec2)
# print findings
print(f'{friendly_name} // {instance["InstanceType"]} is {instance_type}')
break
def get_instance_info(instance_type, ec2):
"""Get hypervisor from the instance type"""
response = ec2.describe_instance_types(
InstanceTypes=[instance_type]
)
return response['InstanceTypes'][0]['Hypervisor']
def get_friendly_name(instance):
"""Get friendly name of the instance"""
tags = instance['Tags']
for tag in tags:
if tag['Key'] == 'Name':
return tag['Value']
return 'Unknown'
def run():
"""Main method to call"""
ec2 = ec2_connection()
reservations = get_reservations(ec2)
process_instances(reservations, ec2)
if name == 'main':
run()
print('Done')
Im using the Method: firewalls.list but when I try filtering I get
"Invalid value for field 'filter': 'sourceranges=[0.0.0.0/0]'. Invalid list filter expression."
from pprint import pprint
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('compute', 'v1', credentials=credentials)
# Project ID for this request.
project = "helloworld-273420" # TODO: Update placeholder value.
sourceRanges = "0.0.0.0/0"
request = service.firewalls().list(project=project, filter=sourceRanges)
while request is not None:
response = request.execute()
for firewall in response['items']:
# TODO: Change code below to process each `firewall` resource:
pprint(firewall)
request = service.firewalls().list_next(previous_request=request, previous_response=response)
You can use instead a gcloud command to filter by sourceRanges:
gcloud compute firewall-rules list --filter="sourceRanges:(0.0.0.0/0)"
For more information check the usage of gcloud compute firewall-rules list:
https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/list
Cheers.
computeService = discovery.build('compute', 'v1', credentials=credentials)
fireruleRequest = computeService.firewalls().list(project=project)
while fireruleRequest is not None:
fireruleList = fireruleRequest.execute()
for firewall in fireruleList['items']:
if firewall['sourceRanges'] == ['0.0.0.0/0']:
pprint(firewall) #prints all fields related to the firewall. Use firewall['field'] for a specific field
# print('error: nothing valid')
fireruleRequest = computeService.firewalls().list_next(previous_request=fireruleRequest, previous_response=fireruleList)
The firewall.list method does not allow filtering of sourceRanges yet
refer to firewall().list docs for more info