How can I mock ECS with moto? - amazon-web-services

I want to create a mock ECS cluster, but it seems not to work properly. Although something is mocked (I don't get a credentials error), it seems not to "save" the cluster.
How can I create a mock cluster with moto?
MVCE
foo.py
import boto3
def print_clusters():
client = boto3.client("ecs")
print(client.list_clusters())
return client.list_clusters()["clusterArns"]
test_foo.py
import boto3
import pytest
from moto import mock_ecs
import foo
#pytest.fixture
def ecs_cluster():
with mock_ecs():
client = boto3.client("ecs", region_name="us-east-1")
response = client.create_cluster(clusterName="test_ecs_cluster")
yield client
def test_foo(ecs_cluster):
assert foo.print_clusters() == ["test_ecs_cluster"]
What happens
$ pytest test_foo.py
Test session starts (platform: linux, Python 3.8.1, pytest 5.3.5, pytest-sugar 0.9.2)
rootdir: /home/math/GitHub
plugins: black-0.3.8, mock-2.0.0, cov-2.8.1, mccabe-1.0, flake8-1.0.4, env-0.6.2, sugar-0.9.2, mypy-0.5.0
collecting ...
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_foo ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ecs_cluster = <botocore.client.ECS object at 0x7fe9b0c73580>
def test_foo(ecs_cluster):
> assert foo.print_clusters() == ["test_ecs_cluster"]
E AssertionError: assert [] == ['test_ecs_cluster']
E Right contains one more item: 'test_ecs_cluster'
E Use -v to get the full diff
test_foo.py:19: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------------
{'clusterArns': [], 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
test_foo.py ⨯
What I expected
I expected the list of cluster ARNs to have one element (not the one in the assert statement, but an ARN). But the list is empty.

When creating a cluster, you're using a mocked ECS client.
When listing the clusters, you're creating a new ECS client outside the scope of moto.
In other words, you're creating a cluster in memory - but then ask AWS itself for a list of clusters.
You could rewrite the foo-method to use the mocked ECS client:
def print_clusters(client):
print(client.list_clusters())
return client.list_clusters()["clusterArns"]
def test_foo(ecs_cluster):
assert foo.print_clusters(ecs_cluster) == ["test_ecs_cluster"]

def test_foo(ecs_cluster):
assert foo.print_clusters(ecs_cluster) == ["test_ecs_cluster"]
#This will cause you a bug but I have fixed this bug ..so the code looks like this:
def cluster_list(ecs_cluster):
assert ecs.print_clusters(ecs_cluster) == ['arn:aws:ecs:us-east-1:123456789012:cluster/test_ecs_cluster']
#Explination
So basically you have passed the incorrect assert values..assert foo.print_clusters(ecs_cluster) --> this containes cluster arns it is in the form of an array which is ['arn:aws:ecs:us-east-1:123456789012:cluster/test_ecs_cluster'] and you are trying to access the [1] index and testing if its == "test_ecs_cluster" which is wrong so instead to that pass the full arn just to test your code ..

Related

Pytest on Flask based API - test by calling the remote API

New to using Pytest on APIs. From my understanding, testing creates another instance of Flask. Additionally, from the tutorials I have seen, they also suggest to create a separate DB table instance to add, fetch and remove data for test purposes. However, I simply plan to use the remote api URL as host to simply make the call.
Now, I set my conftest like this, where the flag --testenv would indicate to make the get/post call on the host listed below:
import pytest
import subprocess
def pytest_addoption(parser):
"""Add option to pass --testenv=api_server to pytest cli command"""
parser.addoption(
"--testenv", action="store", default="exodemo", help="my option: type1 or type2"
)
#pytest.fixture(scope="module")
def testenv(request):
return request.config.getoption("--testenv")
#pytest.fixture(scope="module")
def testurl(testenv):
if testenv == 'api_server':
return 'http://api_url:5000/'
else:
return 'http://locahost:5000'
And my test file is written like this:
import pytest
from app import app
from flask import request
def test_nodes(app):
t_client = app.test_client()
truth = [
{
*body*
}
]
res = t_client.get('/topology/nodes')
print (res)
assert res.status_code == 200
assert truth == json.loads(res.get_data)
I run the code using this:
python3 -m pytest --testenv api_server
The thing I expect is that the test file would simply make a call to the remote api with the creds, fetch the data regardless of how it gets pulled in the remote code, and bring it here for assertion. However, I am getting the 400 BAD REQUEST error, with the error being like this:
assert 400 == 200
E + where 400 = <WrapperTestResponse streamed [400 BAD REQUEST]>.status_code
single_test.py:97: AssertionError
--------------------- Captured stdout call ----------------------
{"timestamp": "2022-07-28 22:11:14,032", "level": "ERROR", "func": "connect_to_mysql_db", "line": 23, "message": "Error connecting to the mysql database (2003, \"Can't connect to MySQL server on 'mysql' ([Errno -3] Temporary failure in name resolution)\")"}
<WrapperTestResponse streamed [400 BAD REQUEST]>
Does this mean that the test file is still trying to lookup the database locally for fetching? I am unable to figure out on which host are they sending the test url as well, so I am kind of stuck here. Looking to get some help around here.
Thanks.

How can I get list of all cloud SQL ( GCP ) instances which are stopped in python, I am using google cloud api for this purpose

from googleapiclient import discovery
PROJECT = gcp-test-1234
sql_client = discovery.build('sqladmin', 'v1beta4')
resp = sql_client.instances().list(project=PROJECT).execute()
print(resp)
But in response, I am getting a state as "RUNNABLE" for stopped instances, so how can I verify that the instance is running or stopped programmatically
I have also check gcloud sql instances describe gcp-test-1234-test-db, it is providing state as "STOPPED"
how can I achieve this programmatically using python
In the Rest API, the RUNNABLE for the state field means that the instance is running, or has been stopped by the owner, as stated here.
You need to read from the activationPolicy field, where ALWAYS means your instance is running and NEVER means it is stopped. Something like the following will work:
from pprint import pprint
from googleapiclient import discovery
service = discovery.build('sqladmin', 'v1beta4')
project = 'gcp-test-1234'
instance = 'gcp-test-1234-test-db'
request = service.instances().get(project=project,instance=instance)
response = request.execute()
pprint(response['settings']['activationPolicy'])
Another option would be to use the Cloud SDK command directly from your python file:
import os
os.system("gcloud sql instances describe gcp-test-1234-test-db | grep state | awk {'print $2'}")
Or with subprocess:
import subprocess
subprocess.run("gcloud sql instances describe gcp-test-1234-test-db | grep state | awk {'print $2'}", shell=True)
Note that when you run gcloud sql instances describe you-instance --log-http on a stopped instance, in the response of the API, you'll see "state": "RUNNABLE", however, the gcloud command will show the status STOPPED. This is because the output of the command gets the status from the activationPolicy of the API response rather than the status, if the status is RUNNABLE.
If you want to check the piece of code that translates the activationPolicy to the status, you can see it in the SDK. The gcloud tool is written in python:
cat $(gcloud info --format "value(config.paths.sdk_root)")/lib/googlecloudsdk/api_lib/sql/instances.py|grep "class DatabaseInstancePresentation(object)" -A 17
You'll se the following:
class DatabaseInstancePresentation(object):
"""Represents a DatabaseInstance message that is modified for user visibility."""
def __init__(self, orig):
for field in orig.all_fields():
if field.name == 'state':
if orig.settings and orig.settings.activationPolicy == messages.Settings.ActivationPolicyValueValuesEnum.NEVER:
self.state = 'STOPPED'
else:
self.state = orig.state
else:
value = getattr(orig, field.name)
if value is not None and not (isinstance(value, list) and not value):
if field.name in ['currentDiskSize', 'maxDiskSize']:
setattr(self, field.name, six.text_type(value))
else:
setattr(self, field.name, value)

How can I combine methods from the EC2 resource and client API?

I'm trying to take in input for stopping and starting instances, but if I use client, it comes up with the error:
'EC2' has no attribute 'instance'
and if I use resource, it says
'ec2.Serviceresource' has no attribute 'Instance'
Is it possible to use both?
#!/usr/bin/env python3
import boto3
import botocore
import sys
print('Enter Instance id: ')
instanceIdIn=input()
ec2=boto3.resource('ec2')
ec2.Instance(instanceIdIn).stop()
stopwait=ec2.get_waiter('instance_stopped')
try:
stopwait.wait(instanceIdIn)
print('Instance Stopped. Starting Instance again.')
except botocore.exceptions.waitError as wex:
logger.error('instance not stopped')
ec2.Instance(instanceIdIn).start()
try:
logger.info('waiting for running state...')
print('Instance Running.')
except botocore.exceptions.waitError as wex2:
logger.error('instance has not been stopped')
In boto3 there's two kinds of APIs for most service - a resource-based API, which is supposed to be an abstraction of the lower level API calls that the client API provides.
You can't directly mix and match calls to these. Instead you should create a separate instance for each of those like this:
import boto3
ec2_resource = boto3.resource("ec2")
ec2_client = boto3.client("ec2")
# Now you can call the client methods on the client
# and resource classes from the resource:
my_instance = ec2_resource.Instance("instance-id")
my_waiter = ec2_client.get_waiter("instance_stopped")

How to access run-property of AWS Glue workflow in Glue job?

I have been working with AWS Glue workflow for orchestrating batch jobs.
we need to pass push-down-predicate in order to limit the processing for batch job.
When we run Glue jobs alone, we can pass push down predicates as a command line argument at run time (i.e. aws glue start-job-run --job-name foo.scala --arguments --arg1-text ${arg1}..). But when we use glue workflow to execute Glue jobs, it is bit unclear.
When we orchestrate Batch jobs using AWS Glue workflows, we can add run properties while creating workflow.
Can I use run properties to pass push down predicate for my Glue Job ?
If yes, then how can I define value for the run property (push down predicate) at run time. The reason I want to define value for push down predicate at run time, is because the predicate arbitrarily changes every day. (i.e. run glue-workflow for past 10 days, past 20 days, past 2 days etc.)
I tried:
aws glue start-workflow-run --name workflow-name | jq -r '.RunId '
aws glue put-workflow-run-properties --name workflow-name --run-id "ID"
--run-properties --pushdownpredicate="some value"
I am able to see the run property I have passed using put-workflow-run-property
aws glue put-workflow-run-properties --name workflow-name --run-id "ID"
But I am not able to detect "pushdownpredicate" in my Glue Job.
Any idea how to access workflow's run property in Glue Job?
If you are using python as programming language for your Glue job then you can issue get_workflow_run_properties API call to retrieve the property and use it inside your Glue job.
response = client.get_workflow_run_properties(
Name='string',
RunId='string'
)
This will give you below response which you can parse and use it:
{
'RunProperties': {
'string': 'string'
}
}
If you are using scala then you can use equivalent AWS SDK.
In first instance you need to be sure that the job is running from a workflow:
def get_worfklow_params(args: Dict[str, str]) -> Dict[str, str]:
"""
get_worfklow_params is delegated to retrieve the WORKFLOW parameters
"""
glue_client = boto3.client("glue")
if "WORKFLOW_NAME" in args and "WORKFLOW_RUN_ID" in args:
workflow_args = glue_client.get_workflow_run_properties(Name=args['WORKFLOW_NAME'], RunId=args['WORKFLOW_RUN_ID'])["RunProperties"]
print("Found the following workflow args: \n{}".format(workflow_args))
return workflow_args
print("Unable to find run properties for this workflow!")
return None
This method will return a map of the workflow input parameter.
Than you can use the following method in order to retrieve a given parameter:
def get_worfklow_param(args: Dict[str, str], arg) -> str:
"""
get_worfklow_param is delegated to verify if the given parameter is present in the job and return it. In case of no presence None will be returned
"""
if args is None:
return None
return args[arg] if arg in args else None
From reuse the code, in my opinion is better to create a python (whl) module and set the module in the script path of your job. By this way, you can retrieve the method with a simple import.
Without the whl module, you can move in the following way:
def MyTransform(glueContext, dfc) -> DynamicFrameCollection:
import boto3
import sys
from typing import Dict
def get_worfklow_params(args: Dict[str, str]) -> Dict[str, str]:
"""
get_worfklow_params is delegated to retrieve the WORKFLOW parameters
"""
glue_client = boto3.client("glue")
if "WORKFLOW_NAME" in args and "WORKFLOW_RUN_ID" in args:
workflow_args = glue_client.get_workflow_run_properties(
Name=args['WORKFLOW_NAME'], RunId=args['WORKFLOW_RUN_ID'])["RunProperties"]
print("Found the following workflow args: \n{}".format(workflow_args))
return workflow_args
print("Unable to find run properties for this workflow!")
return None
def get_worfklow_param(args: Dict[str, str], arg) -> str:
"""
get_worfklow_param is delegated to verify if the given parameter is present in the job and return it. In case of no presence None will be returned
"""
if args is None:
return None
return args[arg] if arg in args else None
_args = getResolvedOptions(sys.argv, ['JOB_NAME', 'WORKFLOW_NAME', 'WORKFLOW_RUN_ID'])
worfklow_params = get_worfklow_params(_args)
job_run_id = get_worfklow_param(_args, "WORKFLOW_RUN_ID")
my_parameter= get_worfklow_param(_args, "WORKFLOW_CUSTOM_PARAMETER")
If you run Glue Job using workflow then sys.argv (which is a list) will contain parameters --WORKFLOW_NAME and --WORKFLOW_RUN_ID in it. You can use this fact to check if a Glue Job is being run by Workflow or not and then retrieve the Workflow Runtime Properties
from awsglue.utils import getResolvedOptions
if '--WORKFLOW_NAME' in sys.argv and '--WORKFLOW_RUN_ID' in sys.argv:
glue_args = getResolvedOptions(
sys.argv, ['WORKFLOW_NAME', 'WORKFLOW_RUN_ID']
)
workflow_args = glue_client.get_workflow_run_properties(
Name=glue_args['WORKFLOW_NAME'], RunId=glue_args['WORKFLOW_RUN_ID']
)["RunProperties"]
return {**workflow_args}
else:
raise Exception("GlueJobNotStartedByWorkflow")

aws boto3 client Stubber help stubbing unit tests

I'm trying to write some unit tests for aws RDS. Currently, the start stop rds api calls have not yet been implemented in moto. I tried just mocking out boto3 but ran into all sorts of weird issues. I did some googling and found http://botocore.readthedocs.io/en/latest/reference/stubber.html
So I have tried to implement the example for rds but the code appears to be behaving like the normal client, even though I have stubbed it. Not sure what's going on or if I am stubbing correctly?
from LambdaRdsStartStop.lambda_function import lambda_handler
from LambdaRdsStartStop.lambda_function import AWS_REGION
def tests_turn_db_on_when_cw_event_matches_tag_value(self, mock_boto):
client = boto3.client('rds', AWS_REGION)
stubber = Stubber(client)
response = {u'DBInstances': [some copy pasted real data here], extra_info_about_call: extra_info}
stubber.add_response('describe_db_instances', response, {})
with stubber:
r = client.describe_db_instances()
lambda_handler({u'AutoStart': u'10:00:00+10:00/mon'}, 'context')
so the mocking WORKS for the first line inside the stubber and the value of r is returned as my stubbed data. When I try and go into my lambda_handler method inside my lambda_function.py and still use the stubbed client it behaves like a normal unstubbed client:
lambda_function.py
def lambda_handler(event, context):
rds_client = boto3.client('rds', region_name=AWS_REGION)
rds_instances = rds_client.describe_db_instances()
error output:
File "D:\dev\projects\virtual_envs\rds_sloth\lib\site-packages\botocore\auth.py", line 340, in add_auth
raise NoCredentialsError
NoCredentialsError: Unable to locate credentials
You will need to patch boto3 where it is called in the routine that you will be testing. Also Stubber responses appear to be consumed on each call and thus will require another add_response for each stubbed call as below:
def tests_turn_db_on_when_cw_event_matches_tag_value(self, mock_boto):
client = boto3.client('rds', AWS_REGION)
stubber = Stubber(client)
# response data below should match aws documentation otherwise more errors due to botocore error handling
response = {u'DBInstances': [{'DBInstanceIdentifier': 'rds_response1'}, {'DBInstanceIdentifierrd': 'rds_response2'}]}
stubber.add_response('describe_db_instances', response, {})
stubber.add_response('describe_db_instances', response, {})
with mock.patch('lambda_handler.boto3') as mock_boto3:
with stubber:
r = client.describe_db_instances() # first_add_response consumed here
mock_boto3.client.return_value = client
response=lambda_handler({u'AutoStart': u'10:00:00+10:00/mon'}, 'context') # second_add_response would be consumed here
# asert.equal(r,response)