I'm trying to take in input for stopping and starting instances, but if I use client, it comes up with the error:
'EC2' has no attribute 'instance'
and if I use resource, it says
'ec2.Serviceresource' has no attribute 'Instance'
Is it possible to use both?
#!/usr/bin/env python3
import boto3
import botocore
import sys
print('Enter Instance id: ')
instanceIdIn=input()
ec2=boto3.resource('ec2')
ec2.Instance(instanceIdIn).stop()
stopwait=ec2.get_waiter('instance_stopped')
try:
stopwait.wait(instanceIdIn)
print('Instance Stopped. Starting Instance again.')
except botocore.exceptions.waitError as wex:
logger.error('instance not stopped')
ec2.Instance(instanceIdIn).start()
try:
logger.info('waiting for running state...')
print('Instance Running.')
except botocore.exceptions.waitError as wex2:
logger.error('instance has not been stopped')
In boto3 there's two kinds of APIs for most service - a resource-based API, which is supposed to be an abstraction of the lower level API calls that the client API provides.
You can't directly mix and match calls to these. Instead you should create a separate instance for each of those like this:
import boto3
ec2_resource = boto3.resource("ec2")
ec2_client = boto3.client("ec2")
# Now you can call the client methods on the client
# and resource classes from the resource:
my_instance = ec2_resource.Instance("instance-id")
my_waiter = ec2_client.get_waiter("instance_stopped")
Related
We are running Airflow via AWS's managed MWAA Offering. As part of their offering they include a tutorial on securely using the SSH Operator in conjunction with AWS Secrets Manager. The gist of how their solution works is described below:
Run a Task that fetches the pem file from a Secrets Manager location and store it on the filesystem at /tmp/mypem.pem.
In the SSH Connection include the extra information that specifies the file location
{"key_file":"/tmp/mypem.pem"}
Use the SSH Connection in the SSHOperator.
In short the workflow is supposed to be:
Task1 gets the pem -> Task2 uses the pem via the SSHOperator
All of this is great in theory, but it doesn't actually work. It doesn't work because Task1 may run on a different node from Task2, which means Task2 can't access the /tmp/mypem.pem file location that Task1 wrote the file to. AWS is aware of this limitation according to AWS Support, but now we need to understand another way to do this.
Question
How can we securely store and access a pem file that can then be used by Tasks running on different nodes via the SSHOperator?
I ran into the same problem. I extended the SSHOperator to do both steps in one call.
In AWS Secrets Manager, two keys are added for airflow to retrieve on execution.
{variables_prefix}/airflow-user-ssh-key : the value of the private key
{connections_prefix}/ssh_airflow_user : ssh://replace.user#replace.remote.host?key_file=%2Ftmp%2Fairflow-user-ssh-key
from typing import Optional, Sequence
from os.path import basename, splitext
from airflow.models import Variable
from airflow.providers.ssh.operators.ssh import SSHOperator
from airflow.providers.ssh.hooks.ssh import SSHHook
class SSHOperator(SSHOperator):
"""
SSHOperator to execute commands on given remote host using the ssh_hook.
:param ssh_conn_id: :ref:`ssh connection id<howto/connection:ssh>`
from airflow Connections.
:param ssh_key_var: name of Variable holding private key.
Creates "/tmp/{variable_name}.pem" to use in SSH connection.
May also be inferred from "key_file" in "extras" in "ssh_conn_id".
:param remote_host: remote host to connect (templated)
Nullable. If provided, it will replace the `remote_host` which was
defined in `ssh_hook` or predefined in the connection of `ssh_conn_id`.
:param command: command to execute on remote host. (templated)
:param timeout: (deprecated) timeout (in seconds) for executing the command. The default is 10 seconds.
Use conn_timeout and cmd_timeout parameters instead.
:param environment: a dict of shell environment variables. Note that the
server will reject them silently if `AcceptEnv` is not set in SSH config.
:param get_pty: request a pseudo-terminal from the server. Set to ``True``
to have the remote process killed upon task timeout.
The default is ``False`` but note that `get_pty` is forced to ``True``
when the `command` starts with ``sudo``.
"""
template_fields: Sequence[str] = ("command", "remote_host")
template_ext: Sequence[str] = (".sh",)
template_fields_renderers = {"command": "bash"}
def __init__(
self,
*,
ssh_conn_id: Optional[str] = None,
ssh_key_var: Optional[str] = None,
remote_host: Optional[str] = None,
command: Optional[str] = None,
timeout: Optional[int] = None,
environment: Optional[dict] = None,
get_pty: bool = False,
**kwargs,
) -> None:
super().__init__(
ssh_conn_id=ssh_conn_id,
remote_host=remote_host,
command=command,
timeout=timeout,
environment=environment,
get_pty=get_pty,
**kwargs,
)
if ssh_key_var is None:
key_file = SSHHook(ssh_conn_id=self.ssh_conn_id).key_file
key_filename = basename(key_file)
key_filename_no_extension = splitext(key_filename)[0]
self.ssh_key_var = key_filename_no_extension
else:
self.ssh_key_var = ssh_key_var
def import_ssh_key(self):
with open(f"/tmp/{self.ssh_key_var}", "w") as file:
file.write(Variable.get(self.ssh_key_var))
def execute(self, context):
self.import_ssh_key()
super().execute(context)
The answer by holly is good. I am sharing a different way I solved this problem. I used the strategy of converting the SSH Connection into a URI and then input that into Secrets Manager under the expected connections path, and everything worked great via the SSH Operator. Below are the general steps I took.
Generate an encoded URI
import json
from airflow.models.connection import Connection
from pathlib import Path
pem = Path(“/my/pem/file”/pem).read_text()
myconn= Connection(
conn_id="connX”,
conn_type="ssh",
host="10.x.y.z,
login=“mylogin”,
extra=json.dumps(dict(private_key=pem)),
print(myconn.get_uri())
Input that URI under the environment's configured path in Secrets Manager. The important note here is to input the value in the plaintext field without including a key. Example:
airflow/connections/connX and under Plaintext only include the URI value
Now in the SSHOperator you can reference this connection Id like any other.
remote_task = SSHOperator(
task_id="ssh_and_execute_command",
ssh_conn_id="connX"
command="whoami",
)
I want to create a mock ECS cluster, but it seems not to work properly. Although something is mocked (I don't get a credentials error), it seems not to "save" the cluster.
How can I create a mock cluster with moto?
MVCE
foo.py
import boto3
def print_clusters():
client = boto3.client("ecs")
print(client.list_clusters())
return client.list_clusters()["clusterArns"]
test_foo.py
import boto3
import pytest
from moto import mock_ecs
import foo
#pytest.fixture
def ecs_cluster():
with mock_ecs():
client = boto3.client("ecs", region_name="us-east-1")
response = client.create_cluster(clusterName="test_ecs_cluster")
yield client
def test_foo(ecs_cluster):
assert foo.print_clusters() == ["test_ecs_cluster"]
What happens
$ pytest test_foo.py
Test session starts (platform: linux, Python 3.8.1, pytest 5.3.5, pytest-sugar 0.9.2)
rootdir: /home/math/GitHub
plugins: black-0.3.8, mock-2.0.0, cov-2.8.1, mccabe-1.0, flake8-1.0.4, env-0.6.2, sugar-0.9.2, mypy-0.5.0
collecting ...
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test_foo ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ecs_cluster = <botocore.client.ECS object at 0x7fe9b0c73580>
def test_foo(ecs_cluster):
> assert foo.print_clusters() == ["test_ecs_cluster"]
E AssertionError: assert [] == ['test_ecs_cluster']
E Right contains one more item: 'test_ecs_cluster'
E Use -v to get the full diff
test_foo.py:19: AssertionError
---------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------------
{'clusterArns': [], 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
test_foo.py ⨯
What I expected
I expected the list of cluster ARNs to have one element (not the one in the assert statement, but an ARN). But the list is empty.
When creating a cluster, you're using a mocked ECS client.
When listing the clusters, you're creating a new ECS client outside the scope of moto.
In other words, you're creating a cluster in memory - but then ask AWS itself for a list of clusters.
You could rewrite the foo-method to use the mocked ECS client:
def print_clusters(client):
print(client.list_clusters())
return client.list_clusters()["clusterArns"]
def test_foo(ecs_cluster):
assert foo.print_clusters(ecs_cluster) == ["test_ecs_cluster"]
def test_foo(ecs_cluster):
assert foo.print_clusters(ecs_cluster) == ["test_ecs_cluster"]
#This will cause you a bug but I have fixed this bug ..so the code looks like this:
def cluster_list(ecs_cluster):
assert ecs.print_clusters(ecs_cluster) == ['arn:aws:ecs:us-east-1:123456789012:cluster/test_ecs_cluster']
#Explination
So basically you have passed the incorrect assert values..assert foo.print_clusters(ecs_cluster) --> this containes cluster arns it is in the form of an array which is ['arn:aws:ecs:us-east-1:123456789012:cluster/test_ecs_cluster'] and you are trying to access the [1] index and testing if its == "test_ecs_cluster" which is wrong so instead to that pass the full arn just to test your code ..
this is the python code I got from github. running it, I got 300. But when I use gcloud to get role number, I got a total of 479 roles. I was told by the GCP support that pageSize needs to be used. where can I find documentation of how and pageSize can be used? so in my code below, where should pageSize go? or perhaps pageToken needs to be used?
(gcptest):$ gcloud iam roles list |grep name |wc -l
479
(gcptest) : $ python quickstart.py
300
def quickstart():
# [START iam_quickstart]
import os
from google.oauth2 import service_account
import googleapiclient.discovery
import pprint
# Get credentials
credentials = service_account.Credentials.from_service_account_file(
filename=os.environ['GOOGLE_APPLICATION_CREDENTIALS'],
scopes=['https://www.googleapis.com/auth/cloud-platform'])
# Create the Cloud IAM service object
service = googleapiclient.discovery.build(
'iam', 'v1', credentials=credentials)
# Call the Cloud IAM Roles API
# If using pylint, disable weak-typing warnings
# pylint: disable=no-member
response = service.roles().list().execute()
roles = response['roles']
print(type(roles))
print(len(roles))
if name == 'main':
quickstart()
You will need to write code similar to this:
roles = service.roles()
request = roles.list()
while request is not None:
role_list = request.execute()
# process each role here
for role in role_list:
print(role)
# Get next page of results
request = roles.list_next(request, role_list)
Documentation link for the list_next method
In addition of the solution of #JohnHanley, you can also add the queries parameters in parameter of your method. Like this
# Page size of 10
response = service.roles().list(pageSize=10).execute()
Here the definition of this list method
I’m reading AWS Python docs such as SNS Client Publish() but can’t find the details of what exceptions a function can throw.
E.g., publish() can throw EndpointDisabledException but I can’t find this documented.
Where can I look up the list of exceptions a BOTO3 function can throw (for Python)
This is how to handle such exceptions:
import boto3
from botocore.exceptions import ClientError
import logging
try:
response = platform_endpoint.publish(
Message=json.dumps(message, ensure_ascii=False),
MessageStructure='json')
logging.info("r = %s" % response)
except ClientError as e:
if e.response['Error']['Code'] == 'EndpointDisabled':
logging.info('EndpointDisabledException thrown')
Almost all exceptions are subclassed from BotoCoreError. I am not able to find a method to list all exceptions. Look at Botocore Exceptions file to get a list of possible exceptions. I can't find EndpointDisabledException. Are you using the latest version?
See: Botocore Exceptions
you can find the list of exceptions for publish(**kwargs) -> here <- (bottom of publish(**kwargs) part)
Every exception is linked to its documentation
SNS.Client.exceptions.InvalidParameterException
SNS.Client.exceptions.InvalidParameterValueException
SNS.Client.exceptions.InternalErrorException
SNS.Client.exceptions.NotFoundException
SNS.Client.exceptions.EndpointDisabledException
SNS.Client.exceptions.PlatformApplicationDisabledException
SNS.Client.exceptions.AuthorizationErrorException
SNS.Client.exceptions.KMSDisabledException
SNS.Client.exceptions.KMSInvalidStateException
SNS.Client.exceptions.KMSNotFoundException
SNS.Client.exceptions.KMSOptInRequired
SNS.Client.exceptions.KMSThrottlingException
SNS.Client.exceptions.KMSAccessDeniedException
SNS.Client.exceptions.InvalidSecurityException
Use the client and then find the exception
example : if we are dealing with the cognito then
client = boto3.client(
'cognito-idp',....)
try:
some code .......
except client.exceptions.UsernameExistsException as ex:
print(ex)
Following code will generate an exhaustive list of exceptions from all supported services that it's boto3 client can throw.
#!/usr/bin/env python3
import boto3
with open("README.md", "w") as f:
f.write("# boto3 client exceptions\n\n")
f.write(
"This is a generated list of all exceptions for each client within the boto3 library\n"
)
f.write("""
```python
import boto3
s3 = boto3.client("s3")
try:
s3.create_bucket("example")
except s3.exceptions.BucketAlreadyOwnedByYou:
raise
```\n\n""")
services = boto3.session.Session().get_available_services()
for service in services:
f.write(f"- [{service}](#{service})\n")
for service in services:
f.write(f"### {service}\n")
for _, value in boto3.client(
service, region_name="us-east-1"
).exceptions.__dict__.items():
for exception in value:
f.write(f"- {exception}\n")
f.write("\n")
Example output for S3 boto3 client:
BucketAlreadyExists
BucketAlreadyOwnedByYou
InvalidObjectState
NoSuchBucket
NoSuchKey
NoSuchUpload
ObjectAlreadyInActiveTierError
ObjectNotInActiveTierError
Reference: https://github.com/jbpratt/boto3-client-exceptions/blob/master/generate
I need to retrieve the DNS name of my Cloudfront instance (eg. 1234567890abcd.cloudfront.net) and was wondering if there is a quick way to get this in Ansible without resorting to the AWS CLI.
From gleaming the Extra Modules source it would appear there is not a module for this. How are other people getting this attribute?
You can either write your own module or you can write a filter plugin in a few lines and accomplish the same thing.
Example of writing a filter in Ansible. Lets name this file aws.py in your filter_plugins/aws.py
import boto3
import botocore
from ansible import errors
def get_cloudfront_dns(region, dist_id):
""" Return the dns name of the cloudfront distribution id.
Args:
region (str): The AWS region.
dist_id (str): distribution id
Basic Usage:
>>> get_cloudfront_dns('us-west-2', 'E123456LHXOD5FK')
'1234567890abcd.cloudfront.net'
"""
client = boto3.client('cloudfront', region)
domain_name = None
try:
domain_name = (
client
.get_distribution(Id=dist_id)['Distribution']['DomainName']
)
except Exception as e:
if isinstance(e, botocore.exceptions.ClientError):
raise e
else:
raise errors.AnsibleFilterError(
'Could not retreive the dns name for CloudFront Dist ID {0}: {1}'.format(dist_id, str(e))
)
return domain_name
class FilterModule(object):
''' Ansible core jinja2 filters '''
def filters(self):
return {'get_cloudfront_dns': get_cloudfront_dns,}
In order to use this plugin, you just need to call it.
dns_entry: "{{ 'us-west-2' | get_cloudfront_dns('123434JHJHJH') }}"
Keep in mind, you will need boto3 and botocore installed, in order to use this this plugin.
I have a bunch of examples located in my repo linuxdynasty ld-ansible-filters repo
I ended up writing a module for this (cloudfront_facts.py) that has been accepted into Ansible 2.3.0.