I'am using moto to mock aws for my application. I wondering if it is possible to create ami in moto with specific image-id (for example: ami-1a2b3c4d).
Thank you!
You want to use preloaded resources like this file: https://github.com/spulec/moto/blob/master/moto/ec2/resources/amis.json
You can use the following environment variable: MOTO_AMIS_PATH=/full/path/to/amis.json
This JSON-file has to be in the same format as the one linked above. Note that the environment variable has to be set before Moto is initialised - these AMI's are loaded the moment you call from moto import mock_ec2, so the environment variable has to be set before that import takes place.
(Copied from https://stackoverflow.com/a/72270977/7224682)
Here is an example coming straight from the docs:
from . import add_servers
from moto import mock_ec2
#mock_ec2
def test_add_servers():
add_servers('ami-XXXXXXX', 2)
client = boto3.client('ec2', region_name='us-west-1')
instances = client.describe_instances()['Reservations'][0]['Instances']
assert len(instances) == 2
instance1 = instances[0]
assert instance1['ImageId'] == 'ami-XXXXXXXX'
You can choose the AMI ID to be whatever you want, there are no restrictions. I'm not sure I understand what the problem is as these are "mock" resources so they can be in any format/contain any name that you want.
Related
Using moto how can you create an EC2 instance since theres no AMIs available to launch an instance with.
This repo seems to be "pre-loading" AMIs that are later used within tests but im not sure how these are being created
https://github.com/spulec/moto/blob/master/tests/test_ec2/test_instances.py#L25
https://github.com/spulec/moto/blob/master/moto/ec2/resources/amis.json
I've tried calling .describe_images(Owners=["amazon"]) however, all the AMIs returned when used in the run_instances call give the below error.
from moto import mock_ec2
#mock_ec2
def Test_create_ec2():
boto3.client('ec2').run_instances(ImageId="ami-1234abcd", MinCount=1, MaxCount=1)
botocore.exceptions.ClientError: An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id '[ami-1234abcd]' does not exist
This issue also relates to the unresolved question How to create ami with specific image-id using moto?
The call to describe_images does give a list of all available ImageId's.
The following test works against the current development branch of Moto:
#mock_ec2
def test_all_images():
client = boto3.client("ec2", region_name="us-east-1")
images = client.describe_images(Owners=["amazon"])["Images"]
for image in images:
client.run_instances(ImageId=image["ImageId"], MinCount=1, MaxCount=1)
The preloaded images come from this file:
https://github.com/spulec/moto/blob/master/moto/ec2/resources/amis.json
If you want to substitute your own AMI's, you can use the following environment variable:
MOTO_AMIS_PATH=/full/path/to/amis.json
This JSON-file has to be in the same format as the one linked above.
Note that the environment variable has to be set before Moto is initialized - these AMI's are loaded the moment you call from moto import mock_ec2, so the environment variable has to be set before that import takes place.
I'm attempting to use the python package awswrangler to access a non-AWS S3 service.
The AWS Data Wranger docs state that you need to create a boto3.Session() object.
The problem is that the boto3.client() supports setting the endpoint_url, but boto3.Session() does not (docs here).
In my previous uses of boto3 I've always used the client for this reason.
Is there a way to create a boto3.Session() with a custom endpoint_url or otherwise configure awswrangler to accept the custom endpoint?
I finally found the configuration for awswrangler:
import awswrangler as wr
wr.config.s3_endpoint_url = 'https://custom.endpoint'
Any configuration variables for awswrangler can be overwritten directly using the wr.config config object as you stated in your answer, but it may be cleaner or preferable in some use cases to use environment variables.
In that case, simply set WR_S3_ENDPOINT_URL to your custom endpoint, and the configuration will reflect that when you import the library.
Once you create your session, you can use client as well. For example:
import boto3
session = boto3.Session()
s3 = session.client('s3', endpoint_url='<custom-endpoint>')
I am integrating my application with AWS parameter store. For local development which may have no access to AWS I need to disable fetching property values from AWS and use values from application.yml. The issue seems to be not application.yml, but the dependencies: as soon as AWS starter appears in POM, AWS integration is being initialized: Spring is trying to use AwsParamStorePropertySourceLocator. I guess what I need to do is to force my application to use Spring's property source locator regardless of AWS jar being on the class path. Not sure how to do that.
For parameter store it is quite easy: AwsParamStoreBootstrapConfiguration bean is conditional on property aws.paramstore.enabled. Creating aws.paramstore.enabled environment variable and setting its value to false will disable AWS parameter store.
I also tried disabling AWS secrets manager and setting aws.secretsmanager.enabled to false is not sufficient. To fully disable it I had to disable auto configuration for few classes:
import org.springframework.cloud.aws.autoconfigure.context.ContextCredentialsAutoConfiguration;
import org.springframework.cloud.aws.autoconfigure.context.ContextInstanceDataAutoConfiguration;
import org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration;
import org.springframework.cloud.aws.autoconfigure.context.ContextResourceLoaderAutoConfiguration;
import org.springframework.cloud.aws.autoconfigure.context.ContextStackAutoConfiguration;
import org.springframework.cloud.aws.autoconfigure.mail.MailSenderAutoConfiguration;
#Configuration
#Profile("local")
#EnableAutoConfiguration(exclude = { ContextCredentialsAutoConfiguration.class,
ContextInstanceDataAutoConfiguration.class, ContextRegionProviderAutoConfiguration.class,
ContextResourceLoaderAutoConfiguration.class, ContextStackAutoConfiguration.class,
MailSenderAutoConfiguration.class })
public class LocalScanConfig {
}
There are great but still limited set of SDK access to GCP APIs within cloud function SDKs. e.g. Node.
I want to call gcloud cli within a cloud function. Is this possible? e.g.
gcloud sql instances patch my-database --activation-policy=NEVER
The goal is nightly shutdown of an SQL instance
I believe you should use the Cloud SQL Admin API. If you're using the Python runtime for example you'd had 'google-api-python-client==1.7.8' (for example) to your requirements file and on the respective client library you would use the method instances.patch with the appropriate parameters.
Hope this helps.
Also you have here a working example with the Python runtime, just be sure to edit the 'projid' and 'instance' variables accordingly.
from googleapiclient.discovery import build
service = build('sqladmin', 'v1beta4')
projid = '' #project id where Cloud SQL instance is
instance = '' #Cloud SQL instance
patch = {'settings': {'activationPolicy':'NEVER'}}
req = service.instances().patch(project=projid, instance=instance, body=patch)
x = req.execute()
print(x)
I want to try image segmentation with deep learning using AWS. I have my data stored on Amazon S3 and I'd like to access it from a Jupyter Notebook which is running on an Amazon EC2 instance.
I'm planning on using Tensorflow for segmentation, therefore it seemed appropriate to me to use options provided by Tensorflow themselves (https://www.tensorflow.org/deploy/s3) as it feels that in the end I want my data to be represented in the format of tf.Dataset. However, it didn't quite work out for me. I've tried the following:
filenames = ["s3://path_to_first_image.png", "s3://path_to_second_image.png"]
dataset = tf.data.TFRecordDataset(filenames)
iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()
with tf.Session() as sess:
for i in range(2):
print(sess.run(next_element))
I get the following error:
OutOfRangeError: End of sequence
[[Node: IteratorGetNext_6 = IteratorGetNext[output_shapes=[[]], output_types=[DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator_6)]]
I'm quite new to tensorflow and have just recently started trying out some stuff with AWS, so I hope that my mistake is gonna be obvious to someone with more experience. I would greatly appreciate any help or advice! Maybe it's even the wrong way and I'm better off with something like boto3 (also stumbled upon it, but thought that tf would be more appropriate in my case) or something else?
P.S. Tensorflow also recommends to test a setup with the following piece:
from tensorflow.python.lib.io import file_io
print (file_io.stat('s3://path_to_image.png'))
For me this leads to Object doesn't exist error, though the object certainly exists and it's being listed among others if I use
for obj in s3.Bucket(name=MY_BUCKET_NAME).objects.all():
print(os.path.join(obj.bucket_name, obj.key))
I also have my credentials filled in /.aws/credentials. What might be the problem here?
Not a direct answer to your question but still something I noticed as to why you can't load data using Tensorflow.
The files in your filenames are .png and not in the .tfrecord file format which is a binary storage format. So, tf.data.TFRecordDataset(filenames) shouldn't work.
I think the following will work. Note: this is for TF2, not sure if it is the same for TF1. A similar example can be found here at TensorFlow's web site tensorflow example
Step 1
Load your files into a TensorFlow dataset with tf.data.Dataset.list_files.
import tensorflow as tf
list_ds = tf.data.Dataset.list_files(filenames)
Step 2
Make a function that will be applied to each element in the dataset by using map; this will use the function on every element in the TF dataset.
def process_path(file_path):
'''reads the path and returns an image.'''
# load the raw data from the file as a string
byteString = tf.io.read_file(file_path)
# convert the compressed string to a 3D uint8 tensor
img = tf.image.decode_png(byteString, channels=3)
return img
dataset = list_ds.map(preprocess_path)
Step 3
Check out the image.
import matplotlib.pyplot as plt
for image in dataset.take(1): plt.imshow(image)
Directly access S3 data from the Ubuntu Deep Learning instance by
cd ~/.aws
aws configure
Then update aws key and secret key for the instance, just to make sure. Check awscli version using the command:
aws --version
Read more on configuration
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
You can type in jupyter
import pandas as pd
from smart_open import smart_open
import os
aws_key = 'aws_key'
aws_secret = 'aws_secret'
bucket_name = 'my_bucket'
object_key = 'data.csv'
path = 's3://{}:{}#{}/{}'.format(aws_key, aws_secret, bucket_name, object_key)
df = pd.read_csv(smart_open(path))
Also, objects stored in the buckets have a unique key value and are retrieved using a HTTP URL address. For example, if an object with a key value
/photos/mygarden.jpg
is stored in the
myawsbucket
bucket, then it is addressable using the URL
http://myawsbucket.s3.amazonaws.com/photos/mygarden.jpg.
If your data is not sensitive, you can use the http option. More details:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html
You can change the setting of the bucket to public. Hope this helps.