aws ec2 request-spot-instances CLI issues - amazon-web-services

Trying to start a couple of spot instances within a simple script, and the syntax supplied in the AWS documentation and aws ec2 request-spot-instances help output is listed in either JAVA or JSON syntax. How does one enter the parameters under the JSON syntax from inside a shell script?
aws --version
aws-cli/1.2.6 Python/2.6.5 Linux/2.6.21.7-2.fc8xen
aws ec2 request-spot-instances help
-- at the start of "launch specification" it lists JSON syntax
--launch-specification (structure)
Specifies additional launch instance information.
JSON Syntax:
{
"ImageId": "string",
"KeyName": "string",
}, ....
"EbsOptimized": true|false,
"SecurityGroupIds": ["string", ...],
"SecurityGroups": ["string", ...]
}
I have tried every possible combination of the following, adding & moving brackets, quotes, changing options, etc, all to no avail. What would be the correct formatting of the variable $launch below to have this work? Other command variations -- "ec2-request-spot-instances" are not working in my environment, nor does it work if I try to substitute --spot-price with -p.
#!/bin/bash
launch="{"ImageId":"ami-a999999","InstanceType":"c1.medium"} "SecurityGroups":"launch-wizard-6""
echo $launch
aws ec2 request-spot-instances --spot-price 0.01 --instance-count 1 --type c1.small --launch-specification $launch
This provides result:
Unknown options: SecurityGroups:launch-wizard-6
Substituting the security group number has the same result.
aws ec2 describe-instances works perfectly, as does aws ec2 start-instance, so the environment and account information are properly setup, but I need to utilize spot pricing.
In fact, nothing is working as listed in this user documentation: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RequestSpotInstances.html
Thank you,

I know this is an old question, but in case somebody runs into it. I had the same issue recently with the CLI. It was very hard to get all the parameters to work correctly for request-spot-instances
#!/bin/bash
AWS_DEFAULT_OUTPUT="text"
UserData=$(base64 < userdata-current)
region="us-west-2"
price="0.03"
zone="us-west-2c"
aws ec2 request-spot-instances --region $region --spot-price $price --launch-specification "{ \"KeyName\": \"YourKey\", \"ImageId\": \"ami-3d50120d\" , \"UserData\": \"$UserData\", \"InstanceType\": \"r3.large\" , \"Placement\": {\"AvailabilityZone\": \"$zone\"}, \"IamInstanceProfile\": {\"Arn\": \"arn:aws:iam::YourAccount:YourProfile\"}, \"SecurityGroupIds\": [\"YourSecurityGroupId\"],\"SubnetId\": \"YourSubnectId\" }"
Basically what I had to do is put my user data in an external file, load it into the UserData variable and then pass that on the command line. Trying to get everything on the command line or using an external file for the ec2-request-spot-instances just kept failing. Note that other commands worked just fine, so this is specific to the ec2-request-spot-instances.
I detailed more about what i ended up doing here.

You have to use a list in this case:
"SecurityGroups": ["string", ...]
so
"SecurityGroups":"launch-wizard-6"
become
"SecurityGroups":["launch-wizard-6"]
Anyway, I'm dealing with the CLI right now and I found more useful to use a external JSON
Here is an example using Python:
myJson="file:///Users/xxx/Documents/Python/xxxxx/spotInstanceInformation.json"
x= subprocess.check_output(["/usr/local/bin/aws ec2 request-spot-instances --spot-price 0.2 --launch-specification "+myJson],shell=True)
print x
And the output is:
"SpotInstanceRequests": [
{
"Status": {
"UpdateTime": "2013-12-09T02:41:41.000Z",
"Code": "pending-evaluation",
"Message": "Your Spot request has been submitted for review, and is pending evaluation."
etc etc ....
Doc is here : http://docs.aws.amazon.com/cli/latest/reference/ec2/request-spot-instances.html
FYI - I'm appending file:/// because I'm using MAC. If you are launching your bash script using Linux, you could just use myJson="/path/to/file/"

The first problem, here, is quoting and formatting:
$ launch="{"ImageId":"ami-a999999","InstanceType":"c1.medium"} "SecurityGroups":"launch-wizard-6""
This isn't going to generate valid JSON, because the block you copied from the help file includes a spurious closing brace from a nested object that you didn't include, the closing brace is missing, and the unescaped double quotes are disappearing.
But we're not really getting to the point where the json is actually being validated, because with that space after the last brace, the cli is assuming that SecurityGroups and launch-wizard-6 are more command line options following the argument to --launch-specification:
$ echo $launch
{ImageId:ami-a999999,InstanceType:c1.medium} SecurityGroups:launch-wizard-6
That's probably not what you expected... so we'll fix the quoting so that it looks like one long argument, after the json is valid:
From the perspective of just generating valid json structures (not necessarily content), the data you are most likely trying to send would actually look like this, based on the docs:
{"ImageId":"ami-a999999","InstanceType":"c1.medium","SecurityGroups":["launch-wizard-6"]}
Check that as structurally valid JSON, here.
Fixing the bracing, commas, and bracketing, the CLI stops throwing that error, with this formatting:
$ launch='{"ImageId":"ami-a999999","InstanceType":"c1.medium","SecurityGroups":["launch-wizard-6"]}'
$ echo $launch
{"ImageId":"ami-a999999","InstanceType":"c1.medium","SecurityGroups":["launch-wizard-6"]}
That isn't to say the API might not subsequently reject the request due to something else incorrect or missing, but you were never actually getting to the point of sending anything to the API; this was failing local validation in the command line tools.

Related

AWS CLI syntax error: Error parsing parameter '--image': Expected: '=', received: ''' for input:

I am trying to follow a walkthrough involving using AWS to use live feed facial recognition (see link below). I have done the 7 previous steps without issue and am currently on step 8, where I have uploaded a picture to a S3 bucket, and I have to do the command in cmd:
aws rekognition index-faces --image '{"S3Object":{"Bucket":"<S3BUCKET>","Name":"<MYFACE_KEY>.jpeg"}}' --collection-id "rekVideoBlog" --detection-attributes "ALL" --external-image-id "<YOURNAME>" --region us-west-2
I have done this, with my own information input in, but I get an error that states:
Error parsing parameter '--image': Expected: '=', received: ''' for input:
'{S3Object:{Bucket:,Name:.jpeg}}'
I have have looked up what could be the issue, with many solutions involving Windows systems having a different style of quotations compared to the walkthrough (i.e. switching single quotes to doubles and doubles to escaped quotes). No matter what I try, I cannot seem to figure out the issue. Does anyone have an idea of what would be the correct syntax for this on a Windows system or if I am doing something else wrong?
Walkthough: https://aws.amazon.com/blogs/machine-learning/easily-perform-facial-analysis-on-live-feeds-by-creating-a-serverless-video-analytics-environment-with-amazon-rekognition-video-and-amazon-kinesis-video-streams/

Using conditionals in AWS SES templates doesn't work (MissingRenderingAttributeException)

I am trying to use conditionals in SES templates by following this guideline:
https://docs.aws.amazon.com/ses/latest/dg/send-personalized-email-advanced.html
I should be able to create a template with dynamic content based on the value of the variable evaluation result. Still, no matter what I do, I keep getting a 'MissingRenderingAttributeException' error.
For local development I use localstack on docker.
The aws-localstack is an alias I set for communicating with the AWS CLI localstack provides
This is the test I'm running:
aws-localstack ses create-template --cli-input-json '{
"Template": {
"TemplateName": "test_conditionals",
"SubjectPart": "TESTING CONDS",
"TextPart": "{{#if lastName}}[{{lastName}}]{{/if}}",
"HtmlPart": "{{#if lastName}}[{{lastName}}]{{/if}}"
}
}'
aws-localstack ses test-render-template --cli-input-json '{
"TemplateName": "test_conditionals",
"TemplateData": "{\"lastName\":\"test-result\"}"
}'
I keep getting this error:
An error occurred (MissingRenderingAttributeException) when calling the TestRenderTemplate operation: Attribute '#if lastName' is not present in the rendering data.
How do I fix it? What am I missing?
Any suggestion would be appreciated :)
I found the problem, it has nothing to do with AWS, but with localstack
There are many features missing in localstack library, one of them is conditionals support
I thought about deleting this topic, but in favor of everyone who expects it to behave the same on local env by using localstack, I thought it is worth having such topic in SO, because I had to work "harder" to figure it out

AWS create-rule AWS CLI giving error "Unknown parameter in Conditions[0]: "PathPatternConfig", must be one of: Field, Values"

I am trying to add a path pattern /images/* to an existing ALB listener rule. Following is the command that I have executed. Please note that the variables $listenerARN and $tgARN has correct values, which I have not shown here due to security reason.
aws elbv2 create-rule --listener-arn "$listenerARN" --priority 5 --conditions "Field=path-pattern,PathPatternConfig={Values="/images/*"}" --actions Type=forward,TargetGroupArn="$tgARN"
When I execute the above command I get the following error:
Unknown parameter in Conditions[0]: "PathPatternConfig", must be one of: Field, Values
I get the same error if I provide the value for --conditions from the external .json file, which has the following content.
[
{
"Field": "path-pattern",
"PathPatternConfig": {
"Values": ["/images/*"]
}
}
]
I read the documentation several time and I am sure I am following the exact syntax, but I cannot get rid of this error.
It looks like you have to use an alternate syntax for complex JSON here:
--conditions file://conditions.json

Creating Kinesis Analytics applications using aws cli

I want to create a kinesis analytics application using aws cli. I use this command to create the application
aws kinesisanalytics create-application --application-name smartfactorytest1 --application-code "CREATE OR REPLACE STREAM DESTINATION_SQL_STREAM ( "device_serial" VARCHAR(16), "uploadRate" INTEGER, "downloadRate" INTEGER);
CREATE OR REPLACE PUMP "STREAM_PUMP"
AS INSERT INTO DESTINATION_SQL_STREAM
SELECT STREAM "device_serial", "uploadRate", "downloadRate"
FROM SOURCE_SQL_STREAM_001
-- LIKE compares a string to a string pattern (_ matches all char, % matches substring)
-- SIMILAR TO compares string to a regex, may use ESCAPE
WHERE "uploadRate" >20000" --inputs NamePrefix="SOURCE_SQL_STREAM",KinesisStreamsInput={ResourceARN="sourcearn",RoleARN="rolearn"}
But I get this error
invalid type for parameter Inputs[0].KinesisStreamsInput, value: ResourceARN=string, type: <class 'str'>, valid types: <class 'dict'>
Can anyone tell me what am I doing wrong? Any help would be appreciated.
I believe the issue is either that you need to take the quotes out in the KinesisStreamsInput section, or you need to add quotes and escape them. The documentation is unclear on which is the correct option.
According to the AWS Kinesis Analytics CLI Reference, https://docs.aws.amazon.com/cli/latest/reference/kinesisanalytics/create-application.html, the syntax for --inputs with KinesisStreamsInput should look like the example provided for KinesisStreamsOutput:
Name=string,KinesisStreamsOutput={ResourceARN=string,RoleARN=string},...
This would mean removing the quotes around your sourcearn and rolearn. However, the documentation isn't clear that this refers to the CLI syntax in all cases.
If that doesn't work, according to this AWS CLI usage guide page, https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-parameters-quoting-strings.html, it specifies adding quotes and escaping the relevant ones, depending on your OS...
"Linux or macOS
Use single quotation marks (' ') to enclose the JSON data structure, as in the following example. You don't have to do anything special with the embedded double quotation marks embedded in the JSON string.
aws ec2 run-instances --image-id ami-12345678 --block-device-mappings '[{"DeviceName":"/dev/sdb","Ebs":{"VolumeSize":20,"DeleteOnTermination":false,"VolumeType":"standard"}}]'
PowerShell
PowerShell requires single quotation marks (' ') to enclose the JSON data structure. Also, because double quotation marks have a special meaning to PowerShell, you must use a backslash () to escape each double quotation mark (") within the JSON structure, as in the following example.
PS C:\> aws ec2 run-instances --image-id ami-12345678 --block-device-mappings '[{\"DeviceName\":\"/dev/sdb\",\"Ebs\":{\"VolumeSize\":20,\"DeleteOnTermination\":false,\"VolumeType\":\"standard\"}}]'
Windows Command Prompt
The Windows command prompt requires double quotation marks (" ") to enclose the JSON data structure. Also, to prevent the command processor from misinterpreting the double quotation marks embedded in the JSON, you must also escape (precede with a backslash [ \ ] character) each double quotation mark (") within the JSON data structure itself, as in the following example.
C:\> aws ec2 run-instances --image-id ami-12345678 --block-device-mappings "[{\"DeviceName\":\"/dev/sdb\",\"Ebs\":{\"VolumeSize\":20,\"DeleteOnTermination\":false,\"VolumeType\":\"standard\"}}]"
Only the outermost double quotation marks are not escaped."
This link also references needing to escape quotes on Windows, and is using the kinesisanalytics command: https://github.com/aws/aws-cli/issues/3103
"Rishi74744 commented on Feb 6, 2018
I got it to work as -
aws kinesisanalytics add-application-reference-data-source --endpoint https://kinesisanalytics.us-east-1.amazonaws.com --region us-east-1 --application-name alerts --reference-data-source "{\"TableName\":\"DeviceData\",\"S3ReferenceDataSource\":{\"BucketARN\":\"arn: aws: s3: : : bucket-name\",\"FileKey\":\"device.csv\",\"ReferenceRoleARN\":\"arn: aws: iam: : account-id: role/role-name\"},\"ReferenceSchema\":{\"RecordFormat\":{\"RecordFormatType\":\"CSV\",\"MappingParameters\":{\"CSVMappingParameters\":{\"RecordRowDelimiter\":\"\n\",\"RecordColumnDelimiter\":\", \"}}},\"RecordEncoding\":\"UTF-8\",\"RecordColumns\":[{\"Name\":\"key1\",\"SqlType\":\"VARCHAR(64)\"},{\"Name\":\"key2\",\"SqlType\":\"VARCHAR(64)\"}]}}" --current-application-version-id 2
But this should be mentioned in the documentation."
One note: it may be preferable to use JSON files as inputs and use this syntax instead: --cli-input-json file://input.json. This is referenced in the AWS Kinesis CLI Command Reference (first link, under 1.) and also mentioned in the GitHub link above. It's also the method used by the majority of the AWS Kinesis documentation. For example, JSON files used for different purposes in Kinesis Analytics:
https://docs.aws.amazon.com/kinesisanalytics/latest/dev/how-it-works-input.html
Please let me know what works, and I will work with my AWS rep to improve the documentation.

Why is there a <script> tag in the UserData field in AWS CloudFormation?

In the CloudFormation JSON, we have a base64-encoded UserData field containing a what looks like a batch script to me (on Windows at least). But it's surrounded by a <script></script>.
This is strange since the format is JSON, not even XML, so what is a XML tag doing there? Is it some legacy thing, or is it mimicking the <script> tag in web browsers?
Example:
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"<script>\n",
"cfn-init.exe -v -s MyStack -r MyResource --region us-west-1b\n",
"</script>\n"
]
]
}
}
The User Data field passes information to the Amazon EC2 instance. It could be a password for a database, an IP address for a back-end server or anything you choose.
However, software installed on the standard Windows AMIs also look for:
<script>...</script> -- Anything within those tags will be executed on the Command Line the first time the instance boots
<powershell>...</powershell> -- Anything within those tags will be executed in PowerShell the first time the instance boots
See: Running Commands on Your Windows Instance at Launch
For Amazon Linux instances, User Data that begins with #! will be executed in the shell.
So... Why is XML inside a string? It's because the string is simply being passed via User Data to the utility software on the instance and the software is looking for those tags. Without them, the script will not be executed.