I am trying to update crawler using this command:
aws glue update-crawler --name my-crawler --configuration '{"Version":1.0,"CrawlerOutput":{"Partitions":{"AddOrUpdateBehavior":"InheritFromTable"}}}' --region us-west-2
As described here
Instead of update I got:
An error occurred (InvalidInputException) when calling the UpdateCrawler operation: Crawler configuration not valid: Error parsing JSON: Received JsonParseException: Unexpected character (''' (code 39)): expected a valid value (number, String, array, object, 'true', 'false' or 'null'). Check that your JSON is well formed. For more information about the crawler configuration structure, see http://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-crawling.html.
The jsonlint tells me that json is ok.
What is wrong? How pass json as parameter for aws cli?
cli is used under windows 10
You have to escape the quotes under Windows:
aws glue update-crawler --name my-crawler --configuration "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"}}}" --region us-west-2
For Windows, you have to do some "special" escaping, which I've learned the hard way. Take the following JSON snippet...
{ "#t": "timestamp" }`
Here's how you'd enter it on Windows...
DOS
aws dynamodb scan --table-name MyTable --region "us-east-1" --profile dev --projection-expression "failureKey, #t" --expression-attribute-names "{ ""#t"": ""timestamp"" }"
For Powershell, it's a little different...
Powershell
aws dynamodb scan --table-name "MyTable" --region "us-east-1" --profile "dev" --projection-expression "failureKey, #t" --expression-attribute-names '{ \"#t\": \"timestamp\" }'
Used an example with a shorter JSON snippet, but you get the idea. Apply the same concept to your string based on the shell your using.
Related
I'm trying to pass in a variable value to map Jarfile name in dynamoDB table using AWS CLI.
aws dynamodb put-item --table-name epis-deployment-history --item "{\"JarFile\":{\"S\":$JarFile}}" --return-consumed-capacity TOTAL
It threw this error. It substitutes the value correctly but the CLI commands fail to run.
Error parsing parameter '--item': Invalid JSON: Expecting value: line 1 column 17 (char 16)
JSON received: {"JarFile":{"S":medallia-dealertrack-93311b0-20210301-133510.jar}}
Deploy to preprod Complete.
Thank you
I am trying to set redrive policy for SQS using the AWS CLI Command below , but seeing an error related to redrive JSON. Can you please let me know how I can fix this?
redrive_policy="{\"RedrivePolicy\":{\"deadLetterTargetArn\":\"$dlq_arn\",\"maxReceiveCount\":\"15\"}}"
AWS CLI COMMAND
aws sqs set-queue-attributes --queue-url https://queue.amazonaws.com/12345678/test-queue --attributes $redrive_policy --region=us-east-1
Error Message
Parameter validation failed: Invalid type for parameter
Attributes.RedrivePolicy, value: OrderedDict([(u'deadLetterTargetArn',
u'arn:aws:sqs:us-east-1:12345678:dlq'), (u'maxReceiveCount', u'15')]),
type: , valid types:
Have you tried just creating the JSON in a separate file and passing it as an argument to your AWS CLI command? I find it's difficult to get all of the escaping correct when passing the JSON as a parameter. So you'd basically do it as the example shows in the AWS documentation:
https://docs.aws.amazon.com/cli/latest/reference/sqs/set-queue-attributes.html#examples
So first you'd create a new file called "set-queue-attributes.json" like so:
{
"DelaySeconds": "10",
"MaximumMessageSize": "131072",
"MessageRetentionPeriod": "259200",
"ReceiveMessageWaitTimeSeconds": "20",
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:80398EXAMPLE:MyDeadLetterQueue\",\"maxReceiveCount\":\"1000\"}",
"VisibilityTimeout": "60"
}
Then run the command like this:
aws sqs set-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyNewQueue --attributes file://set-queue-attributes.json --region=us-east-1
if you want to run in the same command you can use this example:
aws sqs set-queue-attributes \
--queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyNewQueue \
--attributes '{
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:80398EXAMPLE:MyDeadLetterQueue\",\"maxReceiveCount\":\"1000\"}",
"MessageRetentionPeriod": "259200",
"VisibilityTimeout": "90"
}'
Three Methods to achieve this:
Note: The solutions also work on any other AWS CLI commands that require a stringified JSON
1. Using the Command-line JSON processor jq (Recommended)
This method is recommended because of many reasons:
I've found jq a handy tool to use when working with AWS CLI as the need to stringify JSON comes up quite frequently.
Install for Ubuntu: sudo apt install jq
Basic Options:
jq -R: Returns the stringified JSON
jq -c: Eliminates spacing and newline characters
The benefit is that you can write JSON as JSON and Pipe the result into the jq -R command.
Method 1:
aws sqs set-queue-attributes \
--queue-url "https://sqs.ap-south-1.amazonaws.com/IAMEXAMPLE12345678/ExampleQueue" \
--attributes RedrivePolicy=$(echo '{"maxReceiveCount":500,"deadLetterTargetArn":"arn:aws:sqs:ap-south-1:IAMEXAMPLE12345678:ExampleDeadLetterQueue"}' | jq -R)
OR if you have a sqs-redrive-policy.json file:
Method 2:
In sqs-redrive-policy.json,
{
"maxReceiveCount": 500,
"deadLetterTargetArn": "arn:aws:sqs:ap-south-1:IAMEXAMPLE12345678:ExampleDeadLetterQueue"
}
Run in Command Line:
aws sqs set-queue-attributes \
--queue-url "https://sqs.ap-south-1.amazonaws.com/IAMEXAMPLE12345678/ExampleQueue" \
--attributes RedrivePolicy=$(cat ~/path/to/file/sqs-redrive-policy.json | jq -c | jq -R)
As you can see the second benefit is that you can isolately modify only the --redrive-policy without having to touch any of the other attributes.
Common Confusion: A confusion is the name set-queue-attributes (it would be better named put-queue-attributes). as it doesn't overwrite all attributes but only overwrites the attributes mentioned with the command. So if you already set a Policy attribute earlier during create-queue, this will not overwrite the Policy to null. In other words, this is safe to use.
2. Using a stringified JSON
This is a pain to be honest, and I avoid this.
aws sqs set-queue-attributes \
--queue-url "https://sqs.us-east-1.amazonaws.com/IAMEXAMPLE12345678/ExampleQueue" \
--attributes '{
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:IAMEXAMPLE12345678:ExampleDeadLetterQueue\",\"maxReceiveCount\":\"500\"}",
}'
3. Use a filePathURL to the JSON file for attributes.json NOT sqs-redrive-policy.json
This is my last preference.
Reason:
This means setting all the attributes specified in the attributes.json file again at a single go.
Doesn't escape the pain of writing stringified JSON as text.
In attributes.json,
{
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:IAMEXAMPLE12345678:ExampleDeadLetterQueue\", \"maxReceiveCount\":\"5\"}"
}
Run in command line:
aws sqs set-queue-attributes \
--queue-url "https://sqs.ap-south-1.amazonaws.com/IAMEXAMPLE12345678/ExampleQueue" \
--attributes file:///home/yourusername/path/to/file/attributes.json
My CI pipeline will do two things
generate new lambda version and publish
Update an alias to point at that new version
This will be done via cli commands. My question is, how do I access the version number that been generated from the first command. It is returned and posted to the CLI. Can this be access easily via some nifty was command or will I have to parse it myself?
e.g.
version=$(aws lambda publish-version \
--function-name test_lambda --description "updated via cli" --region eu-west-1 \
--query Version \
--output text)
See Controlling Command Output from the AWS Command Line Interface page of AWS CLI User Guide, specifically How to Filter the Output with the --query Option and Text Output Format
This works but still curious if there is a better way.
version=$(aws lambda publish-version --function-name test_lambda --description "updated via cli" --region eu-west-1| jq '.Version')
NEW_LAMBDA_VERSION=$(aws lambda list-versions-by-function --function-name $LAMBDA_NAME_FOR_DEPLOY --no-paginate --query "max_by(Versions, &to_number(to_number(Version) || '0'))")
NEW_LAMBDA_VERSION=$(echo $NEW_LAMBDA_VERSION | jq -r .Version)
echo $NEW_LAMBDA_VERSION
In this case, I use on .gitlab-ci.yml.
i've created a clustering model on sagemaker and i'm invoking it via CLI with this command:
aws sagemaker-runtime invoke-endpoint --endpoint-name myendpoint --body $mydata --content-type text/csv output.json --region eu-west-1
If my data starts with a negative number, i get an error
"usage: aws [options] [ ...] [parameters]
To see help text, you can run:
aws help
aws help
aws help
aws: error: argument --body: expected one argument"
While if it's a positive number, everything works. How can i escape the first minus of the data to make it work?
Thanks in advance
It looks like that aws cli is treating your input data as another option because negative sign and hyphen are the same.
Have you tried to use quotes before and after $mydata?
For example, instead of:
sagemaker-runtime invoke-endpoint --endpoint-name myendpoint --body -2,1,2 --content-type text/csv output.json --region eu-west-1
use:
sagemaker-runtime invoke-endpoint --endpoint-name myendpoint --body "-2,1,2" --content-type text/csv output.json --region eu-west-1
Did you use AWS Sagemaker's provided image for clustering? If you provided your own, you should be able to modify the inference code to expect the input data to have a header row. Then modify $mydata to include column headers which should avoid this issue you're seeing with negative numbers.
I just want to get back a list of function names. Ideally I want to get all functions (just their name) starting with "some-prefix*". Can I do this with the cli?
Really want this as a cli command if possible (I want to avoid python or another sdk). I see there is a --cli-input-json arg, can I use that for filtering?
You can do that. Use the --query option. The CLI would look like this:
aws lambda list-functions --region us-east-1 --query 'Functions[].FunctionName' --output text
To get the list of functions whose name begin with some-prefix:
aws lambda list-functions --region us-east-1 --query 'Functions[?starts_with(FunctionName, `some-prefix`) == `true`].FunctionName' --output text
To get the complete JSON, the CLI would be:
aws lambda list-functions --region us-east-1
Details about the query parameter can be found here.
As the answer is already given by #krishna, but I was looking for a way to print all function name without specifying a prefix. So here you can get all lambda function name in particular region my default is us-west-2.
aws lambda list-functions --query 'Functions[*].[FunctionName]'
Or as I want them out in text format and space separated to use in my bash script so here you can get in text and single line space separated.
aws lambda list-functions --query 'Functions[*].[FunctionName]' --output text | tr '\r\n' ' '
I have come here for some help to clean up all lambda functions that I have created while following an AWS developer certification tutorial. If anyone is in the same boat, I have created a script to programmatically delete all lambda functions in my AWS account (NOT for production use)
#!/bin/bash
# STOP: DON'T USE/RUN THIS SCRIPT UNLESS YOU ARE 100% SURE WHAT YOU ARE DOING
# I am a learner and created 20+ lambda functions while following a tutorial
# I wrote this script to programatically cleanup functions at the end of course
# precondition: your aws cli is configured
# get all functions from aws account
functions=(`aws lambda list-functions --query 'Functions[*].[FunctionName]' --output text`)
for i in "${functions[#]}"
do
#delete functions 1-by-1
aws lambda delete-function --function-name "$i"
echo "deleted $i"
done
Incase, someone is looking for a similar query with string present in the lambda function name as a substring, try below
aws lambda list-functions --region us-east-1 --query 'FunctionName[?contains(FunctionName, 'containing-string'] == 'true'].[FunctionName]' --output text
Note - the '[]' brackets around '.FunctionName' will provide with each fucntionName on a new line.
You can easily get the list of all lambda functions in given region using below command:
aws lambda list-functions --region us-east-1 | jq -r .Functions[].FunctionName
Download the jq (Lightweight and flexible command-line JSON processor) from here:
https://stedolan.github.io/jq/download/