mongoexport --db ucc_prod /host:myserver /port:27017 --username user1 --password password1 /query:'{copysheet: {$regex: "/^.*pdf/"}}' /out:copysheets.csv --type=csv --fields svOrderId,svItemId --collection copies
gives me error
2016-09-02T08:17:34.632-0500 error parsing command line options: unknown option "^.*pdf/}}'"
What syntax am I missing here?
You may use
--query "{ 'copysheet': { '$regex': '^.*pdf', '$options':'' }}"
The point is that you should pass the data to the query argument as JSON.
See reference:
--query <JSON>, -q <JSON>
Provides a JSON document as a query that optionally limits the documents returned in the export. Specify JSON in strict format.
Note: on different systems, you might need to swap single with double quotes.
Related
I'm running this shell command using groovy (which worked in bash):
aws --profile profileName --region us-east-1 dynamodb update-item --table-name tableName --key '{"group_name": {"S": "group_1"}}' --attribute-updates '{"attr1": {"Value": {"S": "STOP"},"Action": "PUT"}}'
This updates the value of an item to STOP in DynamoDB. In my groovy script, I'm running this command like so:
String command = "aws --profile profileName --region us-east-1 dynamodb update-item --table-name tableName --key '{\"group_name\": {\"S\": \"group_1\"}}' --attribute-updates '{\"attr1\": {\"Value\": {\"S\": \"STOP\"},\"Action\": \"PUT\"}}'"
println(command.execute().text)
When I run this with groovy afile.groovy, nothing is printed out and when I check the table in DynamoDB, it's not updated to STOP. There is something wrong with the way I'm escaping the quotes but I'm not sure what. Would appreciate any insights.
Sidenote: When I do a simple aws command like aws s3 ls it works and prints out the results so it's something with this particular command that is throwing it off.
You don't quote for groovy (and the underlying exec) -- you would have to quote for your shell. The execute() on a String does not work like a shell - the underlyting code just splits at whitespace - any quotes are just passed down as part of the argument.
Use ["aws", "--profile", profile, ..., "--key", '{"group_name": ...', ...].execute() and ignore any quoting.
And instead of banging strings together to generate JSON, use groovy.json.JsonOutput.toJson([group_name: [S: "group_1"]])
I wrote a Powershell script that gets a filtered list of cognito-idp identities using AWS CLI. However, I wanted to make this a lambda script and realized that I could not use AWS CLI and instead needed to use the AWS for Powershell Core module.
When I use the AWS CLI command
aws cognito-idp list-users --user-pool-id $user_pool_id --filter 'email=\"foo#bar.com\"'
I get the expected result.
When I use the equivalent cmdlet from the module
Get-CGIPUserList -UserPoolId $user_pool_id -Region $region -Filter 'email=\"foo#bar.com\"'
I get a filter parsing error
Get-CGIPUserList : One or more errors occurred. (Error while parsing filter.)
At line:1 char:9
+ Get-CGIPUserList -UserPoolId "****" -Region "u ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (Amazon.PowerShe...PUserListCmdlet:GetCGIPUserListCmdlet) [Get-CGIPUserList], InvalidOperationException
+ FullyQualifiedErrorId : System.AggregateException,Amazon.PowerShell.Cmdlets.CGIP.GetCGIPUserListCmdlet
According to the module reference here:
https://docs.aws.amazon.com/powershell/latest/reference/items/Get-CGIPUserList.html the syntax for the filter parameter should be the same. What am I doing wrong?
The powershell module is failing to parse your filter string 'email=\"foo#bar.com\"' because of the escaped double quotations.
Simply remove them and you should get past this error, as the single quote ' in powershell expresses content as string literal:
'email="foo#bar.com"'
You could also wrap your filter string in double quotes ". You would generally only need to do this if your string contained a powershell variable that you would like to interpolate. You would need to replace the \ escape character in this case with powershell's escape character ` like so:
"email=`"foo#bar.com`""
I am using AWS Command Line Interface (CLI) to list some AMI Images from AWS.
The Name of an Image is like:
XY_XYZ_Docker_1.13_XYZ_XXYY
When using
aws ec2 describe-images --filters 'Name=name,Values="*_Docker_1.13_*"'
it works as expected.
Now i want to use Regular Expression instead of static value for the Name-Filter.
In the AWS-Docs I read that filtering by RegEx is possible
My approach is:
1:
aws ec2 describe-images --filters 'Name=name,Values="[_]Docker[_][0-9][.][0-9]{2}[_]"'
The result is always null for this. I tried different ways of quoting the RegEx.
2:
[_]Docker[_][0-9][.][0-9]{2}[_]
(without quotes) leads to
Error parsing parameter '--filters': Expected: ',', received: 'D' for input:
Name=name,Values=[]Docker[][0-9][.][0-9]{2}[_]
3:
*[_]Docker[_][0-9][.][0-9]{2}[_]*
(with Asterisk) leads to
Error parsing parameter '--filters': Expected: ',', received: ']' for input:
Name=name,Values=[_]Docker[_][0-9][.][0-9]{2}[_]
I wasn't able to find if Jmespath or the --filters flag can support regex, so instead I just piped to Python to run through regex.
aws ec2 describe-images --filters 'Name=name,Values="*Docker*"' | \
python -c '
import json, sys, re
obj = json.load(sys.stdin)
matched_images = {"Images":[]}
for image in obj["Images"]:
if len(re.findall(r"[Dd]ocker\s?[0-9][.][0-9]{2}", image["Name"])) > 0:
matched_images["Images"].append(image)
print json.dumps(matched_images)
'
You can pipe the output (which is just a JSON string) to your next bash command if needed with a pipe character following the closing quote. Maybe this can address concerns with using grep since it returns a JSON string instead or regular text.
See the gist below.
It covers:
search ECR images sorted descending by imagePushDate
selecting the tag that meets a Regex criteria
using that to replace a key/value pair in a yaml
https://gist.github.com/pprogrammingg/69e7c85abede9822f2480e9b5e1e66fd
I am getting an extra None in aws-cli (version 1.11.160) with --query parameter and --output text when fetching the first element of the query output.
See the examples below.
$ aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo')].TargetKeyId|[0]" --output text
a3a1f9d8-a4de-4d0e-803e-137d633df24a
None
$ aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo-bar')].TargetKeyId|[0]" --output text
None
None
As far as I know this was working till yesterday but from today onwards this extra None comes in and killing our ansible tasks.
Anyone experienced anything similar?
Thanks
I started having this issue in the past few days too. In my case I was querying exports from a cfn stack.
My solution was (since I'll only ever get one result from the query) to change | [0].Value to .Value, which works with --output text.
Some examples:
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | []'
[
{
"ExportingStackId": "arn:aws:cloudformation:ap-southeast-2:111122223333:stack/stack-name/83ea7f30-ba0b-11e8-8b7d-50fae957fc4a",
"Name": "kms-key-arn",
"Value": "arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa"
}
]
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [].Value'
[
"arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa"
]
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [].Value' --output text
arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa
aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [0].Value' --output text
arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa
None
I'm no closer to finding out why it's happening, but it disproves #LHWizard's theory, or at least indicates there are conditions where that explanation isn't sufficient.
The best explanation is that not every match for your query statement has a TargetKeyId. On my account, there are several Aliases that only have AliasArn and AliasName key/value pairs. The None comes from a null value for TargetKeyId, in other words.
I came across the same issue when listing step functions. I consider it to be a bug. I don't like solutions that ignore the first or last element, expecting it will always be None at that position - at some stage the issue will get fixed and your workaround has introduced a nasty bug.
So, in my case, I did this as a safe workaround (adapt to your needs):
#!/usr/bin/env bash
arn="<step function arn goes here>"
arns=()
for arn in $(aws stepfunctions list-executions --state-machine-arn "$arn" --max-items 50 --query 'executions[].executionArn' --output text); do
[[ $arn == 'None' ]] || arns+=("$arn")
done
# process execution arns
for arn in "${arns[#]}"; do
echo "$arn" # or whatever
done
Supposing you need only the first value:
Replace --output text with --output json and you could parsed with jq
Therefore, you'll have something like
Ps. the -r option with jq is to remove the quotes around the response
aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo')].TargetKeyId|[0]" --output | jq -r '.'
Please help me out from the below error.It works fine when checked in oracle but fails when trying through SQOOP import.
version : Hadoop 0.20.2-cdh3u4 and Sqoop 1.3.0-cdh3u5
sqoop import $SQOOP_CONNECTION_STRING
--query 'SELECT st.reference,u.unit,st.reading,st.code,st.read_id,st.avg FROM reading st,tunit `tu,unit u
WHERE st.reference=tu.reference and st.number IN ('218730','123456') and tu.unit_id = u.unit_id
and u.enrolled='Y' AND st.reading <= latest_off and st.reading >= To_Date('20120701','yyyymmdd')
and st.type_id is null and $CONDITIONS'
--split-by u.unit
--target-dir /sample/input
Error:
12/10/10 09:33:21 ERROR manager.SqlManager: Error executing statement:
java.sql.SQLSyntaxErrorException: ORA-00904: "YYYYMMDD": invalid identifier
followed by....
12/10/10 09:33:21 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.NullPointerException
Thanks & Regards,
Tamil
I believe that the problem is actually on Bash side (or your command line interpret). Your query contains for example following fragment u.enrolled='Y'. Please notice that you're escaping character constants with single quotes. You seem to be putting entire query into additional single quotes: --query 'YOUR QUERY'. Which results in something like --query '...u.enrolled='Y'...'. However such string is stripped by bash to '...u.enrolled=Y...'. You can verify that by using "echo" to see what exactly will bash do with your string before it will be passed to Sqoop.
jarcec#jarcec-thinkpad ~ % echo '...u.enrolled='Y'...'
...u.enrolled=Y..
.
I would recommend to either escape all single quotes (\') inside your query or choose double quotes for entire query. Please note that the later option will require escaping $ characters with backslash (\$).