I have some scripts that pipe "kubectl ... -o json" to "jq". I've been converting many of those occurrences to use the "jsonpath" parameter to kubectl so I can skip the jq step.
There's one set of these that I've been unable to convert, because of punctuation issues. The original line looks something like this:
kubectl ... -o json | jq -r '.data["application.properties"]'
In the json, there is a "data" map whose keys can have periods in the names, so I have to index it as an array, with the name as the key. This works fine using jq.
The simple-minded conversion of this to jsonpath looks something like this:
kubectl ... -o=jsonpath="{.data["application.properties"]}"
This fails with:
error: error parsing jsonpath {.data[application.properties]}, invalid array index application.properties
This doesn't appear to be a quoting problem, as it is clearly detecting what my attempted array index is, it just won't let me use it.
Is there any way to do this entirely with jsonpath?
The trick is to convert
{.data["application.properties"]}
To:
{.data.application\.properties}
(with quoting as required)
Related
Why does this work gcloud config list project --format 'value(core.project)' but not this gcloud config list project --format=value(core.project)? The documentation uses the = notation. I get the error number expected when using it. My guess is that it's trying to evaluate the projection value(core.project) as a number and the `` tells it to evaluate as a string.
I'm unfamiliar with zsh but the bash returns a syntax error unrecognized token '(' if you do this.
This is not an issue with gcloud per se.
The issue is that zsh (and bash) shells have their own interpretation of (...). In bash, this is the command for a subshell.
The solution is to ensure that the flag values are passed to the command as-is rather than be evaluated by the shell.
As #hobbs points out correctly, you can use '...' or "..." to wrap the flag value correctly without issue. My preference is --flag=value and, when using bash, '...' when no variable expansion is ever desired and "..." when it is.
My personal preference is to always use = and default to "..."
gcloud config list project \
--format="value(core.project)"
I am trying to build a connection string that requires pulling 3 IP addresses from another config file. When I get those values, I need to replace the port on each. I plan to replace each port using simple Bash find and replace ${string/pattern/replacement} but my problem is I'm stuck on the best way to parse the pattern out of the IP.
Here is what I have so far:
myFile.config:
ip.1=ip-ip-1-address:1234:5678
ip.2=ip-ip-2-address:1234:5678
ip.3=ip-ip-3-address:1234:5678
Copying some other simple process, I found I can pull the value of each IP like this:
IP1=`grep "ip.1=" /path/to/conf/myFile.config | awk -F "=" '{print $2}'`
which gives me ip.1=ip-ip-1-address:1234:5678. However, I need to replace 1234:5678 with 6543 for example. I've been looking around and I found this awesome answer that detailed using Bash prefix substitution but that relies on knowing the parameter. for example, I would have to do it this way:
test=${ip1##ip-ip-1-address:}
which results in $test being 1234:5678. That's fine but maybe I don't know the IP address as the parameter, so I'm back to considering regex unless there's a way for me to use * as the parameter or something, but I have been unsuccessful so far. For regex, I have tried a bunch such as test=${ip1/(?<=:).*/}.
Note that ${ip1/(?<=:).*/} you tried is an example of string manipulation syntax that does not support regex, only specific patterns.
You seem to want
x='ip.1=ip-ip-1-address:1234:5678'
echo "${x%%:*}:6543" # => ip.1=ip-ip-1-address:6543
The ${x%%:*} takes the value of x and removes all chars from the end till the first : including it. :6543 is added to the result of this manipulation using "${x%%:*}:6543".
To extract that value, you may also use
awk '/^ip\.1=/{sub("^[^:]+:", "");print}' myFile.config
The awk command finds lines starting with ip.1= and then removes all text from the start till the first colon including the colon and only prints these values.
I've designed a data transformation in Dataprep and am now attempting to run it by using the template in Dataflow. My flow has several inputs and outputs - the dataflow template provides them as a json object with key/value pairs for each input & location. They look like this (line breaks added for easy reading):
{
"location1": "project:bq_dataset.bq_table1",
#...
"location10": "project:bq_dataset.bq_table10",
"location17": "project:bq_dataset.bq_table17"
}
I have 17 inputs (mostly lookups) and 2 outputs (one csv, one bigquery). I'm passing these to the gcloud CLI like this:
gcloud dataflow jobs run job-201807301630 /
--gcs-location=gs://bucketname/dataprep/dataprep_template /
--parameters inputLocations={"location1":"project..."},outputLocations={"location1":"gs://bucketname/output.csv"}
But I'm getting an error:
ERROR: (gcloud.dataflow.jobs.run) unrecognized arguments:
inputLocations=location1:project:bq_dataset.bq_table1,outputLocations=location2:project:bq_dataset.bq_output1
inputLocations=location10:project:bq_dataset.bq_table10,outputLocations=location1:gs://bucketname/output.csv
From the error message, it looks to be merging the inputs and outputs so that as I have two outputs, each two inputs are paired with the two outputs:
input1:output1
input2:output2
input3:output1
input4:output2
input5:output1
input6:output2
...
I've tried quoting the input/output objects (single and double, plus removing the quotes in the object), wrapping them in [], using tildes but no joy. Has anyone managed to execute a dataflow job with multiple inputs?
I finally found a solution for this via a huge process of trial and error. There are several steps involved.
Format of --parameters
The --parameters argument is a dictionary-type argument. There are details on these in a document you can read by typing gcloud topic escaping in the CLI, but in short it means you'll need an = between --parameters and the arguments, and then the format is key=value pairs with the value enclosed in quote marks ("):
--parameters=inputLocations="object",outputLocations="object"
Escape the objects
Then, the objects need the quotes escaping to avoid ending the value prematurely, so
{"location1":"gcs://bucket/whatever"...
Becomes
{\"location1\":\"gcs://bucket/whatever\"...
Choose a different separator
Next, the CLI gets confused because while the key=value pairs are separated by a comma, the values also have commas in the objects. So you can define a different separator by putting it between carats (^) at the start of the argument and between the key=value pairs:
--parameters=^*^inputLocations="{"\location1\":\"...\"}"*outputLocations="{"\location1\":\"...\"}"
I used * because ; didn't work - maybe because it marks the end of the CLI command? Who knows.
Note also that the gcloud topic escaping info says:
In cmd.exe and PowerShell on Windows, ^ is a special character and
you must escape it by repeating it. In the following examples, every time
you see ^, replace it with ^^^^.
Don't forget customGcsTempLocation
After all that, I'd forgotten that customGcsTempLocation needs adding to the key=value pairs in the --parameters argument. Don't forget to separate it from the others with a * and enclose it in quote marks again:
...}*customGcsTempLocation="gs://bucket/whatever"
Pretty much none of this is explained in the online documentation, so that's several days of my life I won't get back - hopefully I've helped someone else with this.
I want to filter a log file using logcheck, but instead of setting rules for the log messages I want to ignore, I would like to define rules just for the ones that I want to receive in my inbox. Is it possible to do somehting like that using negative logic (like egrep -v)?
This Regex [^abc] equals not a, b, or c
I can't give great examples, but I believe you can ignore everything and then list the patterns you want flagged in either the violations.d or cracking.d directories.
So in ignore.d.server, remove all files, create a single file containing '^.'
Define patterns in violations.d
You can create a separate config directory for this, copying all of the /etc/logcheck contents to a new directory, editing logcheck.conf to list the correct RULEDIR, and then test with a command something like:
sudo -u logcheck logcheck -c /tmp/logcheck/logcheck.conf -o -t
What is the correct syntax to set a custom TextInputFormat delimiter in Pig? I've tried several variations on the following but its treating it as string values instead of Carriage Return Line Feed.
set textinputformat.record.delimiter '\r\n';
Pig Version is 0.12.0-cdh5.9.0 and Hadoop Version is 2.6.0-cdh5.9.0
Not ideal but a workaround:
Create a properties file like myprops.properties which contains the following line: textinputformat.record.delimiter=\r\n
Then run your script like: pig -P ~/myprops.properties -f path/to/pigscript.pig
It looks like this is a known issue as mentioned in the fourth paragraph of the fourth comment of: PIG_4572
Here is the syntax
SET textinputformat.record.delimiter '<delimiter>';
This works for me