I've created a step-function involving 2 lambdas and a choice already, and I'd basically like to create a CloudFormation resource referencing that, or a CloudFormation resource that I can just paste in the definition of my existing step function. I've been searching and found that you seemingly can just paste in the definition of your step function for the DefinitionString property; the problem with that is I use a lot of substitutions/"$" in my definition. I see that some examples are replacing their "$" values with curly braces, but I was wondering if that's required?
Here's one example:
"ResultSelector":{"start_date.$":"$.Payload.body.payload[0]","end_date.$":"$.Payload.body.payload.[1]"
Would I have to replace even the Keys with curly braces? Also, would I have to get rid of the prefixing "." before the rest of the value? I guess I'm basically asking for an example of how I'd properly replace this code snippet for a YAML template
Related
I've designed a data transformation in Dataprep and am now attempting to run it by using the template in Dataflow. My flow has several inputs and outputs - the dataflow template provides them as a json object with key/value pairs for each input & location. They look like this (line breaks added for easy reading):
{
"location1": "project:bq_dataset.bq_table1",
#...
"location10": "project:bq_dataset.bq_table10",
"location17": "project:bq_dataset.bq_table17"
}
I have 17 inputs (mostly lookups) and 2 outputs (one csv, one bigquery). I'm passing these to the gcloud CLI like this:
gcloud dataflow jobs run job-201807301630 /
--gcs-location=gs://bucketname/dataprep/dataprep_template /
--parameters inputLocations={"location1":"project..."},outputLocations={"location1":"gs://bucketname/output.csv"}
But I'm getting an error:
ERROR: (gcloud.dataflow.jobs.run) unrecognized arguments:
inputLocations=location1:project:bq_dataset.bq_table1,outputLocations=location2:project:bq_dataset.bq_output1
inputLocations=location10:project:bq_dataset.bq_table10,outputLocations=location1:gs://bucketname/output.csv
From the error message, it looks to be merging the inputs and outputs so that as I have two outputs, each two inputs are paired with the two outputs:
input1:output1
input2:output2
input3:output1
input4:output2
input5:output1
input6:output2
...
I've tried quoting the input/output objects (single and double, plus removing the quotes in the object), wrapping them in [], using tildes but no joy. Has anyone managed to execute a dataflow job with multiple inputs?
I finally found a solution for this via a huge process of trial and error. There are several steps involved.
Format of --parameters
The --parameters argument is a dictionary-type argument. There are details on these in a document you can read by typing gcloud topic escaping in the CLI, but in short it means you'll need an = between --parameters and the arguments, and then the format is key=value pairs with the value enclosed in quote marks ("):
--parameters=inputLocations="object",outputLocations="object"
Escape the objects
Then, the objects need the quotes escaping to avoid ending the value prematurely, so
{"location1":"gcs://bucket/whatever"...
Becomes
{\"location1\":\"gcs://bucket/whatever\"...
Choose a different separator
Next, the CLI gets confused because while the key=value pairs are separated by a comma, the values also have commas in the objects. So you can define a different separator by putting it between carats (^) at the start of the argument and between the key=value pairs:
--parameters=^*^inputLocations="{"\location1\":\"...\"}"*outputLocations="{"\location1\":\"...\"}"
I used * because ; didn't work - maybe because it marks the end of the CLI command? Who knows.
Note also that the gcloud topic escaping info says:
In cmd.exe and PowerShell on Windows, ^ is a special character and
you must escape it by repeating it. In the following examples, every time
you see ^, replace it with ^^^^.
Don't forget customGcsTempLocation
After all that, I'd forgotten that customGcsTempLocation needs adding to the key=value pairs in the --parameters argument. Don't forget to separate it from the others with a * and enclose it in quote marks again:
...}*customGcsTempLocation="gs://bucket/whatever"
Pretty much none of this is explained in the online documentation, so that's several days of my life I won't get back - hopefully I've helped someone else with this.
I'm using Jenkins with a branch parameter to specify the branch to build from. Other stuff downstream needs the branch name to not have the leading "origin/" -- just "feature/blahblah" or "bugfix/12345" or similar. The advanced settings for the parameter let me specify a branch filter via regex, but I'm a regex newbie and the solutions I've found in searching are language-dependent. The Jenkins documentation for this is sparse.
When a user clicks on "build with parameters", for the branch I want to see branch names that omit the leading "origin/". I'm not sure how to write a regex for Jenkins that will "consume" that part of the branch name before setting the parameter value.
I solved this problem once before, I'm pretty sure using Stack Overflow, but I can't find those hints now.
For the git branch parameter, set Branch Filter to:
origin/(.*)
I found the parentheses to be counter-intuitive, because if you don't specify a filter you get:
.*
(No parens.) If you are filtering stuff out, you use parens to indicate the part to keep.
I usually use a groovy script evaluated before the job, like:
def map = [:]
map['GIT_BRANCH'] = GIT_BRANCH - 'origin/'
return map
This is using the EnvInject plugin, as described in gitlab-plugin issue 444
If you need to filter multiple patterns without origin/ section, try the following.
origin/(develop.*|feature.*|bugfix.*)
This will list the develop, feature and bugfix branches without the leading origin/.
I'm using Doxygen to document C++ code, and am writing a substantial amount of Doxygen doc for the code. In one place I'm making a list of groups in the code, and would like it to appear as follows:
Control Module: the module that controls everything
Slave Module: the module that is the slave of the Control Module
My documentation source looks like this:
- #ref CM: the module that controls everything
- #ref SM: the module that is the slave of the #CM
But, problem: Doxygen seems to be reading the reference name as CM:, not CM, and thus can't find the reference. So, somehow I need to tell Doxygen where the reference name ends. (For example, if I were using Bash, and wanted to echo a variable string with an "s" as a suffix, I'd use echo "${NOUN}s".)
As a workaround, I could add a space between the name and the subsequent colon, but that makes the resulting doc harder to read and I'd like to avoid it.
Under Special Commands, the Doxygen manual includes the following hopeful-sounding information:
Some commands have one or more arguments. Each argument has a certain
range:
If <sharp> braces are used the argument is a single word.
If (round) braces are used the argument extends until the end of the line on
which the command was found.
If {curly} braces are used the argument
extends until the next paragraph. Paragraphs are delimited by a blank
line or by a section indicator.
OK, that's all fine and good, but the documentation doesn't say, and I can't figure out, where those braces are supposed to go. Around the argument alone? Around the entire command and argument? Neither works, and I can't come up with an alternative that does work.
So, how do I indicate the end of a reference name to Doxygen? And if braces are the answer, where do they go?
This works for Doxygen version 1.8.11:
\ref name "":
Apparently, the empty string triggers a fall-back to use the name argument before it.
The Doxygen documentation you quote is describing the syntax of the Doxygen documentation, not of sources to be parsed by your use of Doxygen.
In other words, if <sharp> braces are used when describing a command, it takes a single word; and so on.
Looking at the documentation of #ref:
\ref <name> ["(text)"]
The name argument is in "sharp braces," and so it's just a single word. Unfortunately, Doxygen seems to interpret : as part of that word. Your best bet would be to introduce a space:
#ref CM : the ...
You could also try whether a zero-width character would break the word recognition:
#ref CM: the ...
I'm trying to create a regular expression in sieve. The implementation of sieve that I'm using is Dovecot Pigeonhole
I'm subscribed to github project updates and I receive emails from github with the subject in the format that looks like this:
Re: [Opserver] Create issues on Jira from Exception details page (#77)
There is a project name in square bracket included in the subject line. Here is the relevant part of my sieve script:
if address "From" "notifications#github.com" {
if header :regex "subject" "\\[(.*)\\]" {
set :lower :upperfirst "repository" "${1}";
fileinto :create "Subscribtions.GitHub.${repository}"; stop;
} else {
fileinto :create "Subscribtions.GitHub"; stop;
}
}
As you can see from the above, I'm moving the messages to appropriate project IMAP folders. So the message with the subject above will end up in Subscribtions.Github.Opserver
Unfortunately, there is one small problem with this script. If someone adds square brackets in the title of their github issue, the filter breaks. For example if the subject is:
[Project] [Please look at it] - very weird issue
The above filter will move the message to folder Subscribtions.Github.Project] [please look at it which is completely undesirable. I'd like it to be moved to Subscribtions.Github.Project anyway.
This happens because by default regular expressions are greedy. So they match the longest possible match. However when I try to fix it the usual way changing "\\[(.*)\\]" to "\\[(.*?)\\]" nothing seems to change.
How do I write this regular expression so that it acts as desired?
The answer is to change "\\[(.*)\\]" to "\\[([^]]*)\\]".
By reading regex spec linked in the question we disvover that POSIX regular expression are used. Unfortunately those do not support non-greedy matches.
However there is a work around in this particular case, given above.
Background
I use JScript (Microsoft's ECMAScript implementation) for a lot of Windows system administration needs. This means I use a lot of ActiveX (Automated COM) objects. The methods of these objects often expect Number or Boolean arguments. For example:
var fso = new ActiveXObject("Scripting.FileSystemObject");
var a = fso.CreateTextFile("c:\\testfile.txt", true);
a.WriteLine("This is a test.");
a.Close();
(CreateTextFile Method on MSDN)
On the second line you see that the second argument is one that I'm talking about. A Boolean of "true" doesn't really describe how the method's behavior will change. This isn't a problem for me, but my automation-shy coworkers are easily spooked. Not knowing what an argument does spooks them. Unfortunately a long list of constants (not real constants, of course, since current JScript versions don't support them) will also spook them. So I've taken to documenting some of these basic function calls with inline block comments. The second line in the above example would be written as such:
var a = fso.CreateTextFile("c:\\testfile.txt", /*overwrite*/ true, /*unicode*/ false);
That ends up with a small syntax highlighting dilemma for me, though. I like my comments highlighted vibrantly; both block and line comments. These tiny inline block comments mean little to me, personally, however. I'd like to highlight those particular comments in a more muted fashion (light gray on white, for example). Which brings me to my dilemma.
Dilemma
I'd like to override the default syntax highlighting for block comments when both the beginning and end marks are on the same line. Ideally this is done solely in my vimrc file, and not in a superseding personal copy of the javascript.vim syntax. My initial attempt is pathetic:
hi inlineComment guifg=#bbbbbb
match inlineComment "\/\*.*\*\/"
Straight away you can see the first problem with this regular expression pattern is that it's a greedy search. It's going to match from the first "/*" to the last "*/" on the line, meaning everything between two inline block comments will get this highlight style as well. I can fix that, but I'm really not sure how to deal with my second concern.
Comments can't be defined inside of String literals in ECMAScript. So this syntax highlighting will override String highlighting as well. I've never had a problem with this in system administration scripts, but it does often bite me when I'm examining the source of many javascript libraries intended for browsers (less.js for example).
What regex pattern, syntax definition, or other solution would the amazing StackOverflow community recommend to restore my vimrc zen?
I'm not sure, but from your description it sounds like you don't need a new syntax definition. Vim syntax files usually let you override a particular syntax item with your own choice of highlighting. In this case, the item you want is called javaScriptComment, so a command like this will set its highlighting:-
hi javaScriptComment guifg=#bbbbbb
but you have to do this in your .vimrc file (or somewhere that's sourced from there), so it's evaluated before the syntax file. The syntax file uses the highlight default command, so the syntax file's choice of highlighting only affects syntax items with no highlighting set. See :help :hi-default for more details on that. BTW, it only works on Vim 5.8 and later.
The above command will change all inline /* */ comments, and leave // line comments with their default setting, because line comments are a different syntax item (javaScriptLineComment). You can find the names of all these groups by looking at the javascript.vim file. (The easiest way to do this is :e $VIMRUNTIME/syntax/javascript.vim .)
If you only want to change some inline comments, it's a little more complicated, but still easy to see what to do by looking at javascript.vim . If you do that, you can see that block comments are defined like this:-
syn region javaScriptComment start="/\*" end="\*/" contains=#Spell,javaScriptCommentTodo
See that you can use separate regexes for begin and end markers: you don't need to worry about matching the stuff in between with non-greedy quantifiers, or anything like that. To have a syntax item that works similarly but only on one line, try adding the oneline option (:h :syn-oneline for more details):-
syn region myOnelineComment start="/\*" end="\*/" oneline
I've removed the two contains groups because (1) if you're only using it for parameter names, you probably don't want spell-checking turned on inside these comments, and (2) contained sections that aren't oneline override the oneline in the container region, so you would still match all TODO comments with this region.
You can define this new kind of comment region in your .vimrc, and set the highlighting how you like: it looks like you already know how to do that, so I won't go into more details on that. I haven't tried out this particular example, so you may still need a bit of fiddling to make it work. Give it a try and let me know how it goes.
Why don't you simply add a comment line above the call?
I think that
// fso.CreateTextFile(filename:String, overwrite:Boolean, unicode:Boolean)
var a = fso.CreateTextFile("c:\\testfile.txt", true, false);
is a lot more readable and informative than
var a = fso.CreateTextFile("c:\\testfile.txt", /*overwrite*/ true, /*unicode*/ false);