Here's an example of pretty basic shell alias in git
[alias]
em = "!git branch --merged | egrep -v '\\*\\|master' | xargs -L 1 -n 1 echo"
It echos the branches that are already merged exluding the master branch (and the current branch you're on). Pretty simple. However the results are different when running this alias and just the script. Example:
will ~/example master
$ git branch --merged | egrep -v '\\*\\|master' | xargs -L 1 -n 1 echo
feature
will ~/example master
$ git em
feature
*
master
For some reason it's like egrep clause isn't even run. In fact (since I'm OS X) it's almost like it ran the grep command instead! Like this: (note the grep vs egrep)
will ~/example master
$ git branch --merged | grep -v '\\*\\|master' | xargs -L 1 -n 1 echo
feature
*
master
Has anybody come across this before?
I've read up on these SO questions and they aren't dealing with the same thing I am. The top one is the closest, but has to do with from where shell command alias are run (the toplevel of the repo).
Why does running command as git alias gives different results?
Bash script produces different result when executed from shell prompt than executed by cron
It appears that the alias definition undergoes some amount of shell processing (or at least backslash processing) before it is defined. Add another set of escaped backslashes to the alias.
[alias]
em = "!git branch --merged | egrep -v '\\\\*\\\\|master' | xargs -L 1 -n 1 echo"
Related
I am exporting all AWS ElasticBeanstalk environment variables and piping the output to serve as command line arguments using xargs.
export $(/opt/elasticbeanstalk/bin/get-config --output YAML environment | sed -r 's/: /=/' | xargs)
Most strings in the YAML output from get-config are not quoted, so the above command snippet breaks when it encounters any environment variable with a space in its value, because the export command uses spaces to delimit the beginning of a new key-value pair.
For example, say I have an environment variable called TEST_VARIABLE with a value of THIS STRING. The above command fails with an error:
-bash: export: `THIS STRING': not a valid identifier
In essence, my question is, how do I get /opt/elasticbeanstalk/bin/get-config --output YAML environment to quote all strings?
I'll be using below env.yaml file as example (I don't use AWS BS so I don't know if there would be a heavy syntax difference), next time please provide an edited example:
env.yaml
env1: this the 1st
env2: this the 2nd
In any case, piping to xargs will be hard for quotes to be kept (as they'd end up being interpreted by a shell, but then you'd need to re-quote them).
You should instead try to produce equivalent (several) export lines, to be consumed by the running shell, something on the lines of source <( output with several export x="..." lines) (valid syntax for bash, zsh, and maybe others).
Pasting below two possibilities:
using only sed
Below solution works (I chose single quotes), on the assumption that there's no single-quoted value.
$ sed -E "s/(.+): (.+)/export \1='\2'/" env.yaml
export env1='this the 1st'
export env2='this the 2nd'
$ source <(sed -E "s/(.+): (.+)/export \1='\2'/" env.yaml)
$ env|egrep ^env
env1=this the 1st
env2=this the 2nd
using yq
Use https://github.com/kislyuk/yq for the needed quoting then sed for the : substitution:
$ yq '.. style="single"' env.yaml|sed -e 's/^/export /' -e 's/: /=/'
export env1='this the 1st'
export env2='this the 2nd'
$ source <(yq '.. style="single"' env.yaml|sed -e 's/^/export /' -e 's/: /=/')
$ env|egrep ^env
env1=this the 1st
env2=this the 2nd
I'm trying to compose up a docker-compose file and there is one step where I ran grep -v '^#' .env to get all the uncommented lines from .env file to set environment variables.
In the .env file, it looks like this
DB_ENGINE=django.db.backends.postgresql
However, after running grep -v '^#' .env and I check the environment, variable DB_ENGINE has value of "django.db.backends.postgresql\r" (notice the trailing \r there).
How can I overcome this? I've been doing this for a long time and it has never happened before.
Without trying a dos2unix on all your files, you could simply remove any \r from your grep result, as in here, with:
grep -v '^#' .env| tr -d '\r'
I am trying to put together a simple script that does a regex on a git clone output and captions the directory that the default clone cmd is cloning too and then captures this into a shell for or something that can then be piped into xargs?
E.g:
git clone git#github.com:thorchain/instaswap-sdk.git |& grep "\'\S*\'" | xargs cd
In this example grep is not the right tool as its output is the matching line and not the match. However, for the life of me, I can't seem to find a simple regex matching tool?
Thanks in adv
Luke
Turns out the missing bit was the grep -o that I needed. What I was trying to do was built a bash function the git cloned and cd into the repos directory.
End solution here:
function gitc() {
cd `git clone "$#" |& grep -o "\'\S*\'" | tr -d "'"`
}
So I am writing a few scripts for migrating SVN to GIT, we have a bunch of "old" branches in SVN that still exist but don't need to be moved to GIT. (Branches which happened to have already been merged to trunk).
After a bit of google-fu I've come up with the following:
$(git for-each-ref --format='%(refname:short)' --merged origin/trunk | grep '(?!origin\/trunk)origin\/.*')
To be passed to
git branch -D --remote _previouscommandgoeshere_
If I run just git for-each-ref --format='%(refname:short)' --merged origin/trunk I get the following output:
origin/IR1091
origin/IR1102
origin/IR1105
...
origin/IR932
origin/Software
origin/trunk
origin/trunk#6792
origin/trunk#6850
When I add the grep command I get 0 values.
However, https://regexr.com/3ot1t has thaught me that my regexp is doing exactly what I want to do. Remove all branches except for the trunk branch.
What is wrong with the regexp/grep? (note I am not a linux/grep guru. This is all done in bash that comes with windows git)
The regexp is right, but grep by default does not support PCRE expression constructs like Negative look-ahead (?!. You need to enable the -P flag to enable the PCRE library, without that it just supports the Basic Regular Expression engine
.. | grep -oP '(?!origin\/trunk)origin\/.*'
Or use a perl regex match on the command line for which no flags need to be set up
.. | perl -ne 'print if /(?!origin\/trunk)origin\/.*/'
grep -P 'origin/(?!trunk)'
just this, can match what your wanted
(update from 14th Nov 14:25:
This question can be closed as it faces multiple issues and should not be confused with each other.
Later on I will come back and add the reference to the more specific question.
Inspired by https://stackoverflow.com/a/39408777 I made an table of possible cases and was able to find identify 2 isolated cases:
+------------------+-------------------------------------------+------------------------+---------------------------------------------------+
| | No Entrypoint | Entrypoint (JSON-form) | Entrypoint (shell-form) |
+------------------+-------------------------------------------+------------------------+---------------------------------------------------+
| No CMD | HostConfig.Config.cmd=/bin/bash is called | breaks | ok |
| | (assumption as of docker inspect) | (See Case 1) | |
+------------------+-------------------------------------------+------------------------+---------------------------------------------------+
| CMD (JSON-form) | breaks(See Case 1) | breaks | breaks |
| | | (See Case 1) | (See Case 2) |
+------------------+-------------------------------------------+------------------------+---------------------------------------------------+
| CMD (shell-form) | ok | ok | Breaks [seems to work as designed] |
| | | | (both are called with a shell concatinated) |
| | | | Example: /bin/sh -c <ENTRYPOINT> /bin/sh -c <CMD> |
+------------------+-------------------------------------------+------------------------+---------------------------------------------------+
Case 1 – escaping issue of double quotes for the exec form.
Possible workarounds:
Create a separate shell script and handle all the logic within this script.
Case 2 - Parameter of CMD in exec form get lost if used with Entrypoint
(
used wording:
exec form using json array -> CMD["coommand", "parameter"]
shell form -> CMD command parameter
see: https://docs.docker.com/engine/reference/builder/#cmd
)
(update from 14th Nov 12:16:
I tried to reproduce this behavior with ubuntu:latest and faced a different behavior with bash and dash. After some trial and error I can say that it works in the shellform for both - ENTRYPOINT and CMD.
I will investigate further what happens when I make use of the jsonform. The escape directive didn't changed the behavior)
my question is not focusing on the decision making when to use ENTRYPOINT or CMD but more how it behaves when it comes to the pass through of the parameter to the container. But t.b.h. I'm not sure if this is a docker or alpine environment / shell related question -> tagged both.
Expected behavior
Use ENTRYPOINT to set stable default command and arguments and use CMD to set overwritable parameter.
Those parameter are then passed through correctly to the container.
Actual behavior
The called executable behaves diffently in three different scenarios. Each three cases will be described below.
Case 1:
Parameter are passed through as expected. But I wasn't able to achive it with ENTRYPOINT. Only CMD.
Case 2:
Combination of ENTRYPOINT and CMD. Two chars "/\" with unknown source appear.
Case 3:
Combination of ENTRYPOINT (shell form) and CMD. Parameter from CMD seem to vanish.
Obervations to fix Case 3 with an interactive shell (docker exec -it --entrypoint="/bin/ash"):
escape all parameter with a single quote from
/bin/sh -c /usr/bin/run.sh server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"
to
/bin/sh -c '/usr/bin/run.sh server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"'
But why does it work in Case 1 with CMD only? And the SHLVL is set to the value '2'.
What I tried so far
diff the docker inspect output of relevant container -> nothing suspicious for me (can be posted if required)
wrote a simple wrapper script to print environment variables, ps faux and parameter passed through. (output below)
Read online articles (e.g. Dockerfile reference and https://stackoverflow.com/a/39408777 (possible combinations of ENTRYPOINT and CMD))
Source of referenced run.sh -> link:
Command used to build an run the container:
"sudo docker build -t docker-hugo . && sudo docker run --rm docker-hugo"
Dockerfile CASE 1 - Works but without ENTRYPOINT (what I like to achieve):
FROM alpine:latest
RUN apk --no-cache add hugo
CMD hugo server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"
Relevant output:
Error: Unable to locate Config file. Perhaps you need to create a new site.
Run `hugo help new` for details. (Config File "config" Not Found in "[/src]") <-- "/src" as provieded via CMD
Command within Container (achived via "ps faux" and docker exec -it --entrypoint="/bin/ash")
PID USER TIME COMMAND
1 root 0:00 /bin/sh -c hugo server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"
7 root 0:00 hugo server --watch=true --bind=0.0.0.0 --source=/src --destination=/output
28 root 0:00 ash
34 root 0:00 ps faux
Dockerfile CASE 2 - Don't understand where the "/\" from the stdout is coming from
FROM alpine:latest
RUN apk --no-cache add hugo
#ENTRYPOINT ["/usr/bin/run.sh"]
ENTRYPOINT ["hugo"]
CMD ["server", "--watch=true", "--bind=0.0.0.0", "--source=\"/src\"", "--destination=\"/output\""]
Relevant output:
Error: Unable to locate Config file. Perhaps you need to create a new site.
Run `hugo help new` for details. (Config File "config" Not Found in "[/\"/src\"]") <-- where is the "/\" comming from?)
Command and environment from container (stdout from run.sh)
HOME='/root'
HOSTNAME='805e3dd6110a'
IFS='
'
OPTIND='1'
PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PPID='0'
PS1='\w \$ '
PS2='> '
PS4='+ '
PWD='/'
SHLVL='1'
PID USER TIME COMMAND
1 root 0:00 ash /usr/bin/run.sh server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"
7 root 0:00 ps faux
server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"
Dockerfile CASE 3 - Not OK - CMD Parameter seem to "vanish"
FROM alpine:latest
RUN apk --no-cache add hugo
#ENTRYPOINT ["/usr/bin/run.sh"]
ENTRYPOINT /usr/bin/run.sh
CMD ["server", "--watch=true", "--bind=0.0.0.0", "--source=\"/src\"", "--destination=\"/output\""]
Relevant output:
Error: Unable to locate Config file. Perhaps you need to create a new site.
Run `hugo help new` for details. (Config File "config" Not Found in "[/]") <-- Parameter had no effect
Command and environment from container (stdout from run.sh)
HOME='/root'
HOSTNAME='cd6df15c5028'
IFS='
'
OPTIND='1'
PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PPID='1'
PS1='\w \$ '
PS2='> '
PS4='+ '
PWD='/'
SHLVL='2'
PID USER TIME COMMAND
1 root 0:00 /bin/sh -c /usr/bin/run.sh server --watch=true --bind=0.0.0.0 --source="/src" --destination="/output"
7 root 0:00 ash /usr/bin/run.sh
8 root 0:00 ps faux