Regex in Windows Batch to automate Docker run - regex

I am trying to automate the process of sending my temporary Amazon AWS keys as environment variables to a Docker image using Windows. I have a file, credentials.txt that contains my AWS credentials (the 3 ids are always the same, but the string values change regularly). I am using Windows command prompt.
Input:
(includes 2 empty lines at end) credentials.txt:
[default]
aws_access_key_id = STR/+ing1
aws_secret_access_key = STR/+ing2
aws_session_token = STR/+ing3
Desired output:
I need to issue the following command in order to run a Docker image (substituting the strings with the actual strings):
docker run -e AWS_ACCESS_KEY_ID=STR/+ing1 -e AWS_SECRET_ACCESS_KEY=STR/+ing2 -e AWS_SESSION_TOKEN=STR/+ing3 my-aws-container
My idea is to try to use regex on credentials.txt to convert it to:
SET aws_access_key_id=STR/+ing1
SET aws_secret_access_key=STR/+ing2
SET aws_session_token=STR/+ing3
And then run:
docker run -e AWS_ACCESS_KEY_ID=%aws_access_key_id% -e AWS_SECRET_ACCESS_KEY=%aws_secret_access_key% -e AWS_SESSION_TOKEN=%aws_session_token% my-aws-container
Does anyone have any advice on how to achieve this?

You can parse your credentials.txt with a for /f loop to set the variables (effectively removing the spaces):
for /f "tokens=1,3" %%a in ('type credentials.txt ^| find "="') do set "%%a=%%b"
and then run the last code line from your question:
docker run -e AWS_ACCESS_KEY_ID=%aws_access_key_id% -e AWS_SECRET_ACCESS_KEY=%aws_secret_access_key% -e AWS_SESSION_TOKEN=%aws_session_token% my-aws-container
Note: the values should not contain spaces or commas.

I've had a go in python that seems to work. Someone else may have a better answer.
I create the python file:
docker_run.py
import re
import os
myfile = 'C:/fullpath/credentials'
with open(myfile,'r') as f:
mystr = f.read()
vals = re.findall('=[\s]*([^\n]+)',mystr)
keys = ['AWS_ACCESS_KEY_ID','AWS_SECRET_ACCESS_KEY','AWS_SESSION_TOKEN']
environment_vars = ''.join([' -e ' + k + '=' + v for k,v in zip(keys,vals)])
cmd = 'docker run'+environment_vars+' my-aws-container'
os.system(cmd)
Then from command prompt I run:
python docker_run.py
This succeeds in running docker
(note: I tried using exec() in the final line rather than os.system(), but got the error "SyntaxError: invalid syntax")

Related

Need to convert os.system with subprocess.call

i need to convert os.system() to subprocess.call()
I have 2 complex commands executing inside os.system as (cmd1 +"&&"+cmd2), I need to convert it to subprocess.call() with shell=False(for security reasons)
example of my code :-
cmd1 ="echo paras"
cmd2 ="rm -rf paras1.txt"
os.system as (cmd1 +"&&"+cmd2)
I Tried:-
subprocess.call([cmd3,"&&",cmd4],shell=False)
subprocess.call(shlex.split(cmd3+ "&&" +cmd4),shell=False)
but failed in both cases, i need to know how can i run 2 commands inside subprocess.call() separated by "&&" with shell=False.
You need to execute the commands within a shell (for example bash):
import subprocess
shell = '/bin/bash' # you may need to adjust the path!
cmds = ['echo "foo"', '&&', 'echo "bar"']
cmd = [shell, '-c', ' '.join(cmds)]
print cmd
print subprocess.call(cmd, shell=False)
Output:
['/bin/bash', '-c', 'echo "foo" && echo "bar"']
foo
bar
0

if condition to folder branch in Jenkinsfile

I have branch folder "feature-set" under this folder there's multibranch
I need to run the below script in my Jenkinsfile with a condition if this build runs from any branches under the "feature-set" folder like "feature-set/" then run the script
the script is:
sh """
if [ ${env.BRANCH_NAME} = "feature-set*" ]
then
echo ${env.BRANCH_NAME}
branchName='${env.BRANCH_NAME}' | cut -d'\\/' -f 2
echo \$branchName
npm install
ng build --aot --output-hashing none --sourcemap=false
fi
"""
the current output doesn't get the condition:
[ feature-set/swat5 = feature-set* ]
any help?
I would re-write this to be primarily Jenkins/Groovy syntax and only go to shell when required.
Based on the info you provided I assume your env.BRANCH_NAME always looks like `feature-set/
// Echo first so we can see value if condition fails
echo(env.BRANCH_NAME)
// startsWith better than contains() based on current usecase
if ( (env.BRANCH_NAME).startsWith('feature-set') ) {
// Split branch string into list based on delimiter
List<String> parts = (env.BRANCH_NAME).tokenize('/')
/**
* Grab everything minus the first part
* This handles branches that include additional '/' characters
* e.g. 'feature-set/feat/my-feat'
*/
branchName = parts[1..-1].join('/')
echo(branchName)
sh('npm install && ng build --aot --output-hashing none --sourcemap=false')
}
This seems to be more on shell side. Since you are planning to use shell if condition the below worked for me.
Administrator1#XXXXXXXX:
$ if [[ ${BRANCH_NAME} = feature-set* ]]; then echo "Success"; fi
Success
Remove the quotes and add an additional "[]" at the start and end respectively.
The additional "[]" works as regex

Environment Variables in newest AWS EC2 instance

I am trying to get ENVIRONMENT Variables into the EC2 instance (trying to run a django app on Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type ami-0ff8a91507f77f867 ). How do you get them in the newest version of amazon's linux, or get the logging so it can be traced.
user-data text (modified from here):
#!/bin/bash
#trying to get a file made
touch /tmp/testfile.txt
cat 'This and that' > /tmp/testfile.txt
#trying to log
echo 'Woot!' > /home/ec2-user/user-script-output.txt
#Trying to get the output logged to see what is going wrong
exec > >(tee /var/log/user-data.log|logger -t user-data ) 2>&1
#trying to log
echo "XXXXXXXXXX STARTING USER DATA SCRIPT XXXXXXXXXXXXXX"
#trying to store the ENVIRONMENT VARIABLES
PARAMETER_PATH='/'
REGION='us-east-1'
# Functions
AWS="/usr/local/bin/aws"
get_parameter_store_tags() {
echo $($AWS ssm get-parameters-by-path --with-decryption --path ${PARAMETER_PATH} --region ${REGION})
}
params_to_env () {
params=$1
# If .Ta1gs does not exist we assume ssm Parameteres object.
SELECTOR="Name"
for key in $(echo $params | /usr/bin/jq -r ".[][].${SELECTOR}"); do
value=$(echo $params | /usr/bin/jq -r ".[][] | select(.${SELECTOR}==\"$key\") | .Value")
key=$(echo "${key##*/}" | /usr/bin/tr ':' '_' | /usr/bin/tr '-' '_' | /usr/bin/tr '[:lower:]' '[:upper:]')
export $key="$value"
echo "$key=$value"
done
}
# Get TAGS
if [ -z "$PARAMETER_PATH" ]
then
echo "Please provide a parameter store path. -p option"
exit 1
fi
TAGS=$(get_parameter_store_tags ${PARAMETER_PATH} ${REGION})
echo "Tags fetched via ssm from ${PARAMETER_PATH} ${REGION}"
echo "Adding new variables..."
params_to_env "$TAGS"
Notes -
What i think i know but am unsure
the user-data script is only loaded when it is created, not when I stop and then start mentioned here (although it also says [i think outdated] that the output is logged to /var/log/cloud-init-output.log )
I may not be starting the instance correctly
I don't know where to store the bash script so that it can be executed
What I have verified
the user-data text is on the instance by ssh-ing in and curl http://169.254.169.254/latest/user-data shows the current text (#!/bin/bash …)
What Ive tried
editing rc.local directly to export AWS_ACCESS_KEY_ID='JEFEJEFEJEFEJEFE' … and the like
putting them in the AWS Parameter Store (and can see them via the correct call, I just can't trace getting them into the EC2 instance without logs or confirming if the user-data is getting run)
putting ENV variables in Tags and importing them as mentioned here:
tried outputting the logs to other files as suggested here (Not seeing any log files in the ssh instance or on the system log)
viewing the System Log on the aws webpage to see any errors/logs via selecting the instance -> 'Actions' -> 'Instance Settings' -> 'Get System Log' (not seeing any commands run or log statements [only 1 unrelated word of user])

Regex syntax error with Sed command in Ubuntu 9.04

I have the sed command like this:
radius_clientsfile=clients.conf
iface_netsize="/64"
wireless_prefix=fd04:bd3:80e8:3::
sed -i "/client $wireless_prefix\\$iface_netsize/ {n s/\(\W*secret\W*=\W\).*/\1$key/}" $radius_clientsfile
clients.conf has the content like this:
client fd04:bd3:80e8:3::/64 {
secret = 00000000000000000000000000000001
}
which aim to replace value of secret by key in clients.conf file. For Example, if key is 00000000000000000000000000000002, the content of clients.conf should be changed as following:
client fd04:bd3:80e8:3::/64 {
secret = 00000000000000000000000000000002
}
This script work on OpenWRT attitude adjustment r35400 for armv5tejl
However, it can not work in Ubuntu 9.04 with error:
sed: -e expression #1, char 36: extra characters after command
Could anyone help me for this situation?
I think you need add a ; between command n and command s, like this
sed -i "/client $wireless_prefix\\$iface_netsize/ {n; s/\(\W*secret\W*=\W\).*/\1$key/}" $radius_clientsfile
This working in my cygwin environment.
You need to separate the commands in the command block with a semi-colon, so add a ; after the n command to separate it from the following command.
Like this:
{n;s/\(\W*secret\W*=\W\).*/\1$key/}

VisualSVN post-commit hook with batch file

I'm running VisualSVN on a Windows server.
I'm trying to add a post-commit hook to update our staging project whenever a commit happens.
In VisualSVN, if I type the command in the hook/post-commit dialog, everything works great.
However, if I make a batch file with the exact same command, I get an error that says the post-commit hook has failed. There is no additional information.
My command uses absolute paths.
I've tried putting the batch file in the VisualSVN/bin directory, I get the same error there.
I've made sure VisualSVN has permissions for the directories where the batch file is.
The only thing I can think of is I'm not calling it correctly from VisualSVN. I'm just replacing the svn update command in the hook/post-commit dialog with the batch file name ("c:\VisualSVN\bin\my-batch-file.bat") I've tried it with and without the path (without the path it doesn't find the file at all).
Do I need to use a different syntax in the SVNCommit dialog to call the batch file? What about within the batch file (It just has my svn update command. It works if I run the batch file from the command line.)
Ultimately I want to use a batch file because I want to do a few more things after the commit.
When using VisualSVN > Select the Repo > Properties > Hooks > Post-commit hook.
Where is the code I use for Sending an Email then running a script, which has commands I want to customize
"%VISUALSVN_SERVER%\bin\VisualSVNServerHooks.exe" ^
commit-notification "%1" -r %2 ^
--from support#domainname.com --to "support#domainname.com" ^
--smtp-server mail.domainname.com ^
--no-diffs ^
--detailed-subject
--no-html
set PWSH=%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe
%PWSH% -command $input ^| C:\ServerScripts\SVNScripts\post-commit-wp.ps1 %1 %2
if errorlevel 1 exit %errorlevel%
The script file is located on C:\ServerScripts\SVNScripts\
post-commit-wp.ps1 and I pass in two VisualSVN variables as %1 and %2
%1 = serverpathwithrep
%2 = revision number
The script file is written in Windows PowerShell
# PATH TO SVN.EXE
$svn = "C:\Program Files\VisualSVN Server\bin\svn.exe"
$pathtowebistesWP = "c:\websites-wp\"
# STORE HOOK ARGUMENTS INTO FRIENDLY NAMES
$serverpathwithrep = $args[0]
$revision = $args[1]
# GET DIR NAME ONLY FROM REPO-PATH STRING
# EXAMPLE: C:\REPOSITORIES\DEVHOOKTEST
# RETURNS 'DEVHOOKTEST'
$dirname = ($serverpathwithrep -split '\\')[-1]
# Combine ServerPath with Dir name
$exportpath = -join($pathtowebistesWP, $dirname);
# BUILD URL TO REPOSITORY
$urepos = $serverpathwithrep -replace "\\", "/"
$url = "file:///$urepos/"
# --------------------------------
# SOME TESTING SCRIPTS
# --------------------------------
# STRING BUILDER PATH + DIRNAME
$name = -join($pathtowebistesWP, "testscript.txt");
# CREATE FILE ON SERVER
New-Item $name -ItemType file
# APPEND TEXT TO FILE
Add-Content $name $pathtowebistesWP
Add-Content $name $exportpath
# --------------------------------
# DO EXPORT REPOSITORY REVISION $REVISION TO THE ExportPath
&"$svn" export -r $revision --force "$url" $exportpath
I added comments to explain each line and what it does. In a nutshell, the scripts:
Gets all the parameters
Build a local dir path
Runs SVN export
Places files to a website/publish directory.
Its a simple way of Deploying your newly committed code to a website.
Did you try to execute batch file using 'call' command? I mean:
call C:\Script\myscript.bat
I was trying the same thing and found that you also must have the script in the hooks folder.. the bat file that is.