I am looking for a reliable method for generating and using temporary, disposable folders, as part of tox environment creation.
[testenv:var-test]
description = Try to store output of a shell command
tmpdir = mktemp -d
commands =
echo {[testenv:var-test]tmpdir}
# prints "mktemp -d" (command is not run)
tmpdir = mktemp -d
# ERROR: InvocationError for command could not find executable tmpdir
There may be better ways, but you certainly can call a bash script from within the commands section.
tox.ini
[testenv]
whitelist_externals = bash
commands =
bash {toxinidir}/commands.sh
commands.sh
TEMP=`mktemp -d`
echo $TEMP
output of a tox run
❯ tox
python run-test-pre: PYTHONHASHSEED='562823002'
python run-test: commands[0] | bash /home/jugmac00/stackOverflow/commands.sh
/tmp/tmp.snc2T0Fa6W
_________________________________________________________ summary __________________________________________________________
python: commands succeeded
congratulations :)
Related
I'm trying to use Docker in Windows to create a Gitlab-Runner to build a C++ application. It works so far, but I guess there are better aproaches. Here's what I did:
Here's my initial Docker Container:
FROM mcr.microsoft.com/windows/servercore:2004
# Restore the default Windows shell for correct batch processing.
SHELL ["cmd", "/S", "/C"]
# Download the Build Tools bootstrapper.
ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe
# Install Build Tools with the Microsoft.VisualStudio.Workload.AzureBuildTools workload, excluding workloads and components with known issues.
RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache `
--installPath C:\BuildTools `
--add Microsoft.VisualStudio.Workload.VCTools `
--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 `
--add Microsoft.VisualStudio.Component.VC.CMake.Project `
--add Microsoft.VisualStudio.Component.Windows10SDK.19041 `
--locale en-US `
|| IF "%ERRORLEVEL%"=="3010" EXIT 0
# Define the entry point for the docker container.
# This entry point starts the developer command prompt and launches the PowerShell shell.
ENTRYPOINT ["cmd","/k", "C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]
And my .gitlab-ci.yml looks like this:
build Docker Windows:
image: buildtools2019_core
stage: build
tags:
- win-docker
script:
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=Release -DenableWarnings=ON -G Ninja -DCMAKE_MAKE_PROGRAM=Ninja ../
- ninja
This works so far and everthing builds correctly. The main problem however is that if the build fails the job succeeds anyways. I suspect that my entrypoint is wrong because powershell is executed inside of a cmd and only the exit code of cmd is checked which always succeeds.
So I tried to use powershell directly as entrypoint. I need to set environment variables via vcvars64.bat but that is not that trivial to do. I tried to execute the "Developer Powershell for VS 2019" but I can't execute the link in the entrypoint directly and the link looks like this:
"C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe -noe -c "&{Import-Module """C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"""; Enter-VsDevShell 6f66c5f6}"
Which I don't quit understand what it does and the hash also varies from installation to installation. Also simply using this as entrypoint didn't work.
I then tried to use the Invoke-Environment Script taken from "https://github.com/nightroman/PowerShelf/blob/master/Invoke-Environment.ps1". This allows me to execute the .bat file from powershell like this:
Invoke-Environment C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat
But to do this I need to add this function to my profile, as far as I understood. I did this by copying it to "C:\Windows\system32\WindowsPowerShell\v1.0\profile.ps1" so that it would be accessible by all users.
In my Docker file I added:
COPY Invoke-Environment.ps1 C:\Windows\system32\WindowsPowerShell\v1.0\profile.ps1
and replaced the entrypoint with:
ENTRYPOINT ["C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe", "-NoExit", "-NoLogo", "-ExecutionPolicy", "Bypass", "Invoke-Environment C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat"]
But that didn't initialize the environment variables correctly. Also "Invoke-Environment" is not found by the gitlab-runner. My last resort was to write a small script (Init64.ps1) that executes the Invoke-Environment function with vcvars64.bat:
function Invoke-Environment {
param
(
# Any cmd shell command, normally a configuration batch file.
[Parameter(Mandatory=$true)]
[string] $Command
)
$Command = "`"" + $Command + "`""
cmd /c "$Command > nul 2>&1 && set" | . { process {
if ($_ -match '^([^=]+)=(.*)') {
[System.Environment]::SetEnvironmentVariable($matches[1], $matches[2])
}
}}
}
Invoke-Environment C:\BuildTools\VC\Auxiliary\Build\vcvars64.bat
I copied this in docker via:
COPY Init64.ps1 Init64.ps1
and used this entrypoint:
ENTRYPOINT ["C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"]
In my build script I need to manually call it to setup the variables:
build Docker Windows:
image: buildtools2019_core
stage: build
tags:
- win-docker
script:
- C:\Init64.ps1
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=Release -DenableWarnings=ON -G Ninja -DCMAKE_MAKE_PROGRAM=Ninja ../
- ninja
Now everything works as intended the build works and the job only succeeds if the build succeeds.
However, I would prefer to setup my environment in the entrypoint so that I don't have to do this in my build script.
Is there a better way to do this? Also feel free to suggest any improvements I could make.
Ok, after some struggling, here is my entry.bat that correctly loads the environment exports the error-level/return-value:
REM Load environment
call C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat
REM If there are no parameters call cmd
IF [%1] == [] GOTO NOCALL
REM If there are parameters call cmd with /S for it to exit directly
cmd /S /C "%*"
exit %errorlevel%
:NOCALL
cmd
exit %errorlevel%
If I want to run a specific command (with arguments) under Software Collections, I can use this command:
scl enable python27 "ls /tmp"
However, if I try to make a shell script that has a similar command as its shebang line, I get errors:
$ cat myscript
#!/usr/bin/scl enable python27 "ls /tmp"
echo hello
$ ./myscript
Unable to open /etc/scl/prefixes/"ls!
What am I doing wrong?
You should try using -- instead of surrounding your command with quotes.
scl enable python27 -- ls /tmp
I was able to make a python script that uses the rh-python35 collection with this shebang:
#!/usr/bin/scl enable rh-python35 -- python
import sys
print(sys.version)
The parsing of arguments in the she-bang command is not really defined. From man execve:
The semantics of the optional-arg argument of an interpreter script vary across implementations. On Linux, the entire string following the interpreter name is passed as a single argument to the interpreter, and this string can include white space. However, behavior differs on some other systems. Some systems use the first white space to terminate optional-arg. On some systems, an interpreter script can have multiple arguments, and white spaces in optional-arg are used to delimit the arguments.
No matter what, argument splitting based on quote sis not supported. So when you write:
#!/usr/bin/scl enable python27 "ls /tmp"
It's very possible that what gets invoked is (using bash notation):
'/usr/bin/scl' 'enable' 'python27' '"ls' '/tmp"'
This is probably why it tries to open the "ls file at /etc/scl/prefixes/"ls
But it is just as likely that the shebang evaluates to:
'/usr/bin/scl' 'enable python27 "ls /tmp"'
And that would fail since it wont be able to find a command named enable python27 "ls /tmp" for scl to execute.
There's a few workarounds you can use.
You can call your script via scl:
$ cat myscript
#!/bin/bash
echo hello
$ scl enable python27 ./myscript
hello
You can also use the heredoc notation, but it might lead to subtle issues. I personally avoid this:
$ cat ./myscript
#!/bin/bash
scl enable python27 -- <<EOF
echo hi
echo \$X_SCLS
EOF
$ bash -x myscript
+ scl enable python27 --
hi
python27
You can see one of the gotcha's already: I had to write \$X_SCLS to access the environment variable instead of just $X_SCL.
Edit: Another option is two have two scripts. One that has the actual code, and the second that simply does scl enable python27 $FIRST_SCRIPT. Then you wont have to remember to enter scl ... manually.
The software collections documentation may also be helpful. In particular you can try
cat myscript.sh | scl enable python27 -
As well as permanently enabling a software collection
source scl_source enable python27
./myscript.sh
How do you configure Tox to source a file before it runs your test command?
I tried the obvious:
commands = source /path/to/my/setup.bash; ./mytestcommand
But Tox just reports the ERROR: InvocationError: could not find executable 'source'
I know Tox has a setenv parameter, but I want to use my setup.bash and not have to copy and paste its contents into my tox.ini.
tox uses exec system call to run commands, not shell; and of course exec doesn't know how to run source. You need to explicitly run the command with bash, and you need to whitelist bash to avoid warnings from tox. That is, your tox.ini should be somewhat like this:
[testenv]
commands =
bash -c 'source /path/to/my/setup.bash; ./mytestcommand'
whitelist_externals =
bash
Supposing my lein project is located in /some/location/project and my current location is /another/location/ how can I run lein build without changing to project location cd /some/location/project?
For example, in maven
mvn -f /path/to/pom.xml
As far as I can tell lein doesn't have an option like this. You have to be in the directory. (Perhaps someone else will correct me.) However, you could write a shell script that does what you want. For example, in a unix that provides the getopts utility, you could use the following script, which might be called "leinthere":
#!/bin/sh
if getopts f: option; then
# user supplied -f
if [ -d "$OPTARG" ]; then
# user also supplied name of a real dir, now in $OPTARG
cd "$OPTARG"
shift 2 # get rid of -f and the dir name
else
# user supplied -f, but not a real dir name
echo "usage: $0 [-f project-dir] [lein args]"
exit 1
fi
fi
# now just run lein in the normal way:
lein "$#"
I am following this guide for python 2.7 :
http://docs.python.org/tutorial/interpreter.html
I do all it says: I have a python file:
#! /usr/bin/env python
print "hello world !\n"
And from terminal, in the directory where is the file I type:
chmod +x hello_world.py
The file is name hello_world.py; But nothing happens, it doesn't print "hello world\n".
sorry if this is insultingly obvious, but
> chmod +x hello_world.py
only changes the file so that you can run it. next you need to actually run the file by typing:
> ./hello_world.py
hello world !
To give a bit more description: the chmod command changes the permissions of a file on a Unix-style system. The +x in the command:
chmod +x hello_world.py
Sets the "Executable" bit for the hello_world.py file, thereby making it a script which can be executed. Thus to run the script:
./hello_world.py
The ./ in front indicates that the file is in the current directory. Alternatively, you can always run a script by invoking the python interpreter directly (regardless of permissions) like so:
python hello_world.py