How can I correct this Dockerfile to take arguments properly? - dockerfile

I have this dockerfile content:
FROM python:latest
ARG b=8
ARG r=False
ARG p=1
ENV b=${b}
ENV r=${r}
ENV p=${p}
# many lines of successful setup
ENTRYPOINT python analyze_all.py /analysispath -b $b -p $p -r $r
My intention was to take three arguments at the command line like so:
docker run -it -v c:\analyze containername -e b=16 -e p=22 -e r=False
But unfortunately, I'm misunderstanding something fundamental and simple here instead of something complicated, so I'm helpless :).

If I understood the question correctly, this Dockerfile should do what is required:
FROM python:latest
# test script that prints argv
COPY analyze_all.py /
ENTRYPOINT ["python", "analyze_all.py", "/analysispath"]
Launch:
$ docker run -it test:latest -b 16 -p 22 -r False
sys.argv=['analyze_all.py', '/analysispath', '-b', '16', '-p', '22', '-r', 'False']
Looks like your Dockerfile is designed to build and run a container on Windows. I tested my Dockerfile on Linux, it probably won't be much different to use this approach on Windows.
I think ARG instructions isn't needed in this case because it defines a variable that users can pass at build-time using the docker build command. I would also suggest that you take a look at the Dockerfile reference for the ENTRYPOINT instruction:
Command line arguments to docker run will be appended after all elements in an exec form ENTRYPOINT, and will override all elements specified using CMD. This allows arguments to be passed to the entry point, i.e., docker run -d will pass the -d argument to the entry point.
Also, this question will probably be useful for you: How do I use Docker environment variable in ENTRYPOINT array?

Related

Does Dockerfile support inline comments? [duplicate]

I am writing a Dockerfile. Is there a way to make comments in this file?
Does Docker have a comment option that takes the rest of a line and ignores it?
You can use # at the beginning of a line to start a comment (whitespaces before # are allowed):
# do some stuff
RUN apt-get update \
# install some packages
&& apt-get install -y cron
#'s in the middle of a string are passed to the command itself, e.g.:
RUN echo 'we are running some # of cool things'
As others have mentioned, comments are referenced with a # and are documented here. However, unlike some languages, the # must be at the beginning of the line. If they occur part way through the line, they are interpreted as an argument and may result in unexpected behavior.
# This is a comment
COPY test_dir target_dir # This is not a comment, it is an argument to COPY
RUN echo hello world # This is an argument to RUN but the shell may ignore it
It should also be noted that parser directives have recently been added to the Dockerfile which have the same syntax as a comment. They need to appear at the top of the file, before any other comments or commands. Originally, this directive was added for changing the escape character to support Windows:
# escape=`
FROM microsoft/nanoserver
COPY testfile.txt c:\
RUN dir c:\
The first line, while it appears to be a comment, is a parser directive to change the escape character to a backtick so that the COPY and RUN commands can use the backslash in the path. A parser directive is also used with BuildKit to change the frontend parser with a syntax line. See the experimental syntax for more details on how this is being used in practice.
With a multi-line command, the commented lines are ignored, but you need to comment out every line individually:
$ cat Dockerfile
FROM busybox:latest
RUN echo first command \
# && echo second command disabled \
&& echo third command
$ docker build .
Sending build context to Docker daemon 23.04kB
Step 1/2 : FROM busybox:latest
---> 59788edf1f3e
Step 2/2 : RUN echo first command && echo third command
---> Running in b1177e7b563d
first command
third command
Removing intermediate container b1177e7b563d
---> 5442cfe321ac
Successfully built 5442cfe321ac
Use the # syntax for comments
From: https://docs.docker.com/engine/reference/builder/#format
# My comment here
RUN echo 'we are running some cool things'
Dockerfile comments start with #, just like Python.
kstaken has good examples:
# Install a more-up-to date version of MongoDB than what is included in the default Ubuntu repositories.
FROM ubuntu
MAINTAINER Kimbro Staken
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
RUN echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" | tee -a /etc/apt/sources.list.d/10gen.list
RUN apt-get update
RUN apt-get -y install apt-utils
RUN apt-get -y install mongodb-10gen
#RUN echo "" >> /etc/mongodb.conf
CMD ["/usr/bin/mongod", "--config", "/etc/mongodb.conf"]
Docker treats lines that begin with # as a comment, unless the
line is a valid parser directive. A # marker anywhere else in a line
is treated as an argument.
example code:
# this line is a comment
RUN echo 'we are running some # of cool things'
Output:
we are running some # of cool things
Format
Here is the format of the Dockerfile:
We can use # for commenting purpose, as for example #COMMENT
#FROM microsoft/aspnetcore
FROM microsoft/dotnet
COPY /publish /app
WORKDIR /app
ENTRYPOINT ["dotnet", "WebApp.dll"]
From the above file when we build the docker, it skips the first line and goes to the next line because we have commented it using #
# this is comment
this isn't comment
is the way to do it. You can place it anywhere in the line and anything that comes later will be ignored

Use Docker-Windows for Gitlab-runner

I'm trying to use Docker in Windows to create a Gitlab-Runner to build a C++ application. It works so far, but I guess there are better aproaches. Here's what I did:
Here's my initial Docker Container:
FROM mcr.microsoft.com/windows/servercore:2004
# Restore the default Windows shell for correct batch processing.
SHELL ["cmd", "/S", "/C"]
# Download the Build Tools bootstrapper.
ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe
# Install Build Tools with the Microsoft.VisualStudio.Workload.AzureBuildTools workload, excluding workloads and components with known issues.
RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache `
--installPath C:\BuildTools `
--add Microsoft.VisualStudio.Workload.VCTools `
--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 `
--add Microsoft.VisualStudio.Component.VC.CMake.Project `
--add Microsoft.VisualStudio.Component.Windows10SDK.19041 `
--locale en-US `
|| IF "%ERRORLEVEL%"=="3010" EXIT 0
# Define the entry point for the docker container.
# This entry point starts the developer command prompt and launches the PowerShell shell.
ENTRYPOINT ["cmd","/k", "C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"]
And my .gitlab-ci.yml looks like this:
build Docker Windows:
image: buildtools2019_core
stage: build
tags:
- win-docker
script:
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=Release -DenableWarnings=ON -G Ninja -DCMAKE_MAKE_PROGRAM=Ninja ../
- ninja
This works so far and everthing builds correctly. The main problem however is that if the build fails the job succeeds anyways. I suspect that my entrypoint is wrong because powershell is executed inside of a cmd and only the exit code of cmd is checked which always succeeds.
So I tried to use powershell directly as entrypoint. I need to set environment variables via vcvars64.bat but that is not that trivial to do. I tried to execute the "Developer Powershell for VS 2019" but I can't execute the link in the entrypoint directly and the link looks like this:
"C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe -noe -c "&{Import-Module """C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"""; Enter-VsDevShell 6f66c5f6}"
Which I don't quit understand what it does and the hash also varies from installation to installation. Also simply using this as entrypoint didn't work.
I then tried to use the Invoke-Environment Script taken from "https://github.com/nightroman/PowerShelf/blob/master/Invoke-Environment.ps1". This allows me to execute the .bat file from powershell like this:
Invoke-Environment C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat
But to do this I need to add this function to my profile, as far as I understood. I did this by copying it to "C:\Windows\system32\WindowsPowerShell\v1.0\profile.ps1" so that it would be accessible by all users.
In my Docker file I added:
COPY Invoke-Environment.ps1 C:\Windows\system32\WindowsPowerShell\v1.0\profile.ps1
and replaced the entrypoint with:
ENTRYPOINT ["C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe", "-NoExit", "-NoLogo", "-ExecutionPolicy", "Bypass", "Invoke-Environment C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat"]
But that didn't initialize the environment variables correctly. Also "Invoke-Environment" is not found by the gitlab-runner. My last resort was to write a small script (Init64.ps1) that executes the Invoke-Environment function with vcvars64.bat:
function Invoke-Environment {
param
(
# Any cmd shell command, normally a configuration batch file.
[Parameter(Mandatory=$true)]
[string] $Command
)
$Command = "`"" + $Command + "`""
cmd /c "$Command > nul 2>&1 && set" | . { process {
if ($_ -match '^([^=]+)=(.*)') {
[System.Environment]::SetEnvironmentVariable($matches[1], $matches[2])
}
}}
}
Invoke-Environment C:\BuildTools\VC\Auxiliary\Build\vcvars64.bat
I copied this in docker via:
COPY Init64.ps1 Init64.ps1
and used this entrypoint:
ENTRYPOINT ["C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"]
In my build script I need to manually call it to setup the variables:
build Docker Windows:
image: buildtools2019_core
stage: build
tags:
- win-docker
script:
- C:\Init64.ps1
- mkdir build
- cd build
- cmake -DCMAKE_BUILD_TYPE=Release -DenableWarnings=ON -G Ninja -DCMAKE_MAKE_PROGRAM=Ninja ../
- ninja
Now everything works as intended the build works and the job only succeeds if the build succeeds.
However, I would prefer to setup my environment in the entrypoint so that I don't have to do this in my build script.
Is there a better way to do this? Also feel free to suggest any improvements I could make.
Ok, after some struggling, here is my entry.bat that correctly loads the environment exports the error-level/return-value:
REM Load environment
call C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat
REM If there are no parameters call cmd
IF [%1] == [] GOTO NOCALL
REM If there are parameters call cmd with /S for it to exit directly
cmd /S /C "%*"
exit %errorlevel%
:NOCALL
cmd
exit %errorlevel%

Conditionals in dockerfile?

Is it possible to do the following
docker build --build-arg myvar=yes
f
RUN if ["$myvar" == "yes"]; \
then FROM openjdk \
COPY . . \
RUN java -jar myjarfile.jar \
fi
As you can tell from above i only want to run the specific section in the dockerfile if the build argument is set. I've seen similar thread but they seems to always be running bash commands. If its possible i can't seem to get the syntax correct.
As of now, doing conditional execution in Dockerfiles without the help of the shell is severely limited, see https://medium.com/#tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae
The idea behind existing approaches is to use a Docker multistage build and create different stages for the different outcomes of the IF. Then, at one point, a stage to copy data from is selected based on the value of a variable.
This is an example similar to what you wrote:
# docker build -t test --build-arg MYVAR=yes .
# docker build -t test --build-arg MYVAR=no .
ARG MYVAR=no
FROM openjdk:latest as myvar-yes
COPY . /datadir
RUN java -jar /datadir/myjarfile.jar || true
FROM openjdk:latest as myvar-no
RUN mkdir /datadir
FROM myvar-${MYVAR} as myvar-src
FROM debian:10
COPY --from=myvar-src /datadir/ /
RUN ls /
Stage myvar-no is a variant with an empty /datadir. Stage myvar-yes contains the jarfile and runs it once (remove the || true for actual use, it is just that I did not provide a "real" jarfile in my local tests). Then the last stage copies from the stage myvar-${MYVAR} and invokes ls to be able to see the differences between the two variants.
I have not understand all of the question about syntax: If there are some troubles with getting the bash syntax correctly, that is possibly easier than trying to conditionally run Dockerfile statements.

Including other Docker files in my Docker file

I would like to build a custom Docker file. I start with Ubuntu
FROM ubuntu
But I would also like to add buildpack-deps:stretch
I understand that I am only allowed to use FROM once, so short of copying the contents of buildpack-deps:stretch into my Docker file, how do I add it to my Docker file?
AFAIK, simply "including" another Dockerfile does not work. But, you actually are allowed to use multiple FROM statements, if you use multistage builds (cf. the Docker docs).
For example, you could do something like this:
FROM buildpack-deps:stretch AS build
RUN echo "hello world!" > /tmp/foo
FROM ubuntu
COPY --from=build /tmp/foo .
CMD ["cat", "foo"]
Running docker build --tag foo . && docker run --rm foo results in hello world!. You could replace the first RUN statement with the compilation of something or whatever you are planning to do.
There are more ways for using multistage builds, e.g. using FROM build in our example directly.

How do I use command substition in Dockerfile

In my Dockerfile I need to use command substition to add some environment variables. I want to set
ENV PYTHONPATH /usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')
but it doesn't work. The result is
foo#bar:~$ echo $PYTHONPATH
/usr/local/$(python3 -c from distutils import sysconfig; print(sysconfig.get_python_lib()))
What's wrong?
What went wrong
The $( ... ) command substitution you attempted is for Bash, whereas the Dockerfile is not Bash. So docker doesn't know what to do with that, it's just plain text to docker, docker just spews out what you wrote as-is.
Recommendation
To avoid hard-coding values into a Dockerfile, and instead, to dynamically change a setting or custom variable as PYTHONPATH during the build, perhaps the ARG ... , --build-arg docker features might be most helpful, in combination with ENV ... to ensure it persists.
Within your Dockerfile:
ARG PYTHON_PATH_ARG
ENV PYTHONPATH ${PYTHON_PATH_ARG}
In Bash where you build your container:
python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"
docker build --build-arg PYTHON_PATH_ARG=$python_path .
Explanation
According to documentation, ARG:
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
So, in Bash we first:
python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"
$(...) Bash command substitution is used to dynamically put together a Python path value
this value is stored temporarily in a Bash variable $python_path for clarity
docker build --build-arg PYTHON_PATH_ARG=$python_path .
Bash variable $python_path value is passed to docker's --build-arg PYTHON_PATH_ARG
Within the Dockerfile:
ARG PYTHON_PATH_ARG
so PYTHON_PATH_ARG stores the value from --build-arg PYTHON_PATH_ARG...
ARG variables are not equivalent to ENV variables, so we couldn't merely do ARG PYTHONPATH and be done with it. According to documentation about Using arg variables:
ARG variables are not persisted into the built image as ENV variables are.
So finally:
ENV PYTHONPATH ${PYTHON_PATH_ARG}
We use Dockerfile's ${...} convention to get the value of PYTHON_PATH_ARG, and save it to your originally named PYTHONPATH environment variable
Differences from original code
You originally wrote:
ENV PYTHONPATH /usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')
I re-wrote the Python path finding portion as a Bash command, and tested on my machine:
$ python_path="/usr/local/$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"
$ echo $python_path
/usr/local//usr/lib/python3/dist-packages
Notice there is a double forward slash ... local//usr ... , not sure if that will break anything for you, depends on how you use it in your code.
Instead, I changed it to:
$ python_path="/usr/local$(python3 -c 'from distutils import sysconfig; print(sysconfig.get_python_lib())')"
Result:
$ echo $python_path
/usr/local/usr/lib/python3/dist-packages
So this new code will have no double forward slashes.
You should use ARG if possible. But sometimes you really need to use command substitution for a dynamic variable. As long as you put all the commands in the same RUN statement, then you can still access the value.
RUN foo=$(date) && \
echo $foo