I need to execute on GPU hardware so I have to create a self-hosted runner for github actions to execute my code. The self-hosted runner is hosted on my local machine (ubuntu 20.04).
I'm running the self hosted runner container locally with -v and binding the socks using: docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e GITHUB_OWNER=<xxx> -e GITHUB_REPOSITORY=<xxxx>-e GITHUB_PAT=<xxxx>
This local self-hosted runner executes successfully until I try to build the second "project" container I need for my project code. I get a permission issue with the docker sock when I try to build the container not run the container. I'm about 70% certain that with the -v binding when running the self-hosted runner locally this enables sibling containers versus Docker in Docker (which I've read isn't cool anymore).
Permission error:
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied
I've tried building the project container with -v /var/run/docker.sock:/var/run/docker.sock in the docker build command but it doesn't like the -v and I've also tried the following approaches in the "project" docker container:
Approach 1.
useradd -m cnncontainer && \
usermod -aG sudo cnncontainer && \
echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
curl -sSL https://get.docker.com/ | sh
usermod -aG docker cnncontainer
Approach 2.
sudo groupadd docker && \
sudo usermod -aG docker "$USER" &&\
newgrp docker
docker run hello-world
Approach 3.
sudo usermod -aG docker $USER
sudo setfacl --modify user:$USER:rw /var/run/docker.sock
docker run hello-world
GitHub actions self-hosted runner Dockerfile:
FROM debian:buster
#tensorflow/tensorflow:2.3.4-gpu - this image doesn't work either
ARG RUNNER_VERSION="2.298.2"
ENV GITHUB_PERSONAL_TOKEN ""
ENV GITHUB_OWNER ""
ENV GITHUB_REPOSITORY ""
RUN apt-get update \
&& apt-get install -y \
curl \
sudo \
git \
jq \
tar \
gnupg2 \
apt-transport-https \
ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN useradd -m github && \
usermod -aG sudo github && \
echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
#setup docker runner
RUN curl -sSL https://get.docker.com/ | sh
RUN usermod -aG docker github
USER github
WORKDIR /home/github
#install github actions cli
RUN curl -O -L https://github.com/actions/runner/releases/download/v$RUNNER_VERSION/actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
RUN tar xzf ./actions-runner-linux-x64-$RUNNER_VERSION.tar.gz
RUN sudo ./bin/installdependencies.sh
COPY --chown=github:github entrypoint.sh ./entrypoint.sh
RUN sudo chmod u+x ./entrypoint.sh
ENTRYPOINT ["/home/github/entrypoint.sh"]```
Self-hosted runner entrypoint.sh:
#!/bin/sh
registration_url="https://api.github.com/repos/${GITHUB_OWNER}/${GITHUB_REPOSITORY}/actions/runners/registration-token"
echo "Requesting registration URL at '${registration_url}'"
payload=$(curl -sX POST -H "Authorization: token ${GITHUB_PAT}" ${registration_url})
export RUNNER_TOKEN=$(echo $payload | jq .token --raw-output)
./config.sh \
--name $(hostname) \
--token ${RUNNER_TOKEN} \
--url https://github.com/${GITHUB_OWNER}/${GITHUB_REPOSITORY} \
--work ${RUNNER_WORKDIR} \
--unattended \
--replace
remove() {
./config.sh remove --unattended --token "${RUNNER_TOKEN}"
}
trap 'remove; exit 130' INT
trap 'remove; exit 143' TERM
./run.sh "$*" & #changed from run.sh
### BEGIN
sudo systemctl start docker
sudo systemctl enable docker
export RUNNER_ALLOW_RUNASROOT=true
export AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
mkdir actions-runner
sudo mkdir /opt/hostedtoolcache
cd actions-runner
# Make /actions-runner/_work
mkdir _work
# Link /opt/hostedtoolcache as /actions-runner/_work/_tool
ln -s /opt/hostedtoolcache _work/_tool
### END
wait $!
Dockerfile I want to run in/with the self-hosted runner
FROM tensorflow/tensorflow:2.3.4-gpu
RUN mkdir -p /app
COPY . main.py /app/
WORKDIR /app
RUN sudo apt install -y make && sudo apt-get install python3-pip -y
RUN pip install -r requirements.txt
RUN sudo usermod -aG docker $USER
RUN sudo setfacl --modify user:$USER:rw /var/run/docker.sock
RUN docker run hello-world
CMD [ "main.py" ]
ENTRYPOINT [ "python" ]
I am trying to set up an Oracle database connection on airflow.
I am getting this error:
ModuleNotFoundError: No module named 'airflow.providers.oracle' when using: from airflow.provideres.oracle.hooks.oracle import OracleHook
Part of my dag file:
from airflow.decorators import task
from airflow.providers.oracle.hooks.oracle import OracleHook
def exe_query_oracle_hook():
hook = OracleHook(oracle_conn_id="oracle_conn")
df = hook.get_pandas_df(sql='SELECT * FROM TABLE')
print(df.to_string())
I tried installing pip install apache-airflow-providers-oracle and most were already required, my current version is 2.1.0. I also followed the docs: airflow building custom images. Here is my Dockerfile
FROM apache/airflow:2.1.0
ARG ORACLE_VERSION=11.2.0.4.0
ARG ORACLE_SHORT_VER=11204
ENV CLIENT_ZIP=instantclient-basiclite-linux.x64-${ORACLE_VERSION}.zip
ENV SDK_ZIP=instantclient-sdk-linux.x64-${ORACLE_VERSION}.zip
ENV ORACLE_HOME=/opt/oracle
ENV TNS_ADMIN ${ORACLE_HOME}/network/admin
WORKDIR ${ORACLE_HOME}
USER root
RUN apt-get update \
&& apt-get -yq install unzip curl \
&& apt-get clean
COPY dockerfiles/${CLIENT_ZIP} ${ORACLE_HOME}/${CLIENT_ZIP}
COPY dockerfiles/${SDK_ZIP} ${ORACLE_HOME}/${SDK_ZIP}
RUN unzip ${ORACLE_HOME}/${CLIENT_ZIP} && unzip ${ORACLE_HOME}/${SDK_ZIP} \
&& rm -f *.zip
VOLUME ["${TNS_ADMIN}"]
RUN apt-get -yq install libaio1 \
&& apt-get autoremove \
&& apt-get clean \
&& echo ${ORACLE_HOME} > /etc/ld.so.conf.d/oracle.conf \
&& mkdir -p ${TNS_ADMIN} \
&& ldconfig \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN pip install --no-cache-dir apache-airflow-providers-oracle
USER 1001
Not sure what else to try, can anyone please provide some assistance? Thanks.
You have to run the pip install as airflow user, currently you run it as root.
...
USER 1001
RUN pip install --no-cache-dir apache-airflow-providers-oracle
I've tried a lot of different things found online, but I'm still unable to solve the below timeout error:
2021-11-27T14:51:21.844520452ZTimeout in polling result file: gs://...
when submitting a Dataflow flex-template job. It goes into Queued state and after 14 mins {x} secs goes to Failed state with the above log message. My Dockerfile is as follows:
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}
COPY requirements.txt .
COPY test-beam.py .
# Do not include `apache-beam` in requirements.txt
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/test-beam.py"
# Setting Proxy
ENV http_proxy=http://proxy-web.{company_name}.com:80 \
https_proxy=http://proxy-web.{company_name}.com:80 \
no_proxy=127.0.0.1,localhost,.{company_name}.com,{company_name}.com,.googleapis.com
# Company Cert
RUN apt-get update && apt-get install -y curl \
&& curl http://{company_name}.com/pki/{company_name}%20Issuing%20CA.pem -o - | tr -d '\r' > /usr/local/share/ca-certificates/{company_name}.crt \
&& curl http://{company_name}.com/pki/{company_name}%20Root%20CA.pem -o - | tr -d '\r' > /usr/local/share/ca-certificates/{company_name}-root.crt \
&& update-ca-certificates \
&& apt-get remove -y --purge curl \
&& apt-get autoremove -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Set pip config to point to Company Cert
RUN pip config set global.cert /etc/ssl/certs/ca-certificates.crt
# Install apache-beam and other dependencies to launch the pipeline
RUN pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir apache-beam[gcp]==2.32.0 \
&& pip install --no-cache-dir -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE \
# Download the requirements to s7peed up launching the Dataflow job.
&& pip download --no-cache-dir --dest /tmp/dataflow-requirements-cache -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE
# Since we already downloaded all the dependencies, there's no need to rebuild everything.
ENV PIP_NO_DEPS=True
ENV http_proxy= \
https_proxy= \
no_proxy=
ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"]
And requirements.py:
numpy
setuptools
scipy
wavefile
I know my Python script used above test-beam.py works as it executes successfully locally using a DirectRunner.
I have gone through many SO posts and GCP's own troubleshooting guide here aimed at this error, however to no success. As you can see from my Dockerfile, I have done the following in it:
Installing apache-beam[gcp] separately and not including it in my requirements.txt file.
Pre-downloading all dependencies using pip download --no-cache-dir --dest /tmp/dataflow-requirements-cache -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE.
Setting ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"] explicitly as it seems this is not set in the base image gcr.io/dataflow-templates-base/python3-template-launcher-base as found by executing docker inspect on it (am I correct about this?).
Unsetting company proxy at the end as it seems to be the cause of timeout issues seen in job logs from previous runs.
What am I missing? How can I fix this issue?
I am using dockers to store the dependencies of my c++ program to use when doing CI testing with gitlab CI. I first build a base docker which contains all of the program dependencies (lets call it DOCKER_A):
FROM gcc:5
RUN mkdir -p /usr/src/optimization
WORKDIR /usr/optimization
#COPY . /usr/optimization
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y build-essential && \
apt-get install -y openssh-client && \
apt-get install -y python3 && \
apt-get install -y python3-pip && \
pip3 install --upgrade pip && \
pip3 install virtualenv
RUN wget http://www.cmake.org/files/v3.7/cmake-3.7.2.tar.gz && \
tar xf cmake-3.7.2.tar.gz && \
cd cmake-3.7.2/ && \
./configure && \
make && \
make install && \
export PATH=/usr/local/bin:$PATH && \
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH && \
cd ..
RUN wget -O boost_1_64_0.tar.gz http://sourceforge.net/projects/boost/files/boost/1.64.0/boost_1_64_0.tar.gz/download && \
tar xzvf boost_1_64_0.tar.gz && \
cd boost_1_64_0 && \
./bootstrap.sh --exec-prefix=/usr/local --with-python=python3 && \
./b2 threading=multi && \
./b2 install threading=multi && \
cd .. && \
rm boost_1_64_0.tar.gz && \
rm -r boost_1_64_0 && \
ln -s /usr/lib/x86_64-linux-gnu/libboost_python-py34.so /usr/lib/x86_64-linux-gnu/libboost_python3.so
This docker doesn't change. Then every time I push to gitlab, I build another docker, starting from DOCKER_A:
FROM DOCKER_A
ARG SSH_PRIVATE_KEY
WORKDIR /usr/optimization
COPY . /usr/optimization
RUN chmod +x ADD_KEY.sh
RUN ./ADD_KEY.sh "$SSH_PRIVATE_KEY"
RUN mkdir build && \
cd build && \
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=on .. && \
make && \
cd ..
This builds the code from the new commit (up to this point everything works as expected).
Next, in my YAML file for gitlab CI, I run my tests, which consist of calling the executable files generated by my build process.
before_script:
- docker info
- docker login -u user -p $CI_JOB_TOKEN docker.registry.url
after_script:
- echo "After script section"
- echo "For example you might do some cleanup here"
buildRelease:
stage: build
script:
- echo "Do your build here"
- docker login -u user -p $CI_JOB_TOKEN docker.registry.url
- docker build --pull -i $CONTAINER_IMAGE_PUSH --build-arg SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" .
- docker push $CONTAINER_IMAGE_PUSH
testDispatch:
stage: test
script:
- echo "Do a test here"
- echo "For example run a test suite"
- docker run -t $CONTAINER_IMAGE_PULL ./bin/dispatch
testState:
stage: test
script:
- docker run -t $CONTAINER_IMAGE_PULL ./bin/state-test
testAlgorithm:
stage: test
script:
- docker run -t $CONTAINER_IMAGE_PULL ./bin/algorithm-test
testSystem:
stage: test
script:
- docker run -t $CONTAINER_IMAGE_PULL ./bin/system-test
Each of these the tests in stage test fails, all giving the same error. Here is an example of the output:
$ docker run -t $CONTAINER_IMAGE_PULL ./bin/algorithm-test
./bin/algorithm-test: error while loading shared libraries:
libboost_graph.so.1.64.0: cannot open shared object file: No such file or directory
I don't understand why my binary cannot find libboost graph, as it is installed in the first docker container, which I am inheriting from.
Any help that could be provided would be appreciated.
How can I build a Docker container with Google's Cloud Command Line Tool/SDK?
The script at the url https://sdk.cloud.google.com appears to require user input so doesn't work in a docker file.
Adding the following to my Docker file appears to work.
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
Use this one-liner in your Dockerfile:
RUN curl -sSL https://sdk.cloud.google.com | bash
source:
https://docs.docker.com/v1.8/installation/google/
Doing it with alpine:
FROM alpine:3.6
RUN apk add --update \
python \
curl \
which \
bash
RUN curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin
RUN curl -sSL https://sdk.cloud.google.com > /tmp/gcl && bash /tmp/gcl --install-dir=~/gcloud --disable-prompts
This will download the google cloud sdk installer into /tmp/gcl, and run it with the parameters as follows:
--install-dir=~/gcloud: Extract the binaries into folder gcloud in home folder. Change this to wherever you want, for example /usr/local/bin
--disable-prompts: Don't show any prompts while installing (headless)
To install gcloud inside a docker container please follow the instructions here.
Basically you need to run
RUN apt-get update && \
apt-get install -y curl gnupg && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && \
apt-get update -y && \
apt-get install google-cloud-sdk -y
inside your dockerfile. It's important you are user ROOT when you run this command, so it may necessary to add USER root before the previous command.
As an alternative, you could use the docker image provided by google namely google/cloud-sdk. https://hub.docker.com/r/google/cloud-sdk/
Dockerfile:
FROM centos:7
RUN yum update -y && yum install -y \
curl \
which && \
yum clean all
RUN curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin
Build:
docker build . -t google-cloud-sdk
Then run gcloud:
docker run --rm \
--volume $(pwd)/assets/root/.config:/root/.config \
google-cloud-sdk gcloud
...or run gsutil:
docker run --rm \
--volume $(pwd)/assets/root/.config:/root/.config \
google-cloud-sdk gsutil
The local assets folder will contain the configuration.
apk upgrade --update-cache --available && \
apk add openssl && \
apk add curl python3 py-crcmod bash libc6-compat && \
rm -rf /var/cache/apk/*
curl https://sdk.cloud.google.com | bash > /dev/null
export PATH=$PATH:/root/google-cloud-sdk/bin
gcloud components update kubectl
I was using Python Alpine image python:3.8.6-alpine3.12 as base and this worked for me:
RUN apk add --no-cache bash
RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-327.0.0-linux-x86_64.tar.gz \
-O /tmp/google-cloud-sdk.tar.gz | bash
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvzf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh -q
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
After building and running the image, you can check if google-cloud-sdk is installed by running docker exec -i -t <container_id> /bin/bash and running this:
bash-5.0# gcloud --version
Google Cloud SDK 327.0.0
bq 2.0.64
core 2021.02.05
gsutil 4.58
bash-5.0# gsutil --version
gsutil version: 4.58
If you want a specific version of google-cloud-sdk, you can visit https://storage.cloud.google.com/cloud-sdk-release
curl https://sdk.cloud.google.com | bash -s -- --disable-prompts
and export env
works for me
I got this working with Ubuntu 18.04 using:
RUN apt-get install -y curl && curl -sSL https://sdk.cloud.google.com | bash
ENV PATH="$PATH:/root/google-cloud-sdk/bin"
You can use multi-stage builds to make this simpler and more efficient than solutions using curl.
FROM bitnami/google-cloud-sdk:0.392.0 as gcloud
FROM base-image-for-production:tag
# Do what you need to configure your production image
COPY --from=gcloud /opt/bitnami/google-cloud-sdk/ /google-cloud-sdk
This work for me.
FROM php:7.2-fpm
RUN apt-get update -y
RUN apt-get install -y python && \
curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin
An example using debian as the base image:
FROM debian:stretch
RUN apt-get update && apt-get install -y apt-transport-https gnupg curl lsb-release
RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
echo "cloud SDK repo: $CLOUD_SDK_REPO" && \
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
apt-get update -y && apt-get install google-cloud-sdk -y
I used most of these examples in some form (thanks #KJoe), but I had to do several other things to setup everything so gcloud would work in the environment. Note that it is preferable to limit the number of lines (it limits layers needed to pull)
Here's a more complete example of Dockerfile with gcloud setup and extending a CircleCI image:
FROM circleci/ruby:2.4.1-jessie-node-browsers
# user is circleci in the FROM image, switch to root for system lib installation
USER root
ENV CCI /home/circleci
ENV GTMP /tmp/gcloud-install
ENV GSDK $CCI/google-cloud-sdk
ENV PATH="${GSDK}/bin:${PATH}"
# do all system lib installation in one-line to optimize layers
RUN curl -sSL https://sdk.cloud.google.com > $GTMP && bash $GTMP --install-dir=$CCI --disable-prompts \
&& rm -rf $GTMP \
&& chmod +x $GSDK/bin/* \
\
&& chown -Rf circleci:circleci $CCI
# change back to the user in the FROM image
USER circleci
# setup gcloud specifics to your liking
RUN gcloud config set core/disable_usage_reporting true \
&& gcloud config set component_manager/disable_update_check true \
&& gcloud components install alpha beta kubectl --quiet
My use case was to generate a google bearer token using the service account, so I wanted the docker container to install gcloud this is how my docker file looks like
FROM google/cloud-sdk
# Setting the default directory in container
WORKDIR /usr/src/app
# copies the app source code to the directory in container
COPY . /usr/src/app
CMD ["/bin/bash","/usr/src/app/token.sh"]
If you need to examine a container after it is built but that isn't running use docker run --rm -it <container-build-id> bash -il and type in gcloud --version if installed correctly or not
In Google documentation you can see the best practice
https://cloud.google.com/sdk/docs/install-sdk
search on the page for "Docker Tip"
eg debian use:
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && apt-get update -y && apt-get install google-cloud-cli -y
If you're just interested in getting the gcloud CLI available, add this to your Dockerfile:
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-409.0.0-linux-x86_64.tar.gz > /tmp/google-cloud-cli.tar.gz
# Installing the gcloud cli
RUN mkdir -p /usr/local/gcloud \
&& tar -xf /tmp/google-cloud-cli.tar.gz \
&& ./google-cloud-sdk/install.sh --quiet