Installing gcloud on Travis CI - google-cloud-platform

I'm following this tutorial on how to use Travis CI with Google Cloud for Continuous Deployments:
https://cloud.google.com/solutions/continuous-delivery-with-travis-ci
When Travis builds, it tells me that the gcloud command is not found. Here's my .travis file:
sudo: false
language: python
cache:
directories:
- "$HOME/google-cloud-sdk/"
env:
- GAE_PYTHONPATH=${HOME}/.cache/google_appengine PATH=$PATH:${HOME}/google-cloud-sdk/bin
PYTHONPATH=${PYTHONPATH}:${GAE_PYTHONPATH} CLOUDSDK_CORE_DISABLE_PROMPTS=1
before_install:
- openssl aes-256-cbc -K $encrypted_404aa45a170f_key -iv $encrypted_404aa45a170f_iv
-in credentials.tar.gz.enc -out credentials.tar.gz -d
- if [ ! -d "${GAE_PYTHONPATH}" ]; then python scripts/fetch_gae_sdk.py $(dirname
"${GAE_PYTHONPATH}"); fi
- if [ ! -d ${HOME}/google-cloud-sdk ]; then curl https://sdk.cloud.google.com | bash;
fi
- tar -xzf credentials.tar.gz
- mkdir -p lib
- gcloud auth activate-service-account --key-file client-secret.json
install:
- gcloud config set project continuous-deployment-192112
- gcloud -q components update gae-python
- pip install -r requirements.txt -t lib/
script:
- python test_main.py
- gcloud -q preview app deploy app.yaml --promote
- python e2e_test.py
This is the same file provided by the example repository from the tutorial. The line that fails is:
- gcloud auth activate-service-account --key-file client-secret.json
Even though it's already checked for the SDK and installed it if it isn't there.
I've already tried adding - source ~/.bash_profile after the install, but this doesn't work.
Am I missing a command somewhere?

I ran into the same issue and this has worked for me:
- if [ ! -d "$HOME/google-cloud-sdk" ]; then
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)";
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list;
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - ;
sudo apt-get update && sudo apt-get install google-cloud-sdk;
fi
The only issue however is since it needs sudo, it will run on gce which is much slower then ec2
https://docs.travis-ci.com/user/reference/overview/#Virtualisation-Environment-vs-Operating-System
Updated:
This is the best solution -
How to install Google Cloud SDK on Travis?

Related

Linux commands for login to Github CLI in command line

I want to login into the gh cli with parameters or flags. Im setting up a cloud-init file, with bash commands. So i can't interact manual processes:
That's how it would look like if you do it manually in the console:
ubuntu#ip-172-31-54-112:~/react$ gh auth login
? What account do you want to log into? GitHub.com
? What is your preferred protocol for Git operations? HTTPS
? Authenticate Git with your GitHub credentials? Yes
? How would you like to authenticate GitHub CLI? Paste an authentication token
Tip: you can generate a Personal Access Token here https://github.com/settings/tokens
The minimum required scopes are 'repo', 'read:org', 'workflow'.
? Paste your authentication token: ****************************************
- gh config set -h github.com git_protocol https
✓ Configured git protocol
✓ Logged in as yuuval
I tried it with this line of code which is printed:
- gh config set -h github.com git_protocol https
But it doesnt really log you into github cli. This is my Cloud-init file:
#cloud-config
keyboard:
layout: ch
package_update: true
package_upgrade: true
packages:
- nginx
- git
package_reboot_if_required: true
runcmd:
- type -p curl >/dev/null || sudo apt install curl -y
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
&& sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& sudo apt update \
&& sudo apt install gh -y
- mkdir react
- cd react
- curl -o actions-runner-linux-x64-2.301.1.tar.gz -L https://github.com/actions/runner/releases/download/v2.300.1/actions-runner-linux-x64-2.301.1.tar.gz
- tar xzf ./actions-runner-linux-x64-2.301.1.tar.gz
- yes "" | ./config.sh --url https://github.com/yuuval/react-deploy-aws --token AVYXWHVAXX2TB4J63XBJCIDDYB6TA
- sudo ./svc.sh install
- sudo ./svc.sh start
- //HERE COMES THE LOGIN PART + GH WORKFLOW RUN NODE.JS.YML FILE
- cd _work
- cd react-deploy-aws
- cd react-deploy-aws
- cd /etc/nginx/sites-available
- sudo rm default
-
echo "server {listen 80 default_server;server_name _;location / {root
/home/ubuntu/react/_work/react-deploy-aws/react-deploy-aws/build;try_files
\$uri /index.html;}}" | sudo tee /etc/nginx/sites-available/default
- sudo service nginx restart
- sudo chmod +x /home
- sudo chmod +x /home/ubuntu
- sudo chmod +x /home/ubuntu/react
- sudo chmod +x /home/ubuntu/react/_work
- sudo chmod +x /home/ubuntu/react/_work/react-deploy-aws
- sudo chmod +x /home/ubuntu/react/_work/react-deploy-aws/react-deploy-aws
- sudo chmod +x /home/ubuntu/react/_work/react-deploy-aws/build
Does anyone know how to login via commandline and no user inputs. And if you know how to do the same with the gh workflow run command and then selecting the node.js.yaml file without user interactions needed that would be nice.
Set an environment variable with your generated Personal Access Token e.g. PAT.
Pass it to gh auth login command with --with-token flag:
gh auth login --hostname github.com --with-token <<< $PAT
Reference:
https://cli.github.com/manual/gh_auth_login

Travis -Gcloud crashed (AttributeError): '_RSAPrivateKey' object has no attribute 'sign'

UPDATE
Here is part of travis file
before_install:
#openssl stuff regarding credentials.tar.gz
- if [ ! -d "${GAE_PYTHONPATH}" ]; then python scripts/fetch_gae_sdk.py $(dirname
"${GAE_PYTHONPATH}"); fi
- if [ ! -d $HOME/google-cloud-sdk/bin ]; then rm -rf $HOME/google-cloud-sdk; curl
https://sdk.cloud.google.com | bash; fi
- tar -xzf credentials.tar.gz
- "$HOME/google-cloud-sdk/bin/gcloud components update"
- "pip install -U cryptography"
- "$HOME/google-cloud-sdk/bin/gcloud auth activate-service-account --key-file travis-credentials.json" # ERROR HAPPENS HERE
#ERROR IS = Gcloud crashed (AttributeError): '_RSAPrivateKey' object has no attribute 'sign'
- "$HOME/google-cloud-sdk/bin/gcloud auth configure-docker"
And this is what I can't understand. I would assume if it was something related to a sudden version upgrade from gcloud and it is incompatible with crpyotgraphy then lots of applications would have failed and my fix attempts would fix it. But this used to work until I get the aforementioned email so I am thinking something is messed up after that email but it is just a wild guess.
All of the Travis file
language: python
python: 2.7
branches:
only:
- master
services:
- docker
cache:
directories:
- "$HOME/google-cloud-sdk/"
env:
- GAE_PYTHONPATH=${HOME}/.cache/google_appengine PATH=$PATH:${HOME}/google-cloud-sdk/bin
PYTHONPATH=${PYTHONPATH}:${GAE_PYTHONPATH} CLOUDSDK_CORE_DISABLE_PROMPTS=1
before_install:
#unrelated stuff
- if [ ! -d "${GAE_PYTHONPATH}" ]; then python scripts/fetch_gae_sdk.py $(dirname
"${GAE_PYTHONPATH}"); fi
- if [ ! -d $HOME/google-cloud-sdk/bin ]; then rm -rf $HOME/google-cloud-sdk; curl
https://sdk.cloud.google.com | bash; fi
- tar -xzf credentials.tar.gz
- "$HOME/google-cloud-sdk/bin/gcloud components update"
- "pip install -U cryptography"
- "$HOME/google-cloud-sdk/bin/gcloud auth activate-service-account --key-file travis-credentials.json"
- "$HOME/google-cloud-sdk/bin/gcloud auth configure-docker"
install:
#push image to gcr
script:
- echo "done"
The same question is asked here question but updating cryptography module didn't resolve the issue(I tried 3 different versions from the latest to one listed in the answer -> 2.6.1). 3 days ago I received an email from google which said the following.
Hello Cloud Shell user,
It's been over 120 days since you opened Cloud Shell from the Google
Cloud Platform console. In 7 days, your Cloud Shell home directory
will be automatically scheduled for deletion.
To keep your Cloud Shell home directory and its data, just log in and
open Cloud Shell.
I opened the shell to keep it activated but when I try to deploy my django application with travis I got the following error when I tried to execute a gcloud command.
$HOME/google-cloud-sdk/bin/gcloud auth activate-service-account --key-file travis-credentials.json
ERROR: gcloud crashed (AttributeError): '_RSAPrivateKey' object has no attribute 'sign'
I tried 2.6.1, 2.8 (my previous version), 3.4.1(most updated version), but none of them worked. Any idea on how to fix this? My last build was a month ago and it worked successfully without changing anything in configuration
Apparently, this issue is related to the gcloud version I was using.I always fetch the latest version (which is currently 331.0.0). Although it is not desirable, downgrading gcloud SDK to 330.0.0 resolved the issue.
gcloud components update --version 330.0.0

Docker file needs to install nvm without internet gateway

I'm working on a webapp on AWS CodePipeline, and one of my backend pipeline's stages includes a docker build command, and the Dockerfile includes these commands:
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.37.2/install.sh | bash
RUN /bin/bash -c ". ~/.nvm/nvm.sh && \
nvm install $NODE_VERSION && nvm use $NODE_VERSION && \
npm install -g aws-cdk cdk-assume-role-credential-plugin#1.1.1 && \
nvm alias default node && nvm cache clear"
RUN echo export PATH="\
/root/.nvm/versions/node/${NODE_VERSION}/bin:\
$(python3.8 -m site --user-base)/bin:\
$(python3 -m site --user-base)/bin:\
$PATH" >> ~/.bashrc && \
echo "nvm use ${NODE_VERSION} 1> /dev/null" >> ~/.bashrc
RUN /bin/bash -c ". ~/.nvm/nvm.sh && cdk --version"
ENTRYPOINT [ "/bin/bash", "-c", ". ~/.nvm/nvm.sh && uvicorn cdkproxymain:app --host 0.0.0.0 --port 8080" ]
The problem is that I'm running code in a VPC without an internet gateway (client's policy), so the curl command fails. I have tried to install nvm locally by copying the nvm folder to my src directory, but I lack the skills to script this.
Any advice is welcome, Thank you so much !

sdkman does not install java in a dockerfile

I have this docker file:
# We are going to star from the jhipster image
FROM jhipster/jhipster
# install as root
USER root
### Setup docker cli (don't need docker daemon) ###
# Install some packages
RUN apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y
# Add Dockers official GPG key:
RUN ["/bin/bash", "-c", "set -o pipefail && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -"]
# Add a stable repository
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# Setup aws credentials as environment variables
ENV AWS_ACCESS_KEY_ID "change it!"
ENV AWS_SECRET_ACCESS_KEY "change it!"
# noninteractive install for tzdata
ARG DEBIAN_FRONTEND=noninteractive
# set timezone for tzdata
ENV TZ=America/Sao_Paulo
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
# Install the latest version of Docker Engine - Community and also aws cli
RUN apt-get update && apt-get install docker-ce docker-ce-cli containerd.io awscli -y
# change back to default user
USER jhipster
# install skd and java version 1.8
RUN curl -s "https://get.sdkman.io" | bash
RUN bash $HOME/.sdkman/bin/sdkman-init.sh
RUN bash -c "sdk install java 8.0.222.j9-adpt"
When I run a command to build an image from this dockerfile it fails on the last step with a message:
/bin/sh: 1: sdk: not found
When I install it on my local machine it runs sdkman (sdk) on bash. But on this script it calls it from sh not bash. How can I make it calls skdman (sdk) from sh? What I actually want to do is install a specific java version through sdkman (sdk). Is there another way to do it?
For sdk command to be available you need to run source sdkman-init.sh.
Here is a working sample with java 11 on centos.
FROM centos:latest
ARG CANDIDATE=java
ARG CANDIDATE_VERSION=11.0.6-open
ENV SDKMAN_DIR=/root/.sdkman
# update the image
RUN yum -y upgrade
# install requirements, install and configure sdkman
# see https://sdkman.io/usage for configuration options
RUN yum -y install curl ca-certificates zip unzip openssl which findutils && \
update-ca-trust && \
curl -s "https://get.sdkman.io" | bash && \
echo "sdkman_auto_answer=true" > $SDKMAN_DIR/etc/config && \
echo "sdkman_auto_selfupdate=false" >> $SDKMAN_DIR/etc/config
# Source sdkman to make the sdk command available and install candidate
RUN bash -c "source $SDKMAN_DIR/bin/sdkman-init.sh && sdk install $CANDIDATE $CANDIDATE_VERSION"
# Add candidate path to $PATH environment variable
ENV JAVA_HOME="$SDKMAN_DIR/candidates/java/current"
ENV PATH="$JAVA_HOME/bin:$PATH"
ENTRYPOINT ["/bin/bash", "-c", "source $SDKMAN_DIR/bin/sdkman-init.sh && \"$#\"", "-s"]
CMD ["sdk", "help"]
The problem is every RUN command in Dockerfile is executed within a new bash environment, so you need to put both of your last two commands under the same line to look like this:
RUN bash $HOME/.sdkman/bin/sdkman-init.sh && bash -c "sdk install java 8.0.222.j9-adpt"

How to install the Google Cloud SDK in a Docker Image?

How can I build a Docker container with Google's Cloud Command Line Tool/SDK?
The script at the url https://sdk.cloud.google.com appears to require user input so doesn't work in a docker file.
Adding the following to my Docker file appears to work.
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
Use this one-liner in your Dockerfile:
RUN curl -sSL https://sdk.cloud.google.com | bash
source:
https://docs.docker.com/v1.8/installation/google/
Doing it with alpine:
FROM alpine:3.6
RUN apk add --update \
python \
curl \
which \
bash
RUN curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin
RUN curl -sSL https://sdk.cloud.google.com > /tmp/gcl && bash /tmp/gcl --install-dir=~/gcloud --disable-prompts
This will download the google cloud sdk installer into /tmp/gcl, and run it with the parameters as follows:
--install-dir=~/gcloud: Extract the binaries into folder gcloud in home folder. Change this to wherever you want, for example /usr/local/bin
--disable-prompts: Don't show any prompts while installing (headless)
To install gcloud inside a docker container please follow the instructions here.
Basically you need to run
RUN apt-get update && \
apt-get install -y curl gnupg && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && \
apt-get update -y && \
apt-get install google-cloud-sdk -y
inside your dockerfile. It's important you are user ROOT when you run this command, so it may necessary to add USER root before the previous command.
As an alternative, you could use the docker image provided by google namely google/cloud-sdk. https://hub.docker.com/r/google/cloud-sdk/
Dockerfile:
FROM centos:7
RUN yum update -y && yum install -y \
curl \
which && \
yum clean all
RUN curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin
Build:
docker build . -t google-cloud-sdk
Then run gcloud:
docker run --rm \
--volume $(pwd)/assets/root/.config:/root/.config \
google-cloud-sdk gcloud
...or run gsutil:
docker run --rm \
--volume $(pwd)/assets/root/.config:/root/.config \
google-cloud-sdk gsutil
The local assets folder will contain the configuration.
apk upgrade --update-cache --available && \
apk add openssl && \
apk add curl python3 py-crcmod bash libc6-compat && \
rm -rf /var/cache/apk/*
curl https://sdk.cloud.google.com | bash > /dev/null
export PATH=$PATH:/root/google-cloud-sdk/bin
gcloud components update kubectl
I was using Python Alpine image python:3.8.6-alpine3.12 as base and this worked for me:
RUN apk add --no-cache bash
RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-327.0.0-linux-x86_64.tar.gz \
-O /tmp/google-cloud-sdk.tar.gz | bash
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvzf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh -q
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
After building and running the image, you can check if google-cloud-sdk is installed by running docker exec -i -t <container_id> /bin/bash and running this:
bash-5.0# gcloud --version
Google Cloud SDK 327.0.0
bq 2.0.64
core 2021.02.05
gsutil 4.58
bash-5.0# gsutil --version
gsutil version: 4.58
If you want a specific version of google-cloud-sdk, you can visit https://storage.cloud.google.com/cloud-sdk-release
curl https://sdk.cloud.google.com | bash -s -- --disable-prompts
and export env
works for me
I got this working with Ubuntu 18.04 using:
RUN apt-get install -y curl && curl -sSL https://sdk.cloud.google.com | bash
ENV PATH="$PATH:/root/google-cloud-sdk/bin"
You can use multi-stage builds to make this simpler and more efficient than solutions using curl.
FROM bitnami/google-cloud-sdk:0.392.0 as gcloud
FROM base-image-for-production:tag
# Do what you need to configure your production image
COPY --from=gcloud /opt/bitnami/google-cloud-sdk/ /google-cloud-sdk
This work for me.
FROM php:7.2-fpm
RUN apt-get update -y
RUN apt-get install -y python && \
curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin
An example using debian as the base image:
FROM debian:stretch
RUN apt-get update && apt-get install -y apt-transport-https gnupg curl lsb-release
RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
echo "cloud SDK repo: $CLOUD_SDK_REPO" && \
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
apt-get update -y && apt-get install google-cloud-sdk -y
I used most of these examples in some form (thanks #KJoe), but I had to do several other things to setup everything so gcloud would work in the environment. Note that it is preferable to limit the number of lines (it limits layers needed to pull)
Here's a more complete example of Dockerfile with gcloud setup and extending a CircleCI image:
FROM circleci/ruby:2.4.1-jessie-node-browsers
# user is circleci in the FROM image, switch to root for system lib installation
USER root
ENV CCI /home/circleci
ENV GTMP /tmp/gcloud-install
ENV GSDK $CCI/google-cloud-sdk
ENV PATH="${GSDK}/bin:${PATH}"
# do all system lib installation in one-line to optimize layers
RUN curl -sSL https://sdk.cloud.google.com > $GTMP && bash $GTMP --install-dir=$CCI --disable-prompts \
&& rm -rf $GTMP \
&& chmod +x $GSDK/bin/* \
\
&& chown -Rf circleci:circleci $CCI
# change back to the user in the FROM image
USER circleci
# setup gcloud specifics to your liking
RUN gcloud config set core/disable_usage_reporting true \
&& gcloud config set component_manager/disable_update_check true \
&& gcloud components install alpha beta kubectl --quiet
My use case was to generate a google bearer token using the service account, so I wanted the docker container to install gcloud this is how my docker file looks like
FROM google/cloud-sdk
# Setting the default directory in container
WORKDIR /usr/src/app
# copies the app source code to the directory in container
COPY . /usr/src/app
CMD ["/bin/bash","/usr/src/app/token.sh"]
If you need to examine a container after it is built but that isn't running use docker run --rm -it <container-build-id> bash -il and type in gcloud --version if installed correctly or not
In Google documentation you can see the best practice
https://cloud.google.com/sdk/docs/install-sdk
search on the page for "Docker Tip"
eg debian use:
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && apt-get update -y && apt-get install google-cloud-cli -y
If you're just interested in getting the gcloud CLI available, add this to your Dockerfile:
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-409.0.0-linux-x86_64.tar.gz > /tmp/google-cloud-cli.tar.gz
# Installing the gcloud cli
RUN mkdir -p /usr/local/gcloud \
&& tar -xf /tmp/google-cloud-cli.tar.gz \
&& ./google-cloud-sdk/install.sh --quiet