I am following instructions here to install Google Cloud SDK:
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get install apt-transport-https ca-certificates gnupg
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-sdk
but in updating APT such that
$ sudo apt update
Hit:1 http://ftp.debian.org/debian unstable InRelease
Hit:2 http://deb.debian.org/debian experimental InRelease
Hit:3 https://updates.signal.org/desktop/apt xenial InRelease
Get:5 https://packages.cloud.google.com/apt cloud-sdk InRelease [6,337 B]
Err:5 https://packages.cloud.google.com/apt cloud-sdk InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
Hit:4 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Reading package lists... Done
W: GPG error: https://packages.cloud.google.com/apt cloud-sdk InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6A030B21BA07F4FB
E: The repository 'https://packages.cloud.google.com/apt cloud-sdk InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
and in adding the key
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
with Debian version
$ cat /etc/issue
Debian GNU/Linux 10 \n \l
How can I properly install Google Cloud SDK in Debian 10?
PO's Google support manual here is outdated. The new thread here fixed the issue:
# Add the Cloud SDK distribution URI as a package source
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
# Import the Google Cloud Platform public key
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
# Update the package list and install the Cloud SDK
sudo apt-get update && sudo apt-get install google-cloud-sdk
The easiest way (from within debian VM or container):
apt-get update && apt-get install -y curl python
curl sdk.cloud.google.com -L | bash
Exit the VM (or container) and access it again.
gcloud init
And you are done.
Related
I have this docker file:
# We are going to star from the jhipster image
FROM jhipster/jhipster
# install as root
USER root
### Setup docker cli (don't need docker daemon) ###
# Install some packages
RUN apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y
# Add Dockers official GPG key:
RUN ["/bin/bash", "-c", "set -o pipefail && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -"]
# Add a stable repository
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# Setup aws credentials as environment variables
ENV AWS_ACCESS_KEY_ID "change it!"
ENV AWS_SECRET_ACCESS_KEY "change it!"
# noninteractive install for tzdata
ARG DEBIAN_FRONTEND=noninteractive
# set timezone for tzdata
ENV TZ=America/Sao_Paulo
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
# Install the latest version of Docker Engine - Community and also aws cli
RUN apt-get update && apt-get install docker-ce docker-ce-cli containerd.io awscli -y
# change back to default user
USER jhipster
# install skd and java version 1.8
RUN curl -s "https://get.sdkman.io" | bash
RUN bash $HOME/.sdkman/bin/sdkman-init.sh
RUN bash -c "sdk install java 8.0.222.j9-adpt"
When I run a command to build an image from this dockerfile it fails on the last step with a message:
/bin/sh: 1: sdk: not found
When I install it on my local machine it runs sdkman (sdk) on bash. But on this script it calls it from sh not bash. How can I make it calls skdman (sdk) from sh? What I actually want to do is install a specific java version through sdkman (sdk). Is there another way to do it?
For sdk command to be available you need to run source sdkman-init.sh.
Here is a working sample with java 11 on centos.
FROM centos:latest
ARG CANDIDATE=java
ARG CANDIDATE_VERSION=11.0.6-open
ENV SDKMAN_DIR=/root/.sdkman
# update the image
RUN yum -y upgrade
# install requirements, install and configure sdkman
# see https://sdkman.io/usage for configuration options
RUN yum -y install curl ca-certificates zip unzip openssl which findutils && \
update-ca-trust && \
curl -s "https://get.sdkman.io" | bash && \
echo "sdkman_auto_answer=true" > $SDKMAN_DIR/etc/config && \
echo "sdkman_auto_selfupdate=false" >> $SDKMAN_DIR/etc/config
# Source sdkman to make the sdk command available and install candidate
RUN bash -c "source $SDKMAN_DIR/bin/sdkman-init.sh && sdk install $CANDIDATE $CANDIDATE_VERSION"
# Add candidate path to $PATH environment variable
ENV JAVA_HOME="$SDKMAN_DIR/candidates/java/current"
ENV PATH="$JAVA_HOME/bin:$PATH"
ENTRYPOINT ["/bin/bash", "-c", "source $SDKMAN_DIR/bin/sdkman-init.sh && \"$#\"", "-s"]
CMD ["sdk", "help"]
The problem is every RUN command in Dockerfile is executed within a new bash environment, so you need to put both of your last two commands under the same line to look like this:
RUN bash $HOME/.sdkman/bin/sdkman-init.sh && bash -c "sdk install java 8.0.222.j9-adpt"
I'm trying to deploy a geodjango application on AWS Elastic Beanstalk. The configuration is 64bit Amazon Linux 2017.09 v2.6.6 running Python 3.6. I am getting this error when trying to deploy:
Requires: libpoppler.so.5()(64bit) Error: Package: gdal-java-1.9.2-8.rhel6.x86_64 (pgdg93) Requires: libpoppler.so.5()(64bit)
How do I install the required package? I read through Setting up Django with GeoDjango Support in AWS Beanstalk or EC2 Instance but I am still getting problems. My ebextensions currently looks like:
commands:
01_yum_update:
command: sudo yum -y update
02_epel_repo:
command: sudo yum-config-manager -y --enable epel
03_install_gdal_packages:
command: sudo yum -y install gdal gdal-devel
packages:
yum:
git: []
postgresql95-devel: []
gettext: []
libjpeg-turbo-devel: []
libffi-devel: []
I'm going to answer my own question for the sake my future projects and anyone else trying to get started with geodjango. Updating this answer as of July 2020
Create an ebextensions file to install GDAL on the EC2 instance at deployment:
01_gdal.config
commands:
01_install_gdal:
test: "[ ! -d /usr/local/gdal ]"
command: "/tmp/gdal_install.sh"
files:
"/tmp/gdal_install.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel
# Geos
cd /
sudo mkdir -p /usr/local/geos
cd usr/local/geos/geos-3.7.2
sudo wget geos-3.7.2.tar.bz2 http://download.osgeo.org/geos/geos-3.7.2.tar.bz2
sudo tar -xvf geos-3.7.2.tar.bz2
cd geos-3.7.2
sudo ./configure
sudo make
sudo make install
sudo ldconfig
# Proj4
cd /
sudo mkdir -p /usr/local/proj
cd usr/local/proj
sudo wget -O proj-5.2.0.tar.gz http://download.osgeo.org/proj/proj-5.2.0.tar.gz
sudo wget -O proj-datumgrid-1.8.tar.gz http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz
sudo tar xvf proj-5.2.0.tar.gz
sudo tar xvf proj-datumgrid-1.8.tar.gz
cd proj-5.2.0
sudo ./configure
sudo make
sudo make install
sudo ldconfig
# GDAL
cd /
sudo mkdir -p /usr/local/gdal
cd usr/local/gdal
sudo wget -O gdal-2.4.4.tar.gz http://download.osgeo.org/gdal/2.4.4/gdal-2.4.4.tar.gz
sudo tar xvf gdal-2.4.4.tar.gz
cd gdal-2.4.4
sudo ./configure
sudo make
sudo make install
sudo ldconfig
As shown, the script checks whether gdal already exists using the test function. It then downloads the Geos, Proj, and GDAL libraries and installs them in the usr/local directory. At the time of writing this, geodjango (Django 3.0) supports up to Geos 3.7, Proj 5.2 (which also requires projdatum. Current releases do not require it), and GDAL 2.4 Warning: this installation process can take a long time. Also I am not a Linux professional so some of those commands may be redundant, but it works.
Lastly I add the following two environment variables to my Elastic Beanstalk configuration:
LD_LIBRARY_PATH: /usr/local/lib:$LD_LIBRARY_PATH
PROJ_LIB: usr/local/proj
If you still have troubles I recommend checking the logs and ssh-ing in the EC2 instance to check that installation took place. Original credit to this post
I would like to work Google Cloud SDK on ARM machine.
$ uname -a
Linux myhost 3.14.79-at10 #2 SMP PREEMPT Mon Mar 6 15:38:30 JST 2017 armv7l GNU/Linux
In this page, I can find only for x86 architecture.
Can I work Google Cloud SDK on ARM?
Yes - I was able to install it using the apt-get instructions on an ARM64 (aarch64) Pinebook Pro. If you don't have Ubuntu/Debian, you could use a Docker container. I did it from Manjaro-ARM using an Ubuntu container.
I would think those instructions would work for a Raspberry Pi running Raspbian.
Although the link above, being maintained by Google, may be the best place to obtain these instructions, I will copy in the current minimal set of commands below, just in case the instructions get moved at some point:
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates gnupg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-sdk
gcloud init
You could optionally install any of the following additional packages:
google-cloud-sdk-app-engine-python
google-cloud-sdk-app-engine-python-extras
google-cloud-sdk-app-engine-java
google-cloud-sdk-app-engine-go
google-cloud-sdk-bigtable-emulator
google-cloud-sdk-cbt
google-cloud-sdk-cloud-build-local
google-cloud-sdk-datalab
google-cloud-sdk-datastore-emulator
google-cloud-sdk-firestore-emulator
google-cloud-sdk-pubsub-emulator
kubectl
The answer is No. The SDK is closed source, and it's very not likely that you can hack it to work on ARM. I won't stop you from doing that since it mostly consists of Python scripts.
On the other hand, gsutil, a part of the SDK which handles Cloud Storage operations, is open source and on PyPI. You can install that using pip just as normal.
We organize our local environments around Docker. Unfortunately, there is no official ARM Docker image for the Google Cloud SDK. To get around that, we cloned the official Google Cloud SDK Dockerfile and, after some trial and error, were able to remove the unavailable SDK modules so we can build locally to produce an ARM Docker image. The unavailable modules were not an issue for us as we don't use them so we just commented them out (see the LOCAL_HACK section below). Here is the current hacked Dockerfile we use:
# This is a temporary workaround Dockerfile to allow us to run the Google SDK on Apple Silicon
# For the original #see https://raw.githubusercontent.com/GoogleCloudPlatform/cloud-sdk-docker/master/Dockerfile
FROM docker:19.03.11 as static-docker-source
FROM debian:buster
ARG CLOUD_SDK_VERSION=365.0.1
ENV CLOUD_SDK_VERSION=$CLOUD_SDK_VERSION
ENV PATH "$PATH:/opt/google-cloud-sdk/bin/"
COPY --from=static-docker-source /usr/local/bin/docker /usr/local/bin/docker
RUN groupadd -r -g 1000 cloudsdk && \
useradd -r -u 1000 -m -s /bin/bash -g cloudsdk cloudsdk
RUN apt-get -qqy update && apt-get install -qqy \
curl \
python3-dev \
python3-crcmod \
python-crcmod \
apt-transport-https \
lsb-release \
openssh-client \
git \
make \
gnupg && \
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
echo "deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" > /etc/apt/sources.list.d/google-cloud-sdk.list && \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
apt-get update && \
apt-get install -y google-cloud-sdk=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-app-engine-python=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-app-engine-python-extras=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-app-engine-java=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-datalab=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-datastore-emulator=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-pubsub-emulator=${CLOUD_SDK_VERSION}-0 \
google-cloud-sdk-firestore-emulator=${CLOUD_SDK_VERSION}-0 \
kubectl && \
gcloud --version && \
docker --version && kubectl version --client
# >>> LOCAL HACK START
# #todo Removed the following packages from the `apt-get install` above as we cannot build them locally
#8 29.36 E: Unable to locate package google-cloud-sdk-app-engine-go
#8 29.37 E: Version '339.0.0-0' for 'google-cloud-sdk-bigtable-emulator' was not found
#8 29.37 E: Unable to locate package google-cloud-sdk-spanner-emulator
#8 29.37 E: Unable to locate package google-cloud-sdk-cbt
#8 29.37 E: Unable to locate package google-cloud-sdk-kpt
#8 29.37 E: Unable to locate package google-cloud-sdk-local-extract
# google-cloud-sdk-app-engine-go=${CLOUD_SDK_VERSION}-0 \
# google-cloud-sdk-bigtable-emulator=${CLOUD_SDK_VERSION}-0 \
# google-cloud-sdk-spanner-emulator=${CLOUD_SDK_VERSION}-0 \
# google-cloud-sdk-cbt=${CLOUD_SDK_VERSION}-0 \
# google-cloud-sdk-kpt=${CLOUD_SDK_VERSION}-0 \
# google-cloud-sdk-local-extract=${CLOUD_SDK_VERSION}-0 \
# <<< LOCAL HACK END
RUN apt-get install -qqy \
gcc \
python3-pip
RUN pip3 install --upgrade pip
RUN pip3 install pyopenssl
RUN git config --system credential.'https://source.developers.google.com'.helper gcloud.sh
VOLUME ["/root/.config", "/root/.kube"]
If you were to save this file as Dockerfile.CloudSdk.arm64, you can then run a docker build on an ARM machine (in our case, an Apple M1 machine) to produce your ARM Docker image:
docker build -f Dockerfile.CloudSdk.arm64 -t yourorg.com/cloud-sdk-docker-arm:latest .
Voila! You now have a reasonably featured Google Cloud SDK Docker image that will run beautifully on an ARM architecture :)
If you have python or python3, along with pip and pip3, try:
pip install --upgrade google-cloud
Hope that helps.
tekk#rack:~ $ uname -a
Linux rack 4.9.59-v7+ #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux
It worked for me.
I am currently using Tika to extract text from files uploaded to my Rails app running on AWS Elastic Beanstalk (64bit Amazon Linux 2016.03 v2.1.2 running Ruby 2.2). I'd like to index scanned images as well, so I need to install Tesseract.
I was able to get it to work by installing it from source like so, but it added 10 minutes to my deploys to a fresh instance. Is there a faster way to do this?
.ebextensions/02-tesseract.config
packages:
yum:
autoconf: []
automake: []
libtool: []
libpng-devel: []
libtiff-devel: []
zlib-devel: []
container_commands:
01-command:
command: mkdir -p install
cwd: /home/ec2-user
02-command:
command: cp .ebextensions/scripts/install_tesseract.sh /home/ec2-user/install/
03-command:
command: bash install/install_tesseract.sh
cwd: /home/ec2-user
.ebextensions/scripts/install_tesseract.sh
#!/usr/bin/env bash
cd_to_install () {
cd /home/ec2-user/install
}
cd_to () {
cd /home/ec2-user/install/$1
}
if ! [ -x "$(command -v tesseract)" ]; then
# Add `usr/local/bin` to PATH
echo 'pathmunge /usr/local/bin' > /etc/profile.d/usr_local.sh
chmod +x /etc/profile.d/usr_local.sh
# Install leptonica
cd_to_install
wget http://www.leptonica.org/source/leptonica-1.73.tar.gz
tar -zxvf leptonica-1.73.tar.gz
cd_to leptonica-1.73
./configure
make
make install
rm -rf /home/ec2-user/install/leptonica-1.73.tar.gz
rm -rf /home/ec2-user/install/leptonica-1.73
# Install tesseract ~ the jewel of Odin's treasure room
cd_to_install
wget https://github.com/tesseract-ocr/tesseract/archive/3.04.01.tar.gz
tar -zxvf 3.04.01.tar.gz
cd_to tesseract-3.04.01
./autogen.sh
./configure
make
make install
ldconfig
rm -rf /home/ec2-user/install/3.04.01.tar.gz
rm -rf /home/ec2-user/install/tesseract-3.04.01
# Install tessdata
cd_to_install
wget https://github.com/tesseract-ocr/tessdata/archive/3.04.00.tar.gz
tar -zxvf 3.04.00.tar.gz
cp /home/ec2-user/install/tessdata-3.04.00/eng.* /usr/local/share/tessdata/
rm -rf /home/ec2-user/install/3.04.00.tar.gz
rm -rf /home/ec2-user/install/tessdata-3.04.00
fi
Short answer
.ebextensions/02-tesseract.config
commands:
01-libwebp:
command: "yum --enablerepo=epel --disablerepo=amzn-main -y install libwebp"
02-tesseract:
command: "yum --enablerepo=epel -y install tesseract"
Long answer
I'm not familiar with non-Ubuntu package managers or ebextensions, so after some digging, I found that there are precompiled binaries that can be installed on Amazon Linux in the stable EPEL repo.
The first obstacle was figuring out how to use the EPEL repo. The easiest way is to use the enablerepo option on the yum command.
That gets us here:
yum --enablerepo=epel install tesseract
Next, I had to resolve this dependency error:
[root#ip-10-0-1-193 ec2-user]# yum install --enablerepo=epel tesseract
Loaded plugins: priorities, update-motd, upgrade-helper
951 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package tesseract.x86_64 0:3.04.00-3.el6 will be installed
--> Processing Dependency: liblept.so.4()(64bit) for package: tesseract-3.04.00-3.el6.x86_64
--> Running transaction check
---> Package leptonica.x86_64 0:1.72-2.el6 will be installed
--> Processing Dependency: libwebp.so.5()(64bit) for package: leptonica-1.72-2.el6.x86_64
--> Finished Dependency Resolution
Error: Package: leptonica-1.72-2.el6.x86_64 (epel)
Requires: libwebp.so.5()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
I found the solution here
Just adding the epel repo doesn't solve it, as the packages in the
amzn-main repository seem to overrule those in the epel repository. If
the libwebp package in the amzn-main repo are excluded it should work
The Tesseract install has some dependencies found in the amzn-main repo. This is why I first install libwebp with --disablerepo=amzn-main.
yum --enablerepo=epel --disablerepo=amzn-main install libwebp
yum --enablerepo=epel install tesseract
Finally, here's how you can install yum packages on Elastic Beanstalk with options:
.ebextensions/02-tesseract.config
commands:
01-libwebp:
command: "yum --enablerepo=epel --disablerepo=amzn-main -y install libwebp"
02-tesseract:
command: "yum --enablerepo=epel -y install tesseract"
Fortunately, this is also the easiest way to install Tesseract on Elastic Beanstalk!
I need to:
sudo apt-get install rubygems
sudo gem install <lots of gems>
does the bootstrap action have sudo access?
The answer is yes. You can test your bootstrap script like this:
elastic_mapreduce --create --alive --ssh
This will create a node and give you a ssh connection to it, from which you can test your bootstrap script.
UPDATE: For reference here is what I'm running:
#!/bin/bash
sudo apt-get -y -V install irb1.8 libreadline-ruby1.8 libruby libruby1.8 rdoc1.8 ruby ruby1.8 ruby1.8-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install bson bson_ext json tzinfo i18n activesupport --no-rdoc --no-ri
UPDATE2: to install aws-sdk
#!/bin/bash
# ruby developer packages
sudo apt-get -y -V install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8
sudo apt-get -y -V install libreadline-ruby1.8 libruby1.8 libopenssl-ruby
# nokogiri requirements
sudo apt-get -y -V install libxslt-dev libxml2-dev
wget http://production.cf.rubygems.org/rubygems/rubygems-1.8.11.zip
unzip rubygems-1.8.11.zip
cd rubygems-1.8.11
sudo ruby setup.rb
sudo gem1.8 install aws-sdk --no-rdoc --no-ri
-y on apt-get makes it not prompt you
I wget rubygems because the version you get with apt-get is way out of date, and some gems won't build using an old version.
Yup, and here is a list of commands that I ran to setup my instance (for all the people who google for this question later):
sudo apt-get update
sudo apt-get -y install emacs
sudo apt-get -y install rubygems
sudo gem install fastercsv --source http://rubygems.org
sudo gem install crack --source http://rubygems.org
sudo gem install json_pure --source http://rubygems.org
exit