I have created a Wordpress Service using Cloud Run . I deployed using below command
gcloud beta run deploy wp --image gcr.io/<project>/wp:v1 \
--add-cloudsql-instances <project>:us-central1:mysql2 \
--update-env-vars DB_HOST='127.0.0.1',DB_NAME=mysql2,DB_USER=wordpress,DB_PASSWORD=password,CLOUDSQL_INSTANCE='<project>:us-central1:mysql2'
The service is deployed fine but while trying to access the service it is showing below error
<h1>Error: Forbidden</h1>
<h2>Your client does not have permission to get URL <code>/</code> from this server.</h2>
UPDATES:
Dockerfile is as follows . I am following this...
https://github.com/acadevmy/cloud-run-wordpress
FROM wordpress:5.2.1-php7.3-apache
EXPOSE 80
# Use the PORT environment variable in Apache configuration files.
RUN sed -i 's/80/${PORT}/g' /etc/apache2/sites-available/000-default.conf /etc/apache2/ports.conf
# wordpress conf
COPY wordpress/wp-config.php /var/www/html/wp-config.php
# download and install cloud_sql_proxy
RUN apt-get update && apt-get -y install net-tools wget && \
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O /usr/local/bin/cloud_sql_proxy && \
chmod +x /usr/local/bin/cloud_sql_proxy
COPY wordpress/cloud-run-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["/usr/local/sbin/apache2ctl -D FOREGROUND"]
##docker-entrypoint.sh
#!/usr/bin/env bash
# Start the sql proxy
cloud_sql_proxy -instances=$CLOUDSQL_INSTANCE=tcp:3306 &
# Execute the rest of your ENTRYPOINT and CMD as expected.
Following can be seen in Console Log
We allowed Unauthenticated authentication and now the error is
"Error establishing a database connection"
Additional Updates:
The DB is running with a private IP so using Serverless VPC .
DB information is as follows:
gcloud sql instances list
NAME DATABASE_VERSION LOCATION TIER PRIMARY_ADDRESS PRIVATE_ADDRESS STATUS
mysql2 MYSQL_5_7 us-central1-b db-f1-micro - 10.0.100.5 RUNNABLE
This is Serverless VPC access range
testserverlessvpc kube-shared-vpc us-central1 192.168.60.0/28 200 300
Now I have added an additional parameter as shown below with both gcloud run deploy and gcloud run service command
--vpc-connector projects/< HOST-Project >/locations/us-central1/connectors/testserverlessvpc
But during gcloud run deploy it is failing with below error
⠏ Deploying new service... Internal system error, system will retry.
Related
Follow this link, I can create a pod whose service account's role can access the AWS resources; so the pod can access them either.
Then, inspired by this EKS-Jenkins-Workshop, I change this workshop a little bit. I want to deploy Jenkins Pipeline, this Jenkins Pipeline can create a pod whose account service's role can access aws resources, but the problem is the cdk code in this pod cannot access AWS resources. (I write the cdk code to access AWS resources, reference (Your first AWS CDK app)[https://docs.aws.amazon.com/cdk/latest/guide/hello_world.html])
This is my Jenkinsfile
pipeline {
agent {
kubernetes {
yaml """
apiVersion: v1
kind: Pod
metadata:
name: jenkins-agent
Namespace: default
spec:
serviceAccountName: jenkins
containers:
- name: node-yuvein
image: node
command:
- cat
tty: true
"""
}
}
stages {
stage('Build') {
steps {
container('node-yuvein') {
dir('hello-cdk'){
sh "pwd"
sh 'npm --version'
sh 'node -v'
sh 'npm install -g typescript'
sh 'npm install -g aws-cdk'
sh 'npm install #aws-cdk/aws-s3'
sh 'npm run build'
sh 'cdk deploy'
}
}
}
}
}
}
When I run the pipeline, it has this error:
User: arn:aws:sts::450261875116:assumed-role/eksctl-eksworkshop-eksctl3-nodegr-NodeInstanceRole-1TCVDYSM1QKSO/i-0a4df3778517df0c6 is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-west-2:450261875116:stack/HelloCdkStack/*
I am a beginner of K8s, Jenkins and cdk.
Hope someone can help me.
Thanks a lot.
Further Debugging:
In Jenkins Console, I can get serviceAccountName: "jenkins", and the name of my service account in EKS is jenkins.
the pod also get correct ENV:
+ echo $AWS_ROLE_ARN
arn:aws:iam::450261875116:role/eksctl-eksworkshop-eksctl3-addon-iamservicea-Role1-YYYFXFS0J4M2
+ echo $AWS_WEB_IDENTITY_TOKEN_FILE
/var/run/secrets/eks.amazonaws.com/serviceaccount/token
The node.js and npm I installed are the lastest version.
+ npm --version
6.14.8
+ node -v
v14.13.0
+ aws sts get-caller-identity
{
"UserId": "AROAWRVNS7GWO5C7QJGRF:botocore-session-1601436882",
"Account": "450261875116",
"Arn": "arn:aws:sts::450261875116:assumed-role/eksctl-eksworkshop-eksctl3-addon-iamservicea-Role1-YYYFXFS0J4M2/botocore-session-1601436882"
}
when I run this command, it appears my service account role. But I still get the original error.
Jenkins podTemplate has serviceAccount option:
https://github.com/jenkinsci/kubernetes-plugin#pod-and-container-template-configuration
Create an IAM role mapped to an EKS cluster
Create a ServiceAccount mapped to an IAM role
Pass ServiceAccount name to a podTemplate
Further debugging:
Ensure the pod has correct service account name.
Check if pod got AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE env vars (they are added automatically).
Check if AWS SDK you use is above the minimal version: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
Run aws sts get-caller-identity to see the role, don't waste time on running an actual job.
In the case of working with Jenkins slaves, one needs to customize the container images to use AWS CLI V2 instead of AWS CLI V1. I was running into errors related to authorization like the question poses; my client was using the cluster node roles instead of using the assumed web identity role of my service account attached to my Jenkins-pods for the slave containers.
Apparently V2 of the AWS CLI includes the web identity token file as part of the default credentials chain whereas V1 does not.
Here's a sample Dockerfile that pulls the latest AWS CLI version so this pattern works.
FROM jenkins/inbound-agent
# run updates as root
USER root
# Create docker group
RUN addgroup docker
# Update & Upgrade OS
RUN apt-get update
RUN apt-get -y upgrade
#install python3
RUN apt-get -y install python3
# add AWS Cli version 2 for web_identity_token files
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
# Add Maven
RUN apt-get -y install maven --no-install-recommends
# Add docker
RUN curl -sSL https://get.docker.com/ | sh
RUN usermod -aG docker jenkins
# Add docker compose
RUN curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
RUN chmod +x /usr/local/bin/docker-compose
# Delete cached files we don't need anymore:
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/*
# close root access
USER jenkins
Further, I had to make sure my serviceaccount was created and attached to both the Jenkins master image and the jenkins slaves. This can be accomplished via Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds -> Pod Template Details.
Be sure to edit Namespace and Serviceaccount fields with the appropriate values.
I want to run a service on Google Cloud Run that uses Cloud Memorystore as cache.
I created an Memorystore instance in the same region as Cloud Run and used the example code to connect: https://github.com/GoogleCloudPlatform/golang-samples/blob/master/memorystore/redis/main.go this didn't work.
Next I created a Serverless VPC access Connectore which didn't help. I use Cloud Run without a GKE Cluster so I can't change any configuration.
Is there a way to connect from Cloud Run to Memorystore?
To connect Cloud Run (fully managed) to Memorystore you need to use the mechanism called "Serverless VPC Access" or a "VPC Connector".
As of May 2020, Cloud Run (fully managed) has Beta support for the Serverless VPC Access. See Connecting to a VPC Network for more information.
Alternatives to using this Beta include:
Use Cloud Run for Anthos, where GKE provides the capability to connect to Memorystore if the cluster is configured for it.
Stay within fully managed Serverless but use a GA version of the Serverless VPC Access feature by using App Engine with Memorystore.
While waiting for serverless VPC connectors on Cloud Run - Google said yesterday that announcements would be made in the near term - you can connect to Memorystore from Cloud Run using an SSH tunnel via GCE.
The basic approach is the following.
First, create a forwarder instance on GCE
gcloud compute instances create vpc-forwarder --machine-type=f1-micro --zone=us-central1-a
Don't forget to open port 22 in your firewall policies (it's open by default).
Then install the gcloud CLI via your Dockerfile
Here is an example for a Rails app. The Dockerfile makes use of a script for the entrypoint.
# Use the official lightweight Ruby image.
# https://hub.docker.com/_/ruby
FROM ruby:2.5.5
# Install gcloud
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
# Generate SSH key to be used by the SSH tunnel (see entrypoint.sh)
RUN mkdir -p /home/.ssh && ssh-keygen -b 2048 -t rsa -f /home/.ssh/google_compute_engine -q -N ""
# Install bundler
RUN gem update --system
RUN gem install bundler
# Install production dependencies.
WORKDIR /usr/src/app
COPY Gemfile Gemfile.lock ./
ENV BUNDLE_FROZEN=true
RUN bundle install
# Copy local code to the container image.
COPY . ./
# Run the web service on container startup.
CMD ["bash", "entrypoint.sh"]
Finally open an SSH tunnel to Redis in your entrypoint.sh script
# !/bin/bash
# Memorystore config
MEMORYSTORE_IP=10.0.0.5
MEMORYSTORE_REMOTE_PORT=6379
MEMORYSTORE_LOCAL_PORT=6379
# Forwarder config
FORWARDER_ID=vpc-forwarder
FORWARDER_ZONE=us-central1-a
# Start tunnel to Redis Memorystore in background
gcloud compute ssh \
--zone=${FORWARDER_ZONE} \
--ssh-flag="-N -L ${MEMORYSTORE_LOCAL_PORT}:${MEMORYSTORE_IP}:${MEMORYSTORE_REMOTE_PORT}" \
${FORWARDER_ID} &
# Run migrations and start Puma
bundle exec rake db:migrate && bundle exec puma -p 8080
With the solution above Memorystore will be available to your application on localhost:6379.
There are a few caveats though
This approach requires the service account configured on your Cloud Run service to have the roles/compute.instanceAdmin role, which is quite powerful.
The SSH keys are backed into the image to speedup container boot time. That's not ideal.
There is no failover if your forwarder crashes.
I've written a longer and more elaborated approach in a blog post that improves the overall security and adds failover capabilities. The solution uses plain SSH instead of the gcloud CLI.
If you need something in your VPC, you can also spin up Redis on Compute Engine
It's more costly (especially for a Cluster) than Redis Cloud - but an temp solution if you have to keep the data in your VPC.
This is the user data used:
#!/bin/bash
yum install httpd -y
yum update -y
aws s3 cp s3://YOURBUCKETNAMEHERE/index.html /var/www/html/
service httpd start
chkconfig httpd on
NAT gateway is configured for the private EC2 instance and also s3fullaccess permissions are given.
Please help me troubleshoot!
You can add some code to the start of your user-data script to redirect the output to logs.
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
Then you can use those logs to troubleshoot from the AWS Console. Select the instance, then Actions menu -> Instance settings -> Get system log. Here is more documentation on what to add to your bash script, as well as a video that shows where to find the logs.
I tried to use Google Container Registry, but it did not work for me.
I wrote the following containers.yaml.
$ cat containers.yaml
version: v1
kind: Pod
spec:
containers:
- name: amazonssh
image: asia.gcr.io/<project-id>/amazonssh
imagePullPolicy: Always
restartPolicy: Always
dnsPolicy: Default
I run instance by the following command.
$ gcloud compute instances create containervm-amazonssh --image container-vm --network product-network --metadata-from-file google-container-manifest=containers.yaml --zone asia-east1-a --machine-type f1-micro
I set the following acl permission.
# gsutil acl ch -r -u <project-number>#developer.gserviceaccount.com:R gs://asia.artifacts.<project-id>.appspot.com
But Access denied occurs when docker pull image from Google Container Registry.
# docker pull asia.gcr.io/<project-id>.a/amazonssh
Pulling repository asia.gcr.io/<project-id>.a/amazonssh
FATA[0000] Error: Status 403 trying to pull repository <project-id>/amazonssh: "Access denied."
Can you verify from your instance that you can read data from your Google Cloud Storage bucket? This can be verified by:
$ curl -H 'Metadata-Flavor: Google' $SVC_ACCT/scopes
...
https://www.googleapis.com/auth/devstorage.full_control
https://www.googleapis.com/auth/devstorage.read_write
https://www.googleapis.com/auth/devstorage.read_only
...
If so then try:
On Google Compute Engine you can login without gcloud with:
$ METADATA=http://metadata.google.internal./computeMetadata/v1
$ SVC_ACCT=$METADATA/instance/service-accounts/default
$ ACCESS_TOKEN=$(curl -H 'Metadata-Flavor: Google' $SVC_ACCT/token \
| cut -d'"' -f 4)
$ docker login -e not#val.id -u _token -p $ACCESS_TOKEN https://gcr.io
Then try your docker pull command again.
You have an extra .a after project-id here, not sure if you ran the command that way?
docker pull asia.gcr.io/<project-id>.a/amazonssh
The container-vm has a cron job running gcloud docker -a as root, so you should be able to docker pull as root.
The kubelet, which launches the container-vm Docker containers also understands how to natively authenticate with GCR, so it should just work.
Feel free to reach out to us at gcr-contact#google.com. It would be useful if you could include your project-id, and possibly the /var/log/kubelet.log from your container-vm.
I am trying to download an image from the Google container registry in a CoreOS machine running in other server (not GCE).
I configured a new service account:
core#XXXX ~ $ docker run -t -i -v $(pwd)/keys:/tmp/keys --name gcloud-config ernestoalejo/google-cloud-sdk-with-docker gcloud auth activate-service-account XXXXXXX#developer.gserviceaccount.com --key-file /tmp/keys/key.p12 --project XXXX
Activated service account credentials for: [XXXXXXX#developer.gserviceaccount.com]
The account is active, but when I try to download the container image it returns a forbidden HTTP status.
core#XXXX ~ $ /usr/bin/docker run --volumes-from gcloud-config --rm -v /var/run/docker.sock:/var/run/docker.sock ernestoalejo/google-cloud-sdk-with-docker sh -c "gcloud preview docker pull gcr.io/XXXXX/influxdb"
Pulling repository gcr.io/XXXXX/influxdb
time="2015-05-08T06:38:55Z" level="fatal" msg="HTTP code: 403"
ERROR: (gcloud.preview.docker) A Docker command did not run successfully.
Tried to run: 'docker pull gcr.io/XXXXX/influxdb'
Exit code: 1
There is only one account in the server and is correctly configured:
core#XXXX ~ $ /usr/bin/docker run --volumes-from gcloud-config --rm -v /var/run/docker.sock:/var/run/docker.sock ernestoalejo/google-cloud-sdk-with-docker sh -c "gcloud auth list"
To set the active account, run:
$ gcloud config set account ``ACCOUNT''
Credentialed accounts:
- XXXXXXXXXXXXX#developer.gserviceaccount.com (active)
How can I authorize the external machine to download images from the registry?
NOTE: The image ernestoalejo/google-cloud-sdk-with-docker is the same as google/cloud-sdk but with this issue fixed.
UPDATE: I have also tried the solution of this answer, but it makes no difference.
PROJECT_ID=XXXXXX
ROBOT=XXXXXX#developer.gserviceaccount.com
gsutil acl ch -u $ROBOT:R gs://artifacts.$PROJECT_ID.appspot.com
gsutil -m acl ch -R -u $ROBOT:R gs://artifacts.$PROJECT_ID.appspot.com
gsutil defacl ch -u $ROBOT:R gs://artifacts.$PROJECT_ID.appspot.com
It seems that the new Frankfurt region of Digital Ocean can't access the Google Container Registry at all. It always returns a 403 Forbidden. As soon as I used a server in London everything started working.