A lot of ray commands require a CLUSTER_CONFIG file.
for example
Usage: ray get-head-ip [OPTIONS] CLUSTER_CONFIG_FILE
Options:
-n, --cluster-name TEXT Override the configured cluster name.
--help Show this message and exit.
The example files provided are big and scary.. like..
cluster_name: default
min_workers: 0
max_workers: 0
docker:
image: ""
container_name: ""
target_utilization_fraction: 0.8
idle_timeout_minutes: 5
provider:
type: local
head_ip: YOUR_HEAD_NODE_HOSTNAME
worker_ips: []
auth:
ssh_user: YOUR_USERNAME
ssh_private_key: ~/.ssh/id_rsa
head_node: {}
worker_nodes: {}
file_mounts:
"/tmp/ray_sha": "/YOUR/LOCAL/RAY/REPO/.git/refs/heads/YOUR_BRANCH"
setup_commands: []
head_setup_commands: []
worker_setup_commands: []
setup_commands:
- source activate ray && test -e ray || git clone https://github.com/YOUR_GITHUB/ray.git
- source activate ray && cd ray && git fetch && git reset --hard `cat /tmp/ray_sha`
# - source activate ray && cd ray/python && pip install -e .
head_start_ray_commands:
- source activate ray && ray stop
- source activate ray && ulimit -c unlimited && ray start --head --redis-port=6379 --autoscaling-config=~/ray_bootstrap_config.yaml
worker_start_ray_commands:
- source activate ray && ray stop
- source activate ray && ray start --redis-address=$RAY_HEAD_IP:6379
Say I already have a ray cluster up and running, and just want to do things like, submit a job to it using ray command line. Do I really need all that stuff, or is there a minimal config I can use.
Here's a minimal example.
In the more verbose examples, the defaults should be good so you shouldn't need to change very much.
Also, if you already have a Ray cluster running and you started it with the autoscaler, you can submit jobs via ray exec, see the relevant documentation. See this script for an example of how to use it.
Related
I want to integrate my C++ project with Jenkins using Docker.
First I created a setup for Jenkins using:
Dockerfile:
FROM jenkins/jenkins:latest
USER root
RUN apt-get update && apt-get install git && apt-get install make g++ -y
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
ENV CASC_JENKINS_CONFIG /var/jenkins_home/casc.yaml
ENV JENKINS_ADMIN_ID=admin
ENV JENKINS_ADMIN_PASSWORD=password
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
COPY casc.yaml /var/jenkins_home/casc.yaml
plugins.txt
ant:latest
antisamy-markup-formatter:latest
authorize-project:latest
build-timeout:latest
cloudbees-folder:latest
configuration-as-code:latest
credentials-binding:latest
email-ext:latest
git:latest
github-branch-source:latest
gradle:latest
ldap:latest
mailer:latest
matrix-auth:latest
pam-auth:latest
pipeline-github-lib:latest
pipeline-stage-view:latest
ssh-slaves:latest
timestamper:latest
workflow-aggregator:latest
ws-cleanup:latest
casc.yaml
jenkins:
securityRealm:
local:
allowsSignup: false
users:
- id: ${JENKINS_ADMIN_ID}
password: ${JENKINS_ADMIN_PASSWORD}
authorizationStrategy:
globalMatrix:
permissions:
- "USER:Overall/Administer:admin"
- "GROUP:Overall/Read:authenticated"
remotingSecurity:
enabled: true
security:
queueItemAuthenticator:
authenticators:
- global:
strategy: triggeringUsersAuthorizationStrategy
unclassified:
location:
url: http://127.0.0.1:8080/
So far, so good. I used docker build -t jenkins:jcasc . and docker run --name jenkins --rm -p 8080:8080 jenkins:jcasc and Jenkins starts normally. I entered the user name (admin) and the password and it worked.
There are although 2 issues that Jenkins displays:
-> It appears that your reverse proxy set up is broken.
-> Building on the built-in node can be a security issue. You should set up distributed builds. See the documentation.
I ignored them so far, and continued with my project and installed Blue Ocean.
After installing Blue Ocean I connected it to my GitHub repository and for this I created a token and the pipeline was created, but it is not giving any result.
The log says:
Branch indexing
Connecting to https://api.github.com with no credentials, anonymous access
Jenkins-Imposed API Limiter: Current quota for Github API usage has 37 remaining (1 over budget). Next quota of 60 in 36 min. Sleeping for 4 min 43 sec.
Jenkins is attempting to evenly distribute GitHub API requests. To configure a different rate limiting strategy, such as having Jenkins restrict GitHub API requests only when near or above the GitHub rate limit, go to "GitHub API usage" under "Configure System" in the Jenkins settings.
My C++ project looks like this:
-->HelloWorld
|
-->HelloWorld.cpp
-->scripts
|
-->Linux-Build.sh
|
-->Linux-Run.sh
-->Jenkinsfile
HelloWorld.cpp:
#include <iostream>
int main(){
std::cout << "Hello World!" << std::endl;
return 0;
}
Linux-Build.sh
#!/bin/bash
vendor/bin/premake/premake5 gmake2
make
Linux-Run.sh
#!/bin/bash
bin/Debug/HelloWorld
Jenkinsfile
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'echo "Building..."'
sh 'chmod +x scripts/Linux-Build.sh'
sh 'scripts/Linux-Build.sh'
archiveArtifacts artifacts: 'bin/Debug/*', fingerprint: true
}
}
stage('Test') {
steps {
sh 'echo "Running..."'
sh 'chmod +x scripts/Linux-Run.sh'
sh 'scripts/Linux-Run.sh'
}
}
}
}
So, where might be the problem? I don't know if the problem is in my C++ code, or in the Docker configuration.
I am also a big fan of cmake, instead of make and I would prefer to use cmake for this, but I am not sure how to integrate this and make it work.
EDIT:
In the tutorial that I followed there was a "vendor" directory but the instructor didn't show what was in that directory.
EDIT:
The program took 4 minutes but it eventually did something and the error is:
+ scripts/Linux-Build.sh
scripts/Linux-Build.sh: line 3: vendor/bin/premake/premake5: No such file or directory
make: *** No targets specified and no makefile found. Stop.
script returned exit code 2
So, the problem is in the C++ project, not Docker.
After I upgraded gitlab yesterday, my gitlab-ci.yml file reported an error
The error message is jobs:gitkeep:only configshould be an array of strings or regexps
My config file content is
gitkeep:
stage: eslint
tags:
- chore
only:
- develop
- /^feature\/((?!snapshot$|latest$|release\/).)+?$/
- /^release\/((?!snapshot$|latest$).)+?$/
- /^dev\/((?!snapshot$|latest$).)+?$/
script:
- docker build -t mdf-modify-all .
- docker run --cpus="2.5" --rm mdf-modify-all sh -c "sh release-all.sh && git add . && git commit -m 'release all' && git push origin HEAD:$CI_COMMIT_REF_NAME"
when: manual
In some online verification tools, these configurations are legal. But in CI lint, it prompts that the regularity is illegal. How can I modify this regularity?
After I delete this regular expression, the verification is passed, but I want to keep this regular expression, how can I modify it?
I would like to know what gcloud builds submit does. In my case I am running the GCloud run tutorial.
The official documentation states that it submits a build. This is not a particularly helpful piece of information.
Can someone provide some more context to this?
What is a build? An Image? A jar file? Where is this 'build' being submitted to?
What does 'submitting' mean? Does this 'submit' process push my 'build' over the network.
When I run gcloud builds submit it also seems to be creating a docker image. So this is also creating the build, and then it is submitting it ?!!??
There are several steps that's happening when you run the gcloud builds submit command:
Compresses your application code, Dockerfile, and any other assets in the current directory as indicated by .;
Uploads the files to a Cloud Storage bucket (there's a default bucket but you're free to specify a bucket on your build config);
Initiates a build using the uploaded files as input;
Tags the image using the provided name; and
Pushes the built image to Container Registry.
On your case, a build is a docker container that is pushed/submitted into the Container Registry. Once it's submitted, you'll be able deploy that container on Cloud Run just as specified on the docs you've provided.
Cloud Build is a service that applies one or more container images in series to some initial set of input files and often generating some artifacts, often (not always) another container image, often some source code that was initially submitted that the service builds into a container image.
Cloud Build is somewhat analogous to e.g. Linux pipelines where some input is transformed by piping data through a series of commands: f | g | h | .... Alternatively you may think of it as composited functions: h(g(f(x))).
Cloud Build is described (and named) as a service to build (code into containers) but, as you know, actually the steps may be any container image and often these have side-effects such as deploying container images to other services e.g. Cloud Run.
Cloud Build is much more general-purpose than Google advertises it. Google limits its scope in its documentation to a cloud-based service to build software.
When you run gcloud builds submit... you provide some source code and either a Dockerfile or a configuration file. The former is a simple case of the second, a configuration file containing a single step that runs docker build....
Configuration files (YAML) list a series of container images with parameters that are run in series. Initially Cloud Build copies a designated source (can be the current directory) to a Compute Engine VM (created by the service) as a directory (that's automatically mounted into each container) as /workspace.
Containers (defined as steps in the configuration file) may operate on this file system (e.g. compline code, validate files, anything that you can do in a container). Often, in conclusion, config files store containers that have been created in e.g. Container Registry.
Solving Quadratic equations with Cloud Build
Cloud Build can be confusing to newcomers. In a spirit of fun and as a way to show that Cloud Build is quite general-purpose, here's a Rube Goldberg machine written in Cloud Build that solves quadratic equations:
For the following cloudbuild.yaml:
steps:
- name: busybox
args:
- ash
- -c
- 'echo "Quadratic: $(cat a)x²+$(cat b)x+$(cat c)s=0"'
- name: busybox
args:
- ash
- -c
- 'echo "$(cat b) * $(cat b)" | bc -l > b2'
- name: busybox
args:
- ash
- -c
- 'echo "4 * $(cat a) * $(cat c)" | bc -l > 4ac'
- name: busybox
args:
- ash
- -c
- 'echo "$(cat b2) - $(cat 4ac)" | bc -l > b2-4ac'
- name: busybox
args:
- ash
- -c
- 'echo "sqrt($(cat b2-4ac))" | bc -l > sqrt'
- name: busybox
args:
- ash
- -c
- 'echo "-($(cat b)) + $(cat sqrt)" | bc -l > add'
- name: busybox
args:
- ash
- -c
- 'echo "-($(cat b)) - $(cat sqrt)" | bc -l > sub'
- name: busybox
args:
- ash
- -c
- 'echo "2 * $(cat a)" | bc -l > 2a'
- name: busybox
args:
- ash
- -c
- 'echo "$(cat add)/$(cat 2a)" | bc -l > root1'
- name: busybox
args:
- ash
- -c
- 'echo "$(cat sub)/$(cat 2a)" | bc -l > root2'
- name: busybox
args:
- ash
- -c
- 'echo "Roots are: $(cat root1); $(cat root2)"'
It expects 3 files (a, b, c) in ${PWD} containing the values of ax²+bx+c=0. So, for 8x²-10x+3:
echo "8" > a
echo "-10" > b
echo "3" > c
You can run it with:
gcloud builds submit ${PWD} \
--config=./cloudbuild.yaml \
--project=${PROJECT}
Explanation Rube Goldberg Cloud Build machine for solving Quadratic equations
A build is the process of creating artifacts from a source, and optionally modifying the state of any system you have access to.
An artifact can be a text file, a Docker container image or a Java archive.
To submit a build is to send your build resources (source files) to Cloud Storage, on the one hand. On the other hand, it is also to create a worker, which is a Google-managed GCE instance in a pool of instances dedicated to builds, instances that scale horizontally on demand and are destroyed when their assigned build is finished.
The worker reads the source files from Cloud Storage, and executes each build step in the configuration file, creating a Docker container for each step.
Each container executes the script in each respective step.
The concatenaton of container script executions is the build, properly speaking, and it produces artifacts.
There is always at least one artifact, a text file with the build log, which is pushed to Cloud Storage, and there can be container images or Java archives also produced as artifacts, which are pushed to the Container Registry.
I wouldn't say there is a difference between "create" and "submit" a build, but, if there is, it might be that "creating" a build is all the build from preparation to end, or preparing the source files, having the environment ready (project, permissions, quota, etc.), and "submitting" it would be just issuing the command submit or having a trigger submit it for you.
I would like to run database migrations written in node.js during the Cloud Build process.
Currently, the database migration command is being executed but it seems that the Cloud Build process does not have access to connect to Cloud SQL via an IP address with username/password.
In the case with Cloud SQL and Node.js it would look something like this:
steps:
# Install Node.js dependencies
- id: yarn-install
name: gcr.io/cloud-builders/yarn
waitFor: ["-"]
# Install Cloud SQL proxy
- id: proxy-install
name: gcr.io/cloud-builders/yarn
entrypoint: sh
args:
- "-c"
- "wget https://storage.googleapis.com/cloudsql-proxy/v1.20.1/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy && chmod +x cloud_sql_proxy"
waitFor: ["-"]
# Migrate database schema to the latest version
# https://knexjs.org/#Migrations-CLI
- id: migrate
name: gcr.io/cloud-builders/yarn
entrypoint: sh
args:
- "-c"
- "(./cloud_sql_proxy -dir=/cloudsql -instances=<CLOUD_SQL_CONNECTION> & sleep 2) && yarn run knex migrate:latest"
timeout: "1200s"
waitFor: ["yarn-install", "proxy-install"]
timeout: "1200s"
You would launch yarn install and download Cloud SQL Proxy in parallel. Once these two steps are complete, you run launch the proxy, wait 2 seconds and finally run yarn run knex migrate:latest.
For this to work you would need Cloud SQL Admin API enabled in your GCP project.
Where <CLOUD_SQL_INSTANCE> is your Cloud SQL instance connection name that can be found here. The same name will be used in your SQL connection settings, e.g. host=/cloudsql/example:us-central1:pg13.
Also, make sure that the Cloud Build service account has "Cloud SQL Client" role in the GCP project, where the db instance is located.
As of tag 1.16 of gcr.io/cloudsql-docker/gce-proxy, the currently accepted answer no longer works. Here is a different approach that keeps the proxy in the same step as the commands that need it:
- id: cmd-with-proxy
name: [YOUR-CONTAINER-HERE]
timeout: 100s
entrypoint: sh
args:
- -c
- '(/workspace/cloud_sql_proxy -dir=/workspace -instances=[INSTANCE_CONNECTION_NAME] & sleep 2) && [YOUR-COMMAND-HERE]'
The proxy will automatically exit once the main process exits. Additionally, it'll mark the step as "ERROR" if either the proxy or the command given fails.
This does require the binary is in the /workspace volume, but this can be provided either manually or via a prereq step like this:
- id: proxy-install
name: alpine:3.10
entrypoint: sh
args:
- -c
- 'wget -O /workspace/cloud_sql_proxy https://storage.googleapis.com/cloudsql-proxy/v1.16/cloud_sql_proxy.linux.386 && chmod +x /workspace/cloud_sql_proxy'
Additionally, this should work with TCP since the proxy will be in the same container as the command.
Use google-appengine/exec-wrapper. It is an image to do exactly this. Usage (see README in link):
steps:
- name: "gcr.io/google-appengine/exec-wrapper"
args: ["-i", "gcr.io/my-project/appengine/some-long-name",
"-e", "ENV_VARIABLE_1=value1", "-e", "ENV_2=value2",
"-s", "my-project:us-central1:my_cloudsql_instance",
"--", "bundle", "exec", "rake", "db:migrate"]
The -s sets the proxy target.
Cloud Build runs using a service account and it looks like you need to grant access to Cloud SQL for this account.
You can find additional info about setting service account permissions here.
Here's how to combine Cloud Build + Cloud SQL Proxy + Docker.
If you're running your database migrations/operations within a Docker container in Cloud Build, it won't be able to directly access your proxy, because Docker containers are isolated from the host machine.
Here's what I managed to get up and running:
- id: build
# Build your application
waitFor: ['-']
- id: install-proxy
name: gcr.io/cloud-builders/wget
entrypoint: bash
args:
- -c
- wget -O /workspace/cloud_sql_proxy https://storage.googleapis.com/cloudsql-proxy/v1.15/cloud_sql_proxy.linux.386 && chmod +x /workspace/cloud_sql_proxy
waitFor: ['-']
- id: migrate
name: gcr.io/cloud-builders/docker
entrypoint: bash
args:
- -c
- |
/workspace/cloud_sql_proxy -dir=/workspace -instances=projectid:region:instanceid & sleep 2 && \
docker run -v /workspace:/root \
--env DATABASE_HOST=/root/projectid:region:instanceid \
# Pass other necessary env variables like db username/password, etc.
$_IMAGE_URL:$COMMIT_SHA
timeout: '1200s'
waitFor: [build, install-proxy]
Because our db operations are taking place within the Docker container, I found the best way to provide the access to Cloud SQL by specifying the Unix socket -dir/workspace instead of exposing a TCP port 5432.
Note: I recommend using the directory /workspace instead of /cloudsql for Cloud Build.
Then we mounted the /workspace directory to Docker container's /root directory, which is the default directory where your application code resides. When I tried to mount it to other than /root, nothing seemed to happen (perhaps a permission issue with no error output).
Also: I noticed the proxy version 1.15 works well. I had issues with newer versions. Your mileage may vary.
I am using CodeDeploy to deploy my applications to EC2 instances created by an Auto Scaling Group.
The applications deploy fine and are moved to their correct file mapped locations, however my AfterInstallation script is never executed. I can see in the logs that it tries to make the script executable and passes that stage, but it never gets executed.
Here is my appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /opt/lobby
hooks:
AfterInstall:
- location: bootstrap.sh
timeout: 30
runas: root
Here is the script
#!/bin/bash
nohup nodejs lobby.js &
echo "finished"
I do not see the echo printed nor do I see any processes running related to lobby.js. I can verify that this script works by typing ./bootstrap.sh after the deployment and this works fine. This hook should do this for me though.
This needs to be a background task as I will be running multiple applications in this script but only one is displayed now just to get it working.
Edit:
I have referred to http://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#appspec-hooks-server and tried replacing AfterInstall with ValidateService and AfterAllowTraffic (but I'm not using a LoadBalancer)
Question
Why is my AfterInstallation script not getting called, or so it seems?
AfterInstall: AfterInstall script contains the tasks need to be executed after installing the Application.
Example of BeforeInstall script:
#!/bin/bash
cd /home/ubuntu/production/weone-backend
sudo chown -R ubuntu:ubuntu /home/ubuntu/production
sudo NODE_ENV=production nohup nodejs app.js > /dev/null 2> /dev/null < /dev/null &
In the above script, we are changing the ownership of our application folder
& starting application process.
Note: Use “/dev/null 2> /dev/null < /dev/null &” to get out of nohup shell automatically, else your CodeDeploy would get stuck at AfterInstall event.
https://www.oodlestechnologies.com/blogs/AWS-CodeDeploy