Trying to build the image by using gcloud build submit command with passing the source as GCS bucket as per the syntax but it's not working.
gcloud builds submit gs://bucket/object.zip --tag=gcr.io/my-project/image
Error : -bash: gs://bucket_name/build_files.zip: No such file or directory
This path exists in the GCP project where I'm executing the command but still it says no such file or directory.
What I'm missing here ?
Cloud Build looks for local file or tar.gz file on Google Cloud Storage.
Is the case of a zip file like your case, the solution is to start to download locally the file, UNZIP THE FILE and then launch your Cloud Build.
Indeed, you need to unzip the file. Cloud Build won't do it for you, it can only ungzip and untar files. When you add --tag parameter, Cloud Build looks for a Dockerfile file if your set of file and run a docker build with this file.
Please try with single quotes(') or double quotes(") around gs://bucket/object.zip, and not the back quote (`), so the command would look like this:
gcloud builds submit 'gs://bucket/object.zip' --tag=gcr.io/my-project/image
Looks like there is an issue with the documentation, the changes have now been submitted to Google.
Related
I have a working script that finds the data file when it is in the same directory as the script. This works both on my local machine and Google Colab.
When I try it on GCP though it can not find the file. I tried 3 approaches:
PySpark Notebook:
Upload the .ipynb file which includes a wget command. This downloads the file without error but I am unsure where it saves it to and the script can not find the file either (I assume because I am telling it that the file is in the same directory and pressumably using wget on GCP saves it somewhere else by default.)
PySpark with bucket:
I did the same as the PySpark notebook above but first I uploaded the dataset to the bucket and then used the two links provided in the file details when you click the file name inside the bucket on the console (neither worked). I would like to avoid this though as wget is much faster then downloading on my slow wifi then reuploading to the bucket through the console.
GCP SSH:
Create cluster
Access VM through SSH.
Upload .py file using the cog icon
wget the dataset and move both into the same folder
Run script using python gcp.py
Just gives me an error saying file not found.
Thanks.
As per your first and third approach, if you are running a PySpark code on Dataproc, irrespective of whether you use .ipynb file or .py file, please note the below points:
If you use the ‘wget’ command to download the file, then it will be downloaded in the current working directory where your code is executed.
When you try to access the file through the PySpark code, it will check defaultly in HDFS. If you want to access the downloaded file from the current working directory, use the “ file:///” URI with absolute file path.
If you want to access the file from HDFS, then you have to move the downloaded file to HDFS and then access from there using an absolute HDFS file path. Please refer the below example:
hadoop fs -put <local file_name> </HDFS/path/to/directory>
I am trying to do host this static website on Google App Engine and I am stuck on this crucial part of the process:
-bash: gcloud: command not found
I get into Google Cloud Platform, then login into the SSH, look for the files, then when I try to deploy, nothing happens. The two main files in this equation include: app.yaml and www (www containing the html and files). I am grabbing a file with a html, then making it the index.html. The index.html is what you see when you open the website after compiling the file(s) (with the command "gcloud app deploy"). After a couple other steps, it becomes available to view on the static website.
I have been trying to find a solution for a few hours now.
Here is what the code looks right now when trying to deploy:
vergil11$ cd Files
vergil11$ ls
websitegc
vergil11$ cd websitegc
vergil11$ ls
app.yaml IMD233 Files README.md www
vergil11$ gcloud app deploy
-bash: gcloud: command not found
vergil11$
Any help provided, thanks
You need to add gloud on your %PATH% (Windows) or $PATH (Linux/Mac)
See here for Mac
Or "How To Install Google Cloud GCP Command Line Utility gcloud ?" for Windows.
Here for Linux, modifying your ~/.profile
I have a Go Dockerfile from https://cloud.google.com/run/docs/quickstarts/build-and-deploy with a one line change so that I can tell what version I'm running:
RUN go build -ldflags "-X main.Version=$(git describe --always)" -mod=readonly -v -o server
When I build locally via docker build . and test, there is no problem with git describe, however if I submit the Docker to be built via gcloud builds submit it fails with:
fatal: not a git repository (or any of the parent directories): .git
How do I build my Cloud Run docker image so it has this Git version reference?
When you perform gcloud builds submit, all the project files aren't sent to Cloud Build. The command take into account your .gitignore file and the .gcloudignore file. If you haven't a .gcloudignore a default behavior is enforced in addition of the .gitignore file directive. More detail here
So, to fix this, create a .gcloudignore file with only the file to exclude for your Build. So, let the .git/ (don't add it in the file) and it will work.
I'm trying to run my code of machine learning from images using tensorflow in Google CloudML. However, it seems the submitted job can't access to my files in my cloud shell or in GCS. Even though it is working fine in my local machine, I get the following error once I submit my job using the command gcloud from the cloud shell:
ERROR 2017-12-19 13:52:28 +0100 service IOError: [Errno 2] No such file or directory: '/home/user/pores-project-googleML/trainer/train.txt'
This folder can be found for sure in cloud shell, and I can check it when I type:
ls /home/user/pores-project-googleML/trainer/train.txt
I tried putting my file train.txt in GCS and access to it from my code (by specifying the path gs://my_bucket/my_path), but once the job submitted, I got a 'No such file or directory' error with the corresponding path.
To check where the job I submitted using gcloud is running, I added print(os.getcwd()) in the beginning of my python code trainer/task.py, which printed as a result in the logs: /user_dir. I couldn't find this path using the cloud shell, not even in GCS. So my question is how can I know in which machine my job is running? If it's in a certain container somewhere, how can I access from it to my files using the cloud shell and in GCS?
Before I do all of this, I succesfully completed the 'Image Classification using Flowers Dataset' tutorial.
The command I used to submit my job is:
gcloud ml-engine jobs submit training $JOB_NAME --job-dir $JOB_DIR --packages trainer-0.1.tar.gz --module-name $MAIN_TRAINER_MODULE --region us-central1
where:
TRAINER_PACKAGE_PATH=/home/use/pores-project-googleML/trainer
MAIN_TRAINER_MODULE="trainer.task"
JOB_DIR="gs://pores/AlexNet_CloudML/job_dir/"
JOB_NAME="census$(date +"%Y%m%d_%H%M%S")"
Regular Python IO library is not able to access files on GCS. Instead, you need to use GCS python client or gstuil cli to access GCS files.
Note that TensorFlow itself has native support of GCS (i.e., it can read GCS files directly).
I want to deploy war from Jenkins to Cloud.
Could you please let me know how to deploy war file from Jenkins on my local to AWS Bean Stalk ?
I tried using a Jenkins post-process plugin to copy the artifact to S3, but I get the following error:
ERROR: Failed to upload files java.io.IOException: put Destination [bucketName=https:, objectName=/s3-eu-west-1.amazonaws.com/bucketname/test.war]:
com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connect to s3.amazonaws.com/s3.amazonaws.com/ timed out at hudson.plugins.s3.S3Profile.upload(S3Profile.java:85) at hudson.plugins.s3.S3BucketPublisher.perform(S3BucketPublisher.java:143)
Some work has been done on this.
http://purelyinstinctual.com/2013/03/18/automated-deployment-to-amazon-elastic-beanstalk-using-jenkins-on-ec2-part-2-guide/
Basically, this is just adding a post-build task to run the standard command line deployment scripts.
From the referenced page, assuming you have the post-build task plugin on Jenkins and the AWS command line tools installed:
STEP 1
In a Jenkins job configuration screen, add a “Post-build action” and choose the plugin “Publish artifacts to S3 bucket”, specify the Source (in our case, we use Maven so the source is target/.war and destination is your S3 bucket name)
STEP 2
Then, add a “Post-build task” (if you don’t have it, this is a plugin in Maven repo) to the same section above (“Post-build Actions”) and drag it below the “Publish artifacts to S3 bucket”. This is important that we want to make sure the war file is uploaded to S3 before proceeding with the scripts.
In the Post-build task portion, make sure you check the box “Run script only if all previous steps were successful”
In the script text area, put in the path of the script to automate the deployment (described in step 3 below). For us, we put something like this:
<path_to_script_file>/deploy.sh "$VERSION_NUMBER" "$VERSION_DESCRIPTION"
The $VERSION_NUMBER and $VERSION_DESCRIPTION are Jenkins’ build parameters and must be specified when a deployment is triggered. Both variables will be used for AEB deployment
STEP 3
The script
#!/bin/sh
export AWS_CREDENTIAL_FILE=<path_to_your aws.key file>
export PATH=$PATH:<path to bin file inside the "api" folder inside the AEB Command line tool (A)>
export PATH=$PATH:<path to root folder of s3cmd (B)>
//get the current time and append to the name of .war file that's being deployed.
//This will create a unique identifier for each .war file and allow us to rollback easily.
current_time=$(date +"%Y%m%d%H%M%S")
original_file="app.war"
new_file="app_$current_time.war"
//Rename the deployed war file with the new name.
s3cmd mv "s3://<your S3 bucket>/$original_file" "s3://<your S3 bucket>/$new_file"
//Create application version in AEB and link it with the renamed WAR file
elastic-beanstalk-create-application-version -a "Hoiio App" -l "$1" -d "$2" -s "<your S3 bucket>/$new_file"