I'm running a job on Google's Machine Learning Engine. I issued this job from the Google Cloud SDK Shell on Windows. At some point, I closed my laptop and lost connection to Google Cloud. The job kept running on Google's servers in the mean time. Now that I have reopened my laptop and got connected to the internet again, the shell has output:
ERROR: (gcloud.ml-engine.jobs.submit.training) There was a problem refreshing your current auth tokens: Unable to find the server at www.googleapis.com
Please run:
$gcloud auth login
to obtain new credentials.
So I ran that command. The browser opened, I clicked on my Google account and authenticated. Then I saw:
You are now logged in as [my Google e-mail address].
Your current project is [None]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
I did that also, and then saw the output:
Updated property [core/project].
So everything seems to work. Online, in the Google Cloud Console, I can view the logs of my job while it is running. However, my question is, is it possible to get those logs/stdout to be printed in my shell again?
I guess you are looking for something like what is explained in this documentation page about ML Engine logging.
You can either use the logging service specifying your preferred filter, with gcloud beta logging read or, in order to print ML Engine job logs in the console, you can use this ML Engine-specific command, with the options and flags you need, in order to print the logs of your job:
gcloud ml-engine jobs stream-logs
You can find the reference for that command in this other page.
Related
I'm trying to set up a private Redash instance with Google Cloud. Step 1 is to add the the Redash image to your account so you can boot a VM with it.
When adding the image through Google Cloud Shell, my shell times out before the process completes.
When adding the image through the Console UI, it loads and loads then disappears without a trace.
When adding an image through gcloud CLI, I finally get a response:
➜ gcloud compute images create "redash" --source-uri gs://redash-images/redash.8.0.0-b32245-1.tar.gz
ERROR: (gcloud.compute.images.create) Could not fetch resource:
- Internal error. Please try again or contact Google Support. (Code: '-527xxxxxxxxxx759')
(x = hidden number)
I have extremely slow internet, so I'm thinking this could potentially be the issue. I've contacted Google Support but no response.
I reproduced and executed the command gcloud compute images create "redash" --source-uri gs://redash-images/redash.8.0.0-b32245-1.tar.gz. Even for me also it seems it is taking more time to execute, then I have killed it using CRTL + C but when I checked the Compute Engine > Images , a redash image is created with the same timestamp I had executed the command. With this experiment I assume that even though command is interrupted, the image creation may run in background. I suggest you to check the images section in Compute Engine once.
This is an issue of Redash's GCP account payment status.
If this issue reproduced, I recommend to tell redash admin to check their GCP account payment status.
The following URL talks about this issue with the Redash community.
https://discuss.redash.io/t/cant-pull-redash-image-on-google-cloud/9486
Help definition is not clear for me:
Token used to route traces of service requests for investigation of
issues.
Could you provide simple example how to use it?
I tried:
gcloud compute instances create vm3 --trace-token xyz123
I can find "vm3" string in logs, but not my token xyz123.
The only use of it seems to be in grep:
history| grep xyz123
The flag --trace-token is intended to be used by the support agents when there is some error which is difficult to track from the logs. The Google Cloud Platform Support agent provides a time bound token which will expire after a specified time and asks the user to run the command for the specific product in which the user is facing the issue. Then it gets easier for the support agent to trace the error by using that --trace-token.
For example :
A user faced some error while creating a Compute Engine instance and contacted the Google Cloud Platform Support team. The support agent then inspected the logs and other resources but could not find the root cause of the issue. Then the support agent provides a --trace-token and asks the user to run the below command with the provided --trace-token.
--trace-token = abcdefgh
Command : gcloud compute instances create my-vm --trace-token abcdefgh
After the user runs the above command the support agent could find the error by analysing in depth with the help of the --trace-token
Please note that when a --trace-token flag is used the content of the trace may include sensitive information like auth tokens, the contents of any accessed files. Hence they should only be used for manual testing and should not be used in production environments.
I've had a Cloud SQL instance for about a year now.
I always accessed it the same way:
I would go to my project on the Cloud Console.
Click on the Cloud Shell icon at the top right (a small right pointing arrow).
A black shell screen would pop up where I would type
gcloud sql connect <my instance> --user=root.
Enter my password.
Now, all of a sudden, I am getting an error message saying:
There was no instance found at projects//instances/ or you are not authorized to connect to it.
I am the owner of the project, and also have Admin rights to the Cloud SQL instance. The project and instance are still there, and my app that accesses the data stored in the instances' database is working fine - therefore I know the database is also present, otherwise my app wouldn't work.
I didn't touch or change anything in the Cloud SQL instance. Suddenly, I simply can't access my database using the exact same procedure I have been using almost every day over the past year now.
I am able to access the database using a local Python script on my laptop and the Cloud SQL Proxy, but I would like to access it from the Cloud Shell again.
Any ideas on what could the problem be?
gcloud components update - update all of your installed components to the latest version
gcloud init - reinitialize gcloud shell. It performs the following setup steps:
Authorizes gcloud and other SDK tools to access Google Cloud Platform using your user account credentials, or from an account of your choosing whose credentials are already available.
It seems like there was a problem with the GCP Cloud Shell (even though there was no mention of it on the GCP error tracking page). When I logged back in today and followed the same above process everything worked well.
Looks like GCP Cloud Shell could occasionally go rouge and start producing errors. Word of advice, don't panic when this happens (like I did) and start resetting, rebooting and messing up things. Just wait a day and check back again.
I use the method mentioned in this excellent answer https://stackoverflow.com/a/49515502/10690958 to connect to Google Cloud Shell via ssh on my ubuntu workstation. Occasionally, I need to enable "boost-mode". In that case, I currently have to open the Cloud Shell via firefox (https://console.cloud.google.com/cloudshell/editor?shellonly=true), then login and enable boost mode. After that I can close firefox, and use the gcloud method to access the cloud shell VM in boost mode.
I would like to do this (access boost-mode) purely through the gcloud cli, since using the browser is quite cumbersome.
The official docs dont mention any method of enabling boost mode via gcloud There seem to be only three options i.e. ssh/scp/sshfs via gcloud alpha cloud-shell. Is there perhaps a way to enable this via some configuration option?
thanks
There does not seem to be any option to enable the boost mode from either the v1 or v1alpha1 versions of the Cloud Shell API (both versions undocumented).
The gcloud command actually uses the API to get the status of your Cloud Shell environment, which contains information about how to connect through SSH, updates the SSH keys if needed, and then connects using that info (use gcloud alpha cloud-shell ssh --log-http if you want to check it by yourself).
As far as I can see, when you click the "Boost mode" button, the browser makes a call to https://ssh.cloud.google.com/devshell?boost=true&forceNewVm=true (and some more parameters), but I can't make it work on the command line, so I'm guessing it's doing some other stuff that I can't identify.
If you need this for your workflow, you could raise a feature request on Google's issue tracker.
It is now possible to access the Cloud Shell in boost mode from the CLI with this command: gcloud alpha cloud-shell ssh --boosted. Other possible arguments are documented here. Just a warning: the first time I tried that my home directory became unreadable and started returning "Input/output error", logging out and in again fixed the issue.
I am really new to Google Cloud Shell, and I accidentally closed the tab for the shell...and I cannot find it now.
I know I need to click the highlighted button at the top of the console window to activate Google Cloud Shell, but it is now grey, and no shell is presented on the page (it should be at the bottom).
Can anyone help?
I observed the same issue.
I refreshed the console page ( CTRL + F5 ) and I was able to see the 'Activate Cloud Shell' button again !!
However, if above does not resolves the issue, some tips:
From google docs Google Cloud Shell Limitations
Weekly usage: Cloud Shell also has weekly usage limits. If you reach
your usage limit, you'll need to wait until the specified time (listed
under Usage Quota, found under the three dots menu icon) before you
can use Cloud Shell again.
Also, there are usage limits on this, check your email if you had violated any conditions on shell usage:
Warning: Violating the Terms of Service will result in Cloud Shell
being disabled for your account. This constitutes activity that
adversely impacts Google Cloud Platform services, other customers' or
their end users' use of services, or the Google network used to
provide these services. Coin mining and network scanning using Cloud
Shell are strictly prohibited.
I had the same issue here.
You can install the command line interface in your terminal to access Google Cloud Datalab.
Here you will find the quickstart to configure the environment.
Installing datalab component:
gcloud components install datalab
Connecting with your VM Instance:
datalab connect *instance-name*
Opening the initial page:
http://localhost:8081
See more:
Google Cloud Datalab - Quickstart
It works now. It seems that there was a service problem earlier today, so the page was not well functioning.