I'm running a VM instance (google compute engine) with CENTOS 7, everytime i run the bq command, i keep getting error. I supposed bq is by default in compute engine.
[username#instance-1 ~]$ bq
Error initializing bq client: service_account
Traceback (most recent call last):
File "/usr/lib64/google-cloud-sdk/platform/bq/third_party/pyglib/appcommands.py", line 805, in _CommandsStart
sys.modules['__main__'].main(GetCommandArgv())
File "/usr/lib64/google-cloud-sdk/platform/bq/bq.py", line 6078, in main
if FLAGS.debug_mode or FLAGS.headless:
File "/usr/lib64/google-cloud-sdk/platform/bq/third_party/absl/flags/_flagvalues.py", line 468, in __getattr__
raise AttributeError(name)
AttributeError: debug_mode
FATAL error in main: debug_mode
Run 'bq.py help' to get help
[username#instance-1 ~]$ bq --format=prettyjson dataset.tableid
FATAL Flags parsing error: Unknown command line flag 'use_gce_service_account'
Run 'bq.py help' to get help
[username#instance-1 ~]$
my compute engine account has full full access to all Cloud APIs, in addition just to be sure, i also add bigquery admin in the IAM part.
I'm not really sure what is wrong.
Have a look at the documentation:
The bq authorization flags are deprecated. To configure authorization
for the bq command-line tool, see Authorizing Cloud SDK tools.
and also in this section you can find flag that cause an error:
--use_gce_service_account
I've tried to run at my linux machine with Google Cloud SDK command bq and it works perfectly:
$ bq
Python script for interacting with BigQuery.
USAGE: bq.py [--global_flags] <command> [--command_flags] [args]
Any of the following commands:
cancel, cp, extract, get-iam-policy, head, help, init, insert, load, ls, mk, mkdef, partition, query, rm, set-iam-policy, shell,
show, update, version, wait
At first, you should update your Cloud SDK to the latest version (or reinstall it) and check again. Also, please update your post with version of your Cloud SDK:
$ gcloud info
Google Cloud SDK [277.0.0]
Python Version: [3.7.5rc1 (default, Dec 20 2019, 17:52:56) [GCC 8.3.0]]
Python Location: [/usr/bin/python3]
If you have en error with the latest Cloud SDK, try set up default values for bq in $HOME/.bigqueryrc as it described in the documentation.
In addition, I'd recommend you to check/update Python if nothing helps.
EDIT You can set up defaults in your ~/.bigqueryrc file like this:
project_id = --my-project-id--
in your favorite text editor like nano or vim. Keep in mind that file paths in .bigqueryrc had to be the full path.
EDIT2 Have a look at the Cloud SDK system requirements:
It requires Python 2.7.9 or higher.
and you have 2.7.5.
UPDATE Deleting and reinstalling the Cloud SDK solved the problem.
You might have a bash alias acting wrong or a ~/.bigqueryrc
Check the syntax and the correct service account files inside ~/.bigqueryrc
Related
I am trying to install google-cloud-sdk in ubuntu-18.04. I am following offical docs given here. When I run ./google-cloud-sdk/install.sh I get following error:-
Welcome to the Google Cloud SDK!
To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. This data is
handled in accordance with our privacy policy
<https://policies.google.com/privacy>. You may choose to opt in this
collection now (by choosing 'Y' at the below prompt), or at any time in the
future by running the following command:
gcloud config set disable_usage_reporting false
Do you want to help improve the Google Cloud SDK (y/N)? N
Traceback (most recent call last):
File "/home/vineet/./google-cloud-sdk/bin/bootstrapping/install.py", line 225, in <module>
main()
File "/home/vineet/./google-cloud-sdk/bin/bootstrapping/install.py", line 200, in main
Prompts(pargs.usage_reporting)
File "/home/vineet/./google-cloud-sdk/bin/bootstrapping/install.py", line 123, in Prompts
scope=properties.Scope.INSTALLATION)
File "/home/vineet/google-cloud-sdk/lib/googlecloudsdk/core/properties.py", line 2406, in PersistProperty
config.EnsureSDKWriteAccess()
File "/home/vineet/google-cloud-sdk/lib/googlecloudsdk/core/config.py", line 198, in EnsureSDKWriteAccess
raise exceptions.RequiresAdminRightsError(sdk_root)
googlecloudsdk.core.exceptions.RequiresAdminRightsError: You cannot perform this action because you do not have permission to modify the Google Cloud SDK installation directory [/home/vineet/google-cloud-sdk].
Re-run the command with sudo: sudo /home/vineet/google-cloud-sdk/bin/gcloud ...
I tried to search it on stackoverflow and github-issues but in vain.
Would appreciate any hint to solve it.
As stated on the error message.
Re-run the command with sudo: sudo /home/vineet/google-cloud-sdk/bin/gcloud ...
The install.sh script should be run using sudo.
There are also other alternatives to install the Google Cloud SDK in Ubuntu 18.04 just as installing the package with apt-get as explained on the documentation.
I am trying to do something simple. At least I thought it should be simple.
We are using an Azure DevOps hosted build with a python based toolchain. The source code was checked out with git on this build machine and is working fine.
Inside this toolchain I would like to retrieve some information from git. I tried to use
options = ['show', '-s', '--format=%ae', 'HEAD']
cmd = "git" + options
subprocess.check_output(cmd)
(I hope I did not do a syntax error in this example, I simplified some parts)
The message I get on the build server is:
ERROR git not found
Traceback (most recent call last):
...
in __git return subprocess.check_output(cmd) File d:\a\1\s\tools\python27\lib\subprocess.py", line 566,
...
WindowsError: [Error 2] The system cannot find the file specified
I also tried to use an absolute path with the typical windows pathes but was not able to locate git on the hosted build machine.
I also had a look into https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/git-commands?view=azure-devops&tabs=yaml
This looks to me like it should be possible to just use git. Hmmm. Does anyone have ideas for me?
Check if your Agent has Git capability enabled.
Go to manage agent pool page
Select your agent pool
Select the agent
Go to capabilities tab
Check if git is added.
If not added just click on Add button and set /usr/git capability
Getting this error when trying to sudo apt-get update. E: Malformed entry 7 in list file /etc/apt/sources.list.d/google-cloud-sdk.list (Suite)
E: The list of sources could not be read.
I tried to run sed to remove and no luck.
Please help.
Okay, after following the first 5 steps in the link: cloud.google.com/sdk/docs/quickstart-debian-ubuntu I received the following output:
Your Google Cloud SDK is configured and ready to use!
Commands that require authentication will use cloud#postaprayer.org by default
Commands will reference project post-a-prayer by default
Compute Engine commands will use region us-west2 by default
Compute Engine commands will use zone us-west2-a by default
Run gcloud help config to learn how to change individual settings
This gcloud configuration is called [postaprayerdns]. You can create additional configurations if you work with multiple accounts and/or projects.
Run gcloud topic configurations to learn more.
Some things to try next:
Run gcloud --help to see the Cloud Platform services you can interact with. And run gcloud help COMMAND to get help on any gcloud command.
Run gcloud topic --help to learn about advanced features of the SDK like arg files and output formatting
Okay. I was able to get into the google-cloud-sdk.list file and edit it using sudo nano /etc/apt/sources.list.d/google-cloud-sdk.list
From there I edited the .list file and deleted line 7 (which stated clear)
I edited these instructions to solve this error: https://askubuntu.com/questions/332669/unable-to-edit-etc-apt-sources-list-file
sudo nano /etc/apt/sources.list.d/google-cloud-sdk.list
`
Solved. Used Nano command to edit *.list file, deleted corrupt entry 7, and then saved.
Summary:
sudo nano /etc/apt/sources.list.d/google-cloud-sdk.list
From there I edited the .list file and deleted line 7 (which stated clear)
I edited these instructions to solve this error: https://askubuntu.com/questions/332669/unable-to-edit-etc-apt-sources-list-file
sudo nano /etc/apt/sources.list.d/google-cloud-sdk.list
I'm trying to run my code of machine learning from images using tensorflow in Google CloudML. However, it seems the submitted job can't access to my files in my cloud shell or in GCS. Even though it is working fine in my local machine, I get the following error once I submit my job using the command gcloud from the cloud shell:
ERROR 2017-12-19 13:52:28 +0100 service IOError: [Errno 2] No such file or directory: '/home/user/pores-project-googleML/trainer/train.txt'
This folder can be found for sure in cloud shell, and I can check it when I type:
ls /home/user/pores-project-googleML/trainer/train.txt
I tried putting my file train.txt in GCS and access to it from my code (by specifying the path gs://my_bucket/my_path), but once the job submitted, I got a 'No such file or directory' error with the corresponding path.
To check where the job I submitted using gcloud is running, I added print(os.getcwd()) in the beginning of my python code trainer/task.py, which printed as a result in the logs: /user_dir. I couldn't find this path using the cloud shell, not even in GCS. So my question is how can I know in which machine my job is running? If it's in a certain container somewhere, how can I access from it to my files using the cloud shell and in GCS?
Before I do all of this, I succesfully completed the 'Image Classification using Flowers Dataset' tutorial.
The command I used to submit my job is:
gcloud ml-engine jobs submit training $JOB_NAME --job-dir $JOB_DIR --packages trainer-0.1.tar.gz --module-name $MAIN_TRAINER_MODULE --region us-central1
where:
TRAINER_PACKAGE_PATH=/home/use/pores-project-googleML/trainer
MAIN_TRAINER_MODULE="trainer.task"
JOB_DIR="gs://pores/AlexNet_CloudML/job_dir/"
JOB_NAME="census$(date +"%Y%m%d_%H%M%S")"
Regular Python IO library is not able to access files on GCS. Instead, you need to use GCS python client or gstuil cli to access GCS files.
Note that TensorFlow itself has native support of GCS (i.e., it can read GCS files directly).
First I installed stand-alone gsutil on Fedora 25, it ran nice for months.
Then I installed Cloud SDK, and my Google Cloud credentials have been broken ever since.
I don't need Cloud SDK after all. I just want to use gsutil again.
Is there a way to uninstall Cloud SDK and credentials from Linux?
Or maybe uninstall all Google Cloud products and reinstall the stand-alone gsutil?
To explain the likely reason this is happening:
When you install the Cloud SDK, it takes some steps to make sure that when you type gsutil from the shell, it resolves to the Cloud SDK version (depending on the installation method, it might make some executable scripts in /usr/local/bin/, or put /path/to/cloud/sdk/bin at the front of your PATH environment variable). This Cloud SDK wrapper script for gsutil does some extra auth logic, loading an extra .boto file which contains credentials produced from running gcloud auth login. You can see this extra .boto file when running gcloud version -l:
$ gsutil version -l
[...]
using cloud sdk: True
config path(s): /home/USER/.boto, /home/USER/.config/gcloud/legacy_credentials/USER#gmail.com/.boto
[...]
It's likely that the auth credentials in that extra .boto file are overriding the credentials in your $HOME/.boto file.
How to use standalone gsutil again:
You'll need to ensure that the first gsutil your shell finds is the standalone version. This essentially means that the directory containing the standalone gsutil executable should come before the cloud sdk directory in your PATH environment variable. This can be done via prepending it to your PATH variable, via adding something like this to the end of your .bashrc file:
if [ -d "/path/to/standalone/gsutil/directory" ]; then
PATH="/path/to/standalone/gsutil/directory:$PATH"
fi
After doing this, you can run this command to reload your .bashrc file and check the "using cloud sdk" value of your gsutil info:
$ source "$HOME/.bashrc"; gsutil version -l
If this still shows that you're using the Cloud SDK version of gsutil, you might have an alias defined for gsutil - you can check for this by running:
$ type gsutil
If you still encounter auth issues when using the standalone version of gsutil, you'll need to generate new credentials:
$ gsutil config