I am attempting to run the DatastoreWordCountExample from the Dataflow SDK.
The instructions indicate that I should run the following commands:
$ export CLOUDSDK_EXTRA_SCOPES=https://www.googleapis.com/auth/datastore
$ gcloud auth login
However, when I do, I receive the following error:
$ export CLOUDSDK_EXTRA_SCOPES=https://www.googleapis.com/auth/datastore
$ gcloud auth login
Traceback (most recent call last):
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/gcloud/gcloud.py", line 91, in <module>
from googlecloudsdk.calliope import base
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/calliope/base.py", line 8, in <module>
from googlecloudsdk.calliope import usage_text
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/calliope/usage_text.py", line 12, in <module>
from googlecloudsdk.calliope import arg_parsers
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/calliope/arg_parsers.py", line 42, in <module>
from googlecloudsdk.core import log
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/core/log.py", line 12, in <module>
from googlecloudsdk.core import properties
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/core/properties.py", line 10, in <module>
from googlecloudsdk.core import config
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/core/config.py", line 293, in <module>
_CheckForExtraScopes()
File "/Users/brian/opt/google-cloud-sdk/./lib/googlecloudsdk/core/config.py", line 291, in _CheckForExtraScopes
CLOUDSDK_SCOPES.extend(extra_scopes.split())
AttributeError: 'tuple' object has no attribute 'extend'
Note that gcloud runs find as long as CLOUDSDK_EXTRA_SCOPES is unset in my environment variables.
According to https://code.google.com/p/google-cloud-sdk/issues/detail?id=128 it seems that you no longer need to set this variable for datastore to work. Can you please try that and tell us if it doesn't work?
(meanwhile I'll update the docs)
Related
What I am trying to do: copy contents of a folder to a bucket. I have successfully done this in earlier versions of gsutil. Not sure if its the python version OR the gsutil version that caused this to break now. Any insights?
Command: gsutil -m cp -r . gs://target-bucket-url
Running: MacOS 10.15.5
Got the following error when trying the above command:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 243, in serve_client
request = recv()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 30, in <module>
from gslib.command import Command
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 50, in <module>
from gslib.cloud_api_delegator import CloudApiDelegator
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 26, in <module>
from gslib.cs_api_map import ApiMapConstants
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/cs_api_map.py", line 23, in <module>
from gslib.gcs_json_api import GcsJsonApi
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 72, in <module>
from gslib.third_party.storage_apitools import storage_v1_client as apitools_client
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 26, in <module>
class StorageV1(base_api.BaseApiClient):
File "/Users/user-x/tools/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 38, in StorageV1
_USER_AGENT += gslib.USER_AGENT
AttributeError: module 'gslib' has no attribute 'USER_AGENT'
gsutil version: 4.51
python version: 3.8
This is a known issue and the Google team is taking a look at it at this issue tracker, I recommend you to go there and add more information if possible: AttributeError: module 'gslib' has no attribute 'USER_AGENT'
Setting python version to python 3.7 as suggested at the issue tracker worked for me. Below is the working configuration.
$ cat ~/.bash_profile
alias python='python3.7'
.....
$ source ~/.bash_profile
I had followed the following instructions for setting up google-cloud-ml on LOCAL: MAC/LINUX
google-cloud-ml setup
But I am getting the following errors while verifying the setup with this command
curl https://raw.githubusercontent.com/GoogleCloudPlatform/cloudml-samples/master/tools/check_environment.py | python
Traceback (most recent call last):
File "", line 70, in
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/init.py", line 16, in
from google.cloud.ml.dataflow._analyzer import AnalyzeModel
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/init.py", line 22, in
from _ml_transforms import DeployVersion
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/_ml_transforms.py", line 24, in
import _ml_functions as ml_func
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/_ml_functions.py", line 25, in
from google.cloud.ml.io.coders import TrainingJobResult
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/io/init.py", line 21, in
from transforms import LoadFeatures
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/io/transforms.py", line 23, in
from google.cloud.ml.dataflow.io import tfrecordio
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/io/init.py", line 15, in
import tfrecordio
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/io/tfrecordio.py", line 16, in
_crc32c_fn = snappy._crc32c # pylint: disable=protected-access
AttributeError: 'module' object has no attribute '_crc32c'
Looks like there is some issue with dataflow library of google cloud.
I tried upgrading the dataflow library with the following command
pip install --upgrade google-api-python-client
but now, another error. Following is the stacktrace:
Traceback (most recent call last):
File "", line 70, in
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/init.py", line 16, in
from google.cloud.ml.dataflow._analyzer import AnalyzeModel
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/init.py", line 17, in
from _analyzer import AnalyzeModel
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/google/cloud/ml/dataflow/_analyzer.py", line 19, in
import apache_beam as beam
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/init.py", line 78, in
from apache_beam import io
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/io/init.py", line 21, in
from apache_beam.io.avroio import *
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/io/avroio.py", line 29, in
from apache_beam.io import filebasedsource
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/io/filebasedsource.py", line 32, in
from apache_beam.io import concat_source
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/io/concat_source.py", line 24, in
from apache_beam.io import iobase
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/io/iobase.py", line 853, in
from apache_beam.runners.dataflow.native_io.iobase import *
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/runners/init.py", line 23, in
from apache_beam.runners.dataflow_runner import DataflowRunner
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/runners/dataflow_runner.py", line 32, in
from apache_beam.internal import json_value
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apache_beam/internal/json_value.py", line 20, in
from apitools.base.py import extra_types
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apitools/base/py/init.py", line 23, in
from apitools.base.py.credentials_lib import *
File "/Users/pratyusha/miniconda2/envs/cloudml/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py", line 50, in
from oauth2client import locked_file
This looks like a different version of 'snappy' is expected. Can you check which version of python-snappy you have installed?
You can do this by running:
pip freeze | grep python-snappy
It worked for me with version 0.5, which is the latest version. If you have an earlier version, try running:
pip install --upgrade python-snappy
(using 'sudo' or '--user' as necessary.)
So I without thinking stupidly named a file io.py in my working directory. When I tried to compile I got a traceback error. Having realised what I'd done I renamed my file and updated references to it but I still get the following error:
Traceback (most recent call last):
File "C:\Users\Tom\workspace\Converter\get_file.py", line 9, in <module>
from scipy import complex_
File "C:\Python27\lib\site-packages\scipy\__init__.py", line 70, in <module>
from numpy import show_config as show_numpy_config
File "C:\Python27\lib\site-packages\numpy\__init__.py", line 185, in <module>
from . import add_newdocs
File "C:\Python27\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "C:\Python27\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
from .type_check import *
File "C:\Python27\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "C:\Python27\lib\site-packages\numpy\core\__init__.py", line 46, in <module>
from numpy.testing import Tester
File "C:\Python27\lib\site-packages\numpy\testing\__init__.py", line 13, in <module>
from .utils import *
File "C:\Python27\lib\site-packages\numpy\testing\utils.py", line 15, in <module>
from tempfile import mkdtemp
File "C:\Python27\lib\tempfile.py", line 32, in <module>
import io as _io
File "C:\Users\Tom\workspace\Converter\io.py", line 1, in <module>
"""The io module provides the Python interfaces to stream handling. The
File "C:\Users\Tom\workspace\Converter\get_file.py", line 9, in <module>
from scipy import complex_
ImportError: cannot import name complex_
I have restarted PyDev to no avail. I have refreshed the interpreter to no avail.
I have followed the instructions found in this Python issue but that command returns the correct result on my machine.
Is tempfile.py a temporary file I can remove to resolve this issue?
I have found the solution. Stupidly simple but maybe worth leaving here for other newbies who fall into the same trap.
PyDev compiles the code into *.pyc files. When you update the references and rename the files these aren't immediately recompiled. Deleting the io.pyc file I had allowed compilation too run properly.
I found some more info on common pitfalls too.
I'd like to put a flickr feed on my homepage using django-syncr; I followed the instructions on the homepage (http://code.google.com/p/django-syncr/), and the installation process was pretty smooth. Steps 1-3 work just fine, but Step 4 is where I run into problems.
When I try to run the commands shown:
from syncr.app.flickr import FlickrSyncr
I get the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django_syncr-0.50-py2.7.egg/sync/app/flickr.py", line 11, in <module>
from syncr.flickr.models import *
File "/usr/local/lib/python2.7/dist-packages/django_syncr-0.50-py2.7.egg/syncr/flickr/models.py", line 1, in <module>
from django.db import models
File "/usr/lib/python2.7/dist-packages/django/db/__init__.py", line 14, in <module>
if not settings.DATABASES:
File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 276, in __getattr__self._setup()
File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 40, in _setup
raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
And I'm not really sure what to do. These files are run in my django project folder, but including the settings file (via import settings) doesn't seem to fix this. What files should be included in order to import the right variables?
thanks!
When attempting to set this up, I am encountering the following error:
Traceback (most recent call last):
File "manage.py", line 11, in <module>
import settings
File "/Users/Paul/Documents/shopifywarrantymanager/settings.py", line 7, in <module>
from djangoappengine.settings_base import *
File "/Users/Paul/Documents/shopifywarrantymanager/djangoappengine/settings_base.py", line 6, in <module>
setup_env()
File "/Users/Paul/Documents/shopifywarrantymanager/djangoappengine/boot.py", line 64, in setup_env
setup_project()
File "/Users/Paul/Documents/shopifywarrantymanager/djangoappengine/boot.py", line 114, in setup_project
from .utils import have_appserver, on_production_server
File "/Users/Paul/Documents/shopifywarrantymanager/djangoappengine/utils.py", line 12, in <module>
appconfig, unused = dev_appserver.LoadAppConfig(PROJECT_DIR, {})
ValueError: too many values to unpack
I have not altered the files in any way other than adding my API key/shared secret, and changing the application name in 'app.yaml' to the one I registered with GAE.
I think you are using an old version of Djangoappengine.
The method LoadAppConfig since SDK 1.6 returns a tuple of three values (AppInfoExternal, URLMatcher, from_cache); the code in utils.py is wrong because is expecting just two values from it.
If you look into a more updated version of Djangoappengine you can see that the method is correctly called:
appconfig = dev_appserver.LoadAppConfig(PROJECT_DIR,
{},
default_partition='dev')[0]