Training job fails with gcloud ml-engine training job.
Solutions tried:
Updated setuptools, pip and wheel to latest version on Google
cloud sdk shell
Removed and reinstalled setuptools, pip and wheel on local machine
Updated setuptools, pip and wheel to latest version on Google
cloud terminal
reinstalled cloud sdk tools on local machine
Trackback (most recent call last):
[...]
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py",line 666, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py",line 856, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py",
line 1078, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py",
line 1063, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 68, in r
un_setup
DirectorySandbox(setup_dir).run(runner)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 114, in
run
return func()
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 67, in r
unner
_execfile(setup_script, ns)
File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 43, in _
execfile
exec(code, globals, locals)
File "/tmp/easy_install-Pe5fZg/pytest-runner-4.2/setup.py", line 76, in <mod
ule> File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 266, in __init__
_Distribution.__init__(self,attrs)
File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
self.finalize_options() File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 301, in fin
alize_options
ep.load()(self, ep.name, value)
File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line 10,
in version_keyword
File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _
warn_if_setuptools_outdated
SetuptoolsOutdatedWarning: your setuptools is too old (<12)
Setup.py
from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['keras','hickle','scipy','numpy']
setup(
name='trainer',
version='1.0',
install_requires=REQUIRED_PACKAGES,
packages=find_packages(),
include_package_data=True,
description='My training application package.'
)
Config.yaml
trainingInput:
scaleTier: BASIC_GPU
Related
Hi am trying to run a scraper on ubuntu/windows machine .
I have installed scrapy version- Scrapy 1.8.0 on using python2.
I am able create a project, but when I run a scraper this error in shown.
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/scrapy", line 8, in <module>
sys.exit(execute())
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 146, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 100, in _run_print_help
func(*a, **kw)
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/cmdline.py", line 154, in _run_command
cmd.run(args, opts)
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/commands/shell.py", line 68, in run
crawler.engine = crawler._create_engine()
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/crawler.py", line 111, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/core/engine.py", line 67, in __init__
self.scheduler_cls = load_object(self.settings['SCHEDULER'])
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 46, in load_object
mod = import_module(module)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/ubuntu/.local/lib/python2.7/site-packages/scrapy/core/scheduler.py", line 7, in <module>
from queuelib import PriorityQueue
File "/home/ubuntu/.local/lib/python2.7/site-packages/queuelib/__init__.py", line 1, in <module>
from queuelib.queue import FifoDiskQueue, LifoDiskQueue
File "/home/ubuntu/.local/lib/python2.7/site-packages/queuelib/queue.py", line 7, in <module>
from contextlib import suppress
ImportError: cannot import name suppress
can anyone help to solve this issue?
The contextlib.suppress function supports from Python 3.4 but you are using Python 2.7. To solve this problem, upgrade your python version.
If this is useful for someone else, the solution for me (also using python 2.7) was:
pip uninstall attrs
pip uninstall queuelib
pip install queuelib==1.5.0
pip install attrs
Python: 2.7.12
Pip: 8.1.1
When I sudo pip install pandas, it shows as follows:
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 317, in run
requirement_set.prepare_files(finder)
File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 512, in _prepare_file
finder, self.upgrade, require_hashes)
File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 273, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 442, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 400, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 545, in _get_pages
page = self._get_page(location)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 648, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/usr/lib/python2.7/dist-packages/pip/index.py", line 757, in get_page
"Cache-Control": "max-age=600",
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/download.py", line 378, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 610, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment
total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
download .whl file for pandas
try
pip install pandas.whl
or upgrade or pip and repeat the command
pip install pandas
I think there is some problem with pip command. So you can install Anaconda for python 2.7 (Download). It comes with pip by default.
Once download will be completed (it is script file).
Give the permission to the script file.
sudo chmod 777 Anaconda2-5.1.0-Linux-x86_64.sh
Run the script file on terminal../Anaconda2-5.1.0-Linux-x86_64.sh or bash Anaconda2-5.1.0-Linux-x86_64.sh
Follow the instruction from point 4th to this document here
1.5.0 installs fine, but when I import tensorflow, I get this error:
RuntimeError: module compiled against API version 0xa but this version
of numpy is 0x9 RuntimeError: module compiled against API version 0xa
but this version of numpy is 0x9 Traceback (most recent call last):
File "<stdin>", line 1, in <module> File
"/Library/Python/2.7/site-packages/tensorflow/__init__.py", line 24,
in <module>
from tensorflow.python import * File "/Library/Python/2.7/site-packages/tensorflow/python/__init__.py",
line 63, in <module>
from tensorflow.python.framework.framework_lib import * File "/Library/Python/2.7/site-packages/tensorflow/python/framework/framework_lib.py",
line 81, in <module>
from tensorflow.python.framework.sparse_tensor import SparseTensor File
"/Library/Python/2.7/site-packages/tensorflow/python/framework/sparse_tensor.py",
line 25, in <module>
from tensorflow.python.framework import tensor_util File "/Library/Python/2.7/site-packages/tensorflow/python/framework/tensor_util.py",
line 34, in <module>
from tensorflow.python.framework import fast_tensor_util File "__init__.pxd", line 163, in init
tensorflow.python.framework.fast_tensor_util ValueError: numpy.dtype
has the wrong size, try recompiling. Expected 88, got 96
1.6.0 fails to install with this error:
DEPENDENCY ERROR
The target you are trying to run requires an OpenSSL implementation.
Your system doesn't have one, and either the third_party directory
doesn't have it, or your compiler can't build BoringSSL.
Please consult INSTALL to get more information.
If you need information about why these tests failed, run:
make run_dep_checks
make: Circular /private/tmp/pip-build-Lth8PD/grpcio/libs/opt/libares.a <- /private/tmp/pip-build-Lth8PD/grpcio/libs/opt/libz.a dependency dropped.
make: *** [stop] Error 1
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/tmp/pip-build-Lth8PD/grpcio/setup.py", line 311, in <module>
cmdclass=COMMAND_CLASS,
File "/Library/Python/2.7/site-packages/setuptools/__init__.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Library/Python/2.7/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py", line 573, in run
self.run_command('build')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build.py", line 127, in run
self.run_command(cmd_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Library/Python/2.7/site-packages/setuptools/command/build_ext.py", line 78, in run
_build_ext.run(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 337, in run
self.build_extensions()
File "/private/tmp/pip-build-Lth8PD/grpcio/src/python/grpcio/commands.py", line 278, in build_extensions
raise Exception("make command failed!")
Exception: make command failed!
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools,
tokenize;__file__='/private/tmp/pip-build-Lth8PD/grpcio/setup.py';f=getattr(tokenize, 'open', open)
(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code,
__file__, 'exec'))" install --record /tmp/pip-eSD2il-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-Lth8PD/grpcio/
I ran into the same issue. The installation error is because the new version of tensorflow requires new dependencies (grpcio). Here is how I handle my problem.
Force installing the binary wheels.
$ pip install --no-cache-dir --only-binary :all: grpcio==1.10.1
Then I can upgrade my tensorflow.
$ pip install --upgrade tensorflow # for Python 2.7
$ pip3 install --upgrade tensorflow # for Python 3.n
Hope it helps.
I am trying to install hive on hadoop 2.7.2 with hdp 2.5. My ambari version is 2.4. Every other component is working fine (HDFS,YARN, MR2, ZOOKEEPER etc.).
When i am trying to install hive with new MySql database through ambari UI it give me following error for HCat client installation:
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py", line 85, in <module>
HCatClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py", line 35, in install
self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 567, in install_packages
retry_count=agent_stack_retry_count) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 54, in action_install
self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 83, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 91, in _call_with_retries
code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner
result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call
tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper
result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call
raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install hive_2_5_0_0_1245' returned 1. Error unpacking rpm package hive_2_5_0_0_1245-1.2.1000.2.5.0.0-1245.el6.noarch error: unpacking of archive failed on file /usr/hdp/2.5.0.0-1245/hive/conf: cpio: rename failed - Is a directory
I am trying to install using pip2.7 install cassandra-driver and it fails with the long stack trace. The error is RuntimeError: maximum recursion depth exceeded while calling a Python object. I can install number of things like scikit etc, just fine. Is there something special needed? Here is the tail of the stack trace.
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 837, in obtain
return installer(requirement)
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 272, in fetch_build_egg
dist = self.__class__({'script_args':['easy_install']})
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 225, in __init__
_Distribution.__init__(self,attrs)
File "/usr/local/lib/python2.7/distutils/dist.py", line 287, in __init__
self.finalize_options()
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 257, in finalize_options
ep.require(installer=self.fetch_build_egg)
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 2029, in require
working_set.resolve(self.dist.requires(self.extras),env,installer))
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 579, in resolve
env = Environment(self.entries)
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 748, in __init__
self.scan(search_path)
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 777, in scan
for dist in find_distributions(item):
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 1757, in find_on_path
path_item,entry,metadata,precedence=DEVELOP_DIST
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 2151, in from_location
py_version=py_version, platform=platform, **kw
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 2128, in __init__
self.project_name = safe_name(project_name or 'Unknown')
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 1139, in safe_name
return re.sub('[^A-Za-z0-9.]+', '-', name)
File "/usr/local/lib/python2.7/re.py", line 155, in sub
return _compile(pattern, flags).sub(repl, string, count)
File "/usr/local/lib/python2.7/re.py", line 235, in _compile
cachekey = (type(key[0]),) + key
RuntimeError: maximum recursion depth exceeded while calling a Python object
Changing python installation to scl fixed the problem. I uninstalled the python2.7 but cleaning out /usr/local with all python 2.7 related things in bin and lib. Reinstalled python27 using the following sequence:
yum install centos-release-SCL
yum install python27
scl enable python27 bash
Installed pip using "easy_install-2.7 pip"
Now I can install cassandra driver...