I am using pyspark with Spark 2.2 and Python 2.7.11 and I am trying to extract rules from the decision trees of a random forest classifier using the .toDebugString Method.
I had success with it but since the update from 2.1 to 2.2, Spark crashes after I use this command and I get this error:
Full error:
rfModel.toDebugString
----------------------------------------u'RandomForestClassificationModel (uid=RandomForestClassifier_4e99b85432585e2e32a2) with 30 trees\n Tr'
Exception happened during processing of request from ('127.0.0.1', 42374)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/local/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/local/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/local/lib/python2.7/SocketServer.py", line 655, in init
self.handle()
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/accumulators.py",
line 235, in handle
num_updates = read_int(self.rfile)
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/serializers.py",
line 577, in read_int
----------------------------------------
raise EOFError
EOFError
If I run the command again after this error I get:
Py4JError: An error occurred while calling o1129.toString
My code is fairly straightforward and this confuses me very much.
#Get random forest from pipeline
rfModel = model.stages[1]
rfModel.toDebugString
I know rfModel is a randomForestClassifier object. Any ideas why I get this error?
The solution to my problem was the python version. By using python3+ with spark 2.2 i managed to work .toDebugString method without a problem. I will update with full error log for anyone seeing this in the future.
Related
I am trying to run the SONOS self test for a music service on Sonos.
After getting the dependencies, and filling out the config file, I try to run the python Sonos selftest, however it runs into an error and I have no clue what the underlying issue might be to get it running:
No handlers could be found for logger "smapi"
Traceback (most recent call last):
File "suite_selftest.py", line 226, in <module>
nightly_mode(parser.config_file)
File "suite_selftest.py", line 51, in nightly_mode
development_mode(config_file)
File "suite_selftest.py", line 186, in development_mode
fixtures.append(getlastupdate.PollingIntervalTest(suite.client, suite.smapiservice))
File "/Users/thomas/Desktop/PythonSelfTest/smapi/content_workflow/getlastupdate.py", line 20, in __init__
self.poll_interval = self.smapiservice.get_polling_interval()
File "../../sonos-1.1.0.dev_r300235-py2.7.egg/sonos/smapi/smapiservice.py", line 465, in get_polling_interval
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 362, in getfloat
return self._get(section, float, option)
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 356, in _get
return conv(self.get(section, option))
ValueError: could not convert string to float:
Found the fix already, forgot to add the Polling Interval in the config file...
I hope somebody here can help. I've been googling this error like crazy but haven't found anything.
I have a pipeline that works perfectly when executed locally but it fails when executed on GCP. The following are the error messages that I get.
Workflow failed. Causes: S03:Write transform
fn/WriteMetadata/ResolveBeamFutures/CreateSingleton/Read+Write
transform fn/WriteMetadata/ResolveBeamFutures/ResolveFutures/Do+Write
transform fn/WriteMetadata/WriteMetadata failed., A work item was
attempted 4 times without success. Each time the worker eventually
lost contact with the service. The work item was attempted on:
Traceback (most recent call last): File "preprocess.py", line 491,
in
main() File "preprocess.py", line 487, in main
transform_data(args,pipeline_options,runner) File "preprocess.py", line 451, in transform_data
eval_data |= 'Identity eval' >> beam.ParDo(Identity()) File "/Library/Python/2.7/site-packages/apache_beam/pipeline.py", line 335,
in exit
self.run().wait_until_finish() File "/Library/Python/2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 897, in wait_until_finish
(self.state, getattr(self._runner, 'last_error_msg', None)), self) apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:
Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent
call last): File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 582, in do_work
work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start() File "apache_beam/runners/worker/operations.py", line 294, in apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:10607)
def start(self): File "apache_beam/runners/worker/operations.py", line 295, in
apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:10501)
with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 300, in
apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:9702)
pickler.loads(self.spec.serialized_fn)) File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py",
line 225, in loads
return dill.loads(s) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in
loads
return load(file) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in
load
obj = pik.load() File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.new(cls, *args) TypeError: new() takes exactly 4 arguments (1 given)
Any ideas??
Thanks,
Pedro
If the pipeline works locally but fails on GCP it's possible that you're running into a version mismatch.
What TF, tf.Transform, beam versions are you running locally and on GCP?
My current version of python is 2.7.10 and version of Django is 1.9.1, path.py is 8.1.2 but every time when I'm trying to invoke python shell by command:
$ python manage.py shell I'm getting a lot of errors and the ending of them looks like:
Error in sys.excepthook:
Traceback (most recent call last):
File "W:\SVN\vendors\python\win32\lib\site-packages\IPython\CrashHandler.py", line 157, in __call__
report.write(self.make_report(traceback))
File "W:\SVN\vendors\python\win32\lib\site-packages\IPython\CrashHandler.py", line 215, in make_report
rpt_add('BZR revision : %s \n\n' % Release.revision)
AttributeError: 'module' object has no attribute 'revision'
Original exception was:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\__init__.py", line 353, in execute_from_command_line
utility.execute()
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\base.py", line 399, in execute
output = self.handle(*args, **options)
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\commands\shell.py", line 69, in handle
self.run_shell(shell=options['interface'])
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\commands\shell.py", line 58, in run_shell
return getattr(self, shell)()
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\commands\shell.py", line 41, in ipython
ip()
File "W:\SVN\vendors\python\win32\lib\site-packages\django\core\management\commands\shell.py", line 22, in _ipython_pre_011
shell = IPShell(argv=[])
File "W:\SVN\vendors\python\win32\lib\site-packages\IPython\Shell.py", line 73, in __init__
debug=debug,shell_class=shell_class)
File "W:\SVN\vendors\python\win32\lib\site-packages\IPython\ipmaker.py", line 521, in make_IPython
IP.pre_config_initialization()
File "W:\SVN\vendors\python\win32\lib\site-packages\IPython\iplib.py", line 835, in pre_config_initialization
self.db = pickleshare.PickleShareDB(rc.ipythondir + "/db")
File "W:\SVN\vendors\python\win32\lib\site-packages\IPython\Extensions\pickleshare.py", line 53, in __init__
if not self.root.isdir():
TypeError: _isdir() takes exactly 1 argument (0 given)
How can I overcome the problem?
From doing a quick search, it looks like this error happens when both Django and IPython are installed, since apparently Django will attempt to use the IPython shell.
The first resource with someone who had a similar issue:
run python manage.py shell occurs errors
And in the link above, one of the comments on the OP's question suggested looking at this separate thread for answers:
https://superuser.com/questions/318655/error-running-ipython3-on-xp-typeerror-isdir-takes-exactly-1-argument-0-gi
The winning answer says the following:
Finally decided to give this another stab, and managed to get it to
work. The solution is a two-line change in the
ipython-0.11-py3.2.egg\IPython\utils\pickleshare.py file, line 52:
Before:
if not self.root.isdir():
self.root.makedirs()
After:
if not os.path.isdir(self.root):
os.makedirs(self.root)
Out of the blue today I started having this problem.. in trying to run the Python Shell from my .py file. The shell window opens with a blinking cursor. If i type anything and hit enter the following error hits. Python IDLE doesn't work at all and I only get this error.
IDLE internal error in runcode()
Traceback (most recent call last):
File "C:\Python27\lib\idlelib\rpc.py", line 235, in asyncqueue
self.putmessage((seq, request))
File "C:\Python27\lib\idlelib\rpc.py", line 332, in putmessage
n = self.sock.send(s[:BUFSIZE])
error: [Errno 10053] An established connection was aborted by the software in your host machine
Python (command line) works fine, but the IDLE does not. I've tried rebooting windows and system restore with no luck. I've been googling for answers with little success. My firewall has always been disabled. Trying to start IDLE from command line returns the following error.
C:\Python27>python.exe -m idlelib.idle
Failed to load extension 'CallTips'
Traceback (most recent call last):
File "C:\Python27\lib\idlelib\EditorWindow.py", line 1061, in load_standard_ex
tensions
self.load_extension(name)
File "C:\Python27\lib\idlelib\EditorWindow.py", line 1076, in load_extension
cls = getattr(mod, name)
AttributeError: 'module' object has no attribute 'CallTips'
----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address: ('127.0.0.1', 49552)
Request: <socket._socketobject object at 0x0176BCA8>
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\idlelib\rpc.py", line 503, in __init__
SocketServer.BaseRequestHandler.__init__(self, sock, addr, svr)
File "C:\Python27\lib\SocketServer.py", line 649, in __init__
self.handle()
File "C:\Python27\lib\idlelib\run.py", line 276, in handle
executive = Executive(self)
File "C:\Python27\lib\idlelib\run.py", line 315, in __init__
self.calltip = CallTips.CallTips()
AttributeError: 'module' object has no attribute 'CallTips'
Windows -32bit OS. thanks in advance for your help.
This is really weird. But a great question you posted! Nice to read.
Here could be a solution: https://stackoverflow.com/a/3277996/1320237
Why would it be impossible to import the Calltips module from the standart library? Maybe you have a Python 3 idle open? Then they interfere?
Well my impatience got the best of me. I copied my site-packages file to the desktop and uninstalled/reinstalled Python. Pasted the site-packages back into my Python directory and all is right with the world again.
When using the development server, I get this error:
Validating models...
0 errors found
Django version 1.3 beta 1, using settings 'book_scouts.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 283
, in run
self.finish_response()
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 323
, in finish_response
self.write(data)
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 402
, in write
self.send_headers()
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 467
, in send_headers
self._write(str(self.headers))
File "C:\Python26\lib\socket.py", line 318, in write
self.flush()
File "C:\Python26\lib\socket.py", line 297, in flush
self._sock.sendall(buffer(data, write_offset, buffer_size))
error: [Errno 10053] An established connection was aborted by the software in yo
ur host machine
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 2426)
Traceback (most recent call last):
File "C:\Python26\lib\SocketServer.py", line 283, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python26\lib\SocketServer.py", line 309, in process_request
self.finish_request(request, client_address)
File "C:\Python26\lib\SocketServer.py", line 322, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python26\lib\site-packages\django\core\servers\basehttp.py", line 569
, in __init__
BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
File "C:\Python26\lib\SocketServer.py", line 618, in __init__
self.finish()
File "C:\Python26\lib\SocketServer.py", line 661, in finish
self.wfile.flush()
File "C:\Python26\lib\socket.py", line 297, in flush
self._sock.sendall(buffer(data, write_offset, buffer_size))
error: [Errno 10053] An established connection was aborted by the software in yo
ur host machine
I can't pin it down to any single source. It seems to happen randomly, afaik. It causes the page I'm testing to load up really long, and the page just loads fine.
Using Django 1.3 Beta 1
UPDATE: I'm using Opera/Firefox for testing now, and the problem hasn't come up since. I'm afraid to try again in Chrome.
I got the same thing and asked on django-users google group. Nobody had any idea what's wrong.
It seems only I get this when I refresh the page from browser and not when I click on links inside the page.
Also just as a heads up: I tried on Python 2.6 and 2.7 both stock version (form python.org website) and ActiveState one. Same thing happens randomly.
LE: I'm getting this on the stable version of Django.