ParallelPython created in thread gives - python-2.7

Creating a job server gives the error:
Python int too large to convert to C long
when created in thread (win32), using parallel python version 1.6.5 & Python27.
Here is the traceback:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\ParrallelPython.py", line 91, in __init__
self.job_server = pp.Server(ppservers=self.ppservers, socket_timeout=3600)
File "C:\Python27\lib\site-packages\pp.py", line 341, in __init__
self.set_ncpus(ncpus)
File "C:\Python27\lib\site-packages\pp.py", line 505, in set_ncpus
range(ncpus - len(self.__workers))])
File "C:\Python27\lib\site-packages\pp.py", line 138, in __init__
self.start()
File "C:\Python27\lib\site-packages\pp.py", line 150, in start
self.pid = int(self.t.receive())
File "C:\Python27\lib\site-packages\pptransport.py", line 148, in receive
msg = self.r.read(e_size-r_size)
OverflowError: Python int too large to convert to C long
Similar issue to link below:
ParallelPython Forum Question

Related

MongoDB doesn't work locally because of SRV error (no answer)

Situation
We are working on the backend of our project consisting of two services written in Django and FastAPI. Both services use MongoDB as their database system.
In Django we use djongo==1.3.0 for ORM compatibility.
In FastAPI we use odmantic==0.3.4.
Both of these libraries use pymongo==3.11.3 underneath.
This MongoDB SRV error has been an issue for our Django service ever since we created it, but we managed to somehow work around it by not using the latest packages, such as:
Django==2.2.20
djongo==1.3.0
pymongo==3.11.3
Recently due to security risks we had to upgrade:
urllib3 from 1.25.8 to 1.26.5
pydantic from 1.8.1 to 1.8.2
Django from 2.2.20 to 2.2.22
Those were suggested by GitHub's dependabot.
Problem
When we run any of these services locally now they break with the following base exception:
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
Full log for Django:
> python manage.py test
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/database.py", line 10, in connect
return clients[db]
KeyError: 'djongo_test'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 212, in __init__
rrset = response.find_rrset(response.answer, qname,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/message.py", line 341, in find_rrset
raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 220, in __init__
crrset = response.find_rrset(response.answer,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/message.py", line 341, in find_rrset
raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 72, in _resolve_uri
results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1100, in query
return get_default_resolver().query(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1003, in query
answer = Answer(_qname, rdtype, rdclass, response,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 232, in __init__
raise NoAnswer(response=response)
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.database-test.avxhw.mongodb.net. IN SRV
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
super().run_from_argv(argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 53, in handle
failures = test_runner.run_tests(test_labels)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/runner.py", line 629, in run_tests
old_config = self.setup_databases(aliases=databases)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/runner.py", line 552, in setup_databases
return _setup_databases(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/utils.py", line 170, in setup_databases
connection.creation.create_test_db(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 58, in create_test_db
self._create_test_db(verbosity, autoclobber, keepdb)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 168, in _create_test_db
with self._nodb_connection.cursor() as cursor:
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/base.py", line 185, in get_new_connection
self.client_connection = Database.connect(db=name, **connection_params)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/database.py", line 13, in connect
clients[db] = MongoClient(**kwargs, connect=False)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 639, in __init__
res = uri_parser.parse_uri(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/uri_parser.py", line 500, in parse_uri
nodes = dns_resolver.get_hosts()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 102, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 83, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 79, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: The DNS response does not contain an answer to the question: _mongodb._tcp.database-test.avxhw.mongodb.net. IN SRV
Full log for FastAPI:
> python -m src.main
Traceback (most recent call last):
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 72, in _resolve_uri
results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1321, in query
return resolve(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1305, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1202, in resolve
(answer, done) = resolution.query_result(response, None)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 674, in query_result
raise NoAnswer(response=answer.response)
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/bk/inz/backend/testservice/src/main.py", line 20, in <module>
client = AsyncIOMotorClient(MONGODB_URL)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/motor/core.py", line 156, in __init__
delegate = self.__delegate_class__(*args, **kwargs)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 639, in __init__
res = uri_parser.parse_uri(
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/uri_parser.py", line 500, in parse_uri
nodes = dns_resolver.get_hosts()
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 102, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 83, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 79, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
In our CI setup on GitHub Actions Django service tests are still run successfully. It's clear that the problem lies within the DNS aspect.
Questions
Are there any specific DNS servers which do not support MongoDB? I was using Google's 8.8.8.8 before and after it broke.
Why would the problem be caused by upgrading listed dependencies? Only urllib was upgraded for both services.
Is there any workaround?

Review https://github.com/mongodb/specifications/blob/master/source/initial-dns-seedlist-discovery/initial-dns-seedlist-discovery.rst for how SRV URIs work, then manually issue the relevant DNS queries using a tool such as dig to verify your environment has working DNS.

Q: Sonos Python Self Test error: No handlers could be found for logger "smapi"

I am trying to run the SONOS self test for a music service on Sonos.
After getting the dependencies, and filling out the config file, I try to run the python Sonos selftest, however it runs into an error and I have no clue what the underlying issue might be to get it running:
No handlers could be found for logger "smapi"
Traceback (most recent call last):
File "suite_selftest.py", line 226, in <module>
nightly_mode(parser.config_file)
File "suite_selftest.py", line 51, in nightly_mode
development_mode(config_file)
File "suite_selftest.py", line 186, in development_mode
fixtures.append(getlastupdate.PollingIntervalTest(suite.client, suite.smapiservice))
File "/Users/thomas/Desktop/PythonSelfTest/smapi/content_workflow/getlastupdate.py", line 20, in __init__
self.poll_interval = self.smapiservice.get_polling_interval()
File "../../sonos-1.1.0.dev_r300235-py2.7.egg/sonos/smapi/smapiservice.py", line 465, in get_polling_interval
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 362, in getfloat
return self._get(section, float, option)
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 356, in _get
return conv(self.get(section, option))
ValueError: could not convert string to float:

Found the fix already, forgot to add the Polling Interval in the config file...

happybase hbase table.put command error?

I am trying to connect to hbase-1.2.6 from python code, is as:
import happybase
connection = happybase.Connection(host='localhost',port=16010)
table = connection.table('blogpost')
table.put('1', {'post:title': 'hello world1'})
I manually created the table-"blogpost" in hbase. I am using python-2.7 and happybase-1.1.0.
error log are as:
/usr/bin/python2.7 /home/spark/PycharmProjects/PySpark/hbase.py
Traceback (most recent call last):
File "/home/spark/PycharmProjects/PySpark/hbase.py", line 5, in <module>
table.put('1', {'post:title': 'hello world1'})
File "/usr/local/lib/python2.7/dist-packages/happybase/table.py", line 464, in put
batch.put(row, data)
File "/usr/local/lib/python2.7/dist-packages/happybase/batch.py", line 137, in __exit__
self.send()
File "/usr/local/lib/python2.7/dist-packages/happybase/batch.py", line 60, in send
self._table.connection.client.mutateRows(self._table.name, bms, {})
File "/usr/local/lib/python2.7/dist-packages/thriftpy/thrift.py", line 198, in _req
return self._recv(_api)
File "/usr/local/lib/python2.7/dist-packages/thriftpy/thrift.py", line 210, in _recv
fname, mtype, rseqid = self._iprot.read_message_begin()
File "thriftpy/protocol/cybin/cybin.pyx", line 439, in cybin.TCyBinaryProtocol.read_message_begin (thriftpy/protocol/cybin/cybin.c:6470)
cybin.ProtocolError: No protocol version header
thanks.
Process finished with exit code 1

TypeError when using botocore to read from AWS SQS queue

I'm using a Tornado server with tornado-botocore to connect to Amazon SQS services.
When running stress tests we sometimes get the following exception:
Traceback (most recent call last):
File "/home/app/handlers/WebSocketsHandler.py", line 95, in listen_outgoing_queue
message = yield tornado.gen.Task(self.outgoing_queue.read)
File "/home/local/lib/python2.7/site-packages/tornado/gen.py", line 870, in run
value = future.result()
File "/home/local/lib/python2.7/site-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/home/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
ret = fn(*args, **kwargs)
File "/home/local/lib/python2.7/site-packages/tornado_botocore/base.py", line 70, in prepare_response
response_dict, operation_model.output_shape)
File "/home/local/lib/python2.7/site-packages/botocore/parsers.py", line 155, in parse
return self._do_error_parse(response, shape)
File "/home/.env/local/lib/python2.7/site-packages/botocore/parsers.py", line 314, in _do_error_parse
root = self._parse_xml_string_to_dom(xml_contents)
File "/home/local/lib/python2.7/site-packages/botocore/parsers.py", line 274, in _parse_xml_string_to_dom
parser.feed(xml_string)
TypeError: must be string or read-only buffer, not None
could it be caused by the concurrency?
has anyone encountered such behavior?
We are using tornado 4.2.1, botocore 0.65.0 and tonado-botocore 0.1.6

problem solved once i removed the #tornado.gen.engine decorator from the method

Scrapy cannot handle bad headers properly [ScrapyHTTPPageGetter,client] Unhandled Error

Environment:
Scrapy 0.16.2
Twisted-12.2.0
python 2.7
macosx-10.6
Okey here is my problem:
I try to run
scrapy shell http://aaa.17domn.com/bt9/file.php/MERH77V.html
Error:
[ScrapyHTTPPageGetter,client] Unhandled Error
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/selectreactor.py", line 150, in _doReadOrWrite
why = getattr(selectable, method)()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/tcp.py", line 202, in doRead
return self._dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/tcp.py", line 208, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/protocols/basic.py", line 564, in dataReceived
why = self.lineReceived(line)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Scrapy-0.16.2-py2.7.egg/scrapy/core/downloader/webclient.py", line 50, in lineReceived
return HTTPClient.lineReceived(self, line.rstrip())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/web/http.py", line 450, in lineReceived
self.extractHeader(self._header)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/web/http.py", line 406, in extractHeader
key, val = header.split(':',1)
exceptions.ValueError: need more than 1 value to unpack
I found the solution from https://groups.google.com/forum/#!msg/scrapy-users/xFKo8ggzPxs/VXDl3CZ4V4cJ
They describe this is caused by twisted. Then I patched function extractHeader in /twisted/web/http.py from http://twistedmatrix.com/trac/ticket/2842. Its WORKS
BUT BUT, Hold on NOt yet!!!
I run another web
scrapy shell http://www1.wkdown.info/fs3/file.php/M994ATR.html
Error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.6-intel.egg/twisted/internet/defer.py", line 551, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Scrapy-0.16.2-py2.7.egg/scrapy/core/downloader/webclient.py", line 122, in _build_response
status = int(self.status)
ValueError: invalid literal for int() with base 10: 'html'
I think something happen on response headers. Scrapy cannot handle it well.
Any idea?
Thank you!