Related
Trying to get AWS SAM CLI working to locally test lambda functions. I've installed the helloworld python function, which I can successfully build and validate, until I add the --use-container flag, at which point I get the below errors.
I have Docker Desktop installed and running. I'm using WSL2 with Ubuntu 20.04 on a windows 11 machine.
mylaptop:~/projects/lambda/lambda-python3.8$ sam build --use-container
Starting Build inside a container
Your template contains a resource with logical ID "ServerlessRestApi", which is a reserved logical ID in AWS SAM. It could result in unexpected behaviors and is not recommended.
Building codeuri: /home/projects/lambda/lambda-python3.8/hello_world runtime: python3.8 metadata: {} architecture: x86_64 functions: HelloWorldFunction
<3>init: (15570) ERROR: UtilConnectUnix:467: connect failed 111
Traceback (most recent call last):
File "docker/credentials/store.py", line 80, in _execute
File "subprocess.py", line 411, in check_output
File "subprocess.py", line 512, in run
subprocess.CalledProcessError: Command '['/usr/bin/docker-credential-desktop.exe', 'get']' returned non-zero exit status 255.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/auth.py", line 264, in _resolve_authconfig_credstore
File "docker/credentials/store.py", line 35, in get
File "docker/credentials/store.py", line 93, in _execute
docker.credentials.errors.StoreError: Credentials store docker-credential-desktop.exe exited with "".
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "samcli/__main__.py", line 12, in <module>
File "click/core.py", line 829, in __call__
File "click/core.py", line 782, in main
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1066, in invoke
File "click/core.py", line 610, in invoke
File "click/decorators.py", line 73, in new_func
File "click/core.py", line 610, in invoke
File "samcli/lib/telemetry/metric.py", line 166, in wrapped
File "samcli/lib/telemetry/metric.py", line 124, in wrapped
File "samcli/lib/utils/version_checker.py", line 41, in wrapped
File "samcli/cli/main.py", line 87, in wrapper
File "samcli/commands/build/command.py", line 201, in cli
File "samcli/commands/build/command.py", line 262, in do_cli
File "samcli/commands/build/build_context.py", line 248, in run
File "samcli/lib/build/app_builder.py", line 221, in build
File "samcli/lib/build/build_strategy.py", line 79, in build
File "samcli/lib/build/build_strategy.py", line 89, in _build_functions
File "samcli/lib/build/build_strategy.py", line 171, in build_single_function_definition
File "samcli/lib/build/app_builder.py", line 654, in _build_function
File "samcli/lib/build/app_builder.py", line 819, in _build_function_on_container
File "samcli/local/docker/manager.py", line 115, in run
File "samcli/local/docker/manager.py", line 85, in create
File "samcli/local/docker/manager.py", line 160, in pull_image
File "docker/api/image.py", line 396, in pull
File "docker/auth.py", line 48, in get_config_header
File "docker/auth.py", line 324, in resolve_authconfig
File "docker/auth.py", line 235, in resolve_authconfig
File "docker/auth.py", line 281, in _resolve_authconfig_credstore
docker.errors.DockerException: Credentials store error: StoreError('Credentials store docker-credential-desktop.exe exited with "".')
[15567] Failed to execute script __main__
I ran docker-credential-desktop.exe version which resulted in the 111 error message, so I was able to isolate the issue to something related to docker-desktop-credential.exe. After googling around and trying lots of different suggestions, this finally worked for me, without any restart required.
mv ~/.docker ~/.docker_old
Situation
We are working on the backend of our project consisting of two services written in Django and FastAPI. Both services use MongoDB as their database system.
In Django we use djongo==1.3.0 for ORM compatibility.
In FastAPI we use odmantic==0.3.4.
Both of these libraries use pymongo==3.11.3 underneath.
This MongoDB SRV error has been an issue for our Django service ever since we created it, but we managed to somehow work around it by not using the latest packages, such as:
Django==2.2.20
djongo==1.3.0
pymongo==3.11.3
Recently due to security risks we had to upgrade:
urllib3 from 1.25.8 to 1.26.5
pydantic from 1.8.1 to 1.8.2
Django from 2.2.20 to 2.2.22
Those were suggested by GitHub's dependabot.
Problem
When we run any of these services locally now they break with the following base exception:
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
Full log for Django:
> python manage.py test
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/database.py", line 10, in connect
return clients[db]
KeyError: 'djongo_test'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 212, in __init__
rrset = response.find_rrset(response.answer, qname,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/message.py", line 341, in find_rrset
raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 220, in __init__
crrset = response.find_rrset(response.answer,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/message.py", line 341, in find_rrset
raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 72, in _resolve_uri
results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1100, in query
return get_default_resolver().query(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1003, in query
answer = Answer(_qname, rdtype, rdclass, response,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 232, in __init__
raise NoAnswer(response=response)
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.database-test.avxhw.mongodb.net. IN SRV
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
super().run_from_argv(argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 53, in handle
failures = test_runner.run_tests(test_labels)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/runner.py", line 629, in run_tests
old_config = self.setup_databases(aliases=databases)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/runner.py", line 552, in setup_databases
return _setup_databases(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/utils.py", line 170, in setup_databases
connection.creation.create_test_db(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 58, in create_test_db
self._create_test_db(verbosity, autoclobber, keepdb)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 168, in _create_test_db
with self._nodb_connection.cursor() as cursor:
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/base.py", line 185, in get_new_connection
self.client_connection = Database.connect(db=name, **connection_params)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/database.py", line 13, in connect
clients[db] = MongoClient(**kwargs, connect=False)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 639, in __init__
res = uri_parser.parse_uri(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/uri_parser.py", line 500, in parse_uri
nodes = dns_resolver.get_hosts()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 102, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 83, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 79, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: The DNS response does not contain an answer to the question: _mongodb._tcp.database-test.avxhw.mongodb.net. IN SRV
Full log for FastAPI:
> python -m src.main
Traceback (most recent call last):
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 72, in _resolve_uri
results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1321, in query
return resolve(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1305, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1202, in resolve
(answer, done) = resolution.query_result(response, None)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 674, in query_result
raise NoAnswer(response=answer.response)
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/bk/inz/backend/testservice/src/main.py", line 20, in <module>
client = AsyncIOMotorClient(MONGODB_URL)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/motor/core.py", line 156, in __init__
delegate = self.__delegate_class__(*args, **kwargs)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 639, in __init__
res = uri_parser.parse_uri(
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/uri_parser.py", line 500, in parse_uri
nodes = dns_resolver.get_hosts()
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 102, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 83, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 79, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
In our CI setup on GitHub Actions Django service tests are still run successfully. It's clear that the problem lies within the DNS aspect.
Questions
Are there any specific DNS servers which do not support MongoDB? I was using Google's 8.8.8.8 before and after it broke.
Why would the problem be caused by upgrading listed dependencies? Only urllib was upgraded for both services.
Is there any workaround?
Review https://github.com/mongodb/specifications/blob/master/source/initial-dns-seedlist-discovery/initial-dns-seedlist-discovery.rst for how SRV URIs work, then manually issue the relevant DNS queries using a tool such as dig to verify your environment has working DNS.
HI~ I want to query xml data from Oracle db with cx_Oracle, but it doesn't work with Ora-19011 error message. I think size of query data is larger than string buffer, but i don't know how to solve this problem
Oracle DB is an external DB and it's not my own DB, So i can't access directly. Therefore, I want to fix my problem on my code and print query data on python terminal.
(my software version)
windows 10 64bit
python 2.7 64bit
oracle-instant client 19.3 64bit
cx_oracle 7.2.2
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cx_Oracle
import sys
import csv
import codecs
printHeader = True
conn = cx_Oracle.connect('id/passwd#ip:port/orcl')
print(conn.version)
curs = conn.cursor()
curs.execute('SELECT * FROM tablename')
for record in curs:
print(record)
Error occured at line 18(for record in curs) and here are error messages.
11.2.0.4.0
We've got an error while stopping in unhandled exception: <class 'cx_Oracle.DatabaseError'>.
Traceback (most recent call last):
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1740, in do_stop_on_unhandled_exception
self.do_wait_suspend(thread, frame, 'exception', arg, is_unhandled_exception=True)
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1615, in do_wait_suspend
with self._threads_suspended_single_notification.notify_thread_suspended(thread_id, stop_reason):
File "C:\Python27\lib\contextlib.py", line 17, in __enter__
return self.gen.next()
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 360, in notify_thread_suspended
with AbstractSingleNotificationBehavior.notify_thread_suspended(self, thread_id, stop_reason):
File "C:\Python27\lib\contextlib.py", line 17, in __enter__
return self.gen.next()
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 308, in notify_thread_suspended
self.send_suspend_notification(thread_id, stop_reason)
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 354, in send_suspend_notification
py_db.writer.add_command(py_db.cmd_factory.make_thread_suspend_single_notification(py_db, thread_id, stop_reason))
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_net_command_factory_json.py", line 309, in make_thread_suspend_single_notification
return NetCommand(CMD_THREAD_SUSPEND_SINGLE_NOTIFICATION, 0, event, is_json=True)
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_net_command.py", line 57, in __init__
text = json.dumps(as_dict)
File "C:\Python27\lib\json\__init__.py", line 244, in dumps
return _default_encoder.encode(obj)
File "C:\Python27\lib\json\encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 11: invalid start byte
Traceback (most recent call last):
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\ptvsd_launcher.py", line 43, in <module>
main(ptvsdArgs)
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\__main__.py", line 432, in main
run()
File "c:\Users\goo41\.vscode\extensions\ms-python.python-2019.8.30787\pythonFiles\lib\python\ptvsd\__main__.py", line 316, in run_file
runpy.run_path(target, run_name='__main__')
File "C:\Python27\lib\runpy.py", line 252, in run_path
return _run_module_code(code, init_globals, run_name, path_name)
File "C:\Python27\lib\runpy.py", line 82, in _run_module_code
mod_name, mod_fname, mod_loader, pkg_name)
File "C:\Python27\lib\runpy.py", line 72, in _run_code
exec code in run_globals
File "c:\PythonWorkspace\oraclePrc\test1.py", line 18, in <module>
for record in curs:
cx_Oracle.DatabaseError: ORA-19011: Character string buffer too small
When you connect to the database, try using this code instead:
conn = cx_Oracle.connect('id/passwd#ip:port/orcl', encoding="UTF-8", nencoding="UTF-8")
This will ensure that you are using a universal encoding -- which may eliminate the first error, and possibly the second as well. If not, adjust the code sample and error messages noted above.
I installed habo malware hunter, but when i ran it,i got following errors:
habo#habo-VirtualBox:~/HaboMalHunter$ sudo python AnalyzeControl.py -z -l ./test_0813.zip Traceback (most recent call last):
File "AnalyzeControl.py", line 688, in <module>
ret = main(len(sys.argv), sys.argv)
File "AnalyzeControl.py", line 615, in main
cfg = init_arguments(argv)
File "AnalyzeControl.py", line 156, in init_arguments
cfg = init_cfg(args.config_path,args)
File "AnalyzeControl.py", line 96, in init_cfg
for k,v in conf_parser.items(SECTION_DEF):
File "/usr/lib/python2.7/ConfigParser.py", line 642, in items
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'main'
I hope somebody here can help. I've been googling this error like crazy but haven't found anything.
I have a pipeline that works perfectly when executed locally but it fails when executed on GCP. The following are the error messages that I get.
Workflow failed. Causes: S03:Write transform
fn/WriteMetadata/ResolveBeamFutures/CreateSingleton/Read+Write
transform fn/WriteMetadata/ResolveBeamFutures/ResolveFutures/Do+Write
transform fn/WriteMetadata/WriteMetadata failed., A work item was
attempted 4 times without success. Each time the worker eventually
lost contact with the service. The work item was attempted on:
Traceback (most recent call last): File "preprocess.py", line 491,
in
main() File "preprocess.py", line 487, in main
transform_data(args,pipeline_options,runner) File "preprocess.py", line 451, in transform_data
eval_data |= 'Identity eval' >> beam.ParDo(Identity()) File "/Library/Python/2.7/site-packages/apache_beam/pipeline.py", line 335,
in exit
self.run().wait_until_finish() File "/Library/Python/2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 897, in wait_until_finish
(self.state, getattr(self._runner, 'last_error_msg', None)), self) apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:
Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent
call last): File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 582, in do_work
work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start() File "apache_beam/runners/worker/operations.py", line 294, in apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:10607)
def start(self): File "apache_beam/runners/worker/operations.py", line 295, in
apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:10501)
with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 300, in
apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:9702)
pickler.loads(self.spec.serialized_fn)) File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py",
line 225, in loads
return dill.loads(s) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in
loads
return load(file) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in
load
obj = pik.load() File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.new(cls, *args) TypeError: new() takes exactly 4 arguments (1 given)
Any ideas??
Thanks,
Pedro
If the pipeline works locally but fails on GCP it's possible that you're running into a version mismatch.
What TF, tf.Transform, beam versions are you running locally and on GCP?