I have a problem when I try to execute multiple Tasks within MWAA using POST Requests. I have been using mw1.small tier of MWAA and I schedule around 3 tasks per minute with EventBridge and Lambda. When I see my results I find that some tasks are missing and when I search for logs, I noticed that the Task was triggered but It was never scheduled or queued, and it does not appear on the tree or graph view.
I have 169 rules created on Event Bridge running a certain time everyday and I only see around 165 to 166 executions of the DAG. It is not a problem from Event Bridge or Lambda. I checked the logs for those services and all 169 DAG invocations are working fine.
The lambda function that I mentioned before triggers the DAG using a POST Request for every rule that I have on Event Bridge.
These are my configuration options that I have set.
celery.pool=1
celery.worker_autoscale=1,1
core.dag_file_processor_timeout=150
core.dagbag_import_timeout=90
core.killed_task_cleanup_time=604800
core.min_serialized_dag_update_interval=60
scheduler.dag_dir_list_interval=300
scheduler.min_file_process_interval=300
scheduler.parsing_processes=1
scheduler.processor_poll_interval=60
scheduler.schedule_after_task_execution=false
NOTE: I know I can use Step Functions but this is not an option in my case.
EDIT:
This problem is caused because I have multiple parallel requests made from the lambda function. Airflow 2.2.2 uses dag_id and execution_date as a primary key for the table dag_run.
The two types of traceback that I found are:
/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py:91 DeprecationWarning: Calling `DAG.create_dagrun()` without an explicit data interval is deprecated
Traceback (most recent call last):
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "dag_run_dag_id_execution_date_key"
DETAIL: Key (dag_id, execution_date)=(test_dag, 2023-02-16 20:19:55+00) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 138, in dag_trigger
dag_id=args.dag_id, run_id=args.run_id, conf=args.conf, execution_date=args.exec_date
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/client/local_client.py", line 30, in trigger_dag
dag_id=dag_id, run_id=run_id, conf=conf, execution_date=execution_date
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 125, in trigger_dag
replace_microseconds=replace_microseconds,
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 91, in _trigger_dag
dag_hash=dag_bag.dags_hash.get(dag_id),
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 2359, in create_dagrun
session.flush()
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
uow,
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1136, in _emit_insert_statements
statement, params
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "dag_run_dag_id_execution_date_key"
DETAIL: Key (dag_id, execution_date)=(test_dag, 2023-02-16 20:19:55+00) already exists.
[SQL: INSERT INTO dag_run (dag_id, queued_at, execution_date, start_date, end_date, state, run_id, creating_job_id, external_trigger, run_type, conf, data_interval_start, data_interval_end, last_scheduling_decision, dag_hash) VALUES (%(dag_id)s, %(queued_at)s, %(execution_date)s, %(start_date)s, %(end_date)s, %(state)s, %(run_id)s, %(creating_job_id)s, %(external_trigger)s, %(run_type)s, %(conf)s, %(data_interval_start)s, %(data_interval_end)s, %(last_scheduling_decision)s, %(dag_hash)s) RETURNING dag_run.id]
[parameters: {'dag_id': 'test_dag', 'queued_at': datetime.datetime(2023, 2, 16, 20, 19, 56, 168249, tzinfo=Timezone('UTC')), 'execution_date': DateTime(2023, 2, 16, 20, 19, 55, tzinfo=Timezone('UTC')), 'start_date': None, 'end_date': None, 'state': <TaskInstanceState.QUEUED: 'queued'>, 'run_id': 'test22__2023-02-16T20:19:03+602430', 'creating_job_id': None, 'external_trigger': True, 'run_type': <DagRunType.MANUAL: 'manual'>, 'conf': <psycopg2.extensions.Binary object at 0x7fe5917cc900>, 'data_interval_start': DateTime(2023, 2, 16, 20, 19, 55, tzinfo=Timezone('UTC')), 'data_interval_end': DateTime(2023, 2, 16, 20, 19, 55, tzinfo=Timezone('UTC')), 'last_scheduling_decision': None, 'dag_hash': 'a1c4fce80be1afad038a0ccd8a41efcf'}]
(Background on this error at: http://sqlalche.me/e/13/gkpj)
and
Traceback (most recent call last):
File "/usr/local/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 138, in dag_trigger
dag_id=args.dag_id, run_id=args.run_id, conf=args.conf, execution_date=args.exec_date
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/client/local_client.py", line 30, in trigger_dag
dag_id=dag_id, run_id=run_id, conf=conf, execution_date=execution_date
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 125, in trigger_dag
replace_microseconds=replace_microseconds,
File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 75, in _trigger_dag
f"A Dag Run already exists for dag id {dag_id} at {execution_date} with run id {run_id}"
airflow.exceptions.DagRunAlreadyExists: A Dag Run already exists for dag id test_dag at 2023-02-16 20:20:23+00:00 with run id test21__2023-02-16T20:20:04+061773
Related
I am doing hyperparameter tuning on GCP using this scikit docker image. When I add the aiplatform package as a dependency, things break. The error comes from the bigquery import.
from google.cloud import bigquery
The error message is below.
The replica workerpool0-0 exited with a non-zero status of 1.
Traceback (most recent call last):
[...]
File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 7, in
from google.cloud import storage, bigquery
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/__init__.py", line 35, in
from google.cloud.bigquery.client import Client
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/client.py", line 60, in
from google.cloud.bigquery import _pandas_helpers
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/_pandas_helpers.py", line 40, in
from google.cloud.bigquery import schema
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery/schema.py", line 19, in
from google.cloud.bigquery_v2 import types
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery_v2/__init__.py", line 23, in
from google.cloud.bigquery_v2 import types
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery_v2/types.py", line 23, in
from google.cloud.bigquery_v2.proto import encryption_config_pb2
File "/usr/local/lib/python3.7/dist-packages/google/cloud/bigquery_v2/proto/encryption_config_pb2.py", line 64, in
file=DESCRIPTOR,
File "/root/.local/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
From the logs, I can see the system is downloading google-cloud-aiplatform v1.17.0. According to the scikit docker image, google-cloud-storage v1.35.0 is installed, but google-cloud-aiplatform drags in v2.5.0.
I am thinking I need to downgrade google-cloud-aiplatform to a specific version. Anyone know which version or how to resolve this problem?
UPDATE: FWIW, if I downgrade google-cloud-aiplatform==1.15.1 then the problem above goes away. However, this problem below shows.
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/.local/lib/python3.7/site-packages/trainer/hpt.py", line 170, in
staging_bucket=f'{args.bucket_uri}'
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/initializer.py", line 138, in init
backing_tensorboard=experiment_tensorboard,
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/metadata/metadata.py", line 235, in set_experiment
experiment_name=experiment, description=description
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/metadata/experiment_resources.py", line 247, in get_or_create
project=project, location=location, credentials=credentials
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/metadata/metadata_store.py", line 283, in ensure_default_metadata_store_exists
encryption_spec_key_name=encryption_key_spec_name,
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/metadata/metadata_store.py", line 123, in get_or_create
credentials=credentials,
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/metadata/metadata_store.py", line 241, in _get
credentials=credentials,
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/metadata/metadata_store.py", line 73, in __init__
self._gca_resource = self._get_gca_resource(resource_name=metadata_store_name)
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/base.py", line 617, in _get_gca_resource
return getattr(self.api_client, self._getter_method)(
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/utils/__init__.py", line 425, in __getattr__
return getattr(self._clients[self._default_version], name)
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform/utils/__init__.py", line 359, in __getattr__
client_info=self._client_info,
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/metadata_service/client.py", line 547, in __init__
api_audience=client_options.api_audience,
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/metadata_service/transports/grpc.py", line 190, in __init__
("grpc.max_receive_message_length", -1),
File "/root/.local/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/metadata_service/transports/grpc.py", line 241, in create_channel
**kwargs,
File "/root/.local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 318, in create_channel
default_host=default_host,
File "/root/.local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 239, in _create_composite_credentials
credentials, scopes=scopes, default_scopes=default_scopes
TypeError: with_scopes_if_required() got an unexpected keyword argument 'default_scopes'
Situation
We are working on the backend of our project consisting of two services written in Django and FastAPI. Both services use MongoDB as their database system.
In Django we use djongo==1.3.0 for ORM compatibility.
In FastAPI we use odmantic==0.3.4.
Both of these libraries use pymongo==3.11.3 underneath.
This MongoDB SRV error has been an issue for our Django service ever since we created it, but we managed to somehow work around it by not using the latest packages, such as:
Django==2.2.20
djongo==1.3.0
pymongo==3.11.3
Recently due to security risks we had to upgrade:
urllib3 from 1.25.8 to 1.26.5
pydantic from 1.8.1 to 1.8.2
Django from 2.2.20 to 2.2.22
Those were suggested by GitHub's dependabot.
Problem
When we run any of these services locally now they break with the following base exception:
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
Full log for Django:
> python manage.py test
Creating test database for alias 'default'...
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/database.py", line 10, in connect
return clients[db]
KeyError: 'djongo_test'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 212, in __init__
rrset = response.find_rrset(response.answer, qname,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/message.py", line 341, in find_rrset
raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 220, in __init__
crrset = response.find_rrset(response.answer,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/message.py", line 341, in find_rrset
raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 72, in _resolve_uri
results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1100, in query
return get_default_resolver().query(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1003, in query
answer = Answer(_qname, rdtype, rdclass, response,
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 232, in __init__
raise NoAnswer(response=response)
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.database-test.avxhw.mongodb.net. IN SRV
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
super().run_from_argv(argv)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/core/management/commands/test.py", line 53, in handle
failures = test_runner.run_tests(test_labels)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/runner.py", line 629, in run_tests
old_config = self.setup_databases(aliases=databases)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/runner.py", line 552, in setup_databases
return _setup_databases(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/test/utils.py", line 170, in setup_databases
connection.creation.create_test_db(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 58, in create_test_db
self._create_test_db(verbosity, autoclobber, keepdb)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/creation.py", line 168, in _create_test_db
with self._nodb_connection.cursor() as cursor:
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/base.py", line 185, in get_new_connection
self.client_connection = Database.connect(db=name, **connection_params)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/djongo/database.py", line 13, in connect
clients[db] = MongoClient(**kwargs, connect=False)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 639, in __init__
res = uri_parser.parse_uri(
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/uri_parser.py", line 500, in parse_uri
nodes = dns_resolver.get_hosts()
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 102, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 83, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/home/bk/inz/backend/authservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 79, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: The DNS response does not contain an answer to the question: _mongodb._tcp.database-test.avxhw.mongodb.net. IN SRV
Full log for FastAPI:
> python -m src.main
Traceback (most recent call last):
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 72, in _resolve_uri
results = resolver.query('_mongodb._tcp.' + self.__fqdn, 'SRV',
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1321, in query
return resolve(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1305, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 1202, in resolve
(answer, done) = resolution.query_result(response, None)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/dns/resolver.py", line 674, in query_result
raise NoAnswer(response=answer.response)
dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/bk/inz/backend/testservice/src/main.py", line 20, in <module>
client = AsyncIOMotorClient(MONGODB_URL)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/motor/core.py", line 156, in __init__
delegate = self.__delegate_class__(*args, **kwargs)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 639, in __init__
res = uri_parser.parse_uri(
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/uri_parser.py", line 500, in parse_uri
nodes = dns_resolver.get_hosts()
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 102, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 83, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "/home/bk/inz/backend/testservice/venv/lib/python3.8/site-packages/pymongo/srv_resolver.py", line 79, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: The DNS response does not contain an answer to the question: _mongodb._tcp.cluster0.k1eh0.mongodb.net. IN SRV
In our CI setup on GitHub Actions Django service tests are still run successfully. It's clear that the problem lies within the DNS aspect.
Questions
Are there any specific DNS servers which do not support MongoDB? I was using Google's 8.8.8.8 before and after it broke.
Why would the problem be caused by upgrading listed dependencies? Only urllib was upgraded for both services.
Is there any workaround?
Review https://github.com/mongodb/specifications/blob/master/source/initial-dns-seedlist-discovery/initial-dns-seedlist-discovery.rst for how SRV URIs work, then manually issue the relevant DNS queries using a tool such as dig to verify your environment has working DNS.
I am trying to setup dynamic thumbnail service thumbor and to support s3 as storage, I need to setup this community powered pip library for aws.
Its working well on my local environment but when I am trying to host it on one of our servers, I am getting NoCredentialsError. I am assuming this is because of difference versions of botocore (latest one and one installed by pip library). Here is error log:
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 774, in get_component
# client config from the session
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 174, in <lambda>
self._components.lazy_register_component(
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 453, in get_data
- agent_version is the value of the `user_agent_version`
File "/usr/local/lib/python2.7/dist-packages/botocore/loaders.py", line 119, in _wrapper
data = func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/loaders.py", line 364, in load_data
DataNotFoundError: Unable to load data for: _endpoints
2016-04-24 12:14:34 tornado.application:ERROR Future exception was never retrieved: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 230, in wrapper
yielded = next(result)
File "/usr/local/lib/python2.7/dist-packages/thumbor/handlers/imaging.py", line 31, in check_image
exists = yield gen.maybe_future(self.context.modules.storage.exists(kw['image'][:self.context.config.MAX_ID_LENGTH]))
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 455, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 443, in wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/storage.py", line 107, in exists
self.storage.get(file_abspath, callback=return_data)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 455, in wrapper
future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 443, in wrapper
result = f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tc_aws/aws/bucket.py", line 44, in get
Key=self._clean_key(path),
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 97, in call
return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 60, in _make_api_call
operation_model=operation_model, request_dict=request_dict, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 54, in _make_request
request_dict=request_dict, operation_model=operation_model, callback=callback)
File "/usr/local/lib/python2.7/dist-packages/tornado_botocore/base.py", line 32, in _send_request
request = self.endpoint.create_request(request_dict, operation_model)
File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 126, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python2.7/dist-packages/botocore/signers.py", line 124, in sign
signer.add_auth(request=request)
File "/usr/local/lib/python2.7/dist-packages/botocore/auth.py", line 626, in add_auth
raise NoCredentialsError
NoCredentialsError: Unable to locate credentials
Could it be fixed with proper ordering in which I install libraries? Because the pip library removes existing newer version of botocore and installs an older version.
EDIT:
I am running processes with supervisor and it seems process cant access aws credentials
EDIT 2:
The issue got resolved with proper configuration of supervisor. The user for process started by supervisor did not have access to config file
The issue got resolved with proper configuration of supervisor. The user for subprocess started by supervisor did not have access to aws config file. So it was working with local environment or creating process separately but not with supervisor.
I am using the flask-sqlalchemy extension with alembic for migrations. When I try to add a new migration file and upgrade the schema to the latest one, I get the following error:
AttributeError: 'int' object has no attribute '_compiler_dispatch'
The content of the migration file:
revision = 'ec2c2d40eb1'
down_revision = '28dda873b826'
from alembic import op
import sqlalchemy as sa
def upgrade():
op.alter_column(
'users',
'wiki_permission',
new_column_name='wiki_group',
nullable=False,
existing_nullable=False,
type_=sa.Integer(),
existing_type=sa.Integer(),
server_default=1,
existing_server_default=1 # Line of error - 27
)
def downgrade():
op.alter_column(
'users',
'wiki_group',
new_column_name='wiki_permission',
nullable=False,
existing_nullable=False,
type_=sa.Integer(),
existing_type=sa.Integer(),
server_default=1,
existing_server_default=1
)
Thanks for taking time to help me.
Edit:
The complete error message :
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.migration] Running upgrade 28dda873b826 -> ec2c2d40eb1, users change column wiki_permission to wiki_group
Traceback (most recent call last):
File "/home/kevin/Code/python/flask/terminus/venv/bin/alembic", line 9, in <module>
load_entry_point('alembic==0.6.5', 'console_scripts', 'alembic')()
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/config.py", line 298, in main
CommandLine(prog=prog).main(argv=argv)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/config.py", line 293, in main
self.run_cmd(cfg, options)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/config.py", line 279, in run_cmd
**dict((k, getattr(options, k)) for k in kwarg)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/command.py", line 125, in upgrade
script.run_env()
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/script.py", line 203, in run_env
util.load_python_file(self.dir, 'env.py')
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/util.py", line 212, in load_python_file
module = load_module_py(module_id, path)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/compat.py", line 58, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "alembic/env.py", line 77, in <module>
run_migrations_online()
File "alembic/env.py", line 70, in run_migrations_online
context.run_migrations()
File "<string>", line 7, in run_migrations
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/environment.py", line 688, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/migration.py", line 258, in run_migrations
change(**kw)
File "alembic/versions/ec2c2d40eb1_users_change_column_wiki_permission_to_.py", line 27, in upgrade
existing_server_default=1,
File "<string>", line 7, in alter_column
File "<string>", line 1, in <lambda>
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/util.py", line 329, in go
return fn(*arg, **kw)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/operations.py", line 317, in alter_column
existing_autoincrement=existing_autoincrement
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/ddl/mysql.py", line 44, in alter_column
else existing_autoincrement
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 76, in _exec
conn.execute(construct, *multiparams, **params)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 720, in execute
return meth(self, multiparams, params)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 67, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 768, in _execute_ddl
compiled = ddl.compile(dialect=dialect)
File "<string>", line 1, in <lambda>
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 468, in compile
return self._compiler(dialect, bind=bind, **kw)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 25, in _compiler
return dialect.ddl_compiler(dialect, self, **kw)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 197, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 220, in process
return obj._compiler_dispatch(self, **kwargs)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/ext/compiler.py", line 410, in <lambda>
lambda *arg, **kw: existing(*arg, **kw))
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/ext/compiler.py", line 448, in __call__
return fn(element, compiler, **kw)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/ddl/mysql.py", line 171, in _mysql_change_column
autoincrement=element.autoincrement
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/ddl/mysql.py", line 190, in _mysql_colspec
spec += " DEFAULT %s" % _render_value(compiler, server_default)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/alembic/ddl/mysql.py", line 179, in _render_value
return compiler.sql_compiler.process(expr)
File "/home/kevin/Code/python/flask/terminus/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py", line 220, in process
return obj._compiler_dispatch(self, **kwargs)
AttributeError: 'int' object has no attribute '_compiler_dispatch'
Alright, I just went through this same issue. I'm not using flask-sqlalchemy, just straight alembic, but it should be identical.
Second, it worked for me with sa.Integer with no parentheses, so I would recommend that.
The alembic docs say:
When producing MySQL-compatible migration files, it is recommended that the existing_type, existing_server_default, and existing_nullable parameters be present, if not being altered.
Since it seems you are not altering these columns, the docs suggest they should be present. So:
Remove nullable, type_, and server_default. They are not being altered.
Keep existing_nullable, existing_type, and existing_server_default.
The tech stack looks like this:
Django frontend and backend
Celery worker queue for asynchronously processing time consuming tasks
Within the past day or two I've noticed a lot of these kinds of stack traces both locally and in production:
[2012-07-05 20:31:01,583: CRITICAL/MainProcess] Task site_endpoint.tasks.async_inbound_message[a950736c-ff93-420c-9fbf-6deb2b88ff4d] INTERNAL ERROR: TypeError('__init__() takes at least 3 arguments (1 given)',)
Traceback (most recent call last):
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 192, in trace_task
R = I.handle_error_state(task, eager=eager)
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 91, in handle_error_state
}[self.state](task, store_errors=store_errors)
File "/projects/site/venv/lib/python2.7/site-packages/celery/execute/trace.py", line 114, in handle_failure
task.backend.mark_as_failure(req.id, exc, self.strtb)
File "/projects/site/venv/lib/python2.7/site-packages/celery/backends/base.py", line 96, in mark_as_failure
traceback=traceback)
File "/projects/site/venv/lib/python2.7/site-packages/celery/backends/base.py", line 229, in store_result
return self._store_result(task_id, result, status, traceback, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/backends/database.py", line 26, in _store_result
traceback=traceback)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 40, in _inner
return fun(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 164, in store_result
"traceback": traceback})
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 82, in update_or_create
return self.get_query_set().update_or_create(**kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/djcelery/managers.py", line 66, in update_or_create
obj, created = self.get_or_create(**kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/query.py", line 385, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 28, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 28, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 276, in get_db_prep_save
return self.get_db_prep_value(value, connection=connection, prepared=False)
File "/projects/site/venv/lib/python2.7/site-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/projects/site/venv/lib/python2.7/site-packages/picklefield/fields.py", line 154, in get_db_prep_value
value = force_unicode(dbsafe_encode(value, self.compress, self.protocol))
File "/projects/site/venv/lib/python2.7/site-packages/picklefield/fields.py", line 57, in dbsafe_encode
value = b64encode(dumps(deepcopy(value), pickle_protocol))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 328, in _reconstruct
args = deepcopy(args, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
TypeError: __init__() takes at least 3 arguments (1 given)
Looking at the database calls when this happens, I can see the following queries being executed:
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x4
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x5
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: SELECT "celery_taskmeta"."id", "celery_taskmeta"."task_id", "celery_taskmeta"."status", "celery_taskmeta"."result", "celery_taskmeta"."date_done", "celery_taskmeta"."traceback" FROM "celery_taskmeta" WHERE "celery_taskmeta"."task_id" = 'a950736c-ff93-420c-9fbf-6deb2b88ff4d'
LOG: statement: SAVEPOINT s140735259576672_x6
I am having a tough time understanding what the source of this invalid object initialization is. Anyone have ideas?
It looks like you are using new-relic remote performance monitoring/forensic analysis. I have seen this kind of problem before when old versions of the library are being used. I recommend checking to make sure you are using the latest version of their client library.
This happened to me and I solved it by reading Python multiprocessing pool hangs at join? and http://bugs.python.org/issue9400.
The problem:
Celery somehow pickles the exceptions it gets and sends them somewhere (to the database/broker, I guess). So, if, at any time, a Celery task raises an exception that isn't pickleable, this bug will happen.
The solution:
You must ensure you're handling all the bizarre exceptions your Celery tasks code might be raising. The Traceback can give you hints, but it may not be very exact about everything.
If you really don't know where are your exceptions coming from, try putting your tasks code inside try...except: pass blocks.