I have a Flask app running on Azure with a connection to an Azure SQL db and after the below developments I can't access the database any longer.
Here is what I recently added :
created a .env file to secure the connection string
added a .gitignore
added the following to my main.py:
connection_string = textwrap.dedent(f'''
Driver={driver};
Server={os.getenv("SERVER")};
Database={os.getenv("DB_NAME")};
Uid={os.getenv("DB_USER")};
Pwd={os.getenv("PASSWORD")};
Encrypt=yes;
TrustServerCertificate=no;
Connection Timeout=30;
''')
I've also added the following to run in production :
if __name__ == '__main__':
from waitress import serve
serve(app, host="0.0.0.0", port=8080)
It works perfectly locally but as soon as I deploy it on Azure, I have issues with the docker deployment in the background. It seems the connection strings params are not recognized from the following error (from Kudu) :
Warning
2021-10-15T12:42:51.4960727
[2021-10-15 12:42:51 +0000] [40] [ERROR] Exception in worker process
Ok
2021-10-15T12:42:51.4961108
Traceback (most recent call last):
Ok
2021-10-15T12:42:51.4961177
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
Ok
2021-10-15T12:42:51.4961354
worker.init_process()
Ok
2021-10-15T12:42:51.4961403
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
Ok
2021-10-15T12:42:51.4961451
self.load_wsgi()
Ok
2021-10-15T12:42:51.4961495
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
Ok
2021-10-15T12:42:51.4961542
self.wsgi = self.app.wsgi()
Ok
2021-10-15T12:42:51.4961586
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
Ok
2021-10-15T12:42:51.4961638
self.callable = self.load()
Ok
2021-10-15T12:42:51.4961682
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
Ok
2021-10-15T12:42:51.4961729
return self.load_wsgiapp()
Ok
2021-10-15T12:42:51.4961772
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
Ok
2021-10-15T12:42:51.496182
return util.import_app(self.app_uri)
Ok
2021-10-15T12:42:51.4961864
File "/tmp/8d98fd8a8763182/antenv/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
Ok
2021-10-15T12:42:51.496191
mod = importlib.import_module(module)
Ok
2021-10-15T12:42:51.4961956
File "/opt/python/3.8.6/lib/python3.8/importlib/__init__.py", line 127, in import_module
Ok
2021-10-15T12:42:51.4962002
return _bootstrap._gcd_import(name[level:], package, level)
Ok
2021-10-15T12:42:51.496308
File "", line 1014, in _gcd_import
Ok
2021-10-15T12:42:51.496315
File "", line 991, in _find_and_load
Ok
2021-10-15T12:42:51.4963197
File "", line 975, in _find_and_load_unlocked
Ok
2021-10-15T12:42:51.4963244
File "", line 671, in _load_unlocked
Ok
2021-10-15T12:42:51.496329
File "", line 783, in exec_module
Ok
2021-10-15T12:42:51.4963337
File "", line 219, in _call_with_frames_removed
Ok
2021-10-15T12:42:51.4963383
File "/tmp/8d98fd8a8763182/main.py", line 104, in
Ok
2021-10-15T12:42:51.4965707
con, cur = AZconnect()
Ok
2021-10-15T12:42:51.496577
File "/tmp/8d98fd8a8763182/main.py", line 91, in AZconnect
Ok
2021-10-15T12:42:51.4965817
con: pyodbc.Connection = pyodbc.connect(connection_string)
Warning
2021-10-15T12:42:51.4965863
pyodbc.OperationalError: ('HYT00', '[HYT00] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0) (SQLDriverConnect)')
I'm quite sure I did something wrong with the .env and load_dotenv() but I don't see where.
I understood my mistake ; .gitignore prenvented to use the .env file as it should hence the environment variable couldn't be set.
I then set these variable in Azure application settings and it worked just fine :
Related
I'm deploying Django in Google App Engine.
I get 502 Bad Gateway and in the log I get the following error:
2021-03-08 12:08:18 default[20210308t130512] Traceback (most recent call last): File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/workers/gthread.py", line 92, in init_process super().init_process() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/workers/base.py", line 119, in init_process self.load_wsgi() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi self.wsgi = self.app.wsgi() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 49, in load return self.load_wsgiapp() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp return util.import_app(self.app_uri) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/gunicorn/util.py", line 358, in import_app mod = importlib.import_module(module) File "/opt/python3.9/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 790, in exec_module File "", line 228, in _call_with_frames_removed File "/srv/main.py", line 1, in from django_project.wsgi import application File "/srv/django_project/wsgi.py", line 16, in application = get_wsgi_application() File "/layers/google.python.pip/pip/lib/python3.9/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/django/init.py", line 19, in setup configure_logging(settings.LOGGING_CONFIG, settings.LOGGING) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/django/conf/init.py", line 82, in getattr self._setup(name) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/django/conf/init.py", line 69, in _setup self._wrapped = Settings(settings_module) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/django/conf/init.py", line 170, in init mod = importlib.import_module(self.SETTINGS_MODULE) File "/opt/python3.9/lib/python3.9/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/srv/django_project/settings.py", line 84, in import pymysql # noqa: 402 ModuleNotFoundError: No module named 'pymysql'
The problem is that I already installed pymysql, in fact if I run pip3 install pymysql, I get Requirement already satisfied: ...
Why is that?
Thanks in advance!
Edit:
Here's requirements.txt:
asgiref==3.3.1
attrs==20.3.0
Automat==20.2.0
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
constantly==15.1.0
cryptography==3.4.1
cssselect==1.1.0
Django==3.1.6
django-phonenumber-field==5.0.0
django-widget-tweaks==1.4.8
hyperlink==21.0.0
idna==2.10
incremental==17.5.0
instaloader==4.6.1
itemadapter==0.2.0
itemloaders==1.0.4
jmespath==0.10.0
jsonfield==3.1.0
lxml==4.6.2
parsel==1.6.0
phonenumberslite==8.12.18
progress==1.5
Protego==0.1.16
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
PyDispatcher==2.0.5
PyHamcrest==2.0.2
pyOpenSSL==20.0.1
pytz==2021.1
queuelib==1.5.0
requests==2.25.1
schedule==0.6.0
Scrapy==2.4.1
semantic-version==2.8.5
service-identity==18.1.0
setuptools-rust==0.11.6
six==1.15.0
sqlparse==0.4.1
toml==0.10.2
urllib3==1.26.3
w3lib==1.22.0
zope.interface==5.2.0
If you run pip3 install pymysql in your local computer, this does not mean that when you deploy the app this module is packaged. In fact GAE attempts to install everything at build time using your requirements.txt file so it doesn't matter if you installed everything in your PC since GAE will not use what you have in local (talking about packages installed with pip).
Checking your requirements.txt file I do not see that the package PyMySQL is added. You should add it to that file and attempt to deploy again.
"pymysql" package is missing (not installed) according to the message below from your error:
ModuleNotFoundError: No module named 'pymysql'
So, you need to run this command below to install "pymysql":
pip install pymysql
On my first Django project, I got started with a sqlite database, and now I'm switching to Postgres. The first part of the code that I wrote and tested was an import management command, to read records in from a file and create database records for them, and that's been working great. So to start making the change, I've updated the default database info in settings.py, and I ran migrations for it like so:
python3 manage.py migrate --database=default
I can see the correct table structure when I view the new database in pgAdmin 4 (just with no rows populated yet)- and the field max values appear to all match what's in the model- hooray! I also added some code to get the ENV_ROLE and database password from environment variables, as recommended here: https://ultimatedjango.com/learn-django/lessons/handling-sensitive-keys/
Next I run my import management command, and the problem: I can run this successfully from the terminal, but when I try running it from within PyCharm I'm getting errors I never saw while running it using the sqlite database:
No manage.py file specified in Settings->Django Support
/usr/local/bin/python3.8 /Library/Frameworks/Python.framework/Versions/3.8/bin/django-admin.py importgedcom Test_tree.ged
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/core/management/__init__.py", line 224, in fetch_command
app_name = commands[subcommand]
KeyError: 'importgedcom'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/django-admin.py", line 5, in <module>
management.execute_from_command_line()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/core/management/__init__.py", line 231, in fetch_command
settings.INSTALLED_APPS
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/conf/__init__.py", line 76, in __getattr__
self._setup(name)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/conf/__init__.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/django/conf/__init__.py", line 142, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mysite.settings'
Process finished with exit code 1
I see a few interesting bits in there, likely multiple/different issues in my setup running from PyCharm vs from the terminal:
No manage.py file specified in Settings->Django Support. My configuration for the management command has always included this: DJANGO_SETTINGS_MODULE=mysite.settings; I did make one update to the configuration, just to add ENV_ROLE=development
ModuleNotFoundError: No module named 'mysite.settings'. I didn't rename the project or app, only updated settings.py (default database and environment logic)
UPDATE: I can run this to completion while stepping through in debug mode, I only see the error when I hit PyCharm's green play button to run it.
These errors happen before the code even does anything (I don't make it to print statements in the add_arguments or handle methods), so there's some problem with the setup. Any advice on what to check/do next, or is there more info I can give that would help figure out next steps?
After running PREFIX=/home/hue make install
Towards the end of my hue install via tarball, I get the following error. I'm not sure what is happening. I have the prerequisites installed as well. Using this guide
Command that is failing: /home/hue/hue/build/env/bin/hue makemigrations --noinput
Error:
Traceback (most recent call last):
File "/home/hue/hue/build/env/bin/hue", line 11, in <module>
load_entry_point('desktop', 'console_scripts', 'hue')()
File "/home/hue/hue/desktop/core/src/desktop/manage_entry.py", line 216, in entry
execute_from_command_line(sys.argv)
File "/home/hue/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/home/hue/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/management/__init__.py", line 307, in execute
settings.INSTALLED_APPS
File "/home/hue/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/home/hue/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/conf/__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "/home/hue/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/conf/__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/hue/hue/desktop/core/src/desktop/settings.py", line 271, in <module>
conf.initialize(_desktop_conf_modules, _config_dir)
File "/home/hue/hue/desktop/core/src/desktop/lib/conf.py", line 598, in initialize
conf_data = load_confs(_configs_from_dir(config_dir))
File "/home/hue/hue/desktop/core/src/desktop/lib/conf.py", line 530, in load_confs
for in_conf in conf_source:
File "/home/hue/hue/desktop/core/src/desktop/lib/conf.py", line 511, in _configs_from_dir
conf = ConfigObj(os.path.join(conf_dir, filename))
File "/home/hue/hue/build/env/lib/python2.7/site-packages/configobj-5.0.6-py2.7.egg/configobj.py", line 1229, in __init__
self._load(infile, configspec)
File "/home/hue/hue/build/env/lib/python2.7/site-packages/configobj-5.0.6-py2.7.egg/configobj.py", line 1318, in _load
raise error
configobj.ConfigObjError: Parsing failed with several errors.
First error at line 1046.
In case anyone else runs into this. So the issue stemmed from modifying the hue.ini file before running the install. Rolling back to the original hue.ini resolved the issue. I must have had a typo in the modified version, thus causing the parse error above.
Tarball File location:
hue/desktop/conf/hue.ini
I am seeing a very weired error while debugging my code in PyCharm. The offending lines that cause the exception are
a = db.links.list_indexes()
or the following:
db.links.create_index("created", expireAfterSeconds=settings.EXPIRY_PERIOD, background=True)
My code runs normally inside PyCharm if I simply do run tests, while trying to debug it causes the following error:
======================================================================
ERROR: test_detail_view (sharescreening.tests.TestIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
File "sharescreening/tests.py", line 53, in test_detail_view
response = self.client.get('/')
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/test/client.py", line 500, in get
**extra)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/test/client.py", line 303, in get
return self.generic('GET', path, secure=secure, **r)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/test/client.py", line 379, in generic
return self.request(**r)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/test/client.py", line 466, in request
six.reraise(*exc_info)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 119, in get_response
resolver_match = resolver.resolve(request.path_info)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 365, in resolve
for pattern in self.url_patterns:
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 401, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 395, in urlconf_module
self._urlconf_module = import_module(self.urlconf_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "sharescreening/urls.py", line 17, in <module>
from .views import put_links, index, get_shares, ShowDetails
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/gevent/builtins.py", line 93, in __import__
result = _import(*args, **kwargs)
File "sharescreening/views.py", line 22, in <module>
a = db.links.list_indexes()
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/pymongo/collection.py", line 1269, in list_indexes
with self._socket_for_primary_reads() as (sock_info, slave_ok):
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 699, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 663, in _get_socket
server = self._get_topology().select_server(selector)
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/pymongo/topology.py", line 121, in select_server
address))
File "/home/oznt/.virtualenvs/screenshares/local/lib/python2.7/site-packages/pymongo/topology.py", line 97, in select_servers
self._error_message(selector))
ServerSelectionTimeoutError: No servers found yet
----------------------------------------------------------------------
Ran 4 tests in 32.173s
FAILED (errors=1)
I have no clue why the tests pass in the command line, and in simple running mode but not in debug mode. Can someone shed some light on this problem here?
It turns out that the issue was really a problem with the combination or using gevent with the debugger.
If you this problem, it is resolved by upgrading your pycharm version to 2006.1.
More details can be found in the issue in PYCharm tracker
$ python manage.py run_gunicorn 0.0.0.0:80 --settings=project_name.settings.production
<- It's run, OK.
but,
$ gunicorn_django -b 0.0.0.0:80 project_name/settings/production.py
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 473, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 100, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 106, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/djangoapp.py", line 102, in load
return mod.make_wsgi_application()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/django_wsgi.py", line 36, in make_wsgi_application
if get_validation_errors(s):
File "/usr/local/lib/python2.7/dist-packages/django/core/management/validation.py", line 35, in get_validation_errors
for (app_name, error) in get_app_errors().items():
File "/usr/local/lib/python2.7/dist-packages/django/db/models/loading.py", line 166, in get_app_errors
self._populate()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/loading.py", line 72, in _populate
self.load_app(app_name, True)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/loading.py", line 94, in load_app
app_module = import_module(app_name)
File "/usr/local/lib/python2.7/dist-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
> ImportError: No module named accounts
2013-06-29 01:56:53 [30859] [INFO] Worker exiting (pid: 30859)
2013-06-29 01:56:53 [30854] [INFO] Shutting down: Master
2013-06-29 01:56:53 [30854] [INFO] Reason: Worker failed to boot.
How can I solve this problem?
I want to run on daemon.
Thanks.
This looks like an issue related to your PYTHONPATH. Try adding the path that contains the accounts module to your python path, using gunicorn's --pythonpath.
If you have newer gunicorn/Django versions using gunicorn via management command should be the preffered way - and it should be more safe regarding issues like this.
Try running your django app using gunicorn instead of django_gunicorn:
$ gunicorn mydjangoproject.wsgi:application
There is also an alternative solution to fix the gunicorn_django script, see here. I ran into the issue when upgrading from gunicorn 18.0 to 19.1.1. Cheers!