Gunicorn & Undetected chromedriver not running as a service - flask

When running my Flask application through Flask normally, I encounter no issues and the "undetected_chromedriver" runs smoothly. However, to ensure that my Flask application runs continuously, I have created a system service to run it at startup. When I press a button within the Flask application to scrape data from a website, the "undetected_chromedriver" crashes abruptly. The following is the traceback I receive:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/abel/LeadPartner/app/includes/dependencies.py", line 63, in run_subprocess
subprocess.Popen(target_function(*args, **kwargs))
File "/home/abel/LeadPartner/app/scripts/scheduler/verstuur_project.py", line 15, in verstuur_project
driver = get_chromedriver(email=gebruiker_gegevens.email, headless=True)
File "/home/abel/LeadPartner/app/includes/webdriver.py", line 73, in get_chromedriver
driver = webdriver.Chrome( options=chrome_options, seleniumwire_options=proxy_options, version_main=110 )
File "/home/abel/LeadPartner/env/lib/python3.10/site- packages/seleniumwire/undetected_chromedriver/webdriver.py", line 61, in __init__
super().__init__(*args, **kwargs)
File "/home/abel/LeadPartner/env/lib/python3.10/site-packages/undetected_chromedriver/__init__.py", line 411, in __init__
browser = subprocess.Popen(
File "/usr/lib/python3.10/subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1720, in _execute_child
and os.path.dirname(executable)
File "/usr/lib/python3.10/posixpath.py", line 152, in dirname
p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
The command I use to run gunicorn through the terminal (which works) is: gunicorn --workers 3 --bind unix:leadpartner.sock --reload --preload -t 0 wsgi:app
And the service looks like this:
Unit]
Description=Gunicorn instance to serve Lead Partner
After=network.target
[Service]
User=abel
Group=www-data
WorkingDirectory=/home/abel/LeadPartner
Environment="PATH=/home/abel/LeadPartner/env/bin"
ExecStart=/home/abel/LeadPartner/env/bin/gunicorn --workers 3 --bind unix:leadpartner.sock -m 007 --error-logfile /home/abel/lp_error.log --capture-output --reload --preload -t 0 wsgi:app
[Install]
WantedBy=multi-user.target
The flask app itself runs fine with this service, only the chromedriver doesn’t.
I am having difficulty understanding the issue at hand. The only information I could find is that the error might be related to "undetected_chromedriver" not being able to locate the path of Google Chrome. However, when I run the Flask application through gunicorn in the terminal, it works perfectly fine. Upon examining the "undetected_chromedriver" initialization file, I discovered that it looks for "google-chrome," which is the name of the Google Chrome package I am using. I believe this cannot be the issue.
I hope someone here can help me fix this error.
Thanks in advance!
EDIT:
I've come to find out that if i add:
chrome_options.binary_location = '/usr/bin/google-chrome-stable'
Then the script goes a little further. I am now getting another error:
unknown error: cannot connect to chrome at 127.0.0.1:55843 from chrome not reachable Stacktrace: #0 0x55a3e9b2ad93 #1 0x55a3e98f915d #2 0x55a3e98e7a2b #3 0x55a3e9925c63 #4 0x55a3e991d2fc #5 0x55a3e99624f4 #6 0x55a3e9959353 #7 0x55a3e9928e40 #8 0x55a3e992a038 #9 0x55a3e9b7e8be #10 0x55a3e9b828f0 #11 0x55a3e9b62f90 #12 0x55a3e9b83b7d #13 0x55a3e9b54578 #14 0x55a3e9ba8348 #15 0x55a3e9ba84d6 #16 0x55a3e9bc2341 #17 0x7f05da868b43
Attempts
'luckily' for this error, i can at least find something on google to help me.
There are three different posts/articles created about this error. So of course, i've tried all of them. But nothing works unfortunately.
The posts are:
First post
Second post
Third post

Related

How can I send E-Mails through python django frequently with a cronjob?

I try to run a function in django every 5 minutes. For this I user the django-crontab package. The function which should run, checks for some conditions in the database and if they are met, sends an e-mail to the user of the app.
I have django 4 running on my linux ubuntu 20.04 server. I added the cronjob via python3 manage.py crontab add (in activated virtual environment). But then I wondered why the cronjob is not running. I tried to execute the job by hand and it worked.
I think the problem boils down to this: When I'm in the activated virtual environment and run the crontab with "python3 manage.py crontab run " it works. But when I run it outside of the virtual environment I get the following error:
Failed to complete cronjob at ('*/5 * * * *', 'evaluation_tool.scripts.cron.send_mail_if_classeval_ended')
Traceback (most recent call last):
File "/var/www/amadeus/lib/python3.10/site-packages/django_crontab/crontab.py", line 145, in run_job
func(*job_args, **job_kwargs)
File "/var/www/amadeus/evaluation_tool/scripts/cron.py", line 12, in send_mail_if_classeval_ended
send_mail_time_over_class_evaluation(class_evaluation=class_evaluation.pk,
File "/var/www/amadeus/evaluation_tool/scripts/email_handler.py", line 122, in send_mail_time_over_class_evaluation
send_falko_mail("AMADEUS Evaluation abgeschlossen", message, to_email_address)
File "/var/www/amadeus/evaluation_tool/scripts/email_handler.py", line 34, in send_falko_mail
msg.send(fail_silently=False)
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/message.py", line 298, in send
return self.get_connection(fail_silently).send_messages([self])
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 124, in send_messages
new_conn_created = self.open()
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 87, in open
self.connection.starttls(
File "/usr/lib/python3.10/smtplib.py", line 769, in starttls
self.ehlo_or_helo_if_needed()
File "/usr/lib/python3.10/smtplib.py", line 611, in ehlo_or_helo_if_needed
if not (200 <= self.ehlo()[0] <= 299):
File "/usr/lib/python3.10/smtplib.py", line 451, in ehlo
self.putcmd(self.ehlo_msg, name or self.local_hostname)
File "/usr/lib/python3.10/smtplib.py", line 378, in putcmd
self.send(f'{s}{CRLF}')
File "/usr/lib/python3.10/smtplib.py", line 365, in send
raise SMTPServerDisconnected('please run connect() first')
smtplib.SMTPServerDisconnected: please run connect() first
Why is the connection to the SMTP Server working in the activated virtual environment but not outside, which is the desired way using a cronjob?
I found the problem.
I set certain environment variables for SMTP Host, User and Password. But crontab cannot access them. So there are two possibilities:
Add them directly to the head of cron file via crontab -e
EMAIL_HOST=XXX
EMAIL_HOST_USER=XXX
EMAIL_PASS=XXX
Add them via a django-crontab argument in settings.py:
CRONTAB_COMMAND_PREFIX = 'EMAIL_HOST=XXX\nEMAIL_HOST_USER=XXX\nEMAIL_PASS=XXX'

Gunicorn workers throwing obscure error in log file

A virtual machine hosting my Django web application has recently been acting erratically (set up with gunicorn + nginx reverse proxy). Erratic being the server becomes unavailable for many minutes under high load (~85% CPU utilization). I looked into var/log/upstart/gunicorn.log, and I see the following at some parts of the log. Can anyone tell me what this is?
From cffi callback From cffi callback <function _osrandom_rand_bytes at 0x7fe2dd2b9b18>:
Traceback (most recent call last):
File "/home/myuser/.allenvs/myenv/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 66, in _osra$
<function _osrandom_rand_bytes at 0x7fe2dcc079b0> #ffi_callback("int (*)(unsigned char *, int)",
File "/home/myuser/.allenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 159, in handle_abort
:
Traceback (most recent call last):
File "/home/myuser/.allenvs/myenv/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 66, in _$
sys.exit(1)
SystemExit: 1
#ffi_callback("int (*)(unsigned char *, int)",
File "/home/myuser/.allenvs/myenv/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 159, in handle_abort
sys.exit(1)
SystemExit: 1
I can't find much documentation about this error. Thanks in advance.
The contents of /etc/init/gunicorn.conf are:
description "Gunicorn application server handling myproject"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
setuid myuser
setgid www-data
chdir /home/myuser/projfolder/myproject/
exec /home/myuser/..allenvs/myenv/bin/gunicorn --chdir=/home/myuser/projfolder/ -w 17 --bind unix:/home/myuser/projfolder/myproject/myproject.sock -e ON_AZURE=1 -e HTTPS=on -e awsaccesskeyid=<some_id> -e awssecretkey=<some_key> myproject.wsgi:application

Postgres to Ubuntu Docker container linking not working

[EDIT]: I'm able to reproduce this without any of my own custom code. I just created a fresh Linode image running Ubuntu 14.04 and installed Docker according to the steps on Docker's website.
I then ran:
docker run -d --name db postgres
and can see it running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9d335e8fc70b postgres "/docker-entrypoint. 7 minutes ago Up 7 minutes 5432/tcp db
Then I start an interactive Ubuntu container, linked to db, and try to test the link via nc and curl:
$docker run -it --link db ubuntu /bin/bash
root#eb02f4e7b89e:/# apt-get install curl
...
root#eb02f4e7b89e:/# curl http://$DB_PORT_5432_TCP_ADDR:$DB_PORT_5432_TCP_PORT/
curl: (7) Failed to connect to 172.17.0.2 port 5432: Connection timed out
What am I missing?
I'm trying to link a postgres container with an app container running Django but it doesn't seem to be linking properly.
The commands to start the containers were:
$ docker run -d --name db postgres
$ docker run -d --name web --link db -p 8000:80 test_image
Both containers appear to be running fine:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5838047eb14c test_image "/test/.docker/st 40 minutes ago Up 40 minutes 0.0.0.0:8000->80/tcp web
d2d6754430a2 postgres "/docker-entrypoint. 44 minutes ago Up 44 minutes 5432/tcp db
And seem to be correctly linked:
$ docker inspect -f "{{ .HostConfig.Links }}" web
[/db:/web/db]
However, when I try to run "python manage.py migrate" in the web container it doesn't seem to be able to connect to the postgres container:
# python manage.py migrate
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 390, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 93, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 19, in __init__
self.loader = MigrationLoader(self.connection)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 47, in __init__
self.build_graph()
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 180, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 59, in applied_migrations
self.ensure_schema()
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 49, in ensure_schema
if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
cursor = self.make_cursor(self._cursor())
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 172, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: could not connect to server: Connection timed out
Is the server running on host "db" (172.17.0.42) and accepting
TCP/IP connections on port 5432?
I'm able to connect to the postgres container directly:
$ docker exec -it db bash
root#d2d6754430a2:/# cat /etc/hosts
172.17.0.42 d2d6754430a2
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
root#d2d6754430a2:/# su postgres
$ psql
psql (9.4.4)
Type "help" for help.
postgres=#
But not from inside the web container:
# curl http://$DB_PORT_5432_TCP_ADDR:$DB_PORT_5432_TCP_PORT/
curl: (7) Failed to connect to 172.17.0.42 port 5432: Connection timed out
My host machine is running Ubuntu 14.04.
Any ideas what I'm missing?
Check that your postgresql.conf has listen_addresses = '*'
The default is listen_addresses = 'localhost', which would behave in the manner you describe.
Edit:
This works for me, does it work for you?
$ docker pull postgres
$ docker pull django
$ docker run -d --name db -d postgres
$ docker run -it --link db:db django bash
root#11c767bd3d09:/# psql -h db -U postgres
psql (9.4.3, server 9.4.4)
Type "help" for help.
postgres=#
Edit (Docker 1.7.1 output)
$ docker rm $(docker ps -a -q)
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
django latest 29755dd6751b 3 days ago 434.5 MB
postgres latest f33438ff9aef 3 days ago 265.5 MB
$ docker -v
Docker version 1.7.1, build 786b29d
$ docker run -d --name db -d postgres
$ docker run -it --link db:db django bash
root#11c767bd3d09:/# psql -h db -U postgres
psql (9.4.3, server 9.4.4)
Type "help" for help.
postgres=#

Difference running python in PyCharm and in terminal

I'm wondering what is the difference when I run a python program from PyCharm or from the command line.
Actually I'm using a library called wand-py (ImageMagick binding).
If I run my program from the command line it works.
Though if I use PyCharm Run or debug it doesn't and I get the following traceback.
/Users/alexisbenoist/Documents/python/papyrus/env/bin/python "/Applications/PyCharm CE.app/helpers/pydev/pydevd.py" --multiproc --client 127.0.0.1 --port 58993 --file /Users/alexisbenoist/Documents/python/papyrus/tets.py
Connected to pydev debugger (build 135.973)
pydev debugger: process 73166 is connecting
Traceback (most recent call last):
File "/Applications/PyCharm CE.app/helpers/pydev/pydevd.py", line 1733, in <module>
debugger.run(setup['file'], None, None)
File "/Applications/PyCharm CE.app/helpers/pydev/pydevd.py", line 1226, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Users/alexisbenoist/Documents/python/papyrus/tets.py", line 23, in <module>
blob = image_to_blob(PATH)
File "/Users/alexisbenoist/Documents/python/papyrus/tets.py", line 12, in image_to_blob
pdf.alpha_channel = False
File "/Users/alexisbenoist/Documents/python/papyrus/env/lib/python2.7/site-packages/wand/image.py", line 419, in wrapped
result = function(self, *args, **kwargs)
File "/Users/alexisbenoist/Documents/python/papyrus/env/lib/python2.7/site-packages/wand/image.py", line 992, in alpha_channel
self.raise_exception()
File "/Users/alexisbenoist/Documents/python/papyrus/env/lib/python2.7/site-packages/wand/resource.py", line 218, in raise_exception
raise e
wand.exceptions.WandError: wand contains no images `MagickWand-1' # error/magick-image.c/MagickSetImageAlphaChannel/9504
I'm using the same virtual env in the terminal and PyCharm.
Do you guys know what could cause the problem?
Thanks,
Alexis.

database error on upgrading the site

I have installed the reviewboard in virtual environment and have created site reviewboard on ubuntu 13.10 using apache2 with mod_wsgi.
when i hit the
http://localhost
i was asked to upgarde the site
The version of Review Board running does not match the version the
site was last upgraded to. You are running 1.7.20 and the site was
last upgraded to 1.7.21.
Please upgrade your site to fix this by running:
and on upgraded
$ rb-site upgrade /home/naveen/venv-review/reviewboard
got
i got following message:
Updating database. This may take a while.
The log output below, including warnings and errors, can be ignored
unless upgrade fails.
------------------ ------------------ Creating tables ... Installing custom SQL ... Installing indexes ... Installed
0 object(s) from 0 fixture(s) No evolution required.
------------------- -------------------
Resetting in-database caches. Traceback (most recent call last):
File "/home/naveen/venv-review/bin/rb-site", line 9, in
load_entry_point('ReviewBoard==1.7.21', 'console_scripts', 'rb-site')() File
"/home/naveen/venv-review/local/lib/python2.7/site-packages/ReviewBoard-1.7.21-py2.7.egg/reviewboard/cmdline/rbsite.py", line 2027, in main
command.run() File "/home/naveen/venv-review/local/lib/python2.7/site-packages/ReviewBoard-1.7.21-py2.7.egg/reviewboard/cmdline/rbsite.py", line 1860, in run
site.run_manage_command("fixreviewcounts") File "/home/naveen/venv-review/local/lib/python2.7/site-packages/ReviewBoard-1.7.21-py2.7.egg/reviewboard/cmdline/rbsite.py", line 562, in run_manage_command
execute_manager(reviewboard.settings, [file, cmd] + params) File
"/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/core/management/init.py", line 459, in execute_manager
utility.execute() File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/core/management/init.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/core/management/base.py",
line 196, in run_from_argv
self.execute(*args, **options.dict) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/core/management/base.py",
line 232, in execute
output = self.handle(*args, **options) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/core/management/base.py",
line 371, in handle
return self.handle_noargs(**options) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/ReviewBoard-1.7.21-py2.7.egg/reviewboard/reviews/management/commands/fixreviewcounts.py",
line 16, in handle_noargs
starred_public_request_count=None) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/db/models/manager.py",
line 185, in update
return self.get_query_set().update(*args, **kwargs) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/db/models/query.py",
line 536, in update
rows = query.get_compiler(self.db).execute_sql(None) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/db/models/sql/compiler.py",
line 988, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type) File
"/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/db/models/sql/compiler.py",
line 818, in execute_sql
cursor.execute(sql, params) File "/home/naveen/venv-review/local/lib/python2.7/site-packages/Django-1.4.10-py2.7.egg/django/db/backends/sqlite3/base.py", line 344, in execute
return Database.Cursor.execute(self, query, params) django.db.utils.DatabaseError: attempt to write a readonly database
the sqlite database directory has been given access to server (www-data) like :
sudo chown www-data /home/naveen/venv-review/reviewboard/data
Does www-data should have read/write access of site , reviewboard, if it is the case ?
Update *
checking apache2 user
ps aux | grep apache2
root 1146 0.0 0.0 84208 3928 ? Ss 11:19 0:00 /usr/sbin/apache2 -k start
www-data 3632 0.1 0.6 698464 51056 ? Sl 11:34 0:01 /usr/sbin/apache2 -k start
www-data 3742 0.1 0.6 698480 51032 ? Sl 11:34 0:01 /usr/sbin/apache2 -k start
naveen 7162 0.0 0.0 13652 940 pts/1 S+ 11:53 0:00 grep apache2
The sqlite3 file, not only the directory, have to be owned and have to be writeable by the user running the web server.
so you may try running, from a terminal shell:
sudo chown -R www-data /home/naveen/venv-review/reviewboard/data
sudo chmod -R u+w /home/naveen/venv-review/reviewboard/data/*
the first command recursively change the owner of the files inside the folder, the second one ensures the users have writing permissions on these files.