fabric deploy problem - django

I'm trying to deploy a django app with fabric and get the following
error:
Alexs-MacBook:fabric alex$ fab config:instance=peergw deploy -H <ip> -
u <username> -p <password>
[192.168.2.93] run: cat /etc/issue
Traceback (most recent call last):
File "build/bdist.macosx-10.6-universal/egg/fabric/main.py", line
419, in main
File "/Users/alex/Rabota/server/mx30/scripts/fabric/fab/
commands.py", line 37, in deploy
checkup()
File "/Users/alex/Rabota/server/mx30/scripts/fabric/fab/
commands.py", line 140, in checkup
if not 'Ubuntu' in run('cat /etc/issue'):
File "build/bdist.macosx-10.6-universal/egg/fabric/network.py", line
382, in host_prompting_wrapper
File "build/bdist.macosx-10.6-universal/egg/fabric/operations.py",
line 414, in run
File "build/bdist.macosx-10.6-universal/egg/fabric/network.py", line
65, in __getitem__
File "build/bdist.macosx-10.6-universal/egg/fabric/network.py", line
140, in connect
File "build/bdist.macosx-10.6-universal/egg/paramiko/client.py",
line 149, in load_system_host_keys
File "build/bdist.macosx-10.6-universal/egg/paramiko/hostkeys.py",
line 154, in load
File "build/bdist.macosx-10.6-universal/egg/paramiko/hostkeys.py",
line 66, in from_line
File "build/bdist.macosx-10.6-universal/egg/paramiko/rsakey.py",
line 61, in __init__
paramiko.SSHException: Invalid key
Alexs-MacBook:fabric alex$
I can't connect to the server via ssh. What can be my problem?
Regards, Arshavski Alexander.

Going out on a limb here, I'd say your SSH key is incorrect:
paramiko.SSHException: Invalid key
What does your servers say when you try to ssh into it, using the username and password you were providing to fabric?
On second thought: as you are providing fabric with a password, that might suggest your SSH host key has changed and / or has not yet been added to ~/.ssh/known_hosts.

Yeah, I'd say that the host key on the machine you're connecting to has changed. (Or you are connecting from a machine that never went through the "xxx is an unknown host, do you want to add it to the list of known hosts?" dialogue.)
If you are not concerned about man-in-the-middle attacks or have changed the key yourself a few days ago add the following line somewhere in your env.variables:
env.disable_known_hosts = True
that should take care of it!

Related

How can I send E-Mails through python django frequently with a cronjob?

I try to run a function in django every 5 minutes. For this I user the django-crontab package. The function which should run, checks for some conditions in the database and if they are met, sends an e-mail to the user of the app.
I have django 4 running on my linux ubuntu 20.04 server. I added the cronjob via python3 manage.py crontab add (in activated virtual environment). But then I wondered why the cronjob is not running. I tried to execute the job by hand and it worked.
I think the problem boils down to this: When I'm in the activated virtual environment and run the crontab with "python3 manage.py crontab run " it works. But when I run it outside of the virtual environment I get the following error:
Failed to complete cronjob at ('*/5 * * * *', 'evaluation_tool.scripts.cron.send_mail_if_classeval_ended')
Traceback (most recent call last):
File "/var/www/amadeus/lib/python3.10/site-packages/django_crontab/crontab.py", line 145, in run_job
func(*job_args, **job_kwargs)
File "/var/www/amadeus/evaluation_tool/scripts/cron.py", line 12, in send_mail_if_classeval_ended
send_mail_time_over_class_evaluation(class_evaluation=class_evaluation.pk,
File "/var/www/amadeus/evaluation_tool/scripts/email_handler.py", line 122, in send_mail_time_over_class_evaluation
send_falko_mail("AMADEUS Evaluation abgeschlossen", message, to_email_address)
File "/var/www/amadeus/evaluation_tool/scripts/email_handler.py", line 34, in send_falko_mail
msg.send(fail_silently=False)
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/message.py", line 298, in send
return self.get_connection(fail_silently).send_messages([self])
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 124, in send_messages
new_conn_created = self.open()
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 87, in open
self.connection.starttls(
File "/usr/lib/python3.10/smtplib.py", line 769, in starttls
self.ehlo_or_helo_if_needed()
File "/usr/lib/python3.10/smtplib.py", line 611, in ehlo_or_helo_if_needed
if not (200 <= self.ehlo()[0] <= 299):
File "/usr/lib/python3.10/smtplib.py", line 451, in ehlo
self.putcmd(self.ehlo_msg, name or self.local_hostname)
File "/usr/lib/python3.10/smtplib.py", line 378, in putcmd
self.send(f'{s}{CRLF}')
File "/usr/lib/python3.10/smtplib.py", line 365, in send
raise SMTPServerDisconnected('please run connect() first')
smtplib.SMTPServerDisconnected: please run connect() first
Why is the connection to the SMTP Server working in the activated virtual environment but not outside, which is the desired way using a cronjob?
I found the problem.
I set certain environment variables for SMTP Host, User and Password. But crontab cannot access them. So there are two possibilities:
Add them directly to the head of cron file via crontab -e
EMAIL_HOST=XXX
EMAIL_HOST_USER=XXX
EMAIL_PASS=XXX
Add them via a django-crontab argument in settings.py:
CRONTAB_COMMAND_PREFIX = 'EMAIL_HOST=XXX\nEMAIL_HOST_USER=XXX\nEMAIL_PASS=XXX'

Kubernetes Pod Object crashing in infinite loop when trying to deploy Dockerized Django application with PostgreSQL DB

I am working on my Django project and I am trying to deploy it on Kubernetes cluster ( Google Cloud Provider ). I've managed to create all the files I need for configuring the cluster:
Django App Deployment + Service .yml files
Postgres DB Deployment + Service + PersistentVolumeClaim ( which dynamicly creates PersistentVolume object and uses PD storage on
cloud nodes by default )+ Secret .yml files
Docker File + Docker Compose ( I dont have much experience with docker so there might be something wrong with them, maybe that is the
issue but I cannot figure it out )
For some reason when i try to apply them my PODs, specifically Django Application PODs, are Crashing in an infinite loop. Also my Postgres Pod is not crashing but it is not running eather.
Can anyone help me figure out what am I doing wrong? Here is my github repo: https://github.com/…/mast…/agents/config/kubernetes/postgres
My best guess is that I did not set DATABASES configuration in settings.py file correct, and that my Django Application cannot find DATABASE HOST specified in settings.py. If anyone has any suggestions please leave a comment.
Here is a picture of my Kubernetes Pod list:
Here is a picture where I've described 1st Pod object:
And here is a picture of one of my nodes described:
* UPDATE *
Here is what my ERROR message looks like when i try to run python src/manage.py runserver:
(web_development) cepa995#cepa995-VirtualBox:~/Agentske_Tehnologije/agents$ python src/manage.py runserver
/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Performing system checks...
System check identified no issues (0 silenced).
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7f6b37e9a9d8>
Traceback (most recent call last):
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
self.connect()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection
connection = Database.connect(**conn_params)
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "postgres-service" to address: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/utils/autoreload.py", line 225, in wrapper
fn(*args, **kwargs)
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 124, in inner_run
self.check_migrations()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/core/management/base.py", line 427, in check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
self.loader = MigrationLoader(self.connection)
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
self.build_graph()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/migrations/loader.py", line 206, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations
if self.has_table():
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table
return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
return self._cursor()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
self.ensure_connection()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
self.connect()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
self.connect()
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection
connection = Database.connect(**conn_params)
File "/home/cepa995/anaconda3/envs/web_development/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not translate host name "postgres-service" to address: Name or service not known
Your application cannot connect to the database, because your Postgres server cannot start (you have 0/1 pods in Ready state).
You did not publish its Pod events, but I guess the main reason is its volume.
In your Postgres specs, I see you set the volume type to
ReadWriteMany, but Google Cloud does not provide volumes with that type (documentation, path "Access Modes") and, actually, I don't see the reason why you need it. Change the type to ReadWriteOnce and redeploy the database, it should help.
P.S. To get a log of application in a cluster, you can call kubectl logs $podname, it’s much more helpful
P.P.S. When you publish text information, use a text format, please.

trying to use cookiecutter-django, getting errors and does not create anything

trying to get a Django project started using cookiecutter-django and can't seem to get it to generate anything.
using Python 3.6, Django 2.0.5, cookiecutter 1.6.0 (then created a virtualenv and entered a new, blank directory)
so I enter this command:
cookiecutter https://github.com/pydanny/cookiecutter-django
and get this error traceback:
Traceback (most recent call last):
File "c:\python\python36\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\python\python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\Python\python36\Scripts\cookiecutter.exe\__main__.py", line 9, in
<module>
File "c:\python\python36\lib\site-packages\click\core.py", line 722, in
__call__
return self.main(*args, **kwargs)
File "c:\python\python36\lib\site-packages\click\core.py", line 697, in main
rv = self.invoke(ctx)
File "c:\python\python36\lib\site-packages\click\core.py", line 895, in
invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\python\python36\lib\site-packages\click\core.py", line 535, in
invoke
return callback(*args, **kwargs)
File "c:\python\python36\lib\site-packages\cookiecutter\cli.py", line 120,
in main
password=os.environ.get('COOKIECUTTER_REPO_PASSWORD')
File "c:\python\python36\lib\site-packages\cookiecutter\main.py", line 63,
in cookiecutter
password=password
File "c:\python\python36\lib\site-packages\cookiecutter\repository.py", line
103, in determine_repo_dir
no_input=no_input,
File "c:\python\python36\lib\site-packages\cookiecutter\vcs.py", line 99, in
clone
stderr=subprocess.STDOUT,
File "c:\python\python36\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "c:\python\python36\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone',
'https://github.com/pydanny/cookiecutter-django']' returned non-zero exit
status 128.
OK - figured out how to get this to work.
used Github desktop
from cookiecutter-django repository, right click
open it Git Shell
this opens a Powershell window.
CD to directory where project will be placed in.
cookiecutter https://github.com/pydanny/cookiecutter-django
and it works.
not sure exactly why this works when regular CMD and elevated CMD do not, but this was the only way I could get it to work.
This is a permission issue with github due to the need to setup ssh keys. By the way I'm using ubuntu 12.
https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/ - create a key first in your machine using the instructions in the link. Once you have your ssh key, proceed to step 2. (Step 2 is indicated in the first link as last step)
https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account - add the generated ssh key to your github account.

Trying to setup a Ghost blog with Buster

I'm trying to set up a static Ghost blog with Github hosting using the Buster Static generator. I've tried various instructions including:
https://stefanscherer.github.io/setup-ghost-for-github-pages/
http://blog.sunnyg.io/2015/09/24/ghost-with-github/
But when I get to the "buster generate" command I get the following output in terminal.
It is running fine locally.
Can anyone point me in the right direction?
buster generate
--2016-03-07 23:53:11-- http://localhost:2368/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:2368... failed: Connection refused.
Connecting to localhost|127.0.0.1|:2368... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4508 (4.4K) [text/html]
Saving to: '/Users/philip/Development/Node/ghost-0.7.8/static/index.html'
fixing links in /Users/philip/Development/Node/ghost-0.7.8/static/index.html
Traceback (most recent call last):
File "/usr/local/bin/buster", line 9, in <module>
load_entry_point('buster==0.1.3', 'console_scripts', 'buster')()
File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/buster/buster.py", line 90, in main
newtext = fixLinks(filetext, parser)
File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/buster/buster.py", line 64, in fixLinks
d = PyQuery(bytes(bytearray(text, encoding='utf-8')), parser=parser)
File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyquery/pyquery.py", line 226, in __init__
elements = fromstring(context, self.parser)
File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyquery/pyquery.py", line 90, in fromstring
result = custom_parser(context)
File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/lxml/html/__init__.py", line 867, in fromstring
doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
File "/usr/local/Cellar/python/2.7.8_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/lxml/html/__init__.py", line 752, in document_fromstring
value = etree.fromstring(html, parser, **kw)
File "src/lxml/lxml.etree.pyx", line 3213, in lxml.etree.fromstring (src/lxml/lxml.etree.c:82934)
File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:124533)
File "src/lxml/parser.pxi", line 1707, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:123074)
File "src/lxml/parser.pxi", line 1079, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:117114)
File "src/lxml/parser.pxi", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:110510)
File "src/lxml/parser.pxi", line 683, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:112276)
File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:111367)lxml.etree.XMLSyntaxError: None
As stated in the comment, I did get in working however instead of just doing the 'Quick Install' recommended by many, I went the route of the 'Developer Install' guide here https://github.com/TryGhost/Ghost using the Stable branch.
After you run through that (with your local server running), in another terminal, run
$ buster setup
<Enter git repo>
$ buster generate --domain=localhost:2368
$ buster deploy (or as most sane people prefer, just git push)
Full instructions here: http://phil-a.github.io/getting-ghost-running-on-github-with-buster

"xml.sax._exceptions.SAXReaderNotAvailable: No parsers found" when run in jenkins

So I'm working towards having automated staging deployments via Jenkins and Ansible. Part of this is using a script called ec2.py from ansible in order to dynamically retrieve a list of matching servers to deploy to.
SSH-ing into the Jenkins server and running the script from the jenkins user, the script runs as expected. However, running the script from within jenkins leads to the following error:
ERROR: Inventory script (ec2/ec2.py) had an execution error: Traceback (most recent call last):
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/ec2/ec2.py", line 1262, in <module>
Ec2Inventory()
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/ec2/ec2.py", line 159, in __init__
self.do_api_calls_update_cache()
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/ec2/ec2.py", line 386, in do_api_calls_update_cache
self.get_instances_by_region(region)
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/ec2/ec2.py", line 417, in get_instances_by_region
reservations.extend(conn.get_all_instances(filters = { filter_key : filter_values }))
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/.local/lib/python2.7/site-packages/boto/ec2/connection.py", line 585, in get_all_instances
max_results=max_results)
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/.local/lib/python2.7/site-packages/boto/ec2/connection.py", line 681, in get_all_reservations
[('item', Reservation)], verb='POST')
File "/opt/bitnami/apps/jenkins/jenkins_home/jobs/Deploy API/workspace/deploy/.local/lib/python2.7/site-packages/boto/connection.py", line 1181, in get_list
xml.sax.parseString(body, h)
File "/usr/lib/python2.7/xml/sax/__init__.py", line 43, in parseString
parser = make_parser()
File "/usr/lib/python2.7/xml/sax/__init__.py", line 93, in make_parser
raise SAXReaderNotAvailable("No parsers found", None)
xml.sax._exceptions.SAXReaderNotAvailable: No parsers found
I don't know too much about python, so I'm not sure how to debug this issue further.
So it turns out the issue was to do with Jenkins overwriting the default LD_LIBRARY_PATH variable. By unsetting that variable before running python, I was able to make the python app work!