My short question, how to setup Django-channels v2 on AWS Elasticbeanstalk? The long and less inviting questions underneath.
Thank you in advance!
I am trying to set up an elasticbeanstalk instance using Django Channels 2.02. I followed this https://hackernoon.com/setting-up-django-channels-on-aws-elastic-beanstalk-716fd5a49c4a tutorial. Tutorial made for Channels 1.
The Elasticbeanstalk instance was created with an application load balancer. The Redis 4.0 instance is running 0.0.0.0:6379, confirmed by netstat. Custom TCP, Port Range: 6379, Source: 0.0.0.0/0 on redis instance.
alb_listener.config in .ebekstension:
option_settings: aws:elbv2:listener:80:
DefaultProcess: http
ListenerEnabled: 'true'
Protocol: HTTP aws:elasticbeanstalk:environment:process:http:
Port: '8000'
Protocol: HTTP
Daphne server startet with command "Daphne .asgi:application". Listening on TCP address 0.0.0.0:8000. It returns:
2018-03-12 08:59:30,778 ERROR Exception inside application: [Errno -2] Name or service not known
File "/opt/python/run/venv/local/lib/python3.6/site-packages/channels/consumer.py", line 54, in __call__
await await_many_dispatch([receive, self.channel_receive], self.dispatch)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/channels/utils.py", line 47, in await_many_dispatch
result = task.result()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/channels_redis/core.py", line 167, in receive
task.result()
File "/opt/python/run/venv/local/lib/python3.6/site-packages/channels_redis/core.py", line 187, in receive_loop
real_channel, message = await self.receive_single(general_channel)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/channels_redis/core.py", line 224, in receive_single
async with self.connection(index) as connection:
File "/opt/python/run/venv/local/lib/python3.6/site-packages/channels_redis/core.py", line 403, in __aenter__
self.conn = await aioredis.create_redis(**self.kwargs)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/aioredis/commands/__init__.py", line 174, in create_redis
loop=loop)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/aioredis/connection.py", line 107, in create_connection
timeout, loop=loop)
File "/usr/lib64/python3.6/asyncio/tasks.py", line 339, in wait_for
return (yield from fut)
File "/opt/python/run/venv/local/lib/python3.6/site-packages/aioredis/stream.py", line 19, in open_connection
lambda: protocol, host, port, **kwds)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 733, in create_connection
infos = f1.result()
File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib64/python3.6/socket.py", line 743, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
[Errno -2] Name or service not known
The Websocket returns no error in browser console.
It works on my local runserver setup. What is the problem? Can you please help me write a config script so that I don't need to start Daphne manually?
Very late reply here.
The problem is with config:hosts URL in the tutorial. Remove the redis:// portion from the URL.
For instance,
REDIS_PUBLIC_DNS = "YOUR-REDIS-DNS.amazonaws.com"
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [(REDIS_PUBLIC_DNS, 6379)],
}
}
}
You can check the connection using redis-cli,
redis-cli -h REDIS-PUBLIC-DNS-url ping ;you should expect PONG in response.
Related
I am struggling in sending a email from cloud-composer 2 environment(composer-2.0.11-airflow-2.2.3) via smtp.gmail.com.
Please find my configs below :
Note I tried using smtp_password_cmd and smtp_password_secret but both give the same error.
I have also setup a network connectivity test which shows smtp.gmail.com is reachable from my VPC
Error Msg:
File "/opt/python3.8/lib/python3.8/smtplib.py", line 310, in _get_socket
return socket.create_connection((host, port), timeout,
File "/opt/python3.8/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/opt/python3.8/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
Any help/pointers would be appreciated.
Thanks
I'm trying to run localstack for the first time. I am using link mint 17. However, I keep getting this error:
Starting local dev environment. CTRL-C to quit.
Starting mock ES service (http port 4578)...
Starting local Elasticsearch (http port 4571)...
Starting mock S3 (http port 4572)...
Starting mock SNS (http port 4575)...
Starting mock SQS (http port 4576)...
Starting mock SES (http port 4579)...
Starting mock SSM (http port 4583)...
Starting mock API Gateway (http port 4567)...
Starting mock DynamoDB (http port 4569)...
Starting mock DynamoDB Streams service (http port 4570)...
Starting mock Firehose service (http port 4573)...
Starting mock Lambda service (http port 4574)...
Starting mock Kinesis (http port 4568)...
Starting mock Redshift (http port 4577)...
Starting mock Route53 (http port 4580)...
Starting mock CloudFormation (http port 4581)...
Starting mock CloudWatch (http port 4582)...
2018-10-29T22:06:03:WARNING:infra.py: Service "elasticsearch" not yet available, retrying...
2018-10-29T22:06:11:ERROR:localstack.services.generic_proxy: Error forwarding request: HTTPConnectionPool(host='127.0.0.1', port=4564): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f050959e5c0>: Failed to establish a new connection: [Errno 111] Connection refused',)) Traceback (most recent call last):
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/home/user/venv35/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f050959e5c0>: Failed to establish a new connection: [Errno 111] Connection refused
What should i do to fix this? I already installed the dependencies, like java, maven, npm, pip, and python
Edit to ask who thought that this question had anything to do with the "possibly related" one.
I have a fairly simple django (1.11) project using the rest_framework that works fine when I start it from the command line, typing
nohup python manage.py runserver 0.0.0.0:4448 &
on centos. Connects to a postgresql database, with
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'my_database',
'USER': 'my_user',
#'PASSWORD': 'mypassword',
'HOST': '127.0.0.1',
' PORT': '5432',
}
}
in my settings.py file. However if I set up the runserver command to run at boot time from a cron I get the following when I send a request to the application:
... lot of stuff
django.db.utils.OperationalError: could not connect to server: Connection
refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
and in my pg_hba file I have
IPv4 local connections:
host all all 127.0.0.1/32
trust # local host
host all all 10.2.11.53/32
trust
host all my_user 0.0.0.0/0 trust
and also I have
netstat -plunt | grep post
tcp 0 0 10.2.11.53:5432 0.0.0.0:* LISTEN
867/postmaster
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN
867/postmaster
Any suggestion?
Thanks,
a
PS The full traceback:
Unhandled exception in thread started by <function wrapper at 0x1a11230>
Performing system checks...
[<RegexURLPattern batch-batch-done ^batch/batch-done/$>, <RegexURLPattern batch-load-urls ^batch/load-urls/$>, <RegexURLPattern batch-request-batch ^batch/request-batch/$>, <RegexURLPattern batch-schedule-job ^batch/schedule-job/$>]
System check identified no issues (0 silenced).
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/django/utils/autoreload.py", line 227, in wrapper
fn(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 128, in inner_run
self.check_migrations()
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 422, in check_migrations
executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
File "/usr/lib/python2.7/site-packages/django/db/migrations/executor.py", line 20, in __init__
self.loader = MigrationLoader(self.connection)
File "/usr/lib/python2.7/site-packages/django/db/migrations/loader.py", line 52, in __init__
self.build_graph()
File "/usr/lib/python2.7/site-packages/django/db/migrations/loader.py", line 209, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/usr/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 65, in applied_migrations
self.ensure_schema()
File "/usr/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 52, in ensure_schema
if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
return self._cursor()
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
self.ensure_connection()
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
Could you post the full traceback, as well as the cron job itself?
Also, if I understand your question correctly, you are trying to use cron to start your server on boot? If this is the case, you would be better served using something like supervisord to manage your server processes IMO.
I'm trying to build an app in Django that uses Django Channels. I'm deploying on Elastic Beanstalk. My understanding is that the application load balancer supports websockets and I can route websocket traffic to the appropriate port. The websockets work on my localhost:8000. I'm using the free tier Redis Labs for my channel layer.
I followed this tutorial.
https://blog.mangoforbreakfast.com/2017/02/13/django-channels-on-aws-elastic-beanstalk-using-an-alb/#comment-43
Following this tutorial it appears I can get Daphne and the workers running on the correct ports. I SSHed into the EC2 instance to get the following output.
$ sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf status
Daphne RUNNING pid 4240, uptime 0:05:51
Worker:Worker_00 RUNNING pid 4242, uptime 0:05:51
Worker:Worker_01 RUNNING pid 4243, uptime 0:05:51
Worker:Worker_02 RUNNING pid 4244, uptime 0:05:51
Worker:Worker_03 RUNNING pid 4245, uptime 0:05:51
httpd RUNNING pid 4248, uptime 0:05:51
My daphne.out.log and workers.out.log look fine.
$ cat daphne.out.log
2017-07-20 21:41:43,693 INFO Starting server at tcp:port=5000:interface=0.0.0.0, channel layer mysite.asgi:channel_layer.
2017-07-20 21:41:43,693 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2017-07-20 21:41:43,694 INFO Using busy-loop synchronous mode on channel layer
2017-07-20 21:41:43,694 INFO Listening on endpoint tcp:port=5000:interface=0.0.0.0
$ cat workers.out.log
2017-07-20 21:41:44,114 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,120 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,121 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,121 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,126 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,126 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,126 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,127 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,127 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2017-07-20 21:41:44,133 - INFO - runworker - Using single-threaded worker.
2017-07-20 21:41:44,136 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2017-07-20 21:41:44,136 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
From here I updated the security group settings and configured the load balancer and tested my webpage. Instead of /ws/ as the path for rerouting, I used /table/* because those are the pages that need websockets.
If I use /ws/ I get this error.
WebSocket connection to 'ws://example.com/table/1/' failed: Error during WebSocket handshake: Unexpected response code: 404
If I use /table/* I get this error.
WebSocket connection to 'ws://example.com/table/1/' failed: Error during WebSocket handshake: Unexpected response code: 504
When I changed the security group settings of my load balancer to also allow TCP at port 5000, the worker log changes. With the /ws/ path rule in my load balancer, I now get this in the worker log:
...
2017-07-20 21:41:44,136 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
Not Found: /ws/
If I use the /table/* path, I get a very long error in my log.
2017-07-21 01:22:05,270 - ERROR - worker - Error processing message with consumer deal.consumers.ws_connect:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 193, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/postgresql/base.py", line 175, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/python/run/venv/local/lib64/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/worker.py", line 119, in run
consumer(message, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/sessions.py", line 220, in inner
result = func(message, *args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/channels/auth.py", line 71, in inner
message.user = auth.get_user(fake_request)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 167, in get_user
user_id = _get_user_session_key(request)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/auth/__init__.py", line 59, in _get_user_session_key
return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 48, in __getitem__
return self._session[key]
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/base.py", line 198, in _get_session
self._session_cache = self.load()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/contrib/sessions/backends/db.py", line 33, in load
expire_date__gt=timezone.now()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/manager.py", line 122, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 381, in get
num = len(clone)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 240, in __len__
self._fetch_all()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 1074, in _fetch_all
self._result_cache = list(self.iterator())
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/query.py", line 52, in __iter__
results = compiler.execute_sql()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 846, in execute_sql
cursor = self.connection.cursor()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 231, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 204, in _cursor
self.ensure_connection()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
self.connect()
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/base/base.py", line 171, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/python/run/venv/local/lib/python3.4/site-packages/django/db/backends/postgresql/base.py", line 175, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/python/run/venv/local/lib64/python3.4/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
Unsurprisingly when I try to use the websockets I get this error:
WebSocket is already in CLOSING or CLOSED state.
Any idea what I'm doing wrong?
Ok so problem was when Daphne runs it doesn't see the environment variables. I added the following code to my supervisord.conf file and it worked:
[program:Daphne]
environment=PATH="/opt/python/run/venv/bin"
**command=/bin/bash -c "source /opt/python/current/env && /opt/python/run/venv/bin/daphne -b 0.0.0.0 -p 5000 mysite.asgi:channel_layer"**
directory=/opt/python/current/app
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/daphne.out.log
[program:Worker]
environment=PATH="/opt/python/run/venv/bin"
**command=/bin/bash -c "source /opt/python/current/env && python manage.py runworker"**
directory=/opt/python/current/app
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/tmp/workers.out.log
I have a strange issue here and it's driving me nuts.
I have an agent setup on the another application that works smoothly. However, running same gives the error below.
The application with the issue has Route53 integrated with the url like "name.xxbg3wy.us-east-1.elasticbeanstalk.com " while the other application that works smoothly has the url "name.us-east-1.elasticbeanstalk.com"
Is there anything that I'm missing out on here? Could it be that the endpoint-url has to be set somewhere in the config file? The region is "us-east-1" and I've confirmed severally from the config file.
2017-07-17 19:20:54,998 - cwlogs.threads - ERROR - 28422 - Thread-11 -
Exception caught in <EventBatchPublisher(Thread-11, started daemon
140656144017152)>
Traceback (most recent call last):
File "/var/awslogs/local/lib/python2.7/site-
packages/cwlogs/threads.py", line 58, in run
self._run()
File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/push.py",
line 1402, in _run
self._publish_event_batch()
File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/push.py",
line 1209, in _publish_event_batch
self.sequence_token = self._put_log_events(self.event_batch)
File "/var/awslogs/local/lib/python2.7/site-packages/cwlogs/push.py",
line 1247, in _put_log_events
response = self.logs_service.put_log_events(**params)
File "/var/awslogs/local/lib/python2.7/site-
packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/awslogs/local/lib/python2.7/site-
packages/botocore/client.py", line 530, in _make_api_call
operation_model, request_dict)
File "/var/awslogs/local/lib/python2.7/site-
packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/var/awslogs/local/lib/python2.7/site-
packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
EndpointConnectionError: Could not connect to the endpoint URL:
"https://logs.us-east-1.amazonaws.com/"
2017-07-17 19:21:55,927 - cwlogs.push.reader - INFO - 28422 - Thread-12
- Reader is leaving as requested...
Have you checked your Security Group policies? Looks like you might need to whitelist certain ports.
Try for "All traffic' first and see of its connects.
You could use, telnet service to check the connectivity to ports from your host.