I configured ElastiCache redis with cluster mode enabled.
I want to connect ElastiCache with local Django.
So I configured bastion host.
I connected ElastiCache(non-cluster mode) with local Django. I tried cache.set(), cache.get(). It's OK.
I installed 'Django-redis-cache' and 'settings.py' is like that.
CACHES = {
'default': {
'BACKEND': 'redis_cache.RedisCache',
'LOCATION': 'localhost:6379',
}
}
But I have problem when I connect ElastiCache(cluster mode) with django.
I tried tunneling with ElastiCache configuration endpoint.
When I use the same 'settings.py', error message is like that.
'SELECT is not allowed in cluster mode'
So, I changed 'settings.py'.
CACHES = {
'default': {
'BACKEND': 'redis_cache.RedisCache',
'LOCATION': 'localhost:6379',
'OPTIONS': {
'DB': 0
},
}
}
And then, error message is like that.
'MOVED 4205 xx.xx.xx.xxx:6379'
What I have to do?
There are no example which connect ElastiCache(cluster mode) with Django.
Related
redis.service is active and I can connect to redis-cli but in Django I got the issue when Celery try to get access to redis:6379 (as I understand).
CELERY_RESULT_BACKEND = os.environ.get('REDISCLOUD_URL', 'redis://localhost:6379/0')
CACHEOPS_REDIS = os.environ.get('CACHEOPS_REDIS', 'redis://localhost:6379/9')
CACHES = {
'default': {
'BACKEND': 'redis_lock.django_cache.RedisCache',
'LOCATION': 'redis://127.0.0.1:6379/',
'OPTIONS': {
'CLIENT_CLASS': 'django_redis.client.DefaultClient',
'DB': 3,
}
}
}
It happened after I reinstalled Ubuntu. Before was ok.
What LOCATION should I point Memcached to after deployment on pythonanywhere server? For local I am using this setting and things are working fine.
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
I need to change 'LOCATION' to replace localhost. Any guidance?
You can location set to a path.
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': 'unix:~/memcached.sock', }}
However, I don't think pythonanywhere lets you use memcache, since you can't use 'sudo apt-get' on a pythonanywhere console, and using memcache requires you to install it. (sudo apt-get install memcached)
Another stackoverflow answer says you need to set up a elasticache.config file to create Redis servers with ElastiCache automatically.
However, can I just create a Redis instance on AWS (Elasticache) and add its endpoint into Django settings? Eg, with Django-redis:
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": "redis://<REDIS AWS ENDPOINT AND PORT HERE>",
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
}
}
I suspect the above could cause trouble with multiple beanstalk server instances. Given this, I am tempted to use MemCache and not Redis, given that there is a Django package written explicitly for interfacing with AWS Elasticache for Memcache: django-elasticache.
Thanks,
Andy.
Short answer: yes.
Long answer: I have not used Elastic Beanstalk, however I can confirm that if you create a Redis instance (that is: cluster mode disabled) in ElastiCache it will work fine with django-redis. Just insert the primary_endpoint into the Django config you posted.
N.B. If you plan to use read replicas, set it up like this:
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": [
"redis://<MASTER ENDPOINT>",
"redis://<SLAVE ENDPOINT>",
]
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
}
}
If you spin up a Redis cluster however, you cannot use vanilla django-redis. You'll have to use use redis-py-cluster with it as described in this post. Replicated here:
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': 'redis://XXX.YYY.ZZZ.cache.amazonaws.com/0',
'OPTIONS': {
'REDIS_CLIENT_CLASS': 'rediscluster.RedisCluster',
'CONNECTION_POOL_CLASS': 'rediscluster.connection.ClusterConnectionPool',
'CONNECTION_POOL_KWARGS': {
'skip_full_coverage_check': True # AWS ElasticCache has disabled CONFIG commands
}
}
}
}
I am hoping to use Amazon's Elasticsearch server to power a search of longtext fields in a Django database. However, I also don't want to expose this search to those who don't have a log in and don't want to rely on security through obscurity or some IP restriction tactic (unless it would work well with an existing heroku app, where the Django app is deployed).
Haystack seems to go a long way toward this, but there doesn't seem to be an easy way to configure it to use Amazon's IAM credentials to access the Elasticsearch service. This functionality does exist in elasticsearch-py, whichi it uses.
https://elasticsearch-py.readthedocs.org/en/master/#running-with-aws-elasticsearch-service
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
host = 'YOURHOST.us-east-1.es.amazonaws.com'
awsauth = AWS4Auth(YOUR_ACCESS_KEY, YOUR_SECRET_KEY, REGION, 'es')
es = Elasticsearch(
hosts=[{'host': host, 'port': 443}],
http_auth=awsauth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
print(es.info())
Regarding using HTTP authorization, I found this under issues at https://github.com/django-haystack/django-haystack/issues/1046
from urlparse import urlparse
parsed = urlparse('https://user:pass#host:port')
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': parsed.hostname,
'INDEX_NAME': 'haystack',
'KWARGS': {
'port': parsed.port,
'http_auth': (parsed.username, parsed.password),
'use_ssl': True,
}
}
}
I am wondering if there is a way to combine these two, something like the following (which, as expected, gives an error since it's more than just a user name and password):
from requests_aws4auth import AWS4Auth
awsauth = AWS4Auth([AACCESS_KEY],[SECRET_KEY],[REGION],'es')
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': [AWSHOST],
'INDEX_NAME': 'haystack',
'KWARGS': {
'port': 443,
'http_auth': awsauth,
'use_ssl': True,
'verify_certs': True
}
},
}
The error here:
TypeError at /admin/
must be convertible to a buffer, not AWS4Auth
Request Method: GET
Request URL: http://127.0.0.1:8000/admin/
Django Version: 1.7.7
Exception Type: TypeError
Exception Value:
must be convertible to a buffer, not AWS4Auth
Exception Location: /usr/lib/python2.7/base64.py in b64encode, line 53
Any ideas on how to accomplish this?
You are one step from success, add connection_class to KWARGS and everything should work as expected.
import elasticsearch
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': [AWSHOST],
'INDEX_NAME': 'haystack',
'KWARGS': {
'port': 443,
'http_auth': awsauth,
'use_ssl': True,
'verify_certs': True,
'connection_class': elasticsearch.RequestsHttpConnection,
}
},
}
AWS Identity and Access Management (IAM) allows you to manage users and user permissions for AWS services, to control which AWS resources users of AWS itself can access.
You cannot use IAM credentials to authorize users at the application level via http_auth, as it appears you are trying to do via Haystack here. They are different authentication schemes for different services. They are not compatible.
In your security use case, you have stated the need to 1) restrict access to your application, and 2) to secure the Elasticsearch service port from open access. These two requirements can be met using the following methods:
Restrict access to your application
I also don't want to expose this search to those who don't have a log in
For the front-end search app, you want to use a server level Basic access authentication (HTTP auth) configuration on the web server. This is where you want to control user login access to your app, via a standard http_auth username and password (again, not IAM). This will secure your app at the application level.
Secure the Elasticsearch service port
don't want to rely on security through obscurity or some
IP restriction tactic (unless it would work well with an existing
heroku app, where the Django app is deployed).
IP restriction is exactly what would work here, and consistent with AWS security best practices. You want to use security groups and security group rules as a firewall to control traffic for your EC2 instances.
Given a Haystack configuration of:
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}
you will want to implement an IP restriction at the security group and/or ACL level on that IP and port 127.0.0.1, to restrict access from only your Django host or other authorize hosts. This will secure it from any unauthorized access at the service level.
In your implementation, the URL will likely resolve to a public or private IP, depending on your network architecture.
I'm attempting to use django-redis using Unix sockets rather than a TCP connection:
This is the settings.py configuration:
CACHES = {
'default': {
'BACKEND': 'redis_cache.cache.RedisCache',
'LOCATION': 'unix:/tmp/redis.sock:1',
'OPTIONS': {
'PASSWORD': '',
'PICKLE_VERSION': -1, # default
'PARSER_CLASS': 'redis.connection.HiredisParser',
'CLIENT_CLASS': 'redis_cache.client.DefaultClient',
},
},
}
and this is an extract of the redis config file at /etc/redis/6379.conf:
# Specify the path for the unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
#
unixsocket /tmp/redis.sock
unixsocketperm 755
Still I receive a ConnectionInterrumped exception, which stands for an error during connection. Any ideas about what this configuration's issue is?
P.S. My Django version is 1.5.1, django-redis is 3.3 and hiredis is 0.0.1.
EDIT: Apparently I read the cache provider wrong, the below answer is the solution for django-redis-cache, not django-redis. I'll let the answer remain though, since changing cache provider and using this config seems to have solved the problem.
You should not need the unix: prefix, and the backend setting looks strange;
'default': {
'BACKEND': 'redis_cache.RedisCache',
'LOCATION': '/tmp/redis.sock',
'OPTIONS': { ...