setting up uWSGI and Django app - django

I am trying to follow the steps in this guide: http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html
Before I even get to the nginx part I am trying to make sure that uWSGI works correctly
my folder structure is srv/www/domain/projectdatabank/
the project databank folder contains my manage.py file
my wsgi.py file looks like this:
import os
import sys
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "databank.settings")
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
i go into the projectdatabank folder and run the following command
uwsgi --http :8000 --wsgi-file projectdatabank/databank/wsgi.py
when i go to the web page i get this error
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-3) on 06 July 2013 00:16:13
os: Linux-3.8.4-linode50 #1 SMP Mon Mar 25 15:50:29 EDT 2013
nodename:
machine: i686
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /srv/www/databankinfo.com
detected binary path: /usr/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 1024
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uWSGI http bound on :8000 fd 4
spawned uWSGI http 1 (pid: 10091)
uwsgi socket 0 bound to TCP address 127.0.0.1:47129 (port auto-assigned) fd 3
Python version: 2.6.6 (r266:84292, Feb 21 2013, 23:54:59) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x8cf8598
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 64000 bytes (62 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x8cf8598 pid: 10090 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 10090, cores: 1)
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 236, in __call__
self.load_middleware()
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 45, in load_middleware
for middleware_path in settings.MIDDLEWARE_CLASSES:
File "/usr/lib/python2.6/site-packages/django/conf/__init__.py", line 53, in __getattr__
self._setup(name)
File "/usr/lib/python2.6/site-packages/django/conf/__init__.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "/usr/lib/python2.6/site-packages/django/conf/__init__.py", line 134, in __init__
raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'databank.settings' (Is it on sys.path?): No module named databank.settings
[pid: 10090|app: 0|req: 1/1] 66.56.35.151 () {38 vars in 669 bytes} [Tue Jul 9 17:34:52 2013] GET / => generated 0 bytes in 1 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
however I know that settings.py exists in the same directory as wsgi.py

You need to provide an additional argument to your uwsgi call:
--chdir /path/to/your/project/

Related

OpenEdx error while running python code in Codejail Plugins using Dockerize container services

I have installed a stack of OpexEDX platform using Tutor and installed OpexEdx "Codejail" plugin using below link
pip install git+https://github.com/edunext/tutor-contrib-codejail
https://github.com/eduNEXT/tutor-contrib-codejail
I am facing a problem during working on the code jail while importing python matplotlib library.
importing the same library inside codejail container is working fine. the only problem is import through OpnexEdx code block. > advance black > problem.
I have already installed the Codejail and Matplotlib on docker.
I have to run this code. which gives error
<problem>
<script type="loncapa/python">
import matplotlib
</script>
</problem>
import os works fine
but getting error while
import matplotlib
detail of current stack:
open edx version : openedx-mfe:14.0.1
code jail version : codejailservice:14.1.0
please see the error message below
cannot create LoncapaProblem block-v1:VUP+Math101+2022+type#problem+block#3319c4e42da64a74b0e40f048e3f2599: Error while executing script code: Couldn't execute jailed code: stdout: b'', stderr: b'Traceback (most recent call last):\n File "jailed_code", line 19, in <module>\n exec(code, g_dict)\n File "<string>", line 66, in <module>\n File "/sandbox/venv/lib/python3.8/site-packages/matplotlib/__init__.py", line 921, in <module>\n dict.update(rcParams, rc_params_in_file(matplotlib_fname()))\n File "/sandbox/venv/lib/python3.8/site-packages/matplotlib/__init__.py", line 602, in matplotlib_fname\n for fname in gen_candidates():\n File "/sandbox/venv/lib/python3.8/site-packages/matplotlib/__init__.py", line 599, in gen_candidates\n yield os.path.join(get_configdir(), \'matplotlibrc\')\n File "/sandbox/venv/lib/python3.8/site-packages/matplotlib/__init__.py", line 239, in wrapper\n ret = func(**kwargs)\n File "/sandbox/venv/lib/python3.8/site-packages/matplotlib/__init__.py", line 502, in get_configdir\n return get_config_or_cache_dir(_get_xdg_config_dir())\n File "/sandbox/venv/lib/python3.8/site-packages/matplotlib/__init__.py", line 474, in get_config_or_cache_dir\n tempfile.mkdtemp(prefix="matplotlib-")\n File "/opt/pyenv/versions/3.8.6_sandbox/lib/python3.8/tempfile.py", line 347, in mkdtemp\n prefix, suffix, dir, output_type = sanitize_params(prefix, suffix, dir)\n File "/opt/pyenv/versions/3.8.6_sandbox/lib/python3.8/tempfile.py", line 117, in sanitize_params\n dir = gettempdir()\n File "/opt/pyenv/versions/3.8.6_sandbox/lib/python3.8/tempfile.py", line 286, in gettempdir\n tempdir = get_default_tempdir()\n File "/opt/pyenv/versions/3.8.6_sandbox/lib/python3.8/tempfile.py", line 218, in _get_default_tempdir\n raise FileNotFoundError(_errno.ENOENT,\nFileNotFoundError: [Errno 2] No usable temporary directory found in [\'/tmp\', \'/var/tmp\', \'/usr/tmp\', \'/tmp/codejail-lbfd69da\']\n' with status code: 1. For more information check Codejail Service logs.
Codejail service logs are as follows:
{"log":"[pid: 6|app: 0|req: 20/39] 172.18.0.10 () {36 vars in 483 bytes} [Tue Nov 22 11:24:59 2022] POST /api/v0/code-exec =\u003e generated 1978 bytes in 742 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 0)\n","stream":"stderr","time":"2022-11-22T11:25:00.151315626Z"} {"log":"2022-11-22 11:26:23,304 INFO 9 [codejailservice.app] code_exec_service.py:52 - Running problem_id:53fbaa04859f41989ab967c15a12c013 jailed code for course_id:course-v1:VUP+Math101+2022 ...\n","stream":"stderr","time":"2022-11-22T11:26:23.30489438Z"} {"log":"2022-11-22 11:26:23,343 INFO 9 [codejailservice.app] code_exec_service.py:73 - Jailed code was executed in 0.03849988000001758 seconds.\n","stream":"stderr","time":"2022-11-22T11:26:23.343618965Z"} {"log":"[pid: 9|app: 0|req: 20/40] 172.18.0.10 () {36 vars in 483 bytes} [Tue Nov 22 11:26:23 2022] POST /api/v0/code-exec =\u003e generated 73 bytes in 40 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)\n","stream":"stderr","time":"2022-11-22T11:26:23.344178308Z"} {"log":"2022-11-23 04:15:24,786 INFO 6 [codejailservice.app] code_exec_service.py:52 - Running problem_id:3319c4e42da64a74b0e40f048e3f2599 jailed code for course_id:course-v1:VUP+Math101+2022 ...\n","stream":"stderr","time":"2022-11-23T04:15:24.786287416Z"} {"log":"2022-11-23 04:15:25,582 ERROR 6 [codejailservice.app] code_exec_service.py:70 - Error found while executing jailed code.\n","stream":"stderr","time":"2022-11-23T04:15:25.582527974Z"} {"log":"[pid: 6|app: 0|req: 21/41] 172.18.0.10 () {36 vars in 483 bytes} [Wed Nov 23 04:15:24 2022] POST /api/v0/code-exec =\u003e generated 1978 bytes in 798 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 0)\n","stream":"stderr","time":"2022-11-23T04:15:25.583132326Z"} {"log":"2022-11-23 06:00:15,150 INFO 9 [codejailservice.app] code_exec_service.py:52 - Running problem_id:3319c4e42da64a74b0e40f048e3f2599 jailed code for course_id:course-v1:VUP+Math101+2022 ...\n","stream":"stderr","time":"2022-11-23T06:00:15.15073834Z"} {"log":"2022-11-23 06:00:15,891 ERROR 9 [codejailservice.app] code_exec_service.py:70 - Error found while executing jailed code.\n","stream":"stderr","time":"2022-11-23T06:00:15.8916806Z"} {"log":"[pid: 9|app: 0|req: 21/42] 172.18.0.10 () {36 vars in 483 bytes} [Wed Nov 23 06:00:15 2022] POST /api/v0/code-exec =\u003e generated 1978 bytes in 742 msecs (HTTP/1.1 200) 2 headers in 73 bytes (1 switches on core 0)\n","stream":"stderr","time":"2022-11-23T06:00:15.892225441Z"}

Problems initialising Django 1.8 app using apache 2.4 mod_passenger 5.0 after centos 7 yum update

I have been ussing apache/mod_passenger on Centos 7 for a year now on my server at home to run a basic Django 1.8 app.
No issues, install updates with yum, etc.
Until today, I have noticed the website of my application was throwing an error. In the apache log I can see this:
[ 2016-02-21 19:45:38.8531 15680/7f470c08e880 age/Cor/CoreMain.cpp:707 ]: Passenger core online, PID 15680
[ 2016-02-21 19:45:38.8584 15685/7f6eda27f880 age/Ust/UstRouterMain.cpp:504 ]: Starting Passenger UstRouter...
[ 2016-02-21 19:45:38.8591 15685/7f6eda27f880 age/Ust/UstRouterMain.cpp:317 ]: Passenger UstRouter online, PID 15685
[Sun Feb 21 19:45:38.865561 2016] [mpm_prefork:notice] [pid 15644] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 Phusion_Passenger/5.0.25 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Sun Feb 21 19:45:38.865602 2016] [core:notice] [pid 15644] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sun Feb 21 19:46:58.467864 2016] [autoindex:error] [pid 15701] [client 86.121.21.124:64572] AH01276: Cannot serve directory /var/www/myProjects/testuser66/static/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Options directive
App 15712 stdout:
App 15712 stderr: error: cannot open Packages index using db5 - Permission denied (13)
App 15712 stderr: error: cannot open Packages database in /var/lib/rpm
App 15712 stderr: Traceback (most recent call last):
App 15712 stderr: File "/usr/share/passenger/helper-scripts/wsgi-loader.py", line 325, in <module>
App 15712 stderr: app_module = load_app()
App 15712 stderr: File "/usr/share/passenger/helper-scripts/wsgi-loader.py", line 62, in load_app
App 15712 stderr:
App 15712 stderr: return imp.load_source('passenger_wsgi', startup_file)
App 15712 stderr: File "/var/www/myProjects/testuser66/passenger_wsgi.py", line 8, in <module>
App 15712 stderr: if sys.executable != INTERP: os.execl(INTERP, INTERP, *sys.argv)
App 15712 stderr: File "/usr/lib64/python2.7/os.py", line 312, in execl
App 15712 stderr: execv(file, args)
App 15712 stderr: OSError: [Errno 13] Permission denied
App 15712 stderr:
[ 2016-02-21 19:47:02.9959 15680/7f470bef7700 age/Cor/App/Implementation.cpp:304 ]: Could not spawn process for application /var/www/myProjects/testuser66: An error occurred while starting the web application. It exited before signalling successful startup back to Phusion Passenger.
Error ID: d7ea712a
Error details saved to: /tmp/passenger-error-kqxLLO.html
Message from application: An error occurred while starting the web application. It exited before signalling successful startup back to Phusion Passenger. Please read this article for more information about this problem.<br>
Raw process output:
error: cannot open Packages index using db5 - Permission denied (13)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
File "/usr/share/passenger/helper-scripts/wsgi-loader.py", line 325, in <module>
app_module = load_app()
File "/usr/share/passenger/helper-scripts/wsgi-loader.py", line 62, in load_app
return imp.load_source(&apos;passenger_wsgi&apos;, startup_file)
File "/var/www/myProjects/testuser66/passenger_wsgi.py", line 8, in <module>
if sys.executable != INTERP: os.execl(INTERP, INTERP, *sys.argv)
File "/usr/lib64/python2.7/os.py", line 312, in execl
execv(file, args)
OSError: [Errno 13] Permission denied
[ 2016-02-21 19:47:02.9999 15680/7f4706242700 age/Cor/Con/CheckoutSession.cpp:277 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is d7ea712a. Please see earlier logs for details about the error.
I used to have some trouble with Passenger after running yum update, but usually because of SE Linux. But I have checked permissions to my best knowledge and it is not the case.
Also the "cannot open /var/lib/rpm" is confusiog for me. What does it have to do with Apache/Passenger? I checked the rpm db, deleted yum cache, there are no signs of corupted rpm db.
Any help would be appreciated!

django + celery + uwsgi without supervisor

On dev server Django is working well with celery and django-celery:
python manage.py runserver
celery -A backup worker -l info -B
celerycam --frequency=10.0
On production server I tried to run celery with:
[uwsgi]
...
master = True
smart-attach-daemon = ${path}/${name_project}/.env/bin/python ${path}/manage.py celery -A test worker -l info -B
smart-attach-daemon = ${path}/${name_project}/.env/bin/python ${path}/manage.py celerycam --frequency=10.0
But it does not work.
How can I run it all without the use supervisor?
Update
It does not see Django, but the site works. In logs uwsgi:
Mon Aug 3 16:10:57 2015 - spawned uWSGI master process (pid: 23462)
Mon Aug 3 16:10:57 2015 - spawned uWSGI worker 1 (pid: 23666, cores: 1)
Mon Aug 3 16:10:57 2015 - spawned uWSGI worker 2 (pid: 23667, cores: 1)
Mon Aug 3 16:10:57 2015 - [uwsgi-daemons] spawning "/home/1/2/3/manage.py celery -A backup worker -l info -B"
Mon Aug 3 16:10:57 2015 - [uwsgi-daemons] spawning "/home/1/2/3/manage.py celerycam --frequency=10.0"
Traceback (most recent call last):
File "/home/1/2/3/manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Traceback (most recent call last):
File "/home/1/2/3/manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Mon Aug 3 16:10:58 2015 - subprocess 23668 exited with code 1
Mon Aug 3 16:10:58 2015 - subprocess 23669 exited with code 1
manage.py:
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "backup.settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
You can use either attach-daemon or smart-attach-daemon. But if you are using smart-attach-daemon then you should also start celery with pid file and set path to it in uwsgi:
smart-attach-daemon = ${path}/${name_project}/var/celery-worker.pid ${path}/${name_project}/.env/bin/python ${path}/manage.py celery -A test worker --pidfile=${path}/${name_project}/var/celery-worker.pid -l info -B
Replace "smart-attach-daemon" on "attach-daemon"

Celery timeout exception on Amazon Elastic Beanstalk using RabbitMQ

I'm trying to use Celery on my Beanstalk environment (this is the final piece in order to complete the technology stack of my project :P).
This is what I've done so far:
Since, RabbitMQ is the best broker for Celery and Amazon does not provide a dedicated service I created a custom AMI based on Ubuntu 13 64bit
installed RabbitMQ
removed the default user guest/guest
created a custom user
created a custom virtual host
installed admin plugins
tested my configuration using the http API in order to confirm that my RabbitMQ server is up and running.
So far so good! Then in my beanstalk .config file I added a couple of commands for celery:
04_celery_periodic_tasks:
command: "celery worker --app=com.cygora --loglevel=info --beat --autoreload -n period_tasks_worker.%h"
leader_only: true
05_celery_standard_worker:
command: "celery worker --app=com.cygora --loglevel=info --autoreload -n worker_1.%h"
Once I deployed my app, I didn't find any error related to celery (so I'm assuming it's all ok, from "the Python/Django side")... but as soon as I use a feature of my site that requires sending a message to Rabbit via Celery I get a timeout exception:
[Thu Feb 20 22:01:24 2014] [error] File "/opt/python/run/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 111, in establish_connection
[Thu Feb 20 22:01:24 2014] [error] conn = self.Connection(**opts)
[Thu Feb 20 22:01:24 2014] [error] File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
[Thu Feb 20 22:01:24 2014] [error] self.transport = create_transport(host, connect_timeout, ssl)
[Thu Feb 20 22:01:24 2014] [error] File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/transport.py", line 274, in create_transport
[Thu Feb 20 22:01:24 2014] [error] return TCPTransport(host, connect_timeout)
[Thu Feb 20 22:01:24 2014] [error] File "/opt/python/run/venv/lib/python2.7/site-packages/amqp/transport.py", line 89, in __init__
[Thu Feb 20 22:01:24 2014] [error] raise socket.error(last_err)
[Thu Feb 20 22:01:24 2014] [error] error: timed out
I specified the broker url in settings as:
BROKER_URL = "amqp://myuser:mypassword#myelasticip:5672/myvirtualhost"
What I'm missing or what I did wrong? Why the socket connection can't be established?
I forgot I had asked this question... anyway I solved. It was just a matter of opening the right TCP ports for RabbitMQ:
22
15672
5672
I also changed the way I run celery, by using supervisor + django-supervisor in order to daemonize it properly :)

Celery not connecting to Redis Broker (Django)

This is my first time using Celery and redis so there's probably something obvious that I'm not inferring from the documentation and searching through others' questions on here. Whenever I try to run a worker my connection keeps timing out with:
ResponseError: unknown command 'WATCH'
[2013-06-12 18:25:23,059: ERROR/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
here's my requirements.txt
South==0.7.6
amqp==1.0.11
anyjson==0.3.3
billiard==2.7.3.28
boto==2.9.4
celery==3.0.19
celery-with-redis==3.0
dj-database-url==0.2.1
django-admin-bootstrapped==0.3.2
django-celery==3.0.17
django-jsonfield==0.9.4
django-stripe-payments==2.0b20
mimeparse==0.1.3
oauthlib==0.4.0
paramiko==1.10.1
psycopg2==2.5
pycrypto==2.6
python-dateutil==2.1
python-openid==2.2.5
pytz==2013b
redis==2.7.5
requests==1.2.0
requests-oauthlib==0.3.1
six==1.3.0
stripe==1.7.9
wsgiref==0.1.2
settings.py
import djcelery
djcelery.setup_loader()
INSTALLED_APPS = (
...
'djcelery',
...
)
CACHES = {
"default": {
"BACKEND": "redis_cache.cache.RedisCache",
"LOCATION": "127.0.0.1:6379:1",
"OPTIONS": {
"CLIENT_CLASS": "redis_cache.client.DefaultClient",
}
}
}
BROKER_URL = 'redis://localhost:6379/0'
when I start my redis server and run
./manage.py celeryd -B
My connect just keeps timing out with:
Traceback (most recent call last):
File "/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 395, in start
self.consume_messages()
File "/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 407, in consume_messages
with self.hub as hub:
File "/venv/lib/python2.7/site-packages/celery/worker/hub.py", line 198, in __enter__
self.init()
File "/venv/lib/python2.7/site-packages/celery/worker/hub.py", line 146, in init
callback(self)
File "/venv/lib/python2.7/site-packages/celery/worker/consumer.py", line 401, in on_poll_init
self.connection.transport.on_poll_init(hub.poller)
File "/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 749, in on_poll_init
self.cycle.on_poll_init(poller)
File "/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 266, in on_poll_init
num=channel.unacked_restore_limit,
File "/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 159, in restore_visible
self.restore_by_tag(tag, client)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 94, in Mutex
pipe.watch(name)
File "/venv/lib/python2.7/site-packages/redis/client.py", line 1941, in watch
return self.execute_command('WATCH', *names)
File "/venv/lib/python2.7/site-packages/redis/client.py", line 1760, in execute_command
return self.immediate_execute_command(*args, **kwargs)
File "/venv/lib/python2.7/site-packages/redis/client.py", line 1779, in immediate_execute_command
return self.parse_response(conn, command_name, **options)
File "/venv/lib/python2.7/site-packages/redis/client.py", line 1883, in parse_response
self, connection, command_name, **options)
File "/venv/lib/python2.7/site-packages/redis/client.py", line 388, in parse_response
response = connection.read_response()
File "/venv/lib/python2.7/site-packages/redis/connection.py", line 309, in read_response
raise response
ResponseError: unknown command 'WATCH'
[2013-06-12 18:25:23,059: ERROR/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
redis:
[1197] 12 Jun 18:50:09 * Server started, Redis version 1.3.14
[1197] 12 Jun 18:50:09 * DB loaded from disk: 0 seconds
[1197] 12 Jun 18:50:09 * The server is now ready to accept connections on port 6379
[1197] 12 Jun 18:50:09 - Accepted 127.0.0.1:53061
[1197] 12 Jun 18:50:09 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1197] 12 Jun 18:50:09 - 1 clients connected (0 slaves), 1076976 bytes in use
[1197] 12 Jun 18:50:09 - Accepted 127.0.0.1:53062
[1197] 12 Jun 18:50:09 - Accepted 127.0.0.1:53063
[1197] 12 Jun 18:50:09 - Client closed connection
[1197] 12 Jun 18:50:09 - Accepted 127.0.0.1:53064
[1197] 12 Jun 18:50:09 - Client closed connection
[1197] 12 Jun 18:50:09 - Accepted 127.0.0.1:53065
[1197] 12 Jun 18:50:09 - Client closed connection
[1197] 12 Jun 18:50:09 - Accepted 127.0.0.1:53066
[1197] 12 Jun 18:50:09 - Client closed connection
etc etc.
Any guidance for where I should be looking or what possible culprits are? thanks
Your Redis server is too old (1.3.14) to be used with Celery. From this error you can see Celery is trying to use the WATCH command which was introduced in Redis 2.2.