26 November 2013
Post by Daniel Lombraña
Cover photo by Jakub Michankow
Reading time ~
GIFs & memes ~ 0
arrow_back Back to entries
In the last days I have been working really hard to add a new cache system to PyBossa.
Up to now, PyBossa has been using Flask-Cache, an extension for the Flask micro-framework
that allows you to use several types of caching backends (i.e. memcached or Redis).
Crowdcrafting was using memcached, as it is a robust solution and it is really well documented.
However in the last days due to high loads in the server, I’ve been thinking how I can
improve the situation, and have a cache system that meets the following criteria:
After checking several solutions, the best candidate that meets those requirements is: Redis.
Redis is amazing. Let me repeat it: amazing.
The reason I decided to start using
Redis as the main cache for PyBossa is that it has an incredible set of features,
a very simple master-slave setup, it is asynchronous, and over all those features: it gives you Sentinel.
Sentinel is a system that allows you to monitor your master-slave setup, and discover
Redis services in a very simple way. Fortunately, there is a Python client that uses
the Sentinel mode, and allows you to connect your Python software to Redis, handling behind
the curtains which nodes are alive and which nodes are not.
As Sentinel is perfect for what I wanted to build, I decided to start using it.
The result a new PyBossa caching module based on Redis.
The new module is very simple, it uses Sentinel to discover the master-slave setup
and configure two clients: master and slave. Then, all read actions are loaded from the slave node, and all write actions are done in the master, balancing the
load between the infrastructure.
The most interesting aspect of this setup, is that you can add more slaves in real time,
and Sentinel will handle them for you. If the master node dies, then it will look for
another master to configure it (this needs another Sentinel node), and voile: system
up and running without having to do any special.
Today I’ve deployed this new version (bumping PyBossa version to v0.2.0) in Crowdcrafting
and the speed of the site is really awesome!
The only “issue” with Redis, is that Ubuntu LTS does not have the most recent version
of it (not even 2.6), so you have to download it manually, and compile it. The good news are
that after installing the Ubuntu package build-essential you should have all the requirements to run
a simple make command and build it.
NOTE: in order to add it to the init.d section of Ubuntu, install first the old version via the
package manager, copy the files /etc/init.d/redis-server and /etc/redis/redis.conf to your
home folder, modify them to your needs (use the redis.conf as a guide for your redis.conf 2.6 config
file) and you will be done. Another option is to configure everything with Supervisor
which would be pretty handy (mental note, write a blog post about it).
Share this blog post: