The idea of a queuing service is to offload something that’s potentially resource intensive and takes time to process. Moving a piece of a logic that would originally be executed at visitor run time could potentially speed up the users experience, this alone makes queuing a winner.
Slow websites have become a trending topic lately, with lots of sites writing articles.
Recently I was able to use beanstalkd to offload error prone logic making calls to Third Party Services that were sometimes lengthy. This meant I was able to trigger retries if the code failed to execute successfully for whatever reason.
“Beanstalkd is a simple, fast work queue. It’s interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.” It was originally created to power the Facebook application Causes.
There are more and more queue’s becoming available with more of a SAAS model, potentially making them easier to implement but also having other downsides such as price.
Popular queue services
These services generally follow the AMQP which has produced a standard. These may be more appealing if using a PAAS due to the lack of System Administrator abilities available.
Lots of plugins have become available for popular frameworks such as CakePHP which can potentially be alternates to the previous services mentioned.
Working with the beanstalkd queue service
Setup beanstalkd
apt-get install beanstalkd
Now if you want to alter your beanstlalkd config to support logging of jobs in-case the service dies you will need to alter “/etc/default/beanstalkd”
Changing “DAEMON_OPTS” to use the -b command. ## Debian systems. Append “-b /var/lib/beanstalkd” for persistent.
Uncommnet “START=yes” so beanstalkd starts automaticlly on boot
While here you may need to change “BEANSTALKD_LISTEN_ADDR” to “127.0.0.1”. If you’re running a multiple servers over a loadbalancer and want a dedicated queue server it would be best for you to change the listen address to the servers internal IP Address, grab this by using “ifconfig”.
Save and start beanstalkd.
sudo service beanstalkd start
Example of /etc/init.d/beanstalkd
Defaults for the beanstalkd init script, /etc/init.d/beanstalkd on
Debian systems. Append ``-b /var/lib/beanstalkd'' for persistent
storage.
##BEANSTALKD_LISTEN_ADDR=0.0.0.0
BEANSTALKD_LISTEN_ADDR=127.0.0.1
BEANSTALKD_LISTEN_PORT=11300
DAEMON_OPTS="-l $BEANSTALKD_LISTEN_ADDR -p $BEANSTALKD_LISTEN_PORT -b /var/lib/beanstalkd"
Uncomment to enable startup during boot.
START=yes
Gotcha
When installing beanstalkd make sure you’re running the correct version as working with older versions of Ubuntu Lucid you’ll find an old version of beanstalkd gets installed due to this. This causes issues if you use a tube name with “_” (underscores).
The workaround is to wget the different package and install.
Example:
wget http://netapp.ma.utexas.edu/pub/.snapshot/lucid/ubuntu/pool/universe/b/beanstalkd/beanstalkd_1.4.6-1_amd64.deb
dpkg -i beanstalkd_1.4.6-1_amd64.deb
Creating a daemon service to work with beanstalkd
This is easy to achieve with Debian based systems thanks to Upstart slowly replacing Cron functionality. Generally you’ll find you can place your new service within “/etc/init/blah.conf”
An example of a service:
# Example service
description "Example service triggering a script that emails users"
author "Tom Rothwell <tom@pollenizer.com>"
start on startup
stop on shutdown
respawn
#respawn limit 2 5
script
exec /var/www/example/app/Console/cake -app “/var/www/example/app” EmailFriends -q >> /var/log/example.log 2>&1
end script
With this in place you can start/check the service as per any normal service.
sudo service blah status
blah being the name of the conf file created within “/etc/init/”.
Now we have a daemon that will restart and always be running, as long as the triggered script doesn’t die due to something like syntax errors. In this case the service needs to be restarted. With this setup you now have beanstalkd running and a service kicking off each job. The script that runs should be interfacing with beanstalkd somehow and grabbing a job from the queue. Lots of client libraries exist to help you do this.