Crash or… Monit!

As an application grows it starts demanding more and more work. At some point of the ongoing project, processing a task in the background becomes a necessity. One of the most popular tools to accomplish this in Ruby is Sidekiq.

Now, did it ever happen to you that you get it all up and running on the server and the next thing you know is the server is down and you get a colorful variety of error reports yelling that stuff is not working?

Meet Monit, a tool that’ll let you get ahead of trouble. It automatically restarts programs when they crash and can send email reports on failures!

In case you want to try it out –and make yourself a favour– here’s some aid on installing, configuring and get it up and running.

Install Monit

Depending on your package manager this may differ:

apt-get install monit
yum install monit
brew install monit

Configure monit

1. Edit /etc/monit.conf  or /etc/monitrc

2. Uncomment (or add) these lines:

set httpd port 2812 and
  use address localhost
  allow localhost

Add Sidekiq entry

Create the file /etc/monit.d/sidekiq.monitrc  with:

check process sidekiq_application_production0 with pidfile /path/to/shared/tmp/pids/sidekiq.pid
start program = "/bin/bash -c 'cd /path/to/app/current && source /home/user/.rvm/environments/ruby-2.0.0-p353@global && bundle exec sidekiq -e production -C /path/to/app/config/sidekiq.yml -P tmp/pids/sidekiq.pid -L log/sidekiq.log'"
stop program = "/bin/bash -c 'kill -s INT `cat /path/to/shared/tmp/pids/sidekiq.pid`'" with timeout 90 seconds
if totalmem is greater than 500 MB for 2 cycles then restart # eating up memory?
group sidekiq

Start Monit

Let’s not forget to start Monit: /etc/init.d/monit start

Now a sudo monit status should show the sidekiq_application_production0  entry.

If you use capistrano you will notice the current and shared directories. If you don’t use capistrano just make sure the PID is stored in some place where the path doesn’t change to let Monit find it.

This configuration will also be useful if you use capistrano-sidekiq, just make sure the check process name in the config matches what is used during deploy.

This configuration includes a memory protection that will restart Sidekiq before it eats up all the memory in your server. Yes, I’ve got a server down more than once due to this issue 🙁

Configure email server

To receive Monit’s status via email just add the following to /etc/monit.conf  or /etc/monitrc

Add the email server, e.g.: for gmail’s SMTP add in the following line to the config file

set mailserver smtp.gmail.com port 587 username “notifier@gmail.com” password “********” using tlsv1 with timeout 30 seconds

Subscribe

To subscribe to all emails, including non-error emails, add the following line to the config file:

set alert my_app_notifier@gmail.com

If everything is set up correctly, executing sudo monit reload  should trigger an email.

If you want to get emails for specific events or have more questions regarding Monit’s config check out https://mmonit.com/monit/documentation/monit.html.

A word of warning, the doc is huge. If you are looking for something specific like email, alert or whatever you need, just search it.

That’s pretty much it. Of course Monit doesn’t take care of starting the processes if the server restarts. You’ll have to look for another tool to do that.

Reposted from http://blog.afast.uy/2015/03/keep-sidekiq-running-using-monit.html