Crash or… Monit!
As an application grows it starts demanding more and more work. At some point of the ongoing project, processing a task in the background becomes a necessity. One of the most popular tools to accomplish this in Ruby is Sidekiq.
Now, did it ever happen to you that you get it all up and running on the server and the next thing you know is the server is down and you get a colorful variety of error reports yelling that stuff is not working?
Meet Monit, a tool that’ll let you get ahead of trouble. It automatically restarts programs when they crash and can send email reports on failures!
In case you want to try it out –and make yourself a favour– here’s some aid on installing, configuring and get it up and running.
Depending on your package manager this may differ:
apt-get install monit
yum install monit
brew install monit
- Edit /etc/monit.conf or /etc/monitrc
- Uncomment (or add) these lines:
set httpd port 2812 and
use address localhost
Add Sidekiq entry
Create the file /etc/monit.d/sidekiq.monitrc with:
check process sidekiq_application_production0 with pidfile /path/to/shared/tmp/pids/sidekiq.pid
start program = "/bin/bash -c 'cd /path/to/app/current && source /home/user/.rvm/environments/ruby-2.0.0-p353@global && bundle exec sidekiq -e production -C /path/to/app/config/sidekiq.yml -P tmp/pids/sidekiq.pid -L log/sidekiq.log'"
stop program = "/bin/bash -c 'kill -s INT `cat /path/to/shared/tmp/pids/sidekiq.pid`'" with timeout 90 seconds
if totalmem is greater than 500 MB for 2 cycles then restart # eating up memory?
Let’s not forget to start Monit: /etc/init.d/monit start
Now a sudo monit status should show the sidekiq_application_production0 entry.
If you use capistrano you will notice the current and shared directories. If you don’t use capistrano just make sure the PID is stored in some place where the path doesn’t change to let Monit find it.
This configuration will also be useful if you use capistrano-sidekiq, just make sure the check process name in the config matches what is used during deploy.
This configuration includes a memory protection that will restart Sidekiq before it eats up all the memory in your server. Yes, I’ve got a server down more than once due to this issue 🙁
Configure email server
To receive Monit’s status via email just add the following to /etc/monit.conf or /etc/monitrc
Add the email server, e.g.: for gmail’s SMTP add in the following line to the config file
set mailserver smtp.gmail.com port 587 username "email@example.com" password "********" using tlsv1 with timeout 30 seconds
To subscribe to all emails, including non-error emails, add the following line to the config file:
set alert firstname.lastname@example.org
If everything is set up correctly, executing sudo monit reload should trigger an email.
If you want to get emails for specific events or have more questions regarding Monit’s config check out https://mmonit.com/monit/documentation/monit.html.
A word of warning, the doc is huge. If you are looking for something specific like email, alert or whatever you need, just search it.
That’s pretty much it. Of course Monit doesn’t take care of starting the processes if the server restarts. You’ll have to look for another tool to do that.