Using Passenger, we ran into a strange problem - how to monitor and kill runaway pages which were eating up too much RAM? Our usual monitoring infrastructure, monit, didn’t have the facilities to deal with Passenger since it primarily works with processes that have statically-located PID files.
Ideally, you would want to have a stack trace and suchlike as well, so you could figure out what was killing your server without a bunch of nasty profiling tests and runs and comparative analysis.
Well, here’s a script that does that.
kill -6 $(passenger-memory-stats | grep '[1-9]...\.. MB.*Rails' | awk '{ print $1 }')I put it into a cron that runs every minute. Here’s what that looks like:
*/1 * * * * /usr/local/bin/kill_bad_passengers
So, how’s it work? Well, wherever you’ve got Passenger installed, you can run passenger-memory-stats. This will give you a breakdown of which processes are running and how much memory they take up. We decided to kill anything taking more than a gig of RAM (hence the three dots after the character class in the grep statement up there). Kill -6 sends SIGABRT, which safely kills off the child process and logs an exception. Check the docs on that here.
Inspiration definitely came from Bill Harding (thanks!) and lots of help from fellow HeyZapper Chris. Thanks, guys!