You can also read this and follow us on the Sales Beach Blog

Exception tracking is the to-do list of startup services. It’s very personal how you want exceptions tracked, and depends a lot on your app and business structure. If you’re a social network, errors are much less worrisome than if you’re making bank software.

When we spun out Sales Beach, I had to pick an error tracker. I’ve used a few different ones including Airbrake, Coalmine, and Rollbar, so I know what some of the pitfalls are. I looked through the options and tried to figure out which would work best for us. Here’s a quick summary of my search.

Please note; I’m a Rails guy, and Rails is my go-to framework for when I want to build something. So my analysis comes from that perspective; if you’re using Java or Python, you might want to look closely at how well any given service integrates into your stack.

For the most part, all these services email you when you get a new error and provide stack traces and detailed data on their site. They all have many language-specific integrations that are usually only a few lines of code you can copy + paste. I’ll note if they offer any features I consider differentiating or exceptional, where I found them. Mostly I’m concerned with price for the smallest plan, and how limiting the service is.

The biggest issue is that a lot of these services have a max exceptions / min limit, which means if one commonly-used feature breaks, it will max out the exception limit and you will miss other errors that come in, or it will blow out your monthly limit. That’s a huge problem, and a deal-breaker for us.

The Services

Errplane ★★★☆☆

  • $100 / mo, 50mil max, 5 server limit, 7 day retention

Airbrake ★☆☆☆☆

  • $40 / mo, 5 projects, 60 errors / min, 30 day retention, 50MB limit

Honey Badger ★★★★☆

  • $20 / mo, 3 projects, 7 day retention
  • No rate / storage limits, though they say they may ask you to upgrade if you’re a heavy user.
  • Integrates with tons of services - Hipchat, Asana, and Pivotal
  • The cheapo plan is in a tiny link on their pricing page - just go here.
  • Cheapo plan doesn’t offer service integration, so might be worth the $40 / mo

Raygun★★☆☆☆

  • $15 / mo, 1 project, 14 day retention, 5,000 exceptions / mo limit
  • $40 / mo and higher plans have search

Coalmine ★☆☆☆☆

  • Free, 1 project, 2 exceptions / min
  • $15 / mo for 10 exceptions / min

Bugsnag ★★★★★

  • $30 / mo, no user, project, or rate limits. 25k exceptions / mo, 25¢ per 1k extra
  • Intelligent grouping of exceptions, search, and shows number of users affected
  • Full disclosure: I worked with James & Simon, the founders of Bugsnag. I know them to be smart, awesome, and dedicated. This influenced my decision of which platform to use.

Rollbar ★★☆☆☆

  • Free, 3,000 exceptions / mo, 30 day retention
  • Integrates with Pivotal, Asana, etc
  • Says it has a configurable rate limit, but damned if I can find it. Just a way of hiding the rate limit pricing, in my opinion.
  • Interface is very deep - 10+ clicks to get to any useful settings, usually only a few settings / page

New Relic ???

  • New Relic is amazing for profiling & performance monitoring. It also tracks errors, but I couldn’t find any info on the limits and features it has. Docs for their ruby gem are here

Errbit ★★★☆☆

  • Open source, self-hosted, no limits
  • Github integration
  • Honestly, this looks awesome, but I didn’t want to pay for another production server. I don’t trust the capacity of free Heroku dynos to handle mission-critical data. Might be worth it on a Digital Ocean box, but then you have to invest time managing another server.

exception_notification gem ★★☆☆☆

  • Open source, no limits.
  • Just emails, nothing else. That means all errors & stack traces are searchable in GMail. Pretty gosh darn useful, even in combination with another service.

exceptions_to_hipchat gem ★☆☆☆☆

  • Open source, no limits
  • Posts exceptions in Hipchat. Make a separate room. Not useful if your app generates a lot of errors.

exception_engine gem ☆☆☆☆☆

  • Flat out didn’t work for me - some kind of cursor error.

Our Setup

Ultimately, we went with Bugsnag. While it has monthly limits, we don’t expect our app to exceed them in the near future, and pricing is reasonable. However, I quickly found that reporting errors in the background introduced a lot of repeating code, and put Bugnsag’s name all over our codebase. If we wanted to change services in the future, we’d have to fix all of those. To DRY it up, we refactored it out to use our own module, which handled all error reporting. Here’s the module:

Have a different service? Disagree? I welcome your thoughts on Twitter - or write your own blog post and @ me. :)