Enhancing Oban Job Error Reporting

It’s rough out there on the mean streets of the internet. Application’s invariably run into exceptions. Some are expected, most are unexpected, and either way, they’re unwanted. Oban jobs are no, er, exception—anything doing the real work of running queries, interacting with data, and making requests to the badlands of external networks is guaranteed to hit some snags.

Most production apps, at least those ran by people that care about their well-being, use an error monitor like Sentry, HoneyBadger, AppSignal, etc. to notify attentive devs when an error occurs.

For Oban, identifying precisely which job generated which error can be difficult without additional context and some custom grouping. Error reporters are tailored toward reporting errors for web requests or more blatant exceptions. Fortunately, with a few careful reporting tweaks, we can make job error reports just as detailed and actionable.

🙋 Why Focus on Sentry?

This article uses Sentry for its examples because it’s the most widely used official Elixir client. AppSignal and HoneyBadger are excellent alternatives with comparable mechanisms for all of the tips shown below.

Attaching the Error Handler

The standard Oban playbook outlines how to report exceptions via a standard telemetry event. Oban Pro makes it easier, and more reliable, with a global work hook. In either case, you have all the necessary information normalized and ready to report.

A basic handler looks something like this:

def handle_event([:oban, :job, :exception], _measure, %{job: job}, _conf) do
  %{reason: exception, stacktrace: stacktrace} = job.unsaved_error

  Sentry.capture_exception(exception, stacktrace: stacktrace)
end

Error details including the reason, kind, and stacktrace are stored in a job’s virtual unsaved_error field. That’s ultimately what is formatted and stashed in the errors field that’s saved to the database. The same fields are available in the telemetry event’s meta, but pulling it from unsaved_error works with a global hook as well.

Returned tuples like {:error, :boom} or {:cancel, :boop} are standardized as a Oban.PerformError and crashes are converted to an Oban.CrashError.

Normalization makes reporting simpler and removes the need to carefully inspect the error for reporting. It also means we can always use Sentry.capture_exception/2, because all error reasons are converted to exceptions. Unfortunately, normalization can also lead to incorrectly grouping unrelated exceptions together.

Fingerprinting Errors

Exceptions with the same name, no stacktrace, and slightly differing messages look the same to an error reporter. From the Sentry docs on error grouping:

By default, Sentry will run one of our built-in grouping algorithms to generate a fingerprint based on information available within the event such as stacktrace, exception, and message.

Without better hinting you’ll find yourself with thousands of unrelated PerformError reports from various jobs all grouped together when they don’t have anything in common. That’s not helpful. It makes debugging harder and masks real errors since you only receive an email on an error’s first occurrence.

That’s why Sentry, and other error reporting tools, provide a fingerprint parameter to give hints to the service about how things should be grouped together. A fingerprint built from the job’s worker and the exception module is granular enough to separate the same error from different workers:

fingerprint = [inspect(exception.__struct__), inspect(job.worker)]
# ["Oban.PerformError", "MyApp.BusinessWorker"]

However, in jobs that make liberal use of error tuples, the exception will always be Oban.PerformError, and we can use the exception’s message to be more specific:

fingerprint = [
  inspect(exception.__struct__),
  inspect(job.worker),
  Exception.message(exception)
]
# ["Oban.PerformError", "MyApp.BusinessWorker", "record not found"]

Now add the fingerprint to the context options:

opts = [fingerprint: fingerprint, stacktrace: stacktrace]

Sentry.capture_exception(exception, opts)

Additional Context

Distinguishing between different error notifications is a start. The next step is injecting more contextual details into those notifications to help diagnose the issue and drill down to the root cause. The primary mechanisms for better context are extra and tags maps.

The extra map is for custom, structured data. We can slice off a portion of the job’s fields and pass those along. Any fields that will help identify the job and recognize a pattern are candidates for the extra map:

extra = Map.take(job, ~w(args attempt id max_attempts meta queue tags worker)a)
opts = [extra: extra, fingerprint: fingerprint, stacktrace: stacktrace]

Sentry.capture_exception(exception, opts)

The extra fields are now shown in each report:

Grouping with Tags

Tags are an even better way to identify related events because they’re indexed and searchable. Fields like worker and queue, that are shared among many jobs, are perfect for tags.

tags = %{oban_worker: job.worker, oban_queue: job.queue, oban_state: job.state}
opts = [extra: extra, fingerprint: fingerprint, stacktrace: stacktrace, tags: tags]

Sentry.capture_exception(exception, opts)

While there’s little chance the names will conflict with existing tags, prefixing them with oban_ keeps them grouped and distinct. Now the tags are displayed with the runtime and server information:

Consistent Stacktraces

Erlang/Elixir stacktraces are a finicky beast due to tail call optimization and automatic truncation. They’re elusive outside of a catch block. A useful stacktrace is only provided when an exception or crash is caught. Error tuple returns don’t have any associated stacktrace, and we can’t retrieve one that’s of any use.

Sentry, and other reporters, show the last stacktrace entry in the report title. Fortunately, all we really need is an entry for the worker and process/1 (or perform/1 for standard workers), which is trivial to build manually:

stacktrace =
  case {Oban.Worker.from_string(job.worker), stacktrace} do
    {{:ok, worker}, []} -> [{job.worker, :process, 1, []}]
    _ -> stacktrace
  end

Sentry requires that the first element of a stacktrace is a module, aka an atom. Note the use of from_string/1 to safely convert the worker string to a module name while guarding against a missing module.

Having a relevant stacktrace entry makes it possible to spot the worker while scanning through notices. In the screenshot below, the top entry has a trace and the bottom doesn’t:

Finishing Up

Perhaps you’re thinking “those are all useful tips, why not make a library out of it and save us some time?” Well, there are many error reporting services out there, and we don’t want to play favorites too much.

Besides, there are different takes on the exact components of a fingerprint, various approaches toward tags, and you may have additional context that’s important to submit. A one-size-fits-all solution would be limiting.

Appendix: Putting it All Together

Here’s one last example that puts all the tips together in a single block you can use as a starting point:

def handle_event([:oban, :job, :exception], _measure, %{job: job}, _conf) do
  %{reason: exception, stacktrace: stacktrace} = job.unsaved_error

  fingerprint = [
    inspect(exception.__struct__),
    inspect(job.worker),
    Exception.message(exception)
  ]

  stacktrace =
    case {Oban.Worker.from_string(job.worker), stacktrace} do
      {{:ok, worker}, []} -> [{job.worker, :process, 1, []}]
      _ -> stacktrace
    end
  
  extra = Map.take(job, ~w(args attempt id max_attempts meta queue tags worker)a)
  tags = %{oban_worker: job.worker, oban_queue: job.queue, oban_state: job.state}
  opts = [extra: extra, fingerprint: fingerprint, stacktrace: stacktrace, tags: tags]
  
  Sentry.capture_exception(exception, opts)
end

As usual, if you have any questions or comments, ask in the Elixir Forum or the #oban channel on Elixir Slack. For future announcements and insight into what we're working on next, subscribe to our newsletter.