Settings Changelog for Oban Pro v1.3

This release is entirely dedicated to Smart engine optimizations, from slashing queue transactions to boosting bulk insert performance.

📮 Async Tracking

Rather than synchronously recording updates (acks) in a separate transaction after jobs execute, the Smart engine now bundles acks together to minimize transactions and reduce load on the database.

Async tracking, combined with the other enhancements detailed below, showed the following improvements over the previous Smart engine when executing 1,000 jobs with concurrency set to 20:

• Transactions reduced by 97% (1,501 to 51)
• Queries reduced by 94% (3,153 to 203)

That means less Ecto pool contention, fewer transactions, fewer queries, and fewer writes to the oban_producers table! There are similar, albeit less flashy, improvements over the Basic engine as well.

Notes and Implementation Details

• Acks are stored centrally per queue and flushed with the next transaction using a lock-free mechanism that never drops an operation.

• Acks are grouped and executed as a single query whenever possible. This is most visible in high throughput queues.

• Acks are preserved across transactions to guarantee nothing is lost in the event of a rollback or an exception.

• Acks are flushed on shutdown and when the queue is paused to ensure data remains as consistent as the previous synchronous version.

• Acking is synchronous in testing mode, when draining jobs, and when explicitly enabled by a flag provided to the queue.

See the Smart engine's async tracking section for more details and instructions on how to selectively opt out of async mode.

v1.3.0 — 2024-01-16

Enhancements

• [Smart] Skip extra query to "touch" the producer when acking without global or rate limiting enabled. This change reduces overall producer updates from 1 per job to 2 per minute for standard queues.

• [Smart] Avoid refetching the local producer's data when fetching new jobs.

  Async acking is centralized through the producer, which guarantees global and rate tracking data is up-to-date before fetching without an additional read.

• [Smart] Optimize job insertion with fewer iterations.

  Iterating through job changesets as a map/reduce with fewer conversions improves inserting 1k jobs by 10% while reducing overall memory by 9%.

• [Smart] Efficiently count changesets during insert_all.

  Prevent duplicate iterations through changesets to count unique jobs. Iterating through them once to accumulate multiple counts improved insertion by 3% and reduced overall memory by 2%.

• [Smart] Acking cancelled jobs is done with a single operation and limited to queues with global limiting.

Bug Fixes

• [Smart] Always merge acked meta updates dynamically.

  All meta-updating queries are dynamically merged with existing meta. This prevents recorded jobs from clobbering other meta updates made while the job executed.

• [Smart] Safely extract producer uuid from attempted_by with more than two elements

• [DynamicCron] Preserve stored opts such as args, priority, etc., on reboot when no new opts are set.

• [Relay] Skip attempting relay notifications when the associated Oban pid isn't alive.