# Sidekiq is too fast

FYI: This was presented at "Tokyo Rubyist Meetup" mid 2023.

\_\_\_\_\_\_\_\_\_

On an application I'm working on we added a job after a record has been created.

A record is created in multiple places. We are using Ruby on Rails, an easy way to not miss a place is by using an `after_create` hook.

```ruby
class User < ApplicationRecord
  after_create :schedule_job

  def schedule_job
    UserJob.perform_async(self.id)
  end
end
```

Write the Sidekiq job:

```ruby
class UserJob
  include Sidekiq::Job

  def perform(user_id)
    user = User.find(user_id)
    call_clever_method(user)
  end
end
```

Ok, job done! Shipped and feature is over ... Yes but `User.find(user_id)` raise `id=42 ActiveRecord::RecordNotFound` sometime ... like 20~30% of the time.

There are multiple possibilities:

1. the `user_id` is empty
    
2. the record has already been deleted from the database
    
3. the connection from the database is at fault
    
4. something else, yet unknown
    

Obviously, it's the last option.

Let's understand why the others are not the problem: 1. the error is `Couldn't find User with 'id'=42 (ActiveRecord::RecordNotFound)` the `id` is in the message, 2. I can see the record in the database when investigating and 3. the exception `ActiveRecord::RecordNotFound` is only raised after the database responded.

### What I thought was happening in a timeline:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1700224068396/48bc5304-d9bc-4298-81ac-960cda67251a.png align="center")

The `id` is return from the database, then pushed to Redis for Sidekiq to process. It must exists, right!

Thanks to clever coworker that remind me about database transactions. A transaction in a database is like a branch in git, until the branch is branch to the main branch it like it does not exists.

### Really timeline of the problem:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1700224697742/a73847b1-2938-46fb-b77c-2b9a8e0d55ce.png align="center")

The code was creating a transaction (it was implicit, no `.transaction` to be found) and with the speed of Sidekiq that pickup the job even before the database to commit the transaction, the Sidekiq job was already querying the main "branch" of the database.

Thankfully, it was on a create and the ActiveRecord `.find` raise an exception. On an update, the record would had existed and the job would had got the old data. Depending on the situation, this can have serious implications.

### The solution

We want to push to Sidekiq only when the transaction is committed. In Rails, it's trivial just add `_commit` to get `after_create_commit` . This tiny change ensure it runs after all the changes are available to all.

With that change, here's the new timeline:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1700226482209/e203db50-469f-4e20-9297-cbb43979a068.png align="center")

The full code, not much change but crucial:

```ruby
class User < ApplicationRecord
  after_create_commit :schedule_job # Change here

  def schedule_job
    UserJob.perform_async(self.id)
  end
end
```

### A different solution

I did present this Sidekiq/transaction issue to the Tokyo Rubyist Meetup earlier this year (2023). By publicizing the presentation I got a feedback:

[![](https://cdn.hashnode.com/res/hashnode/image/upload/v1700228023198/97100fa4-290f-4674-9b49-5a241aba1761.png align="center")](https://ruby.social/@getajobmike/110673549020015812)

Mike Perham, the creator of Sidekiq, provided support on Ruby.social (a Mastodon instance for Rubyist).

[Sidekiq 6.5](https://github.com/sidekiq/sidekiq/blob/ed9e01d427ed0503caa73191e32ff18a86e9f35e/Changes.md?plain=1#L253-L258) introduced the `transactional_push` which will automatically push the job to the queue after the database committed the transaction. No more need to remember to call `after_create_commit` instead of `after_create`, it will always work by default. Please note that it works if you use Sidekiq directly, the ActiveJob abstraction might interfere.

For that feature, just add this in the config:

```ruby
# config/initializers/sidekiq.rb:

Sidekiq.transactional_push!
```

That's all for today. I welcome constructive critiques, comments and please share.