Circular Dependency while using threads

Ruby on Rails
1

Hi

We are trying to use threads internally in our app, but it doesn’t seem like rails plays well with it at all. Have I named my files incorrectly, or do I need to change things around in application.rb?

I have created a fresh rails app (no database etc required) to recreate the error (note: Please use the parallel branch)

https://github.com/pratik60/circular_dependency_havoc/tree/parallel

Steps to set up the app

  1. git clone -b parallel git@github.com:pratik60/circular_dependency_havoc.git

  2. bundle install

After you have set up the app, you can recreate the error by doing

  1. bundle exec rake precomputation:fetch_all

In app/models/precomputation.rb, if you change

Parallel.each(ad_accounts, in_threads: 4) do |ad_account_id|
to
Parallel.each(ad_accounts, in_processses: 4) do |ad_account_id|, the error will disappear.

Does anyone know how to use threads and avoid circular dependency in rails?

Here are the 4 files added

lib/tasks/precomputation.rake

namespace :precomputation do
  desc "This fetches data for precomputation"
  task fetch_all: :environment do
      Precomputation.precompute_all_data
    # end
  end
end

app/models/precomputation.rb

class Precomputation
  def self.precompute_all_data
    ad_accounts = [1,2,3]
    Parallel.each(ad_accounts, in_threads: 4) do |ad_account_id|
      ActiveRecord::Base.connection_pool.with_connection do
         MongoPipeline::FbAdCampaignMongoPipeline.new(ad_account_id).fetch_all
      end
    end
  end
end

app/models/mongo_pipeline.rb

module MongoPipeline
  class Base
    def initialize(ad_account_id)
    end

    def insert_data
      puts 'inserting data'
    end

    def fetch_all
      extract_data # Child Class defines this method
      insert_data # Base class defines this method
    end
  end
end

app/models/mongo_pipeline/fb_ad_campaign_mongo_pipeline.rb

module MongoPipeline
  class FbAdCampaignMongoPipeline < MongoPipeline::Base
    def extract_data
      puts 'here is campaign data'
    end
  end
end

I end up with something like this

...
here is campaign data
inserting data
rake aborted!
Circular dependency detected while autoloading constant MongoPipeline::FbAdCampaignMongoPipeline
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:492:in `load_missing_constant'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:184:in `const_missing'
/webapps/circular_dependency_havoc/app/models/precomputation.rb:6:in `block (2 levels) in precompute_all_data'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/webapps/circular_dependency_havoc/app/models/precomputation.rb:5:in `block in precompute_all_data'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/parallel-1.6.2/lib/parallel.rb:410:in `call'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/parallel-1.6.2/lib/parallel.rb:410:in `call_with_index'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/parallel-1.6.2/lib/parallel.rb:288:in `block (2 levels) in work_in_threads'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/parallel-1.6.2/lib/parallel.rb:419:in `with_instrumentation'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/parallel-1.6.2/lib/parallel.rb:287:in `block in work_in_threads'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/parallel-1.6.2/lib/parallel.rb:183:in `block (2 levels) in in_threads'
Tasks: TOP => precomputation:fetch_all

Any ideas on what I’m doing wrong? I don’t think the issue is there with the parallel gem, as even with a custom thread library, I end up with the same.

Would absolutely love anyone’s help!

3

I initially changed this setting

config/environments/development.rb

config.eager_load = true

This didn’t help though!

And I needed to add

config/initializers/eager_load.rb:

Rails.application.eager_load! unless Rails.env.test?

You also need to eager load the entire lib folder, if you are using one (autoload isn’t enough).

config/application.rb

config.eager_load_paths += Dir["#{Rails.root}/lib/**/"]

Also, make sure that if devise is used in initializer it is renamed to 01_devise.rb or something, as initializers are loaded alphabetically, and your User or Admin, will reference it.

Probably suggest that in tests eager_load is skipped as it is suboptimal, and if you are not using threads internally, skip it altogether!