title: FactoryBot refuses to build factory with associations
body: <p>Can't find the reason why FactoryBot refuses to build factory with associations to same model with has_and_belongs_to_many relation.</p>
  
  <p>Product model:</p>
  
  <pre><code>class Product &lt; ApplicationRecord
  #...
    has_and_belongs_to_many :refs,
      class_name: "Product",
       join_table: :refs,
      foreign_key: :from_id,
      association_foreign_key: :to_id,
      dependent: :destroy
    after_destroy :destroy_inverse_refs
  #...
  end
  </code></pre>
  
  <p>Ref model:</p>
  
  <pre><code>class Ref &lt; ActiveRecord::Base
  #...
    belongs_to :from, class_name: 'Product', foreign_key: 'from_id'
    has_one :to, class_name: 'Product', foreign_key: 'id', primary_key: 'to_id'
  end
  </code></pre>
  
  <p>Factory:</p>
  
  <pre><code>FactoryBot.define do
    factory :ref do
      association :from, factory: :product
      association :to, factory: :product
    end
  end
  </code></pre>
  
  <p>Can't find the reason why this factory builds :from association with id and :to association without it:</p>
  
  <pre><code>ref = FactoryBot.build(:ref)
  
  puts ref.inspect
  #&lt;Ref id: nil, from_id: 8965, to_id: nil, created_at: nil, updated_at: nil&gt;
  
  puts ref.from.inspect
  #&lt;Product id: 8965, category_id: 12167, name_en: "Product name 1"&gt;
  
  puts ref.to.inspect
  #&lt;Product id: nil, category_id: 12168, name_en: "Product name 2"&gt;
  </code></pre>