title: Speeding up the conditional filling of huge sparse matrices
body: <p>I was wondering if there is a way of speeding up (maybe via vectorization?) the conditional filling of huge sparse matrices (e.g. ~ 1e10 x 1e10). Here's the sample code where I have a nested loop, and I fill in a sparse matrix only if a certain condition is met:</p>
  
  <pre><code>% We are given the following cell arrays of the same size:
  % all_arrays_1
  % all_arrays_2
  % all_mapping_arrays
  
  N = 1e10; 
  
  % The number of nnz (non-zeros) is unknown until the loop finishes
  huge_sparse_matrix = sparse([],[],[],N,N);
  
  n_iterations = numel(all_arrays_1);
  
  for iteration=1:n_iterations
  
    array_1 = all_arrays_1{iteration};
    array_2 = all_arrays_2{iteration};
  
    mapping_array = all_mapping_arrays{iteration};
  
    n_elements_in_array_1 = numel(array_1);
    n_elements_in_array_2 = numel(array_2);
  
    for element_1 = 1:n_elements_in_array_1
  
      element_2     = mapping_array(element_1);
  
      % Sanity check:
      if element_2 &lt;= n_elements_in_array_2
         item_1 = array_1(element_1);
         item_2 = array_2(element_2);      
  
         huge_sparse_matrix(item_1,item_2) = 1;
      end     
    end   
  end
  </code></pre>
  
  <p>I am struggling to vectorize the nested loop. As far as I understand the filling a sparse matrix element by element is very slow when the number of entries to fill is large (~100M). I need to work with a sparse matrix since it has dimensions in the 10,000M x 10,000M range. However, this way of filling a sparse matrix in MATLAB is very slow.</p>
  
  <p><strong>Edits:</strong></p>
  
  <p>I have updated the names of the variables to reflect their nature better. There are no function calls. </p>
  
  <p><strong>Addendum:</strong></p>
  
  <p>This code builds the matrix adjacency for a huge graph. The variable <code>all_mapping_arrays</code> holds mapping arrays (~ adjacency relationship) between nodes of the graph in a <strong>local representation</strong>, which is why I need <code>array_1</code> and <code>array_2</code> to map the adjacency to a global representation.</p>