title: Working with very big data faster in Matlab?
body: <p>I have to deal with very big data (Point clouds generally more than 30 000 000 points) using Matlab. I can read ascii data using <code>textscan</code> function. After reading, I need to detect invalid data (points with <code>0,0,0</code> coordinates) and then I need to do some mathematical operations on each point or each line in the data. In my way, first I read data with <code>textscan</code> and then I assign this data to a matrix. Secondly, I use <code>for</code> loops for detecting invalid points and doing some mathematical operations on each point or line in the data. A sample of my code is shown as below. According to profile tool of Matlab <code>textscan</code> takes 37% and line</p>
  
  <pre><code>transformed_list((i:i),(1:4)) = coordinate_list((i:i),(1:4))*t_matrix; 
  </code></pre>
  
  <p>takes 35% of all computation time. </p>
  
  <p>I tried it with another point cloud (stores around 5 500 000) and profile tool reported same results. Is there a way of avoiding <code>for</code> loops, or is there another way of speeding up this computation?</p>
  
  <pre><code>fileID = fopen('C:\Users\Mustafa\Desktop\ptx_all_data\dede5.ptx');             
  original_data = textscan(fileID,'%f %f %f %f %f %f %f', 'delimiter',' ');
  fclose(fileID);
  column = original_data{1}(1);
  row = original_data{1}(2);
  t_matrix = [original_data{1}(7) original_data{2}(7) original_data{3}(7) original_data{4}(7)
      original_data{1}(8) original_data{2}(8) original_data{3}(8) original_data{4}(8)
      original_data{1}(9) original_data{2}(9) original_data{3}(9) original_data{4}(9)
      original_data{1}(10) original_data{2}(10) original_data{3}(10) original_data{4}(10)];
  coordinate_list(:,1) = original_data{1}(11:length(original_data{1}));
  coordinate_list(:,2) = original_data{2}(11:length(original_data{2}));
  coordinate_list(:,3) = original_data{3}(11:length(original_data{3}));
  coordinate_list(:,4) = 0;
  coordinate_list(:,5) = original_data{4}(11:length(original_data{4}));
  
  transformed_list = zeros(length(coordinate_list),5);
  for i = 1:length(coordinate_list)
      if coordinate_list(i,1) == 0 &amp;&amp; coordinate_list(i,2) == 0 &amp;&amp; coordinate_list(i,3) == 0
          transformed_list(i,:) = NaN;
      else
          %transformed_list(i,:) = coordinate_list(i,:)*t_matrix;
          transformed_list((i:i),(1:4)) = coordinate_list((i:i),(1:4))*t_matrix;
          transformed_list(i,5) = coordinate_list(i,5);
      end
      %i
  end
  </code></pre>
  
  <p>Thanks in advance</p>