title: Stata: Using loops for keeping and reshaping a subset of variables
body: <p>The raw dat. files of my dataset have  ~281.000 columns which I need to transfer to Stata and reshape. Following the suggestions on SO posted as replies to my previous posts, I am trying to use insheet command, split and reshape the chunks of data and append them. The current “toy” version of my code looks as follows:</p>
  
  <pre><code>insheet using "D:\Trial_Import\products_4_3_2.dat", comma clear
  // extract vars
  
  tempfile orig 1 2 3 
  
  save `orig'
  
  keep simulation v2-v12
  save `1'
  
  use `orig'
  keep simulation v13-v22
  save `2'
  
  use `orig'
  keep simulation v23-v33
  save `3'
  
  // reshape 
  
  use `1'
  reshape long v, i(simulation) j(_count)
  save `1', replace
  
  use `2'
  reshape long v, i(simulation) j(_count)
  save `2', replace
  
  use `3'
  reshape long v, i(simulation) j(_count)
  save `3', replace
  
  
  append using `1'
  </code></pre>
  
  <p>There are two issues with the code:</p>
  
  <p>1)  It appends only the 1st  and 3rd data subsets, and ignores the 2nd. What could be the explanation for this? I suspect it might have something to do with using <code>insheet</code> vs. <code>use</code> when importing the raw data but I cannot figure out the way to fix it.</p>
  
  <p>2)  Since the real dataset contains 281,000 variables, and I was wondering if there is a way to automatize this process (i.e. to create a loop in order to keep and reshape each 10000 vars and then the next 10000 etc…) Does anyone know if it is feasible in Stata?</p>