title: SAS: Improving the speed of a do loop with proc import
body: <p>I have over 3400 CSV files, with size varying between 10kb to 3mb. Each CSV files have this generic filename: <code>stockticker-Ret.csv</code> where <code>stockticker</code> is the stock ticker like AAPL, GOOG, YHOO, and so on and has stock returns for at every minute on a given day. My SAS code first start by loading all the stock ticker names from the <code>stockticker-Ret.csv</code> file in a SAS dataset. I loop over each ticker to load the appropriate <code>.csv</code> file in a SAS dataset called <code>want</code> and apply some datasteps on <code>want</code> and store the final dataset <code>want</code> of each ticker in a SAS dataset called <code>global</code>. As you can imagine, this process takes a long time. Is there a way to improve my <code>DO LOOP</code> code below to make this process go faster?   </p>
  
  <pre><code>/*Record in a sas dataset all the csv file name to extract the stock ticker*/
      data yfiles;
      keep filename;
      length fref $8 filename $80; 
      rc = filename(fref, 'F:\data\'); 
      if rc = 0 then do; did = dopen(fref); 
      rc = filename(fref); end; else do; length msg $200.; msg = sysmsg(); put msg=; did = .; end;
      if did &lt;= 0 then putlog 'ERR' 'OR: Unable to open directory.';
      dnum = dnum(did);
      do i = 1 to dnum; filename = dread(did, i); /* If this entry is a file, then output. */ fid = mopen(did, filename); if fid &gt; 0 then output; end;
      rc = dclose(did);
      run;
  
  /*store in yfiles all the stock tickers*/
      data yfiles(drop=filename1 rename=(filename1=stock));
      set yfiles;
      filename1=tranwrd(filename,'-Ret.csv','');
      run;
  
      proc sql noprint;
      select stock into :name separated by '*' from work.yfiles;
      %let count2 = &amp;sqlobs;
      quit;
  
  
      *Create the template of the desired GLOBAL SAS dataset;
      proc sql;
      create table global
      (stock char(8), time_gap num(5), avg_ret num(5));
      quit;
  
      proc sql;
      insert into global
      (stock, time_gap,avg_ret)
      values('',0,0);
      quit;
  
      %macro y1;
      %do i = 1 %to &amp;count2;
      %let j = %scan(&amp;name,&amp;i,*);
      proc import out = want datafile="F:\data\&amp;j-Ret.csv"
      dbms=csv replace;
      getnames = yes;
      run;
  
  
      data want;
      set want; ....
  
      ....[Here I do 5 Datasteps on the WANT sasfile] 
  
  
  /*Store the want file in a global SAS dataset that will contain all the stock tickers from the want file*/
  
      data global;
      set global want; run;
  
      %end;
      %mend y1;
      %y1()
  </code></pre>
  
  <p>As you can see the global SAS dataset expands for every <code>want</code> dataset that I store in <code>global</code>.        </p>