title: Data Processing In C# - Best Approach?
body: <p>I am still on a learning curve in C# and SQL Server so please forgive my ‘greeness’.</p>
  
  <p>Here is my scenario:</p>
  
  <p>I have an EMPLOYEE table with  10,000 rows. Each of these employees has transactions in a TRANSACTIONS table.</p>
  
  <p>The transaction table has the salary elements like Basic pay, Acting allowance, Overtime hours etc. It also has payroll deductions like advance deductions, some loans (with interest), and savings (pension, social security savings etc.</p>
  
  <p>I need to go through each employee’s transactions and compute taxes, outstanding balances on loans, update balances on savings, convert hours into payments/deductions and some other stuff.</p>
  
  <p>This processing will give me a new set of rows for each employee, with a period marker (eg 2013-04 for April 2013). I need to store this in a HISTORY table for future references.</p>
  
  <p>What is the best approach for processing the entire 10,000 employee table and their transactions?</p>
  
  <p>I am told that pulling the entire table into memory via readers is not good practice and I agree.</p>
  
  <p>Do I keep pulling an employee from the database, process their transactions, and commit the history to the database? And pull the next and so forth?</p>
  
  <p>Too many calls to the back end?</p>
  
  <p>(EF not an option for me, still doing raw SQL in ADO.NET)</p>
  
  <p>I will appreciate any help on this.</p>