Speeding up Parallel.ForEach iterating through datatable and rendering report I am attempting to speed up the processing of 5000 rows received from the database, which I then render as a crystal report, export to a byte stream and save in a database table.

Currently I am using parallel.ForEach on a datatable. This uses 40 parallel processes that then sequentially iterates 125 (i.e 5000/40) records each of the 5000 rows.

It take approximately 5 minutes to do 5000 records at the moment. Is there any way I can speed this up?

DataTable dtBills = new DataTable("dtBills");//I used an OdbcDataAdapter to fill this with 5000 records

private void ParallelProcess()
{
    int numbills = 5000;
    int numprocesses = 40;
    int numsequential = numbills / numprocesses;

    int p=0;

    if (numbills &lt; numprocesses)
        p = numbills;
    else
        p = numprocesses;

    Parallel.For(1, p+1, new ParallelOptions { MaxDegreeOfParallelism = 40 }, i =&gt;
        {
            SequentialProcess(numsequential,i);

        });

}

private void SequentialProcess(int batch,int num)
{
    ReportDocument cryRpt = new ReportDocument();
    cryRpt.Load(Application.StartupPath + "\\" + "Bill.rpt");

    foreach(DataRow drow in dtBills.Rows.Cast&lt;System.Data.DataRow&gt;().Skip((num - 1) * batch).Take(batch).CopyToDataTable().Rows)

    {

        cryRpt.SetParameterValue(..... //here I set Crystal report Parameter values

        Stream stream = cryRpt.ExportToStream(CrystalDecisions.Shared.ExportFormatType.PortableDocFormat);
        byte[] contents = new byte[stream.Length];
        stream.Read(contents, 0, (int)stream.Length);
        stream.Close();

        if (!string.IsNullOrEmpty(sconnstr))
        {
            using (SqlConnection sconn = new SqlConnection(sconnstr))
            {

                String qry = @"INSERT INTO ....."
                using (SqlCommand cmd = new SqlCommand(qry, sconn))
                {

                    cmd.CommandType = CommandType.Text;

                    cmd.Parameters.AddWithValue(// Set values for insert here one of which will be contents

                    cmd.Connection = sconn;

                    sconn.Open();
                    cmd.ExecuteNonQuery();
                    sconn.Close();

                }


            }
        }

    }

    cryRpt.Close();
    cryRpt.Dispose();

}
The first thing here, you shouldn't be using directly the DataTable to read the results, use the method CreateDataReaderto create a Data Reader from it.

The difference is that a DataTable will check back on the database server on each iteration, while the DataReader will get a snapshot of your data and not check back anymore. That saves lots of time when dealing with a big number of records.

In this same channel, don't make the INSERT directly, but create it on memory, like an array or something (I think there is too something in the DataBase class that manages it in memory and commits later, but don't remember its name). If you stop dealing with the database one-at-a-time and doing in chunks, the speed of your code will improve.