Determining if a possible new entry already exists in a dataset For example, if I have the following data table layout:

dt.Columns.Add("ticketid", typeof(Int32));
dt.Columns.Add("createtime", typeof(DateTime));
dt.Columns.Add("creator", typeof(string));
dt.Columns.Add("ticketText", typeof(string));


If I want to add a new entry I need to make sure that there exists none already that has the same primary key (ticketid, createtime, creator).

The check is as follows:

if (
    dt.Rows.Count &lt;= 0 ||
    (
        dt.Rows.Count &gt; 0 &amp;&amp;
                    null == dt.AsEnumerable().Where
                        (
                            r =&gt; r.Field&lt;int&gt;("ticketid") == newEntry.TicketId
                                &amp;&amp; DateTime.Equals(r.Field&lt;DateTime&gt;("createtime"), newEntry.CreateTime)
                                &amp;&amp; r.creator == newEntry.Creator
                         )
                         .FirstOrDefault()
   ))


The code does what it is meant for, BUT my problem is that it is SLOW. In my case I need to read 2 million entries from the file and put them into the database as fast as possible at which this code fails (as soon as I use the code the runtime goes from just 1 minute to over 30 minutes). Is there any faster way to do this?

Site-note: the datatable is read from a file and the above is a check to make sure the data is ok before it is bulk inserted into the db.

Example with happenings:

if (
    dt.Rows.Count &lt;= 0 ||
    (
        dt.Rows.Count &gt; 0 &amp;&amp;
                    null == dt.AsEnumerable().Where
                        (
                            r =&gt; r.Field&lt;int&gt;("ticketid") == newEntry.TicketId
                                &amp;&amp; DateTime.Equals(r.Field&lt;DateTime&gt;("createtime"), newEntry.CreateTime)
                                &amp;&amp; r.creator == newEntry.Creator
                         )
                         .FirstOrDefault()
   ))
{
    // No duplicate found so insert the data into the data table
   dt.Rows.Add(new object[] {newEntry.TicketId, newEntry.CreateTime, newEntry.Creator});
}
Introduce a new object:

public class PrimaryKey
{
    public int TicketId { get; set; }
    public DateTime CreateTime { get; set; }
    public string Creator { get; set; }

    public PrimaryKey(int ticketId, DateTime createTime, string creator)
    {
        TicketId = ticketId;
        CreateTime = createTime;
        Creator = creator;
    }

    public override int GetHashCode()
    {
        int hash = 13;
        hash = (hash * 7) + TicketId.GetHashCode();
        hash = (hash * 7) + CreateTime.GetHashCode();
        hash = (hash * 7) + Creator.GetHashCode();
        return hash;
    }
    public override bool Equals(object value)
    {
        var other = value as PrimaryKey;
        return other != null 
             &amp;&amp; other.TicketId == TicketId 
             // ... etc.
    }
}


Use this in a hashset:

// right at the beginning...
var dedupeList = new Hashset&lt;PrimaryKey&gt;();
// Populate hashset from data table 

if (dedupeList.Add(new PrimaryKey(newEntry.TicketId, newEntry.CreateTime, newEntry.Creator))
{
    // New
}
else
{
    // Already added.
}


You wont have to search through the whole data table each time which should save a lot of time when the data table is large. However, you will use more memory.