Check for duplicates in large vector of strings
<p>I'm attempting to find duplicate instances of strings where I have a vector of ~2.5 million strings.~</p>

<p>At the moment I use something like:</p>

<pre><code>std::vector&lt;string&gt; concatVec; // Holds all of the concatenated strings containing columns C,D,E,J and U.
std::vector&lt;string&gt; dupecheckVec; // Holds all of the unique instances of concatenated columns
std::vector&lt;unsigned int&gt; linenoVec; // Holds the line numbers of the unique instances only

// Copy first element across, it cannot be a duplicate yet
dupecheckVec.push_back(concatVec[0]);
linenoVec.push_back(0);

// Copy across and do the dupecheck
for (unsigned int i = 1; i &lt; concatVec.size(); i++)
{
    bool exists = false;

    for (unsigned int x = 0; x &lt; dupecheckVec.size(); x++)
    {
        if (concatVec[i] == dupecheckVec[x])
        {
            exists = true;
        }
    }

    if (exists == false)
    {
        dupecheckVec.push_back(concatVec[i]);
        linenoVec.push_back(i);
    }
    else
    {
        exists = false;
    }
}
</code></pre>

<p>Which is fine for small files, but obviously ends up taking an extremely long time as filesize grows due to the nested for loop and increasing number of strings contained in dupecheckVec.</p>

<p>What might be a less horrific way to do this in a large file?</p>
