title: Tag all duplicate rows in R as in Stata
body: <p>Following up from my question <a href="https://stackoverflow.com/q/22834725/1414455">here</a>, I am trying to replicate in R the functionality of the Stata command <code>duplicates tag</code>, which allows me to tag all the rows of a dataset that are duplicates in terms of a given set of variables:</p>
  
  <pre><code>clear *
  set obs 16
  g f1 = _n
  expand 104
  bys f1: g f2 = _n
  expand 2
  bys f1 f2: g f3 = _n
  expand 41
  bys f1 f2 f3: g f4 = _n
  des  // describe the dataset in memory
  
  preserve
  sample 10  // draw a 10% random sample
  tempfile sampledata
  save `sampledata', replace
  restore
  
  // append the duplicate rows to the data
  append using `sampledata'
  sort f1-f4
  
  duplicates tag f1-f4, generate(dupvar)
  browse if dupvar == 1  // check that all duplicate rows have been tagged
  </code></pre>
  
  <h2>Edit</h2>
  
  <p>Here is what Stata produces (added on @Arun's request):</p>
  
  <blockquote>
  <pre><code>f1   f2   f3   f4   dupvar  
   1    1    1    1        0  
   1    1    1    2        0  
   1    1    1    3        1  
   1    1    1    3        1  
   1    1    1    4        0  
   1    1    1    5        0  
   1    1    1    6        0  
   1    1    1    7        0  
   1    1    1    8        1  
   1    1    1    8        1
  </code></pre>
  </blockquote>
  
  <p>Note that for <code>(f1, f2, f3, f4) = (1, 1, 1, 3)</code> there are two rows, and both of those are marked <code>dupvar = 1</code>. Similarly, for the two rows that are duplicates for <code>(f1, f2, f3, f4) =(1, 1, 1, 8)</code>.</p>
  
  <h2>R:</h2>
  
  <p>The base function <code>duplicated</code> tags only the second duplicate onwards. So, I wrote a function to replicate the Stata functionality in R, using <code>ddply</code>. </p>
  
  <pre><code># Values of (f1, f2, f3, f4) uniquely identify observations
  dfUnique = expand.grid(f1 = factor(1:16),
              f2 = factor(1:41),
              f3 = factor(1:2),
              f4 = factor(1:104))
  
  # sample some extra rows and rbind them
  dfDup = rbind(dfUnique, dfUnique[sample(1:nrow(dfUnique), 100), ])
  
  # dummy data 
  dfDup$data = rnorm(nrow(dfDup))
  
  # function: use ddply to tag all duplicate rows in the data
  fnDupTag = function(dfX, indexVars) {
    dfDupTag = ddply(dfX, .variables = indexVars, .fun = function(x) {
      if(nrow(x) &gt; 1) x$dup = 1 else x$dup = 0
      return(x)
    })
    return(dfDupTag)
  }
  
  # test the function
  indexVars = paste0('f', 1:4, sep = '')
  dfTemp = fnDupTag(dfDup, indexVars)
  </code></pre>
  
  <p>But as in the linked question, performance is a huge issue. <a href="http://r.789695.n4.nabble.com/Identifying-duplicate-rows-td4642679.html" rel="nofollow noreferrer">Another possible solution</a> is</p>
  
  <pre><code>dfDup$dup = duplicated(dfDup[, indexVars]) | 
    duplicated(dfDup[, indexVars], fromLast = TRUE) 
  dfDupSorted = with(dfDup, dfDup[order(eval(parse(text = indexVars))), ])
  </code></pre>
  
  <p>I have a few questions:<br>
  1. Is it possible to make the <code>ddply</code> version faster?<br>
  2. Is the second version using <code>duplicated</code> correct? For more than two copies of the duplicated rows?
  3. How would I do this using <code>data.table</code>? Would that be faster?</p>