title: Applying non-trivial functions to ordered subsets of data.table
body: <h3>Problem</h3>
  
  <p>I'm trying to use my newfound data.table powers (for good) to compute the frequency content of a bunch of data that looks like this:</p>
  
  <pre><code>|  Sample|  Channel|  Trial|     Voltage|Class  |  Subject|
  |-------:|--------:|------:|-----------:|:------|--------:|
  |       1|        1|      1|  -196.82253|1      |        1|
  |       1|        2|      1|   488.15166|1      |        1|
  |       1|        3|      1|  -311.92386|1      |        1|
  |       1|        4|      1|  -297.06078|1      |        1|
  |       1|        5|      1|  -244.95824|1      |        1|
  |       1|        6|      1|  -265.96525|1      |        1|
  |       1|        7|      1|  -258.93263|1      |        1|
  |       1|        8|      1|  -224.07819|1      |        1|
  |       1|        9|      1|   -87.06051|1      |        1|
  |       1|       10|      1|  -183.72961|1      |        1|
  </code></pre>
  
  <p>There are about 57 million rows--every variable is an integer except Voltage. Sample is an index that goes from 1:350, and Channel goes from 1:118. There are 280 Trials.</p>
  
  <h3>sample data</h3>
  
  <p>Martín's example data is valid, I believe (the numbers of categorical variables are a non-issue with respect to the errors):</p>
  
  <pre><code>big.table &lt;- data.table(Sample = 1:350, Channel = 1:118, Trial = letters,
               Voltage = rnorm(10e5, -150, 100), Class = LETTERS, Subject = 1:20)
  </code></pre>
  
  <h3>process</h3>
  
  <p>The first thing I do is set the key to Sample, because I want anything I do to the individual data series to happen in a sane order:</p>
  
  <pre><code>setkey(big.table,Sample)
  </code></pre>
  
  <p>Then, I do some filtering on the Voltage signals to remove high frequencies. (The filtering function returns a vector of the same length as its second argument):</p>
  
  <pre><code>require(signal)
  high.pass &lt;- cheby1(cheb1ord(Wp = 0.14, Ws = 0.0156, Rp = 0.5, Rs = 10))
  big.table[,Voltage:=filtfilt(high.pass,Voltage),by=Subject]
  </code></pre>
  
  <h3>initial error</h3>
  
  <p>I'd like to see if that processed it properly (i.e. Subject by Subject, Trial by Trial, Channel by Channel, in Sample order), so I add a column containing the spectral content of the Voltage column:</p>
  
  <pre><code>get.spectrum &lt;- function(x) {
      spec.obj &lt;- spectrum(x,method="ar",plot=FALSE)
      outlist &lt;- list()
      outlist$spec &lt;- 20*log10(spec.obj$spec)
      outlist$freq &lt;- spec.obj$freq
      return(outlist)
    }
  big.table[,c("Spectrum","Frequency"):=get.spectrum(Voltage),by=Subject]
  
  Error: cannot allocate vector of size 6.1 Gb
  </code></pre>
  
  <p>I think the issue is that <code>get.spectrum()</code> is trying to eat the whole column at once, considering that the whole table is only around 1.7GB. Is that so? What are my options?</p>
  
  <h2>What have you tried?</h2>
  
  <h3>Increasing the granularity of grouping</h3>
  
  <p>If I make a call to <code>get.spectrum</code> including all of the columns I want to group by, I get a more promising error:</p>
  
  <pre><code>big.table[,c("Spectrum","Frequency"):=get.spectrum(Voltage),
          by=c("Subject","Trial","Channel","Sample")]
  
  Error in ar.yw.default(x, aic = aic, order.max = order.max, na.action = na.action,  : 
    'order.max' must be &gt;= 1
  </code></pre>
  
  <p>That implies the <code>spectrum()</code> function I'm calling is getting data of the wrong shape.</p>
  
  <h3>Cutting points down, trying different 'where' conditions</h3>
  
  <p>Following Roland's advice, I cut the number of points to around 20 million and tried the below:</p>
  
  <pre><code>big.table[,"Spectrum":=get.spectrum(Voltage),
          by=c("Subject","Trial","Channel")]
  
  Error in `[.data.table`(big.table, , `:=`("Spectrum", get.spectrum(Voltage)),  :
    All items in j=list(...) should be atomic vectors or lists. If you are trying something like
    j=list(.SD,newcol=mean(colA)) then use := by group instead (much quicker), or cbind or merge 
    afterwards.
  </code></pre>
  
  <p>My thinking was that I shouldn't group by Sample since I want to apply this function to each group of 350 Samples given by the above <code>by</code> vector.</p>
  
  <p>Improving on that with some things gleaned from section 2.16 of the data.table FAQ, I added the SQL equivalent of an <code>ORDER BY</code>. I know that the Sample column needs to go from 1:350 for each input to the <code>spectrum()</code> function:</p>
  
  <pre><code>&gt; big.table[Sample==c(1:350),c("Spectrum","Frequency"):=as.list(get.spectrum(Voltage)),
  +             by=c("Subject","Trial","Channel")]
  Error in ar.yw.default(x, aic = aic, order.max = order.max, na.action = na.action,  : 
    'order.max' must be &gt;= 1
  </code></pre>
  
  <p>Again, I run into trouble with non-unique inputs.</p>