title: How to detect anomalous resource consumption reliably?
body: <p>This question is about a whole class of similar problems, but I'll ask it as a concrete example.</p>
  
  <p>I have a server with a file system whose contents fluctuate. I need to monitor the available space on this file system to ensure that it doesn't fill up. For the sake of argument, let's suppose that if it fills up, the server goes down.</p>
  
  <p>It doesn't really matter what it is -- it might, for example, be a queue of "work".</p>
  
  <p>During "normal" operation, the available space varies within "normal" limits, but there may be pathologies:</p>
  
  <ul>
  <li>Some other (possibly external)
  component that adds work may run out
  of control</li>
  <li>Some component that removes work seizes up, but remains undetected </li>
  </ul>
  
  <p>The statistical characteristics of the process are basically unknown.</p>
  
  <p>What I'm looking for is an algorithm that takes, as input, timed periodic measurements of the available space (alternative suggestions for input are welcome), and produces as output, an alarm when things are "abnormal" and the file system is "likely to fill up". It is obviously important to avoid false negatives, but almost as important to avoid false positives, to avoid numbing the brain of the sysadmin who gets the alarm. </p>
  
  <p>I appreciate that there are alternative solutions like throwing more storage space at the underlying problem, but I have actually experienced instances where 1000 times wasn't enough.</p>
  
  <p>Algorithms which consider stored historical measurements are fine, although on-the-fly algorithms which minimise the amount of historic data are preferred.</p>
  
  <hr>
  
  <p>I have accepted Frank's answer, and am now going back to the drawing-board to study his references in depth.</p>
  
  <p>There are three cases, I think, of interest, not in order:</p>
  
  <ol>
  <li>The "Harrods' Sale has just started" scenario: a peak of activity that at one-second resolution is "off the dial", but doesn't represent a real danger of resource depletion;</li>
  <li>The "Global Warming" scenario: needing to plan for (relatively) stable growth; and</li>
  <li>The "Google is sending me an unsolicited copy of The Index" scenario: this will deplete all my resources in relatively short order unless I do something to stop it.</li>
  </ol>
  
  <p>It's the last one that's (I think) most interesting, and challenging, from a sysadmin's point of view..</p>