title: Should one test internal implementation, or only test public behaviour?
body: <p>Given software where ...</p>
  
  <ul>
  <li>The system consists of a few subsystems</li>
  <li>Each subsystem consists of a few components</li>
  <li>Each component is implemented using many classes</li>
  </ul>
  
  <p>... I like to write automated tests of each subsystem or component.</p>
  
  <p>I don't write a test for each internal class of a component (except inasmuch as each class contributes to the component's public functionality and is therefore testable/tested from outside via the component's public API).</p>
  
  <p>When I refactor the implementation of a component (which I often do, as part of adding new functionality), I therefore don't need to alter any existing automated tests: because the tests only depend on the component's public API, and the public APIs are typically being expanded rather than altered.</p>
  
  <p>I think this policy contrasts with a document like <a href="http://www.agilealliance.org/system/article/file/995/file.pdf" rel="noreferrer">Refactoring Test Code</a>, which says things like ...</p>
  
  <ul>
  <li>"... unit testing ..."</li>
  <li>"... a test class for every class in the system ..."</li>
  <li>"... test code / production code ratio ... is ideally considered to approach a ratio of 1:1 ..."</li>
  </ul>
  
  <p>... all of which I suppose I disagree with (or at least don't practice).</p>
  
  <p><strong>My question is, if you disagree with my policy, would you explain why? In what scenarios is this degree of testing insufficient?</strong></p>
  
  <p>In summary:</p>
  
  <ul>
  <li>Public interfaces are tested (and retested), and rarely change (they're added to but rarely altered)</li>
  <li>Internal APIs are hidden behind the public APIs, and can be changed without rewriting the test cases which test the public APIs</li>
  </ul>
  
  <hr>
  
  <p>Footnote: some of my 'test cases' are actually implemented as data. For example, test cases for the UI consist of data files which contain various user inputs and the corresponding expected system outputs. Testing the system means having test code which reads each data file, replays the input into the system, and asserts that it gets the corresponding expected output.</p>
  
  <p>Although I rarely need to change test code (because public APIs are usually added to rather than changed), I do find that I sometimes (e.g. twice a week) need to change some existing data files. This can happens when I change the system output for the better (i.e. new functionality improves existing output), which might cause an existing test to 'fail' (because the test code only tries to assert that output hasn't changed). To handle these cases I do the following:</p>
  
  <ul>
  <li>Rerun the automated test suite which a special run-time flag, which tells it to not assert the output, but instead to capture the new output into a new directory</li>
  <li>Use a visual diff tool to see which output data files (i.e. what test cases) have changed, and to verify that these changes are good and as expected given the new functionality</li>
  <li>Update the existing tests by copying new output files from the new directory into the directory from which test cases are run (over-writing the old tests)</li>
  </ul>
  
  <hr>
  
  <p>Footnote: by "component", I mean something like "one DLL" or "one assembly" ... something that's big enough to be visible on an architecture or a deployment diagram of the system, often implemented using dozens or 100 classes, and with a public API that consists of only about 1 or a handful of interfaces ... something that may be assigned to one team of developers (where a different component is assigned to a different team), and which will therefore according to <a href="http://en.wikipedia.org/wiki/Conway&#39;s_Law" rel="noreferrer">Conway's Law</a> having a relatively stable public API.</p>
  
  <hr>
  
  <p>Footnote: The article <em><a href="http://www.rbsc.com/pages/myths.html" rel="noreferrer">Object-Oriented Testing: Myth and Reality</a></em> says,</p>
  
  <blockquote>
    <p><strong>Myth: Black box testing is sufficient.</strong>
    If you do a careful job of test case
    design using the class interface or
    specification, you can be assured that
    the class has been fully exercised.
    White-box testing (looking at a
    method's implementation to design
    tests) violates the very concept of
    encapsulation. </p>
    
    <p><strong>Reality: OO structure matters, part
    II.</strong> Many studies have shown that
    black-box test suites thought to be
    excruciatingly thorough by developers
    only exercise from one-third to a half
    of the statements (let alone paths or
    states) in the implementation under
    test. There are three reasons for
    this. First, the inputs or states
    selected typically exercise normal
    paths, but don't force all possible
    paths/states. Second, black-box
    testing alone cannot reveal surprises.
    Suppose we've tested all of the
    specified behaviors of the system
    under test. To be confident there are
    no unspecified behaviors we need to
    know if any parts of the system have
    not been exercised by the black-box
    test suite. The only way this
    information can be obtained is by code
    instrumentation. Third, it is often
    difficult to exercise exception and
    error-handling without examination of
    the source code.</p>
  </blockquote>
  
  <p>I should add that I'm doing whitebox functional testing: I see the code (in the implementation) and I write functional tests (which drive the public API) to exercise the various code branches (details of the feature's implementation).</p>