<story><title>99% code coverage (2017)</title><url>https://rachelcarmena.github.io/2017/09/01/do-we-have-a-good-safety-net-to-change-this-legacy-code.html</url></story><parent_chain><item><author>austincheney</author><text>&amp;gt; but it&amp;#x27;s likely the test itself will take a significant amount of wall time to execute.&lt;p&gt;Why?  What about your application changed so that it is slower when testing compared to real world use?  If anything it should be dramatically faster because people don&amp;#x27;t provide microsecond accurate automated responses.  If the application naturally executes very quickly I would imagine it would take far long to set up the test scenario than to execute against it.&lt;p&gt;In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect.  There are not many administrative tasks I can complete from start to finish in that frame of time.</text></item><item><author>jeffasinger</author><text>I think your argument is missing the &amp;quot;cost&amp;quot; of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code.&lt;p&gt;For example, I worked on payroll software at some point. After finding a bug, we&amp;#x27;d want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who worked in one city in Ohio, lived in another, previous income above some amount, and with certain tax advantaged benefits. Setting up this test via feature testing is certainly possible, but it&amp;#x27;s likely the test itself will take a significant amount of wall time to execute. It&amp;#x27;s way faster to just test the payroll calculation code, which means you can run the tests more often, and with less developer inconvenience.</text></item><item><author>austincheney</author><text>It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever.&lt;p&gt;In my mind you only need 1 kind of testing: feature tests.  Does the application provide the expected output for a given input and&amp;#x2F;or configuration.  The application does all that it claims to do in a very precise way or it doesn&amp;#x27;t.  Everything else is extraneous, though you may have to test the application in wildly different ways with wildly different means of automation to validate all feature support.&lt;p&gt;Feature tests are executed by running the application with a given input and configuration and comparing the result against a known expected output.  Provided sufficient feature coverage this is enough to test for regression.  While the feature tests are running each test runs against a clock so that dramatic swings in performance can be qualified with numbers.  When that is not enough, as in the case of accessibility, have people (actual users) perform feature tests.&lt;p&gt;Other problems like bad design, redundant features, or poor code quality are qualified using code reviews and validation tools, like a linter.</text></item><item><author>cryptica</author><text>The industry is obsessed with getting 100% unit test code coverage even though it doesn&amp;#x27;t mean anything to the project.
The purpose of unit tests is to lock down the project&amp;#x27;s source code once it&amp;#x27;s essentially completed; to avoid regressions when making minor changes.&lt;p&gt;If you start writing unit tests too early in the project, you&amp;#x27;re effecively locking down units of code which haven&amp;#x27;t yet proved themseves to be useful to your project.
If you build square wheels for example, you may not realize that they&amp;#x27;re not designed correctly until you attach them to the car and realize that the car doesn&amp;#x27;t function well with them.
It makes no sense to preemtively write unit tests for a component which has a very high likelihood of not being in its final desired state; you&amp;#x27;re just giving yourself more work to refactor the unit tests over and over; or worse, you&amp;#x27;re afraid of refactoring the tests and so you lie to yourself thinking that square wheels are fine.&lt;p&gt;Integration tests are by far the most useful tests to have at the beginning and middle stages of the project; integration tests allow you to keep your focus on the real goals of the project and not get stuck on designing the perfect square wheel.</text></item></parent_chain><comment><author>jeffasinger</author><text>Sure, let&amp;#x27;s go back to my payroll example.&lt;p&gt;I need to make 8 HTTP requests to properly setup my test from a clean slate, including setting up the deductions, previous earnings, company location and employee home address. If each of those requests takes 50ms, and then I need to make a request to actually execute my test, and verify everything went well, that could easily be 500ms.&lt;p&gt;And that&amp;#x27;s just what needs to happen to run one end to end test of the payroll calculation feature. It&amp;#x27;s absolutely valuable to have a few such tests, but I&amp;#x27;m not going to run tests for all the weird one off scenarios (like what happens if someone lives in NJ, and works in Yonkers, and the company has a location in NJ as well?) because they&amp;#x27;d take minutes to run. I could run that entire test as a unit test in 2ms for the payroll calculation. That lets me run all of the tests I need very quickly.</text></comment>
<story><title>99% code coverage (2017)</title><url>https://rachelcarmena.github.io/2017/09/01/do-we-have-a-good-safety-net-to-change-this-legacy-code.html</url></story><parent_chain><item><author>austincheney</author><text>&amp;gt; but it&amp;#x27;s likely the test itself will take a significant amount of wall time to execute.&lt;p&gt;Why?  What about your application changed so that it is slower when testing compared to real world use?  If anything it should be dramatically faster because people don&amp;#x27;t provide microsecond accurate automated responses.  If the application naturally executes very quickly I would imagine it would take far long to set up the test scenario than to execute against it.&lt;p&gt;In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect.  There are not many administrative tasks I can complete from start to finish in that frame of time.</text></item><item><author>jeffasinger</author><text>I think your argument is missing the &amp;quot;cost&amp;quot; of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code.&lt;p&gt;For example, I worked on payroll software at some point. After finding a bug, we&amp;#x27;d want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who worked in one city in Ohio, lived in another, previous income above some amount, and with certain tax advantaged benefits. Setting up this test via feature testing is certainly possible, but it&amp;#x27;s likely the test itself will take a significant amount of wall time to execute. It&amp;#x27;s way faster to just test the payroll calculation code, which means you can run the tests more often, and with less developer inconvenience.</text></item><item><author>austincheney</author><text>It drives me nuts during interviews talking about test automation because people are very particular about the type of testing whether its unit testing, integration testing, acceptance testing, or whatever.&lt;p&gt;In my mind you only need 1 kind of testing: feature tests.  Does the application provide the expected output for a given input and&amp;#x2F;or configuration.  The application does all that it claims to do in a very precise way or it doesn&amp;#x27;t.  Everything else is extraneous, though you may have to test the application in wildly different ways with wildly different means of automation to validate all feature support.&lt;p&gt;Feature tests are executed by running the application with a given input and configuration and comparing the result against a known expected output.  Provided sufficient feature coverage this is enough to test for regression.  While the feature tests are running each test runs against a clock so that dramatic swings in performance can be qualified with numbers.  When that is not enough, as in the case of accessibility, have people (actual users) perform feature tests.&lt;p&gt;Other problems like bad design, redundant features, or poor code quality are qualified using code reviews and validation tools, like a linter.</text></item><item><author>cryptica</author><text>The industry is obsessed with getting 100% unit test code coverage even though it doesn&amp;#x27;t mean anything to the project.
The purpose of unit tests is to lock down the project&amp;#x27;s source code once it&amp;#x27;s essentially completed; to avoid regressions when making minor changes.&lt;p&gt;If you start writing unit tests too early in the project, you&amp;#x27;re effecively locking down units of code which haven&amp;#x27;t yet proved themseves to be useful to your project.
If you build square wheels for example, you may not realize that they&amp;#x27;re not designed correctly until you attach them to the car and realize that the car doesn&amp;#x27;t function well with them.
It makes no sense to preemtively write unit tests for a component which has a very high likelihood of not being in its final desired state; you&amp;#x27;re just giving yourself more work to refactor the unit tests over and over; or worse, you&amp;#x27;re afraid of refactoring the tests and so you lie to yourself thinking that square wheels are fine.&lt;p&gt;Integration tests are by far the most useful tests to have at the beginning and middle stages of the project; integration tests allow you to keep your focus on the real goals of the project and not get stuck on designing the perfect square wheel.</text></item></parent_chain><comment><author>nordsieck</author><text>&amp;gt; In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect.&lt;p&gt;An entire unit test suite should complete in well under 2 seconds.</text></comment>