The tragedy of 100% code coverage (2016) - <AUTHOR> http://labs.ig.com/code-coverage-100-percent-tragedy <comment><USER3> I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases.

Just because you ran a function or ran a line doesn't mean it will work for the range of inputs you are allowing. If your function that you are running coverage on calls into the OS or a dependency, you also have to be ready for whatever that might return.

Therefore you can't tell if your code is right just by having run it. Worse, you might be lulled into a false sense of security by saying it works because that line is "covered by testing".

The real answer is to be smart, pick the right kind of testing at the right level to get the most bang for your buck. Unit test your complex logic. Stress test your locking, threading, perf, and io. Integration test your services.

<comment><USER4> I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value.

However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-every-commit test suite just exercises with a selection of values likely to expose obvious problems.

<comment><USER5> Just out of curiosity... What is the point in testing all 2^24 possible color values?

<comment><USER4> The point is being certain I haven't missed an edge case somewhere.

The hard part is the percentage rgb() values, of which there are technically an uncountably infinite number (since any real number in the range 0-100 is a legal percentage value). For those I generate all 16,777,216 integer values, and verify that converting to percentage and back yields the original value.

</comment></comment></comment></comment><comment><USER6> There are a few relevant facts that should be known to everyone (including managers) involved in software development, but which probably are not:

1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program.

2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases from 1) to the point where the latter look tiny in comparison.

3) From 1) and 2), it is completely infeasible to exhaustively test a system of any significant size.

The corollary of 3) is that you cannot avoid being selective about what you test for, so the question becomes, do you want that decision to be an informed one, or will you allow it to be decided by default, as a consequence of your choice to aim for a specific percentage of path coverage?

For example, there are likely to many things that could be unit-tested for, but which could be ruled out as possibilities by tests at a higher level of abstraction. In that case, time spent on the unit tests could probably be better spent elsewhere, especially if (as with some examples from the article) a bug is not likely.

100% path coverage is one of those measures that are superficially attractive for their apparent objectivity and relative ease of measuring, but which don't actually tell you as much as they seem to. Additionally, in this case, the 100% part could be mistaken for a meaningful guarantee of something worthwhile.

<comment><USER7> Also: the fundamental undecidability of the Halting Problem

<comment><USER8> ... is really not relevant to software testing.

</comment></comment></comment><comment><USER9> The worse the developer, the more tests he'll write.

Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there.

Tests will be poorly written, pointless, and give an overall false sense of security to the next sap who breaths a sigh of relief when "nothing is broken". Of course, that house of cards will come down the first time something is in fact broken.

I've worked in plenty of those environments, where there was a test suite, but it couldn't be trusted. In fact, more often than not that is the case. The developers are a constant slave to it, patching it up; keeping it all lubed up. It's like the salt and pepper on a shit cake.

Testing what you do and developing ways to ensure its reliable, fault-tolerant and maintainable should be part of your ethos as a software developer.

But being pedantic about unit tests, chasing after pointless numbers and being obsessed with a certain kind of code is the hallmark of a fool.

<comment><USER10> I inherited a code base with LOTS of test coverage.

Made one small change and everything broke.

Looked into the tests, oh the humanity!!!

Dropped the test suite entirely, haven't had any issues since.

We have VERY good logging and notifications though so anything goes wrong we know all about it.

<comment><USER11> > _Dropped the test suite entirely, haven 't had any issues since._

That you know about.

That bad test suites exist is unarguable.

Bad production code exists too, but we don't give up on writing production code.

</comment></comment></comment><comment><USER12> The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said 70% is where the biggest bang-for-the-buck is. Now maybe you might convince me that something like Ruby needs 100% coverage, and I'd agree with you since some typing errors (for example) are only going to come up at runtime. But a compiled (for some definition of "compiled") language? Meh, you don't need to check every use of a variable at runtime to make sure the data types didn't go haywire.

The real Real Tragedy of 100% coverage is the number of shops who think they're done testing when they hit 100%. I've heard words to that effect out of the mouth of a test manager at Microsoft, as one example. No, code coverage is _a_ metric, not _the_ metric. Code coverage doesn't catch the bugs caused by the code you didn't write but should have, for example. Merely executing code is a simplistic test at best.

<comment><USER13> 70% coverage... I don't know if I'm going to be able to give up my 0% coverage 1-man projects...

<comment><USER14> I'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out.

Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them.

Anything past that is for companies with customers that need to maintain a certain level of quality and service. Worry about it when you get there.

<comment><USER15> How do you know what 10-20% to test?

This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.

<comment><USER16> You should test the thing that makes you money first, and should delay testing supporting functionality. For instance, if I am writing a document converter, the thing that makes me money is the AST -> AST conversion. Testing that should come before testing parsing (bytes -> AST) and rendering (AST -> bytes).

The place where you make money is the place that will have the largest demand for new and changing functionality. And where things change the most is where you need tests to protect against regressions.

</comment></comment></comment></comment></comment><comment><USER17> My main issue with unit testing is what defines a unit?

Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression.

Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box.

That way tests actually assist with refactoring rather than be something that just exactly follows the code and breaks whenever a minor internal detail changes.

However occasionally I do find it helpful map out all input/output for an internal function to cover all edge cases. But that's an exception.

<comment><USER18> I agree with you. That's called functional testing, and it is very useful, but it is not unit testing.

Unit testing: Test all methods and paths of a class, even private ones.

Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests.

ROI of functional testing is high, as it is usually done with real data. In my opinion unit testing is a huge waste of time. Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.

<comment><USER19> Some counterpoints:

\- If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests.

\- If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful.

\- Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up.

I think any test strategy – unit, integration, e2e, acceptance, UI – pursued to exclusion is a bad idea. Different projects and different teams call for different balances between them.

<comment><USER17> > \- If you want to know if your utility classes and functions are sane, unit > testing is far better bang for your buck than trying to figure out whether > they're being adequately exercised in your service tests.

I think we talk about the same thing, sometimes I will test something internal.

I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent introduced is what I'm advocating. I see this as distinct from integration testing. In a micro service architecture, my integration test would be chaining multiple services together into test scenarios.

<comment><USER19> I'm kind of flexible on my terminology, perhaps more flexible than others, since my approach to testing has largely been self-taught from the experience of several shops.

To me, unit tests are when I'm testing the behavior of a "thing" in "isolation", for some definitions of those terms – and yes, I agree that those definitions vary greatly. However, I don't agree with you that the flexibility of the definition is an issue.

Integration testing for me is when I'm testing system interfaces, so I'm focusing on how my systems behave when they come together. Sometimes I do this in isolation, using stub services or even internal stubs to simulate another system's behavior, but generally I do it with an actual system when it's convenient.

I don't generally use the term functional testing because I personally think it's ambiguous – I'm testing functionality in either case! But I suspect our differences really just boil down to how you slice it. If you prefer to divide tests into black-box vs white-box, but don't care as much about the specific level of isolation involved in the test, functional testing is perhaps the term you'd prefer. I prefer to categorize tests in terms of the amount of isolation I'm using, in which case I'm basically thinking unit, integration, e2e.

</comment></comment></comment></comment></comment><comment><USER20> I think a bigger epidemic is we're putting too much emphasis on "do this" and "do that" and "if you don't do this then you're a terrible programmer". While that sometimes may be true, much more importantly is to have competent, properly trained professionals, who can reason and think critically about what they're doing, and who have a few years of experience doing this under their belt. Just like other skilled trades, there's a certain kind of knowledge that you can't just explain or distill into a set of rules, you have to just _know_ it. And I see that in the first example in this article, where the junior programmer is writing terrible tests because he just doesn't know why they're bad tests (yet).

<comment><USER21> It seems to me that management is taught that a dependence on expensive experts, is a problem to be optimized. They want to manage the development process in a way that allows them to easily swap out one developer or team for another, or to ramp up production simply by increasing the number of developers assigned to a task.

It is almost as if they see the success that we have had in dev/ops with the "cattle, not pets" philosophy, and want to apply that in their own field. Making the subject behave consistently and predictably, whether it is a machine or a human professional, would be a prerequisite for that.

<comment><USER13> Will they succeed?

<comment><USER21> Sure, as long as their users unanimously abide by a very large rulebook ;)

<comment><USER20> I'd say they are "succeeding" in the sense that they continue to make money. Whether they're truly successful in the sense of providing proper value to their customers to justify that money, and treating their programmers right, is another story, and I've seen enough cases where they don't do either of these, while the executives make plenty of money to live comfortably for years by exploiting both customers and their own programmers.

</comment></comment></comment></comment></comment><comment><USER22> We've almost stopped unit testing. We still test functionality automatically before releasing anything into production, but we're not doing a unit test in most cases

Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would.

I won't pretend that's a good decision for everyone. But I do think people take test-driven-development a little too religiously and often forget to ask themselves why they are writing a certain unit test.

I mean, before I was a manager I was a developer and I also went to a university where a professor once told me I had to unit test everything. But then, another professor told me to always use the singleton pattern. These days I view both statements as equally false.

<comment><USER23> I'm curious - how old are the systems you're working on?

In my experience, unit tests don't catch many bugs when the code is fresh. But when it's five years old with many modifications over the code base, some dumb little test that you thought was a waste of time is now alerting you to what would have been a horror regression.

In other words: Even though it feels like it's slowing you down now, if you write tests while the functionality is still fresh in your mind, and you capture your assertions well, they'll pay off dividends in the future once you've forgotten what some block of code does.

<comment><USER22> I work in the public sector in Scandinavia and some of our oldest systems still in service run on an old tandem. So some of it is pretty old.

This gives us some unique abilities in terms of modeling our productivity of course, because we started measuring before anyone thought up unit testing.

Over the past 15 years, unit testing has failed to produce anything positive, and test driven development has been an absolute disaster.

That being said, this isn't something which will be universally true. A lot of the software we use isn't build by us, and I'm certainly that a lot of the suppliers on that software use unit tests quite extensively.

For the things we build ourselves, however, there has been almost no value in adopting modern test philosophies.

You say that systems have to be able to be worked on 5 years from now, but the truth is that most of our systems transport data and rarely live 5 years without getting rewritten to deliver better performance, higher levels of security or simply because the business has changed completely. A lot of it hold very few responsibilities as well, making it extremely obvious what to fix when a service breaks.

Don't get me wrong, we've seen problems we wouldn't have with 100% coverage. But that doesn't matter when spending resources fixing them is still a net positive on every account.

<comment><USER24> Do you have any metrics that you can share?

</comment></comment></comment></comment><comment><USER25> I agree (mostly) with the authors standpoints, but his arguments to get there are not convincing:

> _You don 't need to test that. [...] The code is obvious. There are no > conditionals, no loops, no transformations, nothing. The code is just a > little bit of plain old glue code._

The code invokes a user-passed callback to register _another_ callback and specifies some internal logic if _that_ callback is invoked. I personally don't find that obvious at all.

Others may find it obvious. That's why I think, if you start with the notion "this is necessary to test, that isn't", you need to define some objective criteria when things should be tested. Relying on your own gut feeling (or expecting that everyone else magically has the same gut feeling) is not a good strategy.

If I rewrite some java code from vanilla loops-with-conditionals into a stream/filter/map/collect chain, that might make it more obvious, but it wouldn't suddenly remove the need to test it, would it?

> _" But without a test, anybody can come, make a change and break the code!"_

> _" Look, if that imaginary evil/clueless developer comes and breaks that > simple code, what do you think he will do if a related unit test breaks? He > will just delete it."_

You could make that argument against _any_ kind of automated test. So should we get rid of all kinds of testing?

Besides, the argument doesn't even make sense. No one is using tests as a security feature against "evil" developers (I hope). (One of) the points of tests is to be a safeguard for anyone (including yourself) who might change the code in the future and might not be aware of all the implications of that change. In that scenario, it's very likely you change the code but will have a good look at the failed test before deciding what to do.

</comment><comment><USER26> The article illustrates what happens when you have inexperienced or poor developers following a management guideline.

To see how 100% coverage testing can lead to great results, have a look at the SQLite project [1].

In my experience, getting to 100% takes a bit of effort. But once you get there it has the advantage that you have a big incentive to keep it there. There is no way to rationalise that a new function doesn't need testing, because that would mess up the coverage. Going from 85% to 84% coverage is much easier to rationalise.

And of course 100% coverage doesn't mean that there are no bugs, but x% coverage means that 100-x% of the code is not even run by the tests. Do you really want your users to be the first ones to execute the code?

As an anecdote, in one project where I set the goal of 100% coverage, there was a bug in literally the last uncovered statement before getting to 100%.

[1] [https://www.sqlite.org/testing.html](https://www.sqlite.org/testing.html)

<comment><USER27> Doesn't what you're writing actually influence whether 100% coverage is a worthy goal or not?

I mean, SQLite is a good example of something where 100% coverage would actually be useful, because it tries to maintain compatibility with the SQL spec and with Postgres (largely because Postgres complies with the spec). Testing that 100% makes a lot of sense.

Suppose you're instead doing something very UX/UI driven. Why bother trying to cover that 100% with automated tests, when change is going to be driven by the whim and fancy of anyone who sits down in front of it?

<comment><USER26> > Doesn't what you're writing actually influence whether 100% coverage is a > worthy goal or not?

Yes, absolutely.

> Suppose you're instead doing something very UX/UI driven. Why bother trying > to cover that 100% with automated tests, when change is going to be driven > by the whim and fancy of anyone who sits down in front of it?

It might make perfect sense from a business standpoint to have no tests at all.

</comment></comment></comment><comment><USER28> I might be completely wrong on this one, but it seems to me that a lot of the precepts of TDD and full code coverage have a lot to do with the tools that were used by some of the people that popularized this.

Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went untested. A single execution isn't even enough for me: I need a whole lot of testing on most of the code to be comfortable.

When I write Scala at work instead, I test algorithms, but a big percentage of my code is untested, and it all feels fine, because while not every piece of code that compiles works, the kind of bugs that I worry about are far smaller, especially if my code is type heavy, instead of building Map[String,Map[String,Int]] or anything like that. 100% code coverage in Scala rarely feels as valuable as in Ruby.

Also different styles make the value of having tests as a way to try to force good factoring changes by language and paradigm. Most functional Scala doesn't really need redesigning to make it easy to test: Functions without side effects are easy, and are easier to refactor. A deep Ruby inheritance tree with some unnecessary monkey patching just demands testing in comparison, and writing the tests themselves forces better design.

The author's code is Java, and there 95% of the reason for testing that isn't purely based on business requirements comes from runtime dependency injection systems that want you to put mutability everywhere. Those are reasons why 100% code coverage can still sell in a Java shop (I sure worked in some that used too many of the frameworks popular in the 00s), but in practice, there's many cases where the cost of the test is higher than the possible reward.

So if you ask me, whether 100% code coverage is a good idea or not depends a whole lot on your other tooling, and I think we should be moving towards situations where we want to write fewer tests.

<comment><USER29> IMO ruby is the worst because its so easy, and often times encouraged, to write layers of useless tests that ultimately give only a false sense of security. I recently re-did the Michael Hartl tutorial to catch up with Rails5 and it gives examples where you test proper template rendering. I love that book but I'd rather shoot myself in the foot then spend dev time writing tests to check if the right html title attribute was rendered for the page...

<comment><USER30> Curious as to why you're blaming a _language_ (ruby) for a bad _practice_ (writing layers of useless tests). Excuse my pedantry, but it wouldn't it be more accurate to blame a culture that you feel has grown up around that language?

<comment><USER31> In Scala, "does it compile/typecheck" will generally catch typos in code that isn't frequently executed, and raise warnings for unhandled cases. For Ruby, you pretty much need to execute the code to have any inkling if it goes kaboom -- the language (and how people write it) have so much magic that short of executing things, the computer can't tell you anything useful.

<comment><USER30> Valid point.

I don't think I could write low-bug count code faster in statically typed scala than I could in unit tested ruby though. I mean I am well aware of what you're telling me, and it's obvious to me that having the compiler automatically check certain properties is a win. And yet, when push comes to shove, to get something done I'm more likely to reach for ruby.

It's something I've never come up with a good explanation for. Does static typing stunt prototyping and exploration? Do unit tests capture high level goals better?

I guess I don't agree with the assertion that ruby tests are useless. Because you test higher level things than you do with scala types.

<comment><USER32> "Does static typing stunt prototyping and exploration?"

Not in my experience. It is probably about what you personally find hard/uncomfortable/unfamiliar.

<comment><USER30> But you're kind of implying that I feel that way because either I don't use static typing much, or I haven't learned it well, aren't you?

I've used static typing much more than dynamic. I'll admit confusion on things like ocaml polymorphic variants and haskell monads, nevertheless I wouldn't say I find static typing hard as a rule. But surely the point of static type checking is to constrict what you can do for safety and performance reasons. And surely the cost of that is you have less freedom - even when in the exploration phase

<comment><USER32> "you're kind of implying that I feel that way because either I don't use static typing much, or I haven't learned it well, aren't you?"

No. I didn't mean to imply or assert anything of the sort. Apologies if I inadvertently gave offense. You seem to be reading meanings into my reply that aren't there.

I don't know you from Adam. You asked a question in your post. I answered as best as I could.

That said, your latest statement

"I'll admit confusion on things like ocaml polymorphic variants and haskell monads,"

does seems to imply that you don't have much real world experience with static type systems in production (nothing wrong with that) since neither is an arcane concept or particularly difficult to understand.

If you haven't worked extensively with Haskell/Ocaml/SML etc, and you are extrapolating properties of 'static type systems' from those of Java or C++ then your idea of such type systems 'stunt prototyping and exploration' might make sense.

The rest of your comments are extrapolations from misunderstandings - not born of practical experience. My answer was based on (strictly) personal experience. Which is why I said "in my experience". I gladly concede that YMMV.

again, I was just answering your question in your original comment. I didn't mean to "imply" anything and used "probably" to mark my _uncertainty_ about your real world experience with Ocaml/Haskell etc style static type systems.

I have extensive experience with both dynamically typed languages (mostly Python, Lua and Scheme) and statically typed languages (mostly Haskell and SML, besides Java). I answered out of my experience.

YMMV. And that is cool.

<comment><USER30> I'm not offended. I am happy to talk to open minded static typing advocates. I myself am not really sure where I sit.

You're right, I haven't used ocaml or haskell in production. I did use F# though, which seems to be in the same ballpark as those languages in terms of having algebraic data types and inference and all the rest. I suppose the fact that I still don't really grok polymorphic variants after reading the real world ocaml a few times may say something about my ability, motivation, or at the very least how my brain is wired.

Fundamentally though, a type system is a restriction meant to help the programmer. This restriction must inevitably put you down a certain road when you explore stuff, right?

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER33> _But remember nothing is free, nothing is a silver bullet. Stop and think._

I'm going to be the one to point at the elephant in the room and say: Java. More precisely, Java's _culture_. If you ask developers who have been assimilated into a culture of slavish bureaucratic-red-tape adherence to "best practices" and extreme problem-decomposition to step back and ask themselves whether what they're doing makes sense, what else would you expect? These people have been taught --- or perhaps indoctrinated --- that such mindless rule-following is the norm, and to think only about the immediate tiny piece of the whole problem. To ask any more of them is like asking an ostrich to fly.

The method names in the second example are rather WTF-inducing too, but to someone who has only ever been exposed to code like that, it would probably seem normal. (I counted one of them at ~100 characters. It reminds me of [http://steve-yegge.blogspot.com/2006/03/execution-in- kingdom...](http://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of- nouns.html) )

Many years ago I briefly worked with enterprise Java, and found this sort of stifling, anti-intellectual atmosphere entirely unbearable.

<comment><USER34> It might even make sense for many of those enterprise developers to have a set of best practices that they can blindly follow. Not every developer has that many years of experience or is sharp enough to be able to identify the cases where unit testing makes sense, and those where it doesn't.

</comment></comment><comment><USER35> The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs." It isn't.

The purpose of unit tests is to document the intended behaviour of a unit/component (which is not necessarily a single function/method in isolation) in such a way that if someone comes along and makes a change that alters specified behaviour, they are aware that they have done so and prevented from shipping that change unless they consciously alter that specification.

And, if you are doing TDD, as a code structure/design aid. But that is tangential to the article.

<comment><USER36> _The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs."_

More like "in countless articles, comments, talks and projects over, say, the last decade".

<comment><USER35> Yes - it's pretty frustrating to try and have an important discussion when 99% of the time it is framed incorrectly from the start.

</comment></comment></comment><comment><USER37> Unit tests are a poor substitute for correctness. Many unit tests does not a strong argument make.

Unit tests are typically inductive. Developer shows case A, B and C give the expected results for function f. God help us if our expectations are wrong. So, you're saying since A, B and C are correct therefore function f is correct. Well that may be, or maybe A, B and C are trivial cases, in other words, you've made a weak argument.

100% test coverage sounds like lazy management. Alas, the manager may have worked their way via social programming rather than computer programming. In such cases, better to say you have 110% test coverage.

</comment><comment><USER38> Not sure if this is already mentioned but for me the most concise illustration of this fallacy was in The Pragmatic Programmer book. They had a function like this:

double f( double x ) { return 1/ x; }

They pointed out that it is trivial to get 100% coverage in test cases but unless your tests include passing in 0 as the parameter you are going to miss an error case.

<comment><USER5> passing 0 to f returns infinity (IEEE 754)

Not arguing the point here. It's just a terrible example.

<comment><USER39> I thought you'd get NaN, not Inf. Of course I'm probably misremembering.

<comment><USER38> I think what the above returns depends on the language. In java I think there is an actual run time exception for this. The point of the original comment is to show how 100% coverage is easy to achieve but often meaningless.

There is some other more abstract concept related to the classes of input that can possibly be passed to a method. IMHO 100% coverage and "test first" have done more harm than good to the cause of automated testing.

</comment></comment></comment></comment><comment><USER40> /me raises hand on the pro-testing side

I've been programming for a living since 1996, and only recently started to do TDD in the normal sense of writing unit tests before writing code. I've found to it to be an _enormous_ help with keeping my code simple - the tests or the mocking getting difficult is a great indicator that my code can be simplified or generalised somehow

I argued for functional instead of unit testing for years, but it was only when a team-mate convinced me to try unit testing (and writing the tests FIRST) that the scales fell from my eyes. Unit testing isn't really testing, it's a tool for writing better code.

BTW from an operational perspective I've found it's most effective to insist on 100% coverage, but to use annotations to tell the code coverage tool to ignore stuff the team has actively decided not to test - much easier to pick up the uncovered stuff in code review and come to an agreement on whether it's ok to ignore

<comment><USER34> I agree that TDD and unit tests can help you when writing code. But once your code has been written, are all those unit tests still useful? Or do they get in your way when you do large scale refactoring?

I have seen quite a few codebases with loads of unit tests that might once have been useful, but now were just slowing down development. Many of them should have been thrown away and replaced by a couple of functional tests, however the sunk cost fallacy always kicked in, nobody wanted to throw away all the hours of work that had gone into making them and not to forget the precious code coverage as well.

<comment><USER41> > But once your code has been written, are all those unit tests still useful?

The reason for this question is the naming mistake that was made in the beginning. They are not TESTS, they are SPECIFICATIONS.

Once your code has been written, are the specifications still useful? Well, duh... unless the business has changed them then yes, they're still useful.

</comment></comment></comment><comment><USER42> I wish people cared more about the craft of an amazing plugin architecture or an advanced integration between a machine learning system and a UI, but no, more and more of our collective development departments care more about TDD and making sure things look perfect. Don't worry about the fact that there are no integration tests and we keep breaking larger systems, and while there might be 100% code coverage, no developer actually understands the overall system.

<comment><USER43> Yes, unit tests have their place, but you have to understand why they are needed.

To me, it's secondary and has tones of bike shedding in it. Writing tests is easy, mentally. Getting a good, simple, YAGNI/DRY architecture is more challenging and requires several iterations, something that is resisted if you have to rewrite all your unit tests. Let me put it this way, if you write an architecture that you later don't like, you would be more hesitant to scrap it and start over because all the additional work of the unit tests (especially if you're on a deadline). That's bad. To write a good architecture, a developer must be willing to realize he could have done it better and be willing to tear it down instead of build around the bad architecture. That's how you write 20 year code, which should be almost everyone's goal.

</comment></comment><comment><USER44> I've seen projects where management had rules like "you must have 70% code coverage before you check in". Which is crazy, for a lot of reasons.

But the developer response in a couple cases was to puff the code up with layers of fluff that just added levels of abstraction that just passed stuff down to the next layer, unchanged, with a bunch of parameter checking at each new level. This had the effect of adding a bunch of code with no chance of failure, artificially increasing the amount of code covered by the tests (which, by the way, were bullshit).

I got to rip all that junk out. It ran faster, was easier to understand and maintain, and I made sure I never, ever worked with the people who wrote that stuff.

</comment><comment><USER21> If you can prove that your testing process is perfect, then your entire development process can then be reduced to the following, after the test suite is written:

      cat /dev/random | ./build-inline.sh | ./test-inline.sh | tee ./src/blob.c &&
      git commit -Am "I have no idea how this works, but I am certain that it works perfectly, see you all on Monday!" &&
      git push production master --force

When presented like this, relying on human intelligence and experience doesn't seem like such a bad thing after all.

Just so we're clear, my username was not inspired by this scheme.

<comment><USER45> It seems to me that such a "perfect" testing process would basically amount to declarative programming.

<comment><USER46> Indeed it is; a _perfect_ test is equivalent to the code being tested.

</comment></comment></comment><comment><USER47> A lot of people here seem to have strong opinions against 100% coverage, so I'll risk their ire with my strong opinion in favor.

If you have, say, 95% coverage -- and most corporate dev orgs would be thrilled with that number -- and then you commit some new code (with tests) and are still at 95%, you don't know anything about your new code's coverage until you dig into the coverage report. Because your changes could have had 100% coverage of your new thing but masked a path that was previously tested; or had 10% but exercised some of the previously missing 5%.

If you have 100% coverage and you stay at 100% then you know the coverage of your new code: it's 100%. Among other things this lets you use a fall in coverage as a trigger: to block a merge, to go read a coverage report, whatever you think it warrants.

Also, as has been noted elsewhere, anything other than a 100% goal means somebody decides what's "worth" testing... and then you have either unpredictable behavior (what's obvious to whom?) or a set of policies about it, which can quickly become more onerous than a goal of 100%.

It's important to remember that the 100% goal isn't going to save you from bad tests or bad code. It's possible to cheat on the testing as well, and tests need code review too. There's no magic bullet, you still need people who care about their work.

I realize this might not work everywhere, but what I shoot for is 100% coverage using only the public API, with heavy use of mock classes and objects for anything not directly under test and/or not stable in real life. If we can't exercise the code through the public API's then it usually turns out we either didn't rig up the tests right, or the code itself is poorly designed. Fixing either or both is always a good thing.

I don't always hit the 100% goal, especially with legacy code. But it remains the goal, and I haven't seen any convincing arguments against it yet.

 _Open the flame bay doors, Hal..._ :-)

<comment><USER48> The problem with the "100%" goal is that it provides a false sense of safety. There can still be plenty of bugs in code that has "100%" test coverage. The 100% are an illusion. To truly get to 100%, it does not suffice that every line of code is executed, you must also test all valid inputs - which is utopic.

<comment><USER47> I concede that for people who don't understand what unit testing does, 100% might provide a false sense of security. But I don't think that's a problem with the goal... that would be like saying the problem with staying fit is that it provides a false sense of longevity, whereas you could still get brain cancer or be hit by a car.

If you know what's going on -- and hopefully you do if you're making decisions like whether to pursue a 100% coverage goal -- then your point is (I hope) self-evident.

It's also why code review is necessary and not just the coverage metric. 100% coverage is not "100% of all things that could happen," it's "100% of all code execution paths" and (hopefully) a sample of all known _types_ of valid and invalid inputs.

Anyway, so far I've found that most developers respond pretty well to the 100% idea if you explain its utility and what it does and doesn't get you. From a ridiculously small sample, granted.

The utopian risk, I think, is much more around the hope that people high up the food chain in your organization actually understand this stuff. Consider how many VPs of Engineering did a year or two of engineering work (before unit testing was in vogue) then got that MBA and have been busy taking meetings ever since.

</comment></comment></comment><comment><USER49> _You don 't need to test that. ... The code is obvious. There are no conditionals, no loops, no transformations, nothing. The code is just a little bit of plain old glue code._

Here's the code:

      @Override
      public void initialize(WatchlistDao watchlistDao) {
        watchlistDao.loadAll(watchListRow -> watchlists.add(watchListRow));
      }

Maybe I'm dense, but this code raises at least one question that I would prefer to see answered by tests.

The parameter watchlists appears to be defined in a scope above the one under test. What happens if watchlists is null for some reason? What should be the behavior?

Then there's the tricky question of what to do as this method evolves. Next month, a watchListRow might need to be updated with a value before being added to watchlists. Later, a check might be added to ensure some property exists on watchListRow. At what point will a test be written for this method?

</comment><comment><USER50> I once joined a company that had 90% code coverage. After a while it became clear that there were all vanity tests: I could delete huge swathes of code with zero test failure. We let the contractors that wrote it move on, and we formed a solid team in house. We don't run code coverage any more because it makes the build run four times slower. Instead, I trust our teams to write the good tests. Sometimes that means <100% coverage, and the teams are able to justify it.

Some feedback on the article:

>Test-driven development, or as it used to be called: test-first approach

Test-first is not the same as Test-Driven. The test-first approach includes situations where a QA dev writes 20 tests, and then hands them to an engineer who implements them. Thats not TDD.

>"But my boss expects me to write test for all classes," he replied.

That's very unlikely to be TDD. "Writing tests because I've been told to" is never likely to be "I'm writing the tests that I know to be necessary", and that's all TDD is: writing necessary tests. If the test isn't necessary, then neither is the code.

>Look, if that imaginary evil/clueless developer comes and breaks that simple code, what do you think he will do if a related unit test breaks? He will just delete it.

Sure. But then their name is on that act in the commit log. The test is a warning. I've been lucky not to have worked with evil developers, but I have worked with some clueless ones, and indeed some have just deleted tests. Thats an opportunity for education, and quality has steadily improved.

>The tragedy is that once a "good practice" becomes mainstream we seem to forget how it came to be, what its benefits are, and most importantly, what the cost of using it is.

Totally agree. So many programmers and teams practice cargo cult behaviors. Unfortunately, this article is one of them: making claims about TDD, and unit tests in general, without understanding "why" TDD is effective.

<comment><USER51> > If the test isn't necessary, then neither is the code.

How did you come to this conclusion?

What if the code is just glue code for different libraries?

</comment></comment><comment><USER52> Of course any metric can be rendered useless if one "works the metric" rather than the intent of the metric.

But in my experience, code covering unit tests have correlated strongly with faster development and far fewer bugs being uncovered in the field.

<comment><USER53> > Of course any metric can be rendered useless if one "works the metric" > rather than the intent of the metric.

If the metric actually measures the figure of merit rather than a proxy with no necessary linkage, this is not true; working the metric then _is_ working the intent of the metric.

The extent to which a metric can be "worked" distinct from its intent is exactly the extent to which it is measuring something disconnected from what it is used to assess.

<comment><USER54> Even if the measure is accurate, there's an opportunity cost for spending time on things that are also important, but not measured. You'd need a measure that covered everything, from sufficient speed optimisations to fulfilling customer desires to complying with all relevant legal requirements, which basically can't exist.

</comment></comment></comment><comment><USER55> He's right, but he's conflating 100% code coverage with using mocks with writing tests.

Always write tests. And strive for maxmium coverage. But make sure you write the right kinds of tests:

\- Don't overuse mocks. Mocks don't represent real conditions you would actually see in production. Favor using real dependencies over mocks.

\- Don't overspecify your tests. Test only publicly specified parts of the contract. Things that you need to be true and that the callers of the module expect to be true. And yes, you will change the test when the contract changes.

<comment><USER56> Only mock things that would be difficult to setup in a test, is my approach. Then also set up some other ci jobs for the difficult stuff when we have time.

</comment></comment><comment><USER57> My version of this is working on a team with 100% coverage that still saw a steady and heavy influx of bugs. 100% coverage does not mean bug free.

I advocate spending time on identifying/inventing the correct abstractions over coverage.

</comment><comment><USER23> I think you should write a test.

Naming the test just "initialise" is not very useful as it doesn't assert what you expect the method under test to do. Given that the purpose of the initialise function is to populate a watchlists collection variable from the parameter, i'd name the test something like "initialise_daoRecordCountIs9_watchlistCountIs9". The pattern I generally use is <method_name>_<assertion_under_test>_<expected_result>.

Then, my test would be the following:

* Set up / mock the dao parameter to have 9 rows

* Create an instance of the class under test and push in the dao parameter

* Verify / Assert that the class under test now has 9 items in the watchlists variable - I'm assuming there is a public method to access that.

</comment><comment><USER30> I feel like this high test coverage thing can only work if you have tight modules, tight interfaces, and you only bother testing at module boundaries. So the test cases almost function as a bit of executable API documentation - here's the method name, here's what it does, here's the contracts and/or static types, and.... given this input, you should get this output.

Do it for the high level bits you actually expose. If you're exposing everything, tests won't really save you - architecture and modularity are more fundamental and should be tackled first. If you're writing a big ball of mud, what benefit do you get testing a mudball?

</comment><comment><USER58> 100% code coverage and even TDD'd doesn't and shouldn't mean 100% unit tested. Glue code and declaration doesn't need a unit test. Some functional tests should provide all the coverage needed to give you confidence to refractor that code in the future.

Edit: while I'm a huge TDD advocate, I'm not a big advocate of measuring code coverage. That should only be necessary if you are trying to get a code base under coverage that wasn't TDD'd. Even then I'd rather add the coverage as I'm touching uncovered code. If it works and I'm not touching it, it doesn't need tests.

</comment><comment><USER59> The tragedy he mentions aren't 100% code coverage. It's more about people using the wrong tools for the job and using 100% coverage as an indication that everything is fine.

100% coverage for my team means that we were intentional about our code. It's not hard at all to have 100% coverage in a Ruby application as it is possible to do a lot with very little code.

Furthermore it allows us to bring in a junior on the team because we know they have a safety net.

Also for the record we do code reviews and are very thoughtful about the code we write. 100% coverage does not stop the possibility of some bugs inserting themselves somewhere.

<comment><USER60> Exactly this. One problem with TDD, etc. is that you are limited by the quality of your tests. 100% means nothing if you don't have a try/catch around a cache lookup that could should never fail but could and not get handled.

<comment><USER61> No, that isn't a limitation with TDD. That is a limitation of imagination, and can happen to any programmer. (proving the code correct is most likely to catch this - but even that can fail in some cases)

TDD just means that when discover the bug and add the missing try/catch you don't accidentally break something else.

</comment></comment></comment><comment><USER62> There's a human tendency to overemphasize things you can quantify. So we try to figure out how to test every code path rather than what we should do: try to figure out which inputs we should test against.

</comment><comment><USER63> I use the following list to decide on creating a unit test or not. More yeses means a unit test is a good idea.

1\. Is it hard to instantly test the code when implementing it? (Might be the case for library code)

2\. Is there a chance the underlying implementation might change (and so might break in the future)?

3\. Will the interface of the class remain stable? (If not unit test needs to be rewritten too)

4\. Will functional tests pass when something breaks in this class?

<comment><USER64> I often try to write a unit test just to see whether it's possible. If not (i.e. one of the questions was no) there is always a chance that the design can be improved. E.g. can the interface be minimized to something that is stable, and simpler to test?

Testable code tends to be better (have less dependencies, less state etc) than non testable code. So making it testable is a goal in itself, even if the tests are never even committed.

</comment></comment><comment><USER65> Having 100% code coverage is like having 0 warnings (although it certainly is a lot harder). In this situation, your tools are _not_ telling you "all's good", but rather "I can't detect anything suspect here".

There's a good chance that the dev time needed to go from 90% coverage to 100% coverage might be better spent somewhere else.

</comment><comment><USER66> Agreed. I would rather have 5% test coverage that checks against all risky edge cases/inputs than 100% test coverage that checks against arbitrary, low- risk inputs.

Writing tests to confirm the simplest, most predictable use cases is a waste of time - Those cases can be figured out very quickly without automated testing because they are trivial to reproduce manually.

</comment><comment><USER67> Code coverage is an illusion, since what you want is actually "possible states coverage". You can cover all the lines of your code and still cover a minority of the possible states of the program, and _especially_ a minority of the most probable states of the program when the actual users execute it, or you can cover 50% of your lines of code and yet cover much more real world states, and states for which it is more likely to find a bug. I think that more than stressing single features with unit tests, it is more useful to write higher level stress tests (fuzz tests, basically) that have the effect of testing lines of code as a side effect of exploring many states of the program. Specific unit tests are still useful, but mostly in order to ensure that edge cases and main normal behavior corresponds to the specification. As in everything it is develooper's sensibility that should drive what test to write.

</comment><comment><USER68> Has anyone seen a fuzzer that creates variants based on a test suite with 100% coverage? Hmm... the fuzzer still wouldn't necessarily know how to create the correct invariants. #lazyweb

</comment><comment><USER69> I've seen a project with 100% unit test coverage, yet no e2e tests. Nobody knew whether the product worked at all.

</comment><comment><USER70> I wish more people cared about path coverage as opposed to "line coverage".

<comment><USER71> It's slightly more meaningful in the right hands, but still meaningless in the wrong ones. You can get 100% line coverage and never have a single assert.

</comment></comment><comment><USER72> One point already made by several people on this thread is that code coverage, while helpful, is not enough (and perhaps is not even the best bang for the buck).

In hardware verification (where I come from, and where the cost of bugs is usually higher), "functional coverage" is considered more important. This is usually achieved via constraint-based randomization (somewhat similar in spirit to QuickCheck, already mentioned in this thread).

I tried to cover (ahem) this whole how-to-use-and-improve-coverage topic in the following post: [https://blog.foretellix.com/2016/12/23/verification- coverage...](https://blog.foretellix.com/2016/12/23/verification-coverage-and- maximization-the-big-picture/)

</comment><comment><USER73> Back when I learnt Haskell we had a lecturer named John Hughes who had co- authored a tool named QuickCheck[1]. We used this tool extensively throughout the course, with it testing were quite simple and writing elegant generators were a breeze. In my experience, these test did a much more greater job at finding edge cases then many unit tests I've seen in larger close to full coverage TDD projects.

As with much else TDD should be a tool with the ultimate goal of aiding us in writing correct and less bug riddled code, once the tool adds more work it's no longer offering much aid.

[1] . [https://en.wikipedia.org/wiki/QuickCheck](https://en.wikipedia.org/wiki/QuickCheck)

<comment><USER74> Also see

[https://github.com/hedgehogqa](https://github.com/hedgehogqa) [https://hackage.haskell.org/package/smallcheck](https://hackage.haskell.org/package/smallcheck)

</comment></comment><comment><USER75> This remembers me of recent projects where developers started to mock every piece of code.. The result was that all tests passed while the codebase exploded in real environments.

In my opinion the best advice is to force developers to use their brains. I know, there are a lot of sh*tty CTO/CEO/HoIT/SomeOther"Important"Position people out there seeing them as code monkeys and saying that developers are not paid to think but in that case the best thing developers could do is learn to say "NO"... My experience with that kind of people is that they need to learn the meaning of "NO" instead of wasting time and money in the end of the day.

</comment><comment><USER76> What you actually want to do is test the methods with the highest cyclomatic complexity first (where it's greater than 1)

IntelliJ has a plugin

<comment><USER54> Most standard unit coverage tools will do this for you, often with the CRAP (Change-risk anti-patterns) score:

[http://www.ncover.com/blog/change-risk-anti-patterns-code- co...](http://www.ncover.com/blog/change-risk-anti-patterns-code-complexity/)

 _" This is based on the methodology that the more complex code is the more likely it is to have errors, hence the need for greater code coverage.

To better understand, let's assume a scenario where you wanted to keep your change risk anti-patterns score below 30. To maintain this level with a code set that has a cyclomatic complexity of 10, you would need to achieve a code coverage (as defined by branch coverage) ratio of approximately 42%. If however, the complexity of your code was greater and you had a cyclomatic complexity of 20, you would need almost 72% code coverage if you wanted to maintain the same risk level.

Complexity increases risk. Testing decreases risk. The change risk anti- patterns score give you a metric to measure the correlation between the two."_

</comment></comment><comment><USER77> I've heard Kent Beck talk about having the smallest amount of tests that give you confidence.

Which he also gave as an answer here:

[http://stackoverflow.com/questions/153234/how-deep-are- your-...](http://stackoverflow.com/questions/153234/how-deep-are-your-unit- tests/153565#153565)

But I know a lot of people in the early days of XP went to extremes, 100% code coverage, mutation tools for every condition to ensure unit tests broke in expected ways, etc. But they were more experiments in pushing the limits rather than things that gave productivity gains.

</comment><comment><USER78> I have 100% code coverage on a couple of projects. It has two benefits:

Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected.

New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the standard "If your contribution doesn't have 100% coverage it won't be accepted". I don't think this is a bad default position.

<comment><USER79> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security.

So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually test any known edge cases.

<comment><USER80> Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not _test_ it.

<comment><USER78> If you write unit tests like this, it's your own fault when the bridge collapses from driving a car over it.

</comment></comment></comment></comment><comment><USER81> I don't think I know anyone that do TDD. Uncle Bob has indoctrinated a few zealots into that mindset, but it all comes off as crazy to me. A germ of a good idea taken way too far.

People of that school tend to write tests that test implementation rather than functionality. As a result you get fragile tests that break not telling you what went wrong but how the implementation has changed.

Good tests should test behavior. A change in implementation shouldn't break the test.

<comment><USER82> >Uncle Bob has indoctrinated a few zealots into that mindset

Even though Uncle Bob's advice agrees with this submission:

[https://news.ycombinator.com/item?id=14301466](https://news.ycombinator.com/item?id=14301466)

>People of that school tend to write tests that test implementation rather than functionality. As a result you get fragile tests that break not telling you what went wrong but how the implementation has changed.

Also another thing Uncle Bob does not advocate. From his stance, if most of your failing tests are due to refactors (changing implementation), and not bugs, you need to redesign your tests. Abstract out your interfaces so tests are not too dependent on the actual implementation. Tests are first class code, not to be treated differently from your main code base - they should abide by code standards, and they need to be architected as much as your regular code.

(Easier said than done).

The thing is, I'm not even an Uncle Bob fan. But it's crazy how neither his fans nor his detractors see the nuances in what he says - be it his videos or his blog posts.

What is it about programmer types that makes them see everything in binary?

</comment></comment><comment><USER83> IMO, if I ever have 100% code coverage, I did something wrong. The best I can usually achieve is 95-98%, because of my defensive coding to warn about the "impossible" use cases.

Escape a `while True` loop? Log it, along with the current state of the program, and blow up (so we can be restarted). Memory allocation error? Log it. The big "unexpected exception" clause around my main function? Log it.

If I do hit those in testing, my _code_ is wrong.

</comment><comment><USER84> I noticed the author was speechless in two situations, both of which involved "but we write all our tests in <test-framework>." This is legitimate and should be taken more seriously by the author.

Codebases serve businesses and businesses value _legibility_ over efficacy. It's more important to them to have control over their assets than to have better assets. Using _one_ test framework is in perfect service of that goal.

It's inefficient in that it will take future developers more time to understand that code. But fewer architectural elements means that you can get by with less senior programmers.

Imagine if you went onto a software project and they were using 6 different databases because every time they had a new kind of data that they wanted to access differently, they reached for another database rather than use the one they had.

Of course nobody would ever do that, well I hope anyway, but I do see a lot of unnecessary architectural complication in projects in service of "using the right tool for the job." And it can balloon. A new test framework has to work in your CI framework. You need to decide how to handle data. It's not a huge decision, but it's more complicated then most devs would think and it'll take up more of your time than you'll expect.

You can generalize this to the the main thrust of the article. 100% code coverage is not a bad goal to want to hit. Sure, you're going to get a lot of waste. But you're not paying for it, your employer is. And your employer might have a different idea of which side of the tradeoff he wants to be on and where to draw the line. You know the code way better than they will, but they know the economics far better than you ever could.

</comment><comment><USER85> Build tests against your app`s public interface. On a web app, that would be your controllers or API.

That will give you good coverage, while avoiding too simple to be useful unit tests.

It's really hard to foresee all possible input variations and business logic validations, but that doesn't mean your test suite is useless.

It just means it will grow everytime you find a new bug and you are guaranteed that one won't happen again...

<comment><USER61> On a small app that is a good idea. However on a massive app you need to break things down more. I work on a project with > 10 million lines of code, there is no way any human can understand everything in detail. As a result I need to test at a smaller scale which is not my public interfaces, but my interfaces to other people.

</comment></comment><comment><USER86> Coverage isn't the goal. The goal is understanding.

Write a test if you don't feel confident that a piece of code does what you think it does. If you're not sure what it does now, there's little chance that you or anyone else will in the future, so write a test to understand it and to make that understanding explicit.

Use curiosity as a driver.

<comment><USER87> Also, the code of a test could be used as the documentation. Just recently I started using bleve([https://github.com/blevesearch/bleve](https://github.com/blevesearch/bleve)). Documentation is quite poor, but I was able to learn the product entirely from the code of its unit tests. That's not normal though.

</comment></comment><comment><USER88> Automated tests are code, and come with all the engineering and maintenance concerns of 'real' code. They don't do anything for your customers, though, so are only appropriate when they actually make your work faster or safer.

Automated tests are a spec, and are exactly as hard to write completely and correctly, and as easy to get wrong in ignorance, as a 'real' spec. If you find them easy to write, odds are good you would find the code easy to visually verify as well - which is to say, you're working on a trivial problem.

They have their place, but that place is not everywhere. It is where they are efficient and valuable. I particularly look for places where they are like the P half of an NP problem, an independent estimate of the answer to a math problem. If you ever find yourself writing the same code twice, unless it's a safety-critical system or something, that's a moment to stop and reflect on the value of what you are doing.

</comment><comment><USER89> The title does not mean anything and is basically a click bait in my opinion as it sounds cool to trash some ideal. That said, the magic 100% number is far removed from reality and does not represent anything by itself.

100% coverage on project of which size? Imagine you have a single script project that does exactly one thing and 2 test are enough to verify that it works without doing it manually? That is not the same as writing test for file system or tests which consists mostly of mocks upon mocks upon mocks.

I think the real problem is someone comes up with an idea, like TDD, tells people about it, some people hear about, start _preaching_ it, some people start _believing_ it and nobody actually think things through, usually cos they don't have experience (it's not a fetish as someone said). Like everything in life, you have to think things through before doing them, ask your self is this worth doing and when it is worth doing. You can't just say: "Oh we are doing TDD thus _everything must_ be done in TDD way".

For people that say tests are useless, or good code does not need tests, I ask, when you make a change do you still make sure your code works by hand? And if you do make sure, why don't you automate that? You are a programmer after all.

And for those that say you need to test everything, well you don't, specially if you need mock most of it or it is really not that important piece of code as it is dev tool or something. What you want to make sure works is customer/user facing stuff that _must work for you to get paid_ and you want to be able to verify this at any time of day without losing hours clicking around checking for stuff.

So this is not straight forward, 100% means nothing without context and doing anything in excess and without valid reasons is pointless or even harmful. And this has nothing to do with programming but life in general.

</comment><comment><USER90> Most of "we should go for 100% coverage" is simply cargo-culting (pushed by "gurus" like Uncle Bob - the negative aspects of the word guru implied)

Not to mention 100% coverage is not guarantee the system works, in practice _quite the opposite_

Not to mention this BDD crap which only makes my blood boil, it's syntactic yuck disguised as syntactic sugar

<comment><USER82> >Most of "we should go for 100% coverage" is simply cargo-culting (pushed by "gurus" like Uncle Bob - the negative aspects of the word guru implied)

I came here just to search for Uncle Bob to see the fun comments talking about him.

No, he does not recommend 100% coverage.

He is against writing tests for "basic" code like getters/setters.

I can understand the irritation with Uncle Bob - especially if you've read his blog. But everyone seems to get his stance on unit tests wrong - <USER2> Uncle Bob fans.

His fans say he advocates for 100% code coverage.

His detractors say he advocates for 100% code coverage.

All while Uncle Bob is saying he doesn't cover "trivial" code.

</comment></comment><comment><USER91> I think I'd rather focus on documenting the information flow. Of having the tools to track down where things start to go wrong when there's a problem and I ask things to run with more verbosity.

Initial "complete coverage" should probably start from mockups that test an entire API. The complete part should be that, in some way, the tests cover expected successes AND failures (successfully return failure) of every part of the API, but there's no need to test things individually if they've already been tested by other test cases.

Invariably reality will come up with more cases and someone will notice an area that wasn't quite fully tested. That's where a bug exists, but the golden test cases probably wouldn't have located it anyway. It'll take thousands or millions of users to hit that combination and notice it. Then you get to add another test case while you're fixing the problem.

</comment><comment><USER92> I've found that some bootcamps are responsible for this attitude as they preach to have 100% coverage. And no one really questions the experienced and heavily opinionated teacher.

It's good to use hyperbole black and white when teaching so the point comes across easier. But they should be made aware of caveats before they graduate at least.

</comment><comment><USER93> >Testing is usually regarded as an important stage of the software development cycle. Testing will never be a substitute for reasoning. Testing may not be used as evidence of correctness for any but the most trivial of programs. Software engineers some times refer to "exhaustive" testing when in fact they mean "exhausting" testing. Tests are almost never exhaustive. Having lots of tests which give the right results may be reassuring but it can never be convincing. Rather than relying on testing we should be relying in reasoning. We should be relying on arguments which can convince the reader using logic. [http://www.soc.napier.ac.uk/course- notes/sml/introfp.htm](http://www.soc.napier.ac.uk/course- notes/sml/introfp.htm)

</comment><comment><USER94> Property-based testing has made testing more productive and fun for me. You write a few lines of code that produce a large amount of tests. The idea is obviously so useful, I'm surprised it's uncommon in practice. When you think about coverage in terms of inputs applied instead of statements executed, property-based testing is far more productive than writing tests by hand.

It's not a silver bullet though. Some property-based tests are easy to write but offer little value. Sometimes you spend more time writing code to generate the correct inputs than the value of the test warrants. It has a learning curve. Still, I think it is the most powerful tool you can master for testing.

</comment><comment><USER95> I recently broke a unit test by adding one entry to a hash constant (a list of acceptable mime types and their corresponding file extensions). I looked at the test, and it was just comparing the defined constant, to a hardcoded version of itself.

I rewrote the test by converting the constant to a string, taking a checksum of it, and comparing _that_ to a short hardcoded value. Now the test is just 1 line of code, instead of 41! Then I put it through code review, and my team said "What a ridiculous test." But they didn't see any problem in the previous version that compared it to a 40-line hardcoded hash.

It's a weird world.

</comment><comment><USER96> To paraphrase: "Premature testing is the root of all evil".

How I do it is going from rough testing of pages and components to granular testing of those parts which had some error.

For pages, I just run them to see if they display without producing errors, same goes for critical components. This gets me the feeling of roughly tested and from the user perspective working system with little time investment.

Then I test critical business logic, but usually only after some error was reported.

Mind though that I am freelance developer unconstrained by organizational rules.

</comment><comment><USER97> Many of us have made similar mistakes (especially early in our careers) when taking on new techniques for which we became particularly enthralled. That's why it's a good idea to have a couple 'elders' on-staff so as to not allow youthful passion to wreck havoc. They tend to keep teams pragmatic and lazy (a good thing, in programming).

For instance, I remember all the bad code that I wrote and read circa 1997-1999, after design patterns became the rage.

</comment><comment><USER98> While 100% code coverage doesn't guarantee 0% bug, it's useful to easily detect new untested code addition and possible bug addition. Another point is that the code looks obviously right by visual inspection, but we want to automate the check. Relaxing the 100% coverage is a lazy slippery slope I don't take with my code.

The danger of 100% percent coverage is that the goal of tests becomes the 100% code coverage and not bug detection anymore.

</comment><comment><USER65> See the paper "On the Danger of Coverage Directed Test Case Generation". [http://link.springer.com/chapter/10.1007%2F978-3-642-28872-2...](http://link.springer.com/chapter/10.1007%2F978-3-642-28872-2_28) The idea is that a test suite can have 100% coverage and still be a very bad test suite.

</comment><comment><USER99> I find that, as I'm building something from scratch, the vast majority of the errors I make are just things I didn't think of. Tests don't help there because I can't test on input that I don't even imagine happening. So I generally write few tests, because, to be honest, most code is trivial and algorithm-light. Sure, if I have to write a parser or something a bit more fiddly, I'll write a unit test to be sure that it's doing what I expect, but that tends to be the exception, not the rule. I do write my code with an eye toward later testability if it turns out to be necessary, but I find that to be fairly easy, and also a good measure of if I'm doing the write thing: most code that isn't testable is probably code that's difficult to read and maintain, anyway, so if I look at something and think "oof, how would I ever write a test for that?" I'll usually delete it and start over.

When I have something that should be working, I test it in a more functional/integrative manner, and move on.

Later, I'll write unit tests when I need to. If I want to refactor something, or drastically change the implementation of something, I'll write out some tests beforehand to be sure that the pre and post behaviors match.

I've always thought that TDD is just premature optimization. You're optimizing for the idea that you -- or someone -- will later need to make large enough changes to your code that you'd worry about breaking it. In my experience that's fairly rare, and you spend less time overall if you just write the tests as you need them, not up-front. Yes, writing a test when the code is fresh in your mind will be faster than writing it much later, but then you're writing a ton of test code that likely won't be necessary.

An objection I hear to this is that you're not just writing tests for yourself, you're writing tests for the others who will need to help maintain your code, perhaps after you're gone. I'm somewhat sympathetic to this, but I would also say that if someone else needs to modify my code, they damn well better first understand it well enough such that _they_ could write tests before changing it (if they deem it necessary). Anything else is just irresponsible.

(Note that I primarily work in strongly statically typed languages. If I were writing anything of complexity in ruby/python/JS/etc., I don't think I'd feel comfortable without testing a lot of things I'd consider trivial in other languages.)

(Also note that some things are just different: if you're writing a crypto library, then you absolutely need to write tests to verify behaviors, in part because you're building something that must conform to a formal spec, or else it's less than worthless.)

<comment><USER100> > An objection I hear to this is that you're not just writing tests for > yourself, you're writing tests for the others who will need to help maintain > your code, perhaps after you're gone. I'm somewhat sympathetic to this, but > I would also say that if someone else needs to modify my code, they damn > well better first understand it well enough such that they could write tests > before changing it (if they deem it necessary). Anything else is just > irresponsible.

As someone who has had to fix plenty of legacy code, I have truly appreciated the people who have left me at least some working test suites to run - or just look at - and cursed many others. At the same time, I have generally been handed over code bases with tens or hundreds of thousands of lines, some of which had no useful tests.

If it is irresponsible to try to refactor or fix a codebase without first understanding all of it, it may be even more irresponsible to expect that those who follow in our footsteps will be able to do that, even if they are "gods of programming".

The reason why this is so was hammered home very strongly in Peter Naur's "Programming as Theory Building"[1].

Unless a code base is trivially small or simple, leaving it without meaningful tests instantly creates legacy code. I'll close with an excerpt from the back cover of "Working Effectively with Legacy Code[2].

> Is your code easy to change? Can you get nearly instantaneous feedback when > you do change it? Do you understand it? If the answer to any of these > questions is no, you have legacy code, and it is draining time and money > away from your development efforts.

[1]: [https://news.ycombinator.com/item?id=10833278](https://news.ycombinator.com/item?id=10833278)

[2]: [https://www.amazon.com/Working-Effectively-Legacy-Robert- Mar...](https://www.amazon.com/Working-Effectively-Legacy-Robert-Martin- ebook/dp/B005OYHF0A)

<comment><USER99> Yeah, I don't disagree with that, and one of the things I do before I hand off a code base to others is beef up test coverage somewhat, especially in places where the expected behavior of a bit of code might need to be codified to a certain extent.

Regarding your "legacy code" quotes: I don't write code that isn't easy to change, or easy to read. If I write code that's hard to read, I delete it and start again. If I absolutely cannot write something that's easy to read, I write tests around it and document the hell out of it (I'm iffy on comments and docs as well, because they _always_ end up out of date, and then are more of a hindrance than no comments at all).

I think I'm pretty good at what I do, but I wouldn't consider myself a "god of programming". Writing clear, concise code isn't hard. It really really really isn't. In my experience, the main blocker to that is ridiculous time lines and pressure to ship. I know it can be hard to push back against that pressure, but you owe it to yourself, future maintainers of your code, and the company you work for (even if the company doesn't realize or appreciate it at the time) to slow things down and do things the right way. A former colleague used to say, "It's not right because it works; it works because it's right". Just because a bit of code produces the output you want, it doesn't mean it's right. Write the right code -- readable, maintainable, verifiable, testable -- and you don't even need to worry about it working, because of course it will.

I lean on type systems heavily. If I were writing python or ruby, I'd have 10x as much test code as application code, because I just do not believe you can trust a dynamic/weakly-typed language without them. This is why I avoid such languages; I think any gains in rapid development that you get from such languages are quickly wiped out by the need to write extensive tests, or, lacking those, all the bugs that come up because you don't have them.

These days I write most things in scala, if I can. No, it's not a perfect language (honestly, I'd say a half to two thirds of it is crap, but the rest of it is _amazing_ ), but it has a strong type system that lets you lean on the compiler _so much_ more than many other languages. Simply the fact that it compiles gives me much higher confidence than with most other languages.

</comment></comment></comment><comment><USER92> Striving for 100% coverage is an expensive mistake because as a testing indicator it gives you a false sense of security. But someone has to pay for the time spent writing and maintaining those tests, and fixing the bugs that are still there.

I much prefer to use code coverage as a weak indicator for finding dead code.

</comment><comment><USER101> Good example of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." [https://en.wikipedia.org/wiki/Goodhart%27s_law](https://en.wikipedia.org/wiki/Goodhart%27s_law)

</comment><comment><USER102> > The tragedy is that once a "good practice" becomes mainstream we seem to > forget how it came to be, what its benefits are, and most importantly, what > the cost of using it is.

Totally agree. You can say this about lots of things really and not just tests.

</comment><comment><USER103> So generally, I write a test when I want to make an assumption an certainty. If I can't be certain that something is doing what it's supposed to I write a test for it, make sense?

So

``` Int => add(x, y) => x + y; ```

Doesn't get a test, however

``` Int => formulateIt(x, y) => (x * y)^y ```

Does

</comment><comment><USER104> I think pursuing 100% test coverage is not a fixed state, it is a must have process to learn how to write tests.

Think about one question first: why did the manager force develop to achieve 100% coverage? There must have some benefits, or the manager might come from the competitor. When standing at a higher position, think of time and organization factors, it might be a good choice. If every engineer in the corporate has the deeply understanding of test coverage as the author, they really do not need to pursue 100% coverage. But in reality, we can see many companies which do not pursue test coverage, their coverage tend to be 0. That's why we need force 100% test coverage in a short time. Engineers need time to form the habit of test their code, and then experience the pain of bad tests. Then they start to think what kind of tests are valuable.

</comment><comment><USER105> It's a case of convention over common sense.

</comment><comment><USER106> Has anyone read any papers about the relationship between code coverage and defects?

<comment><USER107> Not exactly what you asked for but this compares defects and different types of testing. Unit testing is one of the less effective ways of finding bugs.

[https://kev.inburke.com/kevin/the-best-ways-to-find-bugs- in-...](https://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-your- code/)

</comment></comment><comment><USER108> Laziness is a kind of populism. Such articles will be always upvoted.

<comment><USER109> He's not preaching laziness, he's against hitting metrics solely for the sake of hitting the metric when the metric is irrelevant. 100% code coverage? Good. 100% code coverage in a single test framework at 10 times the development cost when it's still not really 100%? Stupid.

Working smarter isn't always in opposition to working harder.

<comment><USER108> No, he is exactly against covering simple code by tests. It's explained below first sentence.

      It is funny how things turn around.  For fifteen years I
      have been preaching TDD (Test-driven development, or as it
      used to be called: test-first approach), or at least for
      developers to write some unit tests. However, in recent
      times I have found myself saying more often, "Why did you
      write that test?" instead of, "You should write a test."

Answer to that "funny turn around" sentence is simple - he is getting older and more lazy. Each of us will, each of us should fight with it.

And after that, example of what I'm talking about:

      It seems that he had trouble using Mockito to test the
      following piece of code:
      ...
      I think he was very surprised with my response: "You
      don't need to test that."

<comment><USER110> A concise word for 'older and lazy' might be 'experienced'

<comment><USER108> Not always, not in this case definitely.

<comment><USER110> I don't know; disagreement over whether everything must be tested isn't enough to call it 'lazy'? Its an easy dart to throw. But remember, everything you don't do leaves room for more progress elsewhere. Its an optimization problem.

</comment></comment></comment></comment></comment></comment><comment><USER31> One of the pressures for 100% coverage is working in a non-typesafe language. The gospel of coverage largely evolved in the Ruby community, where I often see test suites that look like a handrolled typechecker.

</comment><comment><USER111> I think unit testing makes sense when you have a function doing some math that can't be easily verified to be sensible by merely glancing at the code for two minutes.

I'm not sure there's much use for it in other scenarios.

</comment><comment><USER112> I'd suggest the tragedy here is an absence of kaizen and team processes that foster continuous improvement. If folks are doing inefficient things, that should be caught by the team in a retro or similar.</comment>


Tilemill: Maps done right - <AUTHOR> http://tilemill.com/pages/index.html

<comment><USER2> I've used TileMill pretty extensively, and while TileMill is great for generating static maps (PNGs) for infographics, it won't replace Google Maps any time soon for _serving_ dynamic maps.

For example, you can serve a dynamic map of Washington DC on a website using TileStream and TileMap, but if you wanted to also view Fairfax, VA or Baltimore, MD, you're out of luck unless the tileset includes those tiles. With Google Maps, you always have access to all the tiles.

That's not to say that TileMill isn't a great product. The choice to use CSS is brilliant. Plus they're based in Washington DC, where I'm currently living! Nice to see tech outside of the Bay Area, NYC and Boston.

<comment><USER3> You're right by definition - TileMill doesn't serve maps. Right now it generates them, and TileStream serves generated maps. It's extremely fast and stable, but inherently limiting.

That's not to say this won't change in the future :)

<comment><USER2> Thanks for releasing a "native" app for OSX! This makes it much easier to use TileMill. The previous installation method was not the most user friendly-- this coming from a Node.js developer.

<comment><USER4> Glad you like the Mac app. We're definitely shooting for a first class OS X app experience. Feedback on ways to improve it further are welcome, too.

<comment><USER5> Did you package it yourself or used some package/library?

<comment><USER4> We packaged it ourselves. It's open source, too. Check out <https://github.com/mapbox/tilemill/tree/master/platforms/osx>

<comment><USER5> Amazing, thanks!

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> This looks really, really slick. Next time I'm creating a map-based site I'm going to have a play around and see what customisations I can do. It's very difficult to match the default Google Maps style for clarity, but sometimes it would be good to have more options than their styles permit.

<comment><USER7> Indeed. That css-clone looks like what SLD styling should have been.

<comment><USER8> BTW, geoserver (which uses SLD) has a CSS-styling extension which is much easier to use.

</comment></comment></comment><comment><USER9> I never thought I would see the day where there is OS X and Linux downloads, but only a VM image for Windows. Very cool.

<comment><USER10> I've started saying that the only way for windows to continue to be successful is for windows 9 to be written on top of a common unix.

They have lost the hackers completely, but if they can make windows 9 a platform that will autospin up a vm of the windows platform for things like Office, etc. They _might_ have a chance.

My house mate is a civil construction engineer. He is so envious of my Ubuntu setup.

"Wow, that looks as cool as Android"

"Android is actually the same thing only differently packaged"

"Really! Can I try out your computer?"

"Sure, let me make you an account and show you some hotkeys!"

 _After some time passes_

"Wow! This is awesome! Can you install it on my laptop?"

"Since you use AutoCAD and MS Office I'm afraid it would be too much of a hassle for you, but if you are willing to learn I can."

Never thought I'd have someone ask me to install linux because they find it more usable and attractive.

<comment><USER11> That's the problem I have. I used to primarily use Ubuntu, but as an aerospace engineering student I often need use Solidworks or other CAD applications. Linux/Mac? Out of luck. Maybe that'll change one day (Dassault have released their free 2D CAD application DraftSight on linux, and they develop Solidworks), but we're probably at least 5 years off a decent linux/mac CAD application for designers.

</comment></comment></comment><comment><USER12> I'm consistently impressed by the work DevSeed puts out with such a small team. What's the secret? :-)

<comment><USER13> I couldn't have said it better myself. Hands-down, they're one of the most impressive organizations I've come across on the web both in the products they create and the contribution to society their creations address.

</comment></comment><comment><USER14> I think the screenshot should use a bit higher-contrast color scheme for the map. I was slouching in my chair a bit and couldn't really see what it was at first :)

</comment><comment><USER15> Was just about to download and start using this. No Windows version though. Maybe the start of a weekend project for me.

<comment><USER3> There's a VirtualBox image as well. I wouldn't recommend cygwin unless you want a weekend of hell.

</comment></comment><comment><USER16> The main MapBox site (<http://mapbox.com/>) is definitely worth checking out as well. They have a pretty cool iPad app and a nice ~6 minute video demoing everything (<http://vimeo.com/20006926>)

</comment><comment><USER17> The Google Maps API lets you style maps too [1]. You can change colors and show/hide different map features. There's also a wizard [2] to make styling easier. You can get some very interesting effects [3].

[1] [http://googlegeodevelopers.blogspot.com/2010/05/add-touch- of...](http://googlegeodevelopers.blogspot.com/2010/05/add-touch-of-style-to- your-maps.html)

[2] [http://gmaps- samples-v3.googlecode.com/svn/trunk/styledmaps/...](http://gmaps- samples-v3.googlecode.com/svn/trunk/styledmaps/wizard/index.html)

[3] [http://googlegeodevelopers.blogspot.com/2010/10/five- great-s...](http://googlegeodevelopers.blogspot.com/2010/10/five-great-styled- maps-examples.html)

</comment><comment><USER18> The ruleset screenshot reminds me of <http://www.maperitive.net/> which is an excellent tool too (not web-based).

</comment><comment><USER19> In case anyone is wondering about the shell script for Ubuntu install it checks for maverick or natty and then installs like so:

        apt-add-repository ppa:developmentseed/mapbox
        apt-add-repository ppa:chris-lea/node.js
        apt-get update
        apt-get install tilemill

</comment><comment><USER20> Love TileMill, love TileStream, love the tilesets. One day I'll move to DC...

</comment><comment><USER21> Really incredible work. Is there anything we can do to speed up exporting, short of manually parallelizing across multiple computers? Distributed export might be a nice feature.

<comment><USER3> Recent updates (if you haven't updated since 0.3.0 or so) have doubled & tripled export speed by a bunch of optimizations, especially metatiling.

If you really want to parallelize, you can run the ./index.js command in the source version of TileMill to do exports programmatically.

</comment></comment><comment><USER22> I have started to play around with this foR custom OpenStreetMap rendering. Carto, the css format, seems a bit easier to play with than the raw mapnik xml.

</comment><comment><USER23> This looks great, thank you.</comment>


How many people are there in space right now? - <AUTHOR> http://www.howmanypeopleareinspacerightnow.com/ <comment><USER2> Everybody. Over a big fat floating rock.</comment>


Parable of the Polygons – a playable post on the shape of society - <AUTHOR> http://ncase.me/polygons/ <comment><USER2> This is a fun simulation that I see pop up here on HN periodically. Even this post from 2 years ago was marked as as a [dupe] :)

[https://news.ycombinator.com/item?id=9672219](https://news.ycombinator.com/item?id=9672219)

</comment><comment><USER3> > world starts segregated. what happens when you lower the bias?

There is a hack to fix the situation. Move slider right, up to 100% intolerance, and let them jump around. Wait till chaotic jumps make society chaotic, than move slider to zero. All are happy now.</comment>


Why Garamond Won't Save The Government $467 Million A Year - <AUTHOR> http://www.fastcodesign.com/3028436/why-garamond-wont-save-the-government-467-million-a-year

<comment><USER2> Good lord.</comment>


Good advice for all C programmers out there. - <AUTHOR> http://www.embedded.com/columns/barrcode/216200567?printable=true

<comment><USER2> It is not a good idea to use
volatile
 for creating multi-threaded applications like the author is explaining it will just end up creating more problems than solving one. Better use a library.

The problem with volatile is that if mark everything volatile you program is just going to run a lot slower. If you don't make all of them volatile it is still possible the hardware could reorder references even if the compiler doesn't.

<comment><USER3> The hardware will reorder references regardless of how many of them are marked volatile - CPU doesn't know about volatile directives. The other related problem is caching - reading the same address from different CPUs will occasionally yield different results and may produce _inconsistent_ results. Like so:

      a=b=c=1;a=2;b=2;c=2;

Reading from a different CPU may produce any combination of values 1 and 2, including the seemingly impossible 2,1,2.

Just use a library.</comment></comment>


How I write and curate the cron.weekly newsletter - <AUTHOR> https://ma.ttias.be/how-to-cron-weekly-newsletter/ <comment><USER2> I'm subbed to too many newsletters currently, and this is pretty much the only one I always read 100% of the way through.

There's something about it, the way it gives me the link to the thing first, the short summary with the occasional personal comment. It's just right.

Good job, Mattias!

<comment><USER3> This and devop-ish are the only ones I usually read 80+% every week.

</comment></comment><comment><USER4> Thanks for pulling back the curtain! BTW, I love the McAfee Shillbot you made. Has this been a productive strategy?

[https://github.com/mattiasgeniar/McAfee-Shill- Bot](https://github.com/mattiasgeniar/McAfee-Shill-Bot)

</comment><comment><USER5> This is one of the few newsletters I subscribe to, interesting to see how it's done!

<comment><AUTHOR> Thanks!

</comment></comment><comment><USER6> Is there a reason the link to Mailcoach contains google analytics campaign info? Sponsorship?

<comment><AUTHOR> Author here: definitely not a sponsorship, I'm not getting paid in any way. I _am_ however curious to see what this might do in terms of clicks, hence the parameters.

<comment><USER6> Coolio, thank you for the response. How _did_ it go?</comment></comment></comment>