Test-Driven Development is Stupid - <AUTHOR> http://geometrian.com/programming/tutorials/testing/test-first.php <comment><USER3> > _You are writing code to test something that doesn 't even exist yet. I am > not rightly able to apprehend the kind of confusion of ideas that could > provoke such a method._

Yeah, the fact that you can't comprehend why people do this is very clear; if you could, you wouldn't have written this terrible rant. It feels like the author is criticizing this before coming anywhere close to understanding why people do it.

I really wanted some kind of point to find to disprove, bu there aren't any in this post. The strongest proof the author offers to his point is using one word tautological sentences to repeat what he just said: "Tests don't work because they just. don't. work."

The more I read, the more I realized this post isn't about testing, it's about the author letting the world know how fucking awesome and smart they are. They build huge software suites (from the sound of it, completely alone) with no tests, and everything works out fine, even better than fine, spectacular. It seems like the author deleted this line from the post in response to ridicule in the comments:

> "As it happens, I do write code others depend on--as it happens, a lot of it > --and, my code has never, even once, failed in production: a record I am > extremely proud of."

I don't even particularly ascribe to TDD, but the arrogance and dismissiveness and contempt of this guy made want to see him proved completely and utterly wrong.

<comment><USER4> >You are writing code to test something that doesn't even exist yet.

To piggyback on this... sure, the _code_ doesn't exist yet, but the project specs do. And unit tests can help by making the required specification explicit.

<comment><USER5> In any other field, specifying a function by its value at a handful of points would be a bad joke.

If you don't know what you want your code to do then tests will just make it harder to experiment, and if you do know then there's no harm in writing them after the fact.

<comment><USER6> Bench vices pin down a board at a handful of points. Other things about the environment (shape of board, material properties of wood) help me make sure it generalizes in the ways I care about.

</comment></comment></comment></comment><comment><USER7> > I am against writing unit tests in general, since experience shows that they > actually prevent high-quality code from emerging from development

And he's lost me in his first sentence.

Unit testing, good unit testing at least, is not just about developing as it is about preventing regressions. A unit test that runs on every build ensure that something is true and that it stays true forever.

<comment><USER8> I think I see part of his point, though. By its nature, a unit test is (often, albeit not necessarily) tightly coupled with the thing it is testing--which means that anyone who changes the implementation must change the test, which increases the complexity of code changes.

I know that in my own work, the single most important thing for me is to be able to massively refactor, restructure and redesign my and others' code, as I'm working with it (this is probably why I like dynamic languages and macros so much, and probably why I benefit from static typing so much); anything which gets in the way of deleting, reorganising, clarifying, duplicating, altering, reducing and shifting code is going to slow me down and make the resulting code worse.

Perhaps, though, there is a middle ground: while the internals of a library must be free to mutate, the external interface ought not to change nearly as much. Perhaps all unit tests should be written to test the external interface, not the internal details. That might also help avoid the tests-which-test- that-the-code-does-what-it-does disease.

I do really appreciate tests, and they do help detect and prevent certain types of regressions.

<comment><USER9> Which is why testing behaviour not implementation is so important. Admittedly this sounds easier than it is in practice. Personally I've found that in- memory coarse-grained tests more akin to integration tests, that test from the outside in are helpful and easier to maintain because they tend to be less concerned with the how. I can then choose to write finer-grained unit tests where appropraite

<comment><USER7> Agreed! A code base I recently inherited (and exorcised) contained unit tests with roughly 90% coverage... And not a single assert. They didn't understand the idea of testing behavior, they just copied the implementation as tests.

I reduced 20,000 lines of tests to 5,000, and caught a dozen bugs.

</comment></comment></comment></comment><comment><USER10> I love the first quote for how (ironically) true it is:

"Trying to improve software quality by increasing the amount of testing is like try[ing] to lose weight by weighing yourself more often."

From someone who has lost over 15kg in the past in few months, one of the things that helped most was starting weighing myself as I didn't do that previously. It kept reminding me that I wasn't still there and gave me more motivation

<comment><USER11> What is "more often" ? Weighing yourself daily is useless, since the weight fluctuates too much. Not to mention that "weight" is useless as well, while you're actually trying to loose FAT. Fat percentage and resulting lean body mass (body weight minus body fat) are the metrics to measure. But still a waste of time doing it daily, twice per week to gain insight on the delta and where it's trending is enough.

<comment><USER12> Weighing yourself daily is _not_ useless. The idea of doing it weekly is being spread explicitly because GenPop doesn't understand the concepts of a moving average and a low-pass filter - instead, they freak out over those daily fluctuations. If you weigh yourself daily and remember to always look at the average of the last few samples, you get a more useful indicator of your current weight trend.

</comment></comment></comment><comment><USER13> There're already people speaking about the limits of a too strict TDD:

[http://david.heinemeierhansson.com/2014/tdd-is-dead-long- liv...](http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live- testing.html)

And yes, it may be stupid to test before a design emerges - but only if you start with a very fine-grained test. Usually when I'm coding from scratch I write a very, very, VERY coarse-grained test that "tests something", and when I reach the point of passing it (which may involved creating and designing multiple classes) I probably have a working design and I may begin creating other, smaller unit tests for individual components. The initial test may disappear or become an integration or acceptance test.

By the way there's little content in the article. It's just a rant. And the article about not writing test cases at all is simply ridiculous - since an error can exist in a test, we shouldn't write test cases?!?

<comment><USER14> > Usually when I'm coding from scratch I write a very, very, VERY coarse- > grained test that "tests something", and when I reach the point of passing > it (which may involved creating and designing multiple classes) I probably > have a working design and I may begin creating other, smaller unit tests for > individual components.

That is exactly how I start coding something from scratch, except I don't have that initial test. I don't think such a vague starting test adds anything real.

<comment><USER13> It offers the value of having an initial target, otherwise you risk coding too much without stopping.

</comment></comment></comment><comment><USER15> If you're doing TDD (or software development) like this, you're doing it wrong. Yes, yes, I know. No True Scotsman [1]. But when many - I mean LOTS OF - people say "I get benefits form [Technique]", you can't just say: "It cannot work. I tried it, it sucked.".

I mean, you can say that. But doing so makes you look... ignorant - at best. You know, there is a possibility that you just got it wrong.

So, many great programmers say that they get benefits from TDD. _They_ get benefits. Not the suits. Not their co-workers. _They_.

TDD is hard. I had to learn it and practice it. And I'm still learning and practicing it, even though I'm now also teaching it to others and helping teams implement it. But I get benefits from it. Writing tests helps me to think about problems and to _actually improve my designs_ [2]. And writing tests helps me to know when to stop - To not gold-plate my designs.

Sorry, but this "article" is just an angry rant, with no real arguments. Please, don't give up on TDD early because of rants like this. If you have any questions about TDD or need help getting started, feel free to ask (here or in private - you'll find my email address in my profile)...

[1] [http://davidtanzer.net/no_true_scotsman_in_agile](http://davidtanzer.net/no_true_scotsman_in_agile)

[2] But you have to refactor ruthlessly. And learn and practice refactoring, which is hard.

<comment><USER16> >If you're doing TDD (or software development) like this, you're doing it wrong.

What do you think is the right way to do it, then?

<comment><USER15> My point is: If TDD leads you to _bad design_ you're doing it wrong. You are probably not listening to your tests and you are probably not taking care to refactor towards a better design. Maybe you are even writing bad unit tests [1].

BTW, if _any technique_ in software development leads you to bad design, and you don't stop and try to improve something, you're doing _software development_ wrong. If a technique does not help you, you have two possibilities: Trying to do it better (maybe with outside help), or trying something different.

[1] [http://www.makinggoodsoftware.com/2012/01/27/the-evil- unit-t...](http://www.makinggoodsoftware.com/2012/01/27/the-evil-unit-test/)

<comment><USER12> But that's a general counterargument for criticism against anything - "if it doesn't work for you, you're doing it wrong"!

Ultimately, we're too young a field to be able to replace common sense with a process.

</comment></comment></comment></comment><comment><USER12> _Way_ stronger than I'd write, and I don't agree with it entirely, but the author has a point.

Personally, what annoys me the most about TDD I've seen in the wild are two things: designing for tests instead of actual problems, and tests affecting the structure of "real" code.

Designing for tests - the standard TDD approach, first we write tests, then we write code to pass the tests. Quite often the consideration of _the problem being solved_ disappears. It's a fine approach when your task is to write a small black box that takes some data in and transforms it into something else. But I've never seen a case where someone made it work for complex tasks. It always ends up the same - your tests become more complicated than the tested code. It happens with any non-trivial problem, because the thinking you have to do to write tests that make sense is _the same_ as the thinking you need to do to solve the problem in the first place. So you're basically writing the program twice, only in a convoluted way, and without considerations for global design.

Tests affecting the structure - this is IMO a strong code smell. If you're modifying your design to accomodate for tests, by e.g. adding superfluous dependencies, hooks or injection points, you've screwed up. It only makes the code more complicated and less reliable.

The only tests I've found valuable so far (in terms of effect for effort spent) are regression tests - the ones you write to catch bugs in order to make sure they won't happen again. Everything else in TDD seems to be easily replaceable by proper iterative programming.

Maybe that's why TDD is popular in the languages without a sane REPL.

<comment><USER17> > Tests affecting the structure […] is […] a strong code smell

On the other hand, code being hard or impossible to test is often thought of as a code smell as well. Code that is easy to test is easier to understand — not in the least because there are tests demonstrating its use. Techniques such as dependency injection help a lot here.

<comment><USER18> Techniques like dependency injection can be really useful -- see Angular -- but too often I see a perfectly understandable piece of code expand into a mess of multiple constructors (only one ever called in production) and helper methods all so that strict unit testing can be done. The post's commit story was unsurprising. If TDD is being done this mangling often just happens upfront. DI is great in the same way interfaces are great, but if your code _actually_ just cares about a particular implemention you instantiate wherever, or even better is built-in to the language, it's easiest to reason with that implementation. Taken to the extreme, you get Enterprise FizzBuzz.

</comment></comment></comment><comment><USER19> Just wait until he has to write a system with several hundred web services that have been documented to perform in a very precise manner given particular data sets, and have hundreds of customers who have integrated to that API, and absolutely _required_ that there be no variance in the output in the API, or there (extraordinarily expensive) system integration will fail, at great costs to their business systems.

I'm not saying TDD makes sense everywhere, but being able to confirm that your 3000+ continuous integration tests all passed green before shipping a new version to all of your customer is a huge way to avoid embarrassingly predictable bugs. (Leaving, of course, the opportunity to ship the embarrassingly non-obvious bugs)

And, honestly, it's not clear that he was opposed to TDD, as he was opposed to being constrained to having to write huge test frameworks during the exploratory phase of code development, in which you may want to have a bit more freedom to write/discard while you feel out requirements/solutions.

<comment><USER12> What you described _having tests_ , not _doing TDD_. That is, it's important to have those test cases verifying that your APIs still work, but that doesn't mean you have to _design_ the APIs by writing tests, as opposed to designing by thinking about what you actually want to achieve.

<comment><USER19> Fair point. I was more referring to his comment here: _I am against writing unit tests in general, since experience shows that they actually prevent high- quality code from emerging from development--a better strategy is to focus on competence and good design._

<comment><USER12> I disagree about that with the author too. I think _testing_ and _TDD_ should be clearly separated as two different things.

</comment></comment></comment></comment><comment><USER20> I genuinely wanted a good critique (It's nice to get dissenting viewpoints) but reading text so charged with anger and so scant in information is stressing me out.

<comment><USER21> agreed - there are certainly some disadvantages to TDD, and I was hoping for an discussion raising some ideas I'm yet to formulate fully.

Instead there were lots of parallels that I don't recognise.

It was disappointing the author didn't seem to try and find the advantages in TDD and discuss a counter argument or alternative, such as modifying a foreign code base.

</comment></comment><comment><USER22> There have been one or two HN posts somewhat recently indicate that developers should stop calling themselves "engineers." Enter examples of how engineered solutions generally work the first time round, and how software almost universally does not. There's a fundamental reason for that: it's computer _science._

One of the facets of science is that you have a hypothesis. Writing a unit test before you start is analogous to having the hypothesis: "the solution I come up with for this problem will work." The rest of the exercise of TDD is a scientific process of creating a solution and _proving_ that your solution works (and adjusting your hypothesis if you find it to be incorrect), albeit in a slightly strange way.

I could have a hypothesis about how earth is really traveling through the stars on the back of a turtle. We know that is not true because Einstein came up with a solution, followed by him and others _proving_ that solution. Relatively is only valuable because _proof_ of it exists.

If computer science is a science (which it is) we must put our solutions to the same degree of rigor that other scientists do.

While I no longer follow the strictest form of TDD (starting with tests that fail to compile) I'll never forget the singular most important lesson that it has taught me:

        A solution is worthless if you cannot prove that it works.

I'd take this article more seriously if Mr. Mallett provided an alternative for correctness proofing. I can't honestly sell software to someone if I don't know if it works myself.

<comment><USER12> I disagree on multiple levels. First of all, programming is _not_ computer science. Sure, we probably do not deserve to be called "engineers", but what we do goes in completely other direction - _away_ from science and towards art.

Secondly, I wouldn't try to fit unit testing into scientific process - <USER2> if the test is a hypothesis, then what you're doing is the exact opposite of how science is done. You do not design your experiment to make your hypothesis come out true!

While the quote you cited is interesting, I'd treat it with a grain of salt, given that you can't _prove_ that your solution works - and if you think you did, it usually turns out the _proof itself_ is wrong. It happened even for formally proven algorithms.

Testing gives you increased confidence. It's a worthwhile goal. But IMO, driving your design by tests is going a bit too far, and something you don't need to do to have tests that ensure your code works.

<comment><USER23> "Secondly, I wouldn't try to fit unit testing into scientific process - because if the test is a hypothesis, then what you're doing is the exact opposite of how science is done. You do not design your experiment to make your hypothesis come out true!"

True, but I think that objection can be trivially fixed by simply... doing that. I write tests that confirm the code does what I designed, but I also write code that tries to break my design, and tests that verify that it errors as expected. I do approach it with a scientific mindset. (And I tend to consider "scientific mindset" to be the more important part of science vs. some overprivileging some checkpoint list of specific techniques, which ought to have come from the scientific mindset in the first place.)

Of course it remains true that you can't do that perfectly, but that's a null objection in the end. Nothing ever can be, but at least I try.

I can't even count how many times I've tested a code's error case, only to discover that it unexpectedly "worked". Usually that's because there's a bug and I need to fix the error case... every once in a while it turns out my code corrects my own understanding when it reveals what I thought was an error case is actually perfectly valid and sensible, though. It's important to _try_ to break the code.

<comment><USER12> > _I do approach it with a scientific mindset. (And I tend to consider > "scientific mindset" to be the more important part of science vs. some > overprivileging some checkpoint list of specific techniques, which ought to > have come from the scientific mindset in the first place.)_

True. If you just follow the checklist without following the spirit of the scientific method, you end up doing socio^H^H^H^H^Hcargo-cult science.

As for scientific mindset in programming, I think it's a very valuable thing to have on both larger scale - in various forms of testing - and smaller scale. I found that, when running your code, it's good to just ask yourself what exactly do you expect to happen beforehand, and if you see _any_ deviation, immediately go figure it out, or at least note it down. If a program does something unexpected it means you don't understand it.

</comment></comment></comment></comment><comment><USER24> > Let me emphasize: you write the test cases for your program, and then you > write your program. You are writing code to test something that doesn't even > exist yet. I am not rightly able to apprehend the kind of confusion of ideas > that could provoke such a method.

I don't understand this. What is so absurd about specifying facts about the program you will write? When we have tools that can prove facts, we will be doing formal specifications instead of random sampling. But still, testing is a way of statistically specifying facts about your program, for which it seems sensible to be written before the program.

<comment><USER25> Because like the author says, there is no substitute for competence. If you can't write good code, then your assumptions about the future design will be wrong and/or bad, and your tests will be as poorly written. In other words, TDD (anecdotally) doesn't improve your code but only introduces extra levels of complexity. Which in the hands of an incompetent developer becomes an even bigger problem than if there was no TDD.

<comment><USER26> If you can't write good code, then you will not write good code.

But, there is another common case where unit tests (written before or after) also come in tremendously handy: if you can't write good mistake-free code _100% of the time_.

Which describes all programmers who have ever existed.

</comment></comment></comment><comment><USER27> Honestly this doesn't belong on the front of HN. There is no discussion value, or sense of professionalism. It's just a rant riddled with misplaced anger.

There are valid reasons to be against TDD, but it is not stupid. To call a methodology "stupid" is needlessly judgmental and really has _NO_ place in the pragmatism and trade offs that so often go along with writing software. Anyone that calls a software methodology "stupid" so casually has no business writing software at all, nor blog posts about software.

I don't care how smart you are, or how right you are. If you want people to listen to you and consider your ideas, don't write posts like this, in this tone, at this length, with so little actual substance.

</comment><comment><USER28> When I read these kind of articles I'm always curious to see the professional background of the author. Not to criticize, but to see if he/she's talking about something he/she saw in scale or not. Because if you're working with a very small code base then I may even understand sentences like "I am against unit tests in general". I've never met people who work (or worked) in very large companies being against, at least, unit test. Where hundred/thousands of people touch the same code .. not having unit tests, in the long term, is a suicide.

<comment><USER29> None. He's still in university. [0]

[0] [http://geometrian.com/about/cv.pdf](http://geometrian.com/about/cv.pdf)

<comment><USER30> I think you're being slightly disingenuous there with the "still in university" dismissal. That CV is a lot more impressive than many people I've worked with in the "real world".

</comment></comment></comment><comment><AUTHOR> A while back I described the benefits I think unit tests give you:

1\. Well-tested parts

2\. Decoupled design

3\. Rapid feedback

4\. Local context to test in.

At the same time, there are many cases where they don't help much. More here: [http://henrikwarne.com/2014/09/04/a-response-to-why-most- uni...](http://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing- is-waste/)

<comment><USER31> I think you're missing the most important one: ensuring you don't unexpectedly break your own code in the future (when you come back in 6 months and forget why exactly it's arr[1:n-1] not arr[0:n-1] or arr[1:n]).

<comment><USER32> Sounds like a well placed comment would be of more value than a test there.

<comment><USER33> Comments are easy to miss though. We arnt perfect about going through our code and testing gives us objective measures.

<comment><USER32> A test won't say why code is the way it is; it will simply provide some sample input and expected output to that code. This may be enough, but not always.

</comment></comment></comment></comment></comment><comment><USER34> His experience might be a lot different from mine, but I've found TDD enormously helpful when tackling legacy systems. Maybe that's not "test-driven development" and more like "test-driven refactoring", but working on complex legacy systems where the original developers are basically all gone is _scary_ , and TDD has helped me make some sense of it and feel a lot better about making changes.

</comment><comment><USER35> I don't necessarily use TDD all the time, but I think it provides significant value. A key one is to help guide developers break down complexity.

This is quite apparent especially when I conduct pair programming interviews. Developers who were exposed to TDD (or to projects with significant test code) approach the problem in a far more structured manner, and their code is much more pleasant to look at :)

</comment><comment><USER36> No you're stupid. :P

This is just a rant. I was hoping for an actual study, but the title should have given it away.

</comment><comment><USER37> I'm not clear what technique this article is trying to describe, but anyone that spends two weeks pre-writing tests is not performing TDD. That simply isn't how it works.

The cycle is: red, green, refactor, repeat. That's per test. It shouldn't take long. It works nicely.

</comment><comment><USER38> TDD assumes you know what all the interfaces are going to be beforehand. If you discover that an interface needs to change you have to refactor all of your tests. Unit tests are momentum against change.

It also assumes a bug free app with 100% code coverage is the objective, regardless of the cost it takes to write all that test code. An app that has a few bugs but takes half the resources to develop can make better business sense.

<comment><USER39> > Unit tests are momentum against change.

I get the point, but feel different. When a project becomes large enough, changes can result in breakage in the weirdest places. Proper testing can help discovering this breakage before shipping. For me, tests are a safety net for changes in the tested code.

Large projects without any tests are effectively unmaintainable, often times even for the original author, and most certainly for others.

<comment><USER38> I do agree with this, and have felt it from both ends. If you need to change a core class or interface which cascades throughout the system, those unit tests are sure nice to have. I'm not sure I'd be confident enough to change the core class without the unit tests. In that sense, lack of unit tests is momentum against change.

</comment></comment></comment><comment><USER40> Buried in the rant are a couple of sound points:

>\- You're trying to make a design before you learn anything about it.

>\- People write something one way, but then are afraid to change it because they'll have to rewrite the testing code that goes along with it.

Both of these suggest tests of the wrong granularity; that people are, as he says, writing silly little fencepost error checks for every single function.

</comment><comment><USER41> > Week 3-4: Write tests.

> Week 5-10: Write code.

??? what

The feedback cycle is one test, implement code. Not implement test suite for the entire program, then implement your code base.

</comment><comment><USER42> Please, correct me if I am wrong, but the process he described is completely against TDD and no wonder it did not work (they wrote the function first, and added tests later, lot of tests, in TDD you would have rather more shorter functions with a few tests for each):

 _\- Function A is 147 lines long. It is the simple core of the program.

\- Function A is committed to the repository on June 26th, 2002. Function A has four test cases. Nevertheless, a bug is found in Function A on the 28th and a patch is uploaded on July 6th. It contains two new test cases.

\- This continues a bit. However, by August 2002, function A is mostly stable and has no fewer than thirteen test cases--mainly for fencepost errors and other idiotic things anyone can find with a stack trace. Except for a blip in early 2003, function A, now 152 lines long, is unchanged until mid-2006._

</comment><comment><USER43> I'm writing an SMTP server, and when I started it I looked at the problem, and for the first time since I stopped being a student, I had a well specified problem where I could try TDD.

At the time I rationalized it and said - nah, I'm too lazy for doing that on a side project. But as soon as I tried to use it in the real world, I discovered that all those RFCs are just crude approximations of how people used the server (most of them are even incomplete). There's absolutely no compliant client or server out there, and TDD would catch none of the bugs I discovered at the time. And that's with email - a protocol that people are working on standardizing since before the Internet existed.

Sorry, but nowadays I'm extremely skeptical about TDD having any application at all. Not even for reinventing the wheel.

</comment><comment><USER41> TDD is truly cult like in it's extreme. Usually it's managers and slightly weaker technical folk with a scrum certification that spew the absolutist TDD path.

I'm a contract java developer and I know how to 'play the game'. The hypocrisy in the amount of fizbuz clones I've written in a TDD fashion on interviews, to then see the production code has no little to no unit tests.

I once got negative feedback on an interview because I wasn't TDD enough after expressing the opinion that TDD works great for most use cases but there are limits, e.g. positioning of a front end element isn't always best done TDD. The guy interviewing was non technical and of course any dissent to TDD meant I was a bad fit.

Anyways - end rant. Like I said I just learnt to play the game.

</comment><comment><USER44> It's not really a rant, it is a very strongly communicated opinion. And I agree with a lot of it.

I don't mind writing a few test cases during or after I am done with my work. This is mostly to protect someone (including me) to change my perfect design after I have found it, though ;-)

</comment><comment><USER45> Oh look another article with a provocative (e.g. link-bait) title. I'm sure this will be a well-balanced, nuanced piece on the costs and benefits of TDD, when it makes sense, when it does not make sense, and will leave me with a better understanding of the subject.

Oh.

</comment><comment><USER46> Does anyone have a good counter-point worth reading? I understand that it's more of a anecdotal rant, but it's not the first time I've read something like this about TDD.

I've tried TDD, but I've found that it just stifles my productivity too much.

</comment><comment><USER47> TDD isn't about writing tests. It's about designing architecture.

[http://www.drdobbs.com/tdd-is-about-design-not- testing/22921...](http://www.drdobbs.com/tdd-is-about-design-not- testing/229218691)

</comment><comment><USER48> It seems a lot of these discussions surrounding testing are very anecdotal; this link being a prime example. I remember hearing an interview with Greg Wilson regarding his book _Making Software,_ [1] the premise of which is to apply a more rigorous methodology to understanding what makes software work.

If I remember correctly from the interview (I think it is here[2]), one conclusion was that TDD doesn't have a clear benefit when you add it to a project. On the other hand, in a survey, TDD projects are more likely to succeed because it is a habit common to good developers. I hope I am capturing the subtlety there. Essentially, TDD is not a silver bullet, but rather a good habit shared by many good developers. That was enough to convince me of the merits.

It's another problem altogether to try to institute TDD for a project, especially for a team. Like so many things in programming, TDD could be used and abused. The same could be said for JavaScript or [insert proper noun here]. If misunderstood or used incorrectly, TDD could be a drain on the project. A benefit--and this ties back into the idea of TDD as a habit--is that it forces the code you write to have at least one other client. This requirement would alter the way you write code and arguably for the better.

[1] [http://shop.oreilly.com/product/9780596808303.do](http://shop.oreilly.com/product/9780596808303.do)

[2] [https://blog.stackoverflow.com/2011/06/se- podcast-09/](https://blog.stackoverflow.com/2011/06/se-podcast-09/)

</comment><comment><USER49> > I was once one of these newly educated kids. So, for a few years, I worked > exclusively with the Test-First strategy. When it was over, the results were > undeniable. The code--all of it--was horrible. Class projects, research > code, contract programs, indie games, everything I'd written during that > time--it was all slow, hard to read, and so buggy I probably should have > cried. It passed the tests, but not much else.

Sounds like most everyone's first years :)

</comment><comment><USER50> TDD has always looked to me like a good idea taken to ridiculously dogmatic extremes (a very common occurrence in software development, IMHO), but I think many of this author's indiscriminate potshots miss the most important problems.

In order to write a test for something, you need to know what it is supposed to do, and testing will not tell you that. In order to make something that passes your tests, you have to design it, and a test does not tell you what that design should be - it can tell you if you failed in a specific way, but not what to do about it. There is a whole lot of analytical reasoning and technical judgement to software development that is ignored by TDD, and while thinking about test cases can help with this, it is an insufficiently powerful method to complete the job.

Agile methods have not rescinded this fact. Insofar as they form a complete development methodology (and I am not sure about that), they offer an empirical approach to discovering your requirements (which may or may not be appropriate to your situation), and the use of short-cycle iteration to gauge how you are progressing, but they are largely silent on the issues I raised above.

</comment><comment><USER51> Well it is not a great rant but ultimately if you just read the conclusion he does have some good points. There really is no substitute for competence, the process can and does hinder good design, it is used as a false crutch against incompetence. And I do think that test first and in general a large overburden of existing tests does subconsciously limit refactoring and better design.

Thats not to say all testing is bad. I do some unit test and some functional/integration tests but by no means do I strive for some arbitrary % of coverage as if that means anything. You can have 100% coverage with completely brainless useless tests that are testing lots of simple low risk code. Any sense of security from that is beside the point if your team is incompetent, the overall design calcifies and becomes a mess and nobody fully understands how it all works.

Targeted testing I guess is how I think of it, very targeted. I work on a small project in a small team, in a much larger project with lots of devs I would have to rethink maybe but then again with properly sized teams (two pizzas) that don't even come into existence.

</comment><comment><USER52> Ok so I read the first sentence and felt it set the tone of the paper. Continued reading and it pretty much played out exactly as I thought it would.

>"Trying to improve software quality by increasing the amount of testing is like try[ing] to lose weight by weighing yourself more often."

So this is really funny to me because I just read an article that in fact states that weighing yourself more often does help people lost weight. [http://www.sciencedaily.com/releases/2015/06/150617134622.ht...](http://www.sciencedaily.com/releases/2015/06/150617134622.htm)

Just by looking at the page and the style of reading you can tell this is someone who has not progressed in their career skills since the early 90's

</comment><comment><USER53> In my first big project after university, a RPC API, I used test driven development and it felt really good. I had about 300 tests in the end and if I changed something, a few would blow up and I could fix it.

But it slowed development down massively.

In the start because I had to set up the testing as I had to setup the real software.

Then I had to mess around with the testing framework as I had with the projects frameworks.

Also I had to write features AND write their tests.

And later in the project, when new features broke old stuff not only the features had to be fixed but also the tests.

I don't know, but I had the feeling that the time I spent with the test-code was the same I would have spent with fixing possible bugs later.

</comment><comment><USER54> In my oppinion maybe it's too harsh to dismiss TDD. It's certainly good in some cases. But if you start using it everywhere everything will look like a nail when all you have is hammer. Clearly not everything is easily unit testable.

What I like about unit testing that it's forcing you to separate irrelevant code and prevents you from writing spagetti code.

In my personal oppinion you should test core functions as someone already said, which validates 90% code with 10% of time. And leave parts which can fail gracefully in case of a bug. It's not practical/impossible to test 100% of use cases of a big codebase. But you can test critical parts.

</comment><comment><USER55> Starting out with a demonstrably false quote...

[http://www.hindawi.com/journals/jobe/2015/763680/](http://www.hindawi.com/journals/jobe/2015/763680/)

<comment><USER56> Ha that is hilarious. I am actually trying to gain some weight and i find i weigh myself too often. I think i can could come up with the study that shows the opposite effect. After all it doesn't have to be true, just significant.

</comment></comment><comment><USER57> > I am against writing unit tests in general

> How often have you seen a program crash? If it was developed by a large > software company, chances are it was written using TDD. Clearly, TDD is not > a magic bullet. So, TDD does not "prove your code works".

> Developing software is like a painting commission

Several reasons I gave up a third of the way in. I rarely write tests first, but I can appreciate that it works for plenty of people - my brain just doesn't work that way.

It's hard to take anything in this article seriously because of the nerdrage and the dismissal of anything he disagrees with as "stupid".

</comment><comment><USER26> This is a bad article, based on a fundamental misunderstanding of TDD, as virtually all the comments on the blog post itself and here on HN attest.

People don't spend weeks writing tests for all the functionality and then write the tests. That's not what even the most die-hard TDD advocates do.

I think many people here (like me) clicked through to read a well-reasoned article about how TDD enthusiasm may have gone too far, but sadly, this isn't that.

HN readers would do well to just move on, and the author would probably be well served by the advice of some of the commenters on his blog to take this post offline.

</comment><comment><USER58> The tone of the article is way over the top, but I 100% agree with this quote: "[E]xperience has shown repeatedly that good designs arise only from evolutionary, exploratory interaction between one (or at most a small handful of) exceptionally able designer(s) and an active user population--and that the first try at a big new idea is always wrong."

I agree so much so that the best systems I've built have always been the 2nd revision of a build one to throw it away prototype.

That said, I like to surround my code with lots of unit tests to eliminate absolutely stupid bugs.

</comment><comment><USER59> There is a way out that he has not found (yet). To design with complete testability as a design goal. TDD with no supporting design can indeed lead to disaster. Hope he can paddle past the rant pond.

</comment><comment><USER60> > "Trying to improve software quality by increasing the amount of testing is > like try[ing] to lose weight by weighing yourself more often."

I realize this is probably here to provoke a reaction, but it's still retarded and it's obviously wrong for the obvious reasons.

> You are writing code to test something that doesn't even exist yet. I am not > rightly able to apprehend the kind of confusion of ideas that could provoke > such a method.

I agree TDD can be taken too far and be enforced too strict (although I've yet to encounter that in practice).

That said: It's not stupid to write tests before the code which you are supposed to test. The order of these things are here for a reason: If you don't write your tests before you have working code, how do you know your tests will detect a failure mode, and thus can be used to prove that your code is now working?

Just the other day I thought I had fixed a bug, and _then_ proceeded to write a unit-test for it. The unit-test went green and I was happy.

But then I decided to comment out my fix and re-run the test. I assumed it probably wasn't needed. I was confident and knew after all that I had "fixed the bug".

But that way I could at least say I had followed the TDD mantra, which claims to be there for a reason: 1. write test to reproduce bug, 2. write fix, 3. rerun test and if green 4. commit.

Upon commenting out my fix and "needlessly" rerunning the tests, lo and behold: The test was still green.

My test failed to detect the error-condition. Which meant that my patch had probably not fixed the reported bug. Quality had not improved.

I rewrote the tests and managed to make them go red, that is, detecting the failure mode. Uncommenting my fix and rebuilding, my test was still red. My fix was indeed invalid!

Another round of investigation showed that I had misinterpreted the error condition and produced a "fix" which didn't solve the real problem.

And the "stupid" principles of TDD helped me detect a invalid fix, find the real issue and verify a real fix for it. TDD helped me increase quality.

Stupid indeed, eh?

</comment><comment><USER61> > "Trying to improve software quality by increasing the amount of testing is > like try[ing] to lose weight by weighing yourself more often."

Giving the benefit of the doubt that this quote is in fact an accurate metaphor – to begin with this quote is to begin with the assumption that TDD's single motivation is improving software quality. TDD is a fantastic practice for ensuring conformity to a specification; which one might describe as software's correctness. A more objective metric than "quality".

</comment><comment><USER62> What the author describes, writing the tests for an entire program before writing any of the program, may actually be stupid. But it is not TDD. According to Wikipedia it's a feature at a time, not a program at a time. I usually see it done one class or function at a time.

If the author had bad results writing tests for an entire program before writing any of the program, I'm not terribly suprised. But since doing so isn't TDD, it doesn't add even a single data point to the TDD discussion.

</comment><comment><USER63> >So this is the first reason TDD fails: You're trying to make a design before you learn anything about it.

Well yes, I use my unit tests to learn what works and what doesn't. You're allowed to rewrite or scrap tests, if they are no longer relevant. In the end I feel that having unit tests result in a better final design.

Yes, sometime unit tests are a little contrived. But they can also help you design cleaner interfaces and increase your code reuse.

</comment><comment><USER64> "I am against writing unit tests in general..."

After this sentence I abandoned the article.

What kind of serious software don't need automated tests? Maybe the kind of no one uses.

</comment><comment><USER65> I tend to agree with this. First, no I am not against writing unit tests, they are great way to trap regressions. But test driven DESIGN? really?

Do you drive your car (software) by banging against rail guards (unit tests) to reach a destination? They are there so that you do drive off the road accidentally not guide your way.

</comment><comment><USER66> I'd love to hear both sides of this story from experienced people (and not the ones who speak loudly one way or another).

We'll probably find experience on both side of "good" and "bad". I'm curious about the "bad" and how that comes about!

</comment><comment><USER67> > _Week 3-4: Write tests._

> _Week 5-10: Write code._

I'm not necessarily a TDD advocate but I've literally _never_ heard anyone say you should spend two solid weeks writing tests and then six solid weeks writing code against those tests.

I mean, that's just idiotic.

</comment><comment> UK-AL I don't recognise most this criticism. TDD is designing up front. But your designing only a little bit up front. You implement the design, if it doesn't fit, you scrap it and try again. That doesn't matter since its only a little bit.

</comment><comment><USER68> _Painting is a science and should be pursued as an inquiry into the laws of nature. Why, then, may not a landscape be considered as a branch of natural philosophy, of which pictures are but experiments?_ \-- John Constable.

</comment><comment><USER47> TDD is about design, not writing tests.

[http://www.drdobbs.com/tdd-is-about-design-not- testing/22921...](http://www.drdobbs.com/tdd-is-about-design-not- testing/229218691)

</comment><comment><USER69> Good observations! At that company, sure, TDD is ridiculous.

But other companies don't do it that way, and require TDD, so the article took a narrow, biased view without considering pros and cons and different scenarios.

</comment><comment><USER70> Let's see author's bio. Only a few years of programming experience - check. Never had a job doing programming outside of academic research - check. Everything checks out, folks!

</comment><comment><USER71> just a sorry rant about how OP tried and failed TDD, and then he points to some project from 2004 that to me seems to be missing code ownership (ppl afraid of changing code they dont fully understand), but to OP seems to be doing TDD wrong.

it's a waste of time to read, as it don't bring anything new to the table. i wish the OP would at least attempt to reflect on what TDD is, what it's not and why oh why, so many devs seems to like it.

</comment><comment><USER72> I hope I never work with the author..

</comment><comment><USER73> > I am against writing unit tests in general, since experience shows that they > actually prevent high-quality code from emerging from development--a better > strategy is to focus on competence and good design.

False dilemma. Are competence and good design exclusively from people that don't TDD?

> The basic idea--amazingly, one of the most popular methods of software > engineering (and growing in popularity!)--is that, after you figure out what > you want to do and how you want to do it, you write up a set of small test > programs--each testing some tiny piece of the program that will exist. Then, > you write the program.

No. You are wrong. It is not that.

> Let me emphasize: you write the test cases for your program, and then you > write your program. You are writing code to test something that doesn't even > exist yet. I am not rightly able to apprehend the kind of confusion of ideas > that could provoke such a method.

You are writing specifications for a program that doesn't exist yet!? Nuts.

> The most important argument is a practical one: Test-First doesn't work.

It doesn't work because it doesn't work. Sad argument.

> I was once one of these newly educated kids. So, for a few years, I worked > exclusively with the Test-First strategy. When it was over, the results were > undeniable. The code--all of it--was horrible. Class projects, research > code, contract programs, indie games, everything I'd written during that > time--it was all slow, hard to read, and so buggy I probably should have > cried. It passed the tests, but not much else.

So it didn't work for you so it is everybody else's problem but yours.

> So this is the first reason TDD fails: You're trying to make a design before > you learn anything about it.

No. You are not. Indeed, depending on how you decide to do your tests you should be able to change your design and the underlying implementation and no test should break. Please, take a look to the outside in approach.

> Week 3-4: Write tests.

> Week 5-10: Write code.

Oh my god. This is terrible. And no, this is not TDD either. This is inverse waterfall :P

> Even if you somehow succeed, TDD prevents incremental drafts by functionally > requiring all tests for a module to pass before you get any real results.

Probably this is a misconception too. You don't write all the tests before. One test, small implementation pass, another test, small implementation pass... You get the results incrementally.

> There is literally no substitute for competence. If your coders don't have > it, TDD won't fix it.

I fully agree on this. And probably this is a really common misconception. But false dilemma again. I don't think TDD came in this world to turn bad programmers into good programmers. It is just another tool in the toolset. You can have terrible programmers that do TDD and excellent programmers that do it.

My recommendation: [https://vimeo.com/68375232](https://vimeo.com/68375232)

</comment><comment><USER74> Up next, on 2003's hottest takes...

</comment><comment><USER75> _> I am against writing unit tests in general, since experience shows that they actually prevent high-quality code from emerging from development_

The essay doesn't have the high quality of content I'd expect from PhD candidate in computer science. I'd expect less outbursts of passion and more discussion of pros/cons/tradeoffs.

First, without getting into "TDD" itself, let's just isolate whether "tests" are valuable. I think the examples of SQLite's rock solid reliability (extensive Tcl regression tests[1]) and NASA's disciplined software verification prove that tests help discover bugs and increase code quality.

Tests are valuable -- but they also have a "cost" to write. Let's cover the cost issue at the end.

When to write the tests. If the sequence is: write the code first and then the tests, you might call them regression tests. If you write the tests first and then the code (e.g. iterations to turn red FAILED into green PASSED), you can call it "TDD". And those TDD tests can still function later as regression tests.

To me, TDD acts as a "design" step or "outline" 10,000 foot view. Before any plumbing code is written, what do you want the REST API or group of functions to "look like". Is there unity and coherence to the collection of fucntions modeling the abstraction? The subsequent of fleshing out TDD with "expects()" and "asserts()" is just mechanical work to glue the edit+compile cycles to a verification target but it's not the most interesting aspect philosophically.

However, even though tests have a benefit, there is a cost. The cost-benefit works in some cases but not others:

In my experience, I'm completely sold on TDD (or regression tests) for foundational library type of code. If you're writing a core string library that 100 developers at the company (or open source community) will link into their projects, I prefer seeing extensive regression tests covering all edge cases that proves that it actually works the way the developer intended. It's not strange at all if the code for regression tests outnumber the actual code 10-to-1.

On the other hand, TDD that is mostly UI verification is extremely brittle. If you have TDD that simluates mouse clicks and has "expects()" on reading webpage UI elements to check if things like sales tax calc is correct, you could easily get overwhelmed by all the extra work that synchronizing the actual code and the TDD scenarios generates. (E.g. a UX designer moves an icon 2 pixels or adds a row to table and ends up breaking the entire TDD validation suite for developers.) I could see where TDD at that level would be counterproductive.

[1][https://www.sqlite.org/testing.html](https://www.sqlite.org/testing.html)

[2][http://www.fastcompany.com/28121/they-write-right- stuff](http://www.fastcompany.com/28121/they-write-right-stuff)

</comment><comment><USER76> The "problem" with TDD is that it can only tell you that your expectations have been met. It can't tell you if you are doing the right thing. Only that the things you guessed at are working in the way you decided they should work.

If, on the other hand, your problem is not very well defined, then your tests have a good chance of eventually becoming a liability. If you ever discover that your domain model, interfaces, or even selection of algorithms are insufficient (or just plain _wrong_ ), it's more likely that your pre-existing test code will be unusable, rather than the sort of guide for refactoring that they're touted to be. If you had guessed at the interface or algorithms correctly, but merely implemented them incorrectly, yes, the tests will guide you back to correctness, but I think that tends to be a big "if". Given that you don't understand the problem, the likelihood that you've made mistakes in the design are high.

Of course, this isn't the fault of TDD, that's exactly what it's meant to be. The problem is in people thinking that is equivalent to verification.

TDD is ultimately a design tool, one that is useful in cases where we have a very good idea of constraints and requirements of a problem, on where the problem is very, very well defined.

And that's why I say it's a code-smell. The problems that are well-defined are often that way because several people have created several implementations of it already. If I'm TDDing something, it usually means I'm rewriting code that already exists somewhere.

Now, I might have good reason to do that. Perhaps I have constraints that nobody else has ever considered. I generally hate the phrase "don't reinvent the wheel". I can think of at least 3 times off the top of my head that the wheel was successfully and usefully reinvented in the 20th century alone. But it's very important that you understand that is what you're doing. If you are aware you're reimplementing a known solution, you can now choose to study your forebears and get an even better understanding of the problem.

Personally, I think saved REPL sessions are better than TDD for problems that are not very well understood or well defined. If I were to try to build an MPEG encoder in JavaScript (for whatever reason), I'd certainly use TDD, because MPEG encoders have specifically well-known inputs and outputs. But if I am trying to invent a completely new UI paradigm for virtual reality, then TDD is just not an applicable tool.

Also, it's a lot easier to tell myself to just discard a saved REPL session that ceases to be valuable after a major rethink in how the problem works than it is to discard tests. Plus, I find them to be a little more informative in terms of demonstrating behavior to other developers than test code.

</comment><comment><USER77>
        If you want to change what a function does, you have to change all the tests you wrote for it.

This guy doesn't know how to write good tests at all. The whole point of testing is that you can change those function without any fear of breaking anything because the tests will prove it works the same way as before.</comment>


Cultivating a Tech Team Culture That Lasts – a Founder's POV - <AUTHOR> https://medium.com/app-partners-how-to-start-your-app-startup/cultivating-a-tech-team-culture-that-lasts-a-founders-pov-f4d1914123c4 <comment><AUTHOR> I'd love to hear what kind of things any founders here are doing for their culture.

My all time favorite example of great culture is Valve the video game company. This is their employee handbook [http://www.valvesoftware.com/company/Valve_Handbook_LowRes.p...](http://www.valvesoftware.com/company/Valve_Handbook_LowRes.pdf)</comment>


I want to build a "market research tool" - <AUTHOR>

Hello guys,
I want to build a tool which does market research for you(such as scrapps potential customer emails, lists up numbers...)
Does such a tool currently exists? What features should such a tool have? When you have a magic wand, how should your perfect market research software look like?
greetings and thx in advance for your replies! <comment><AUTHOR> _push_</comment>


First Private Moon Landing Gears Up for Launch by Year's End - <AUTHOR> http://www.popularmechanics.com/space/moon-mars/a26702/moon-express-lunar-landing-launch-years-end/ <comment><USER2> Gee, this guy is slightly over-selling:

> "It will be the space equivalent of the four-minute mile. I think we're > going to redefine the possible," MoonEx co-founder and CEO Bob Richards > tells Popular Mechanics. "We've seen this throughout history—everybody > thinks something is impossible until they see it done."

I'm skeptical there are many people left in the space industry that think a private moon landing is impossible. It's expensive and difficult. It's also a question of time at this point; as whether it's a company like SpaceX or Blue Origin that does it, it's going to happen relatively soon.

<comment><USER3> I read it as him saying that it's considered too expensive and difficult for a private company at this point. In other words, "impossible".

Likewise, people thought the 4-minute mile was impossible until it was done, even though there were probably some people who knew it would eventually be possible as our diets change and people grow up stronger and healthier.

</comment></comment><comment><USER4> Didn't realize the Rocket Labs Electron had its first launch about a week ago:

[https://www.nasaspaceflight.com/2017/05/rocket-labs- electron...](https://www.nasaspaceflight.com/2017/05/rocket-labs-electron- inaugural-flight-new-zealand/)

"It was a great flight. We had a great first stage burn, stage separation, second stage ignition and fairing separation. We didn't quite reach orbit and we'll be investigating why.." said CEO Peter Beck.

<comment><USER5> It's apparently a fully 3D printed engine with an electric turbopump;

      "Use of this kind of electric motor system produces a 95%
      efficiency as compared to the 50% efficiency achieved
      through standard gas-generator cycle engines."

I can't imagine that statement means what I think it means...?

<comment><USER6> My understanding is that it's just talking about the pumps. 95% of the energy from the battery is turned into rotational energy by a motor in the pumps. This pushes the fuel into the reaction chamber of the rocket engine. Contrast this to a gas-generator cycle that uses some of the fuel/oxidizer to run the pumps. It's not perfect combustion (you don't want it to get too hot) turbines aren't perfectly efficient, etc. so you only get 50% of the energy out of the fuel that you use in the turbo pumps.

However, batteries are heavy. I'm hesitant to believe that electric pumps will scale well to larger rockets, but I would love to be proven wrong.

</comment></comment></comment><comment><USER7> One of my bucket list items is to watch a space launch. Is there a calendar of space launches somewhere that I could plan to go see? Or would I have to research individual launches?

<comment><USER8> This is generally the most detailed and reliable: [https://spaceflightnow.com/launch- schedule/](https://spaceflightnow.com/launch-schedule/)</comment></comment>


A Better System of Economics in light of technology - <AUTHOR> http://ieet.org/index.php/IEET/more/tarifi20110908

<comment><USER2> yawn, another central planner who doesn't grasp basic economics and knows what everybody should be doing. saying that "short term thinking invalidates economics" is like saying that "low gravity invalidates physics" and claiming that we need "a new system of physics on the moon".

it's called a variable for a reason.

what annoys me most is that all of his ideas have been suggested and disproven before, but these kind of people never even do basic research on that, they just assume that their ideas are unique snowflakes that nobody else could ever come up with.

<comment><AUTHOR> Please back up what you say with evidence. I happen to have several close friends with PhDs in economics who support this system.

<comment><USER2> Why would I back my claims up with evidence? Those guys should back their claim up, they're making it! I'm the guy who needs to be convinced here.</comment></comment></comment>


Integrating GTA V into Universe - <AUTHOR> https://openai.com/blog/GTA-V-plus-Universe/ <comment> squeaky-clean I love GTAV so much, it's the most real single player world I've ever experienced. I once rear-ended an NPC because they suddenly stopped in the middle of the street. I got out of my car (because it's GTA and you have to shoot everyone, duh) I saw the NPC had stopped because he hit a cat with his car!

I'd be lying if I said OpenAI Universe / GTAV wasn't one of the primary drivers in me trying to learn ML.

Anyone know what kind of hardware I will need to play with this? The AMI page on github mentions an AWS g2.2xlarge (iirc a GTX 1060 is slightly better than this?). And it seems like GTA is actually running on a different system you use VNC to share the screen/input? Any estimates about whether a 1060 or 1080 on a beefy gaming PC could handle both at once?

<comment><USER2> Hmpth. Conversely, while playing GTAV in completely innocent explore-the-world mode, I accidentally bumped a police boat with my jetski. Apparently this warranted being shot in the head.

Most reviews I've read actually complain that GTAV is a step back in realism from GTA4 (for example, body-part specific injuries were dropped), but let's not beat about the bush - no GTA deserves the title of "realistic", because as you say "it's GTA and you have to shoot everyone, duh". They're made to _feel_ realistic, but in fact they powerfully reward violence and punish being law- abiding, in precisely the opposite way that the real world does.

<comment><USER3> I enjoyed GTA V because it was so hilariously over the top.

It didn't try to be realistic. The whole point is that the police response is always massively disproportionate to the crime.

It's a pastiche of society, everything is exaggerated: they have a legalize medicinal cocaine campaign, you can buy supercars online, rocket launchers and machine guns at your local gun shop, you can launch your car at 100 miles an hour of the edge of the highway and survive.

They even reference the outrageousness of it in a scene where Michael goes to the therapist and comments that he may have even killed someone on the way there.

<comment><USER4> I think it tried to be super realistic in many ways and deserves some credit for it. Its like they made a realistic world simulation, and then tweaked a few parameters so it would be funner to play as a game.

Like obviously no real person would be able to survive the injuries my character gets every 10 seconds. But if I was dying all the time it wouldn't be fun to play.

The one thing that really bothers me, as far as realism goes, is lack of persistence. Leave a car somewhere, walk 10 feet away, turn around, and it vanishes. Cars and NPCs vanish when you aren't looking. Damage you do gets quickly reset as if it never happens. And a number of small details like this. It's not just that it's unrealistic, it really limits what you can do with an otherwise awesome sandbox.

Still, I think the game that deserves most credit for realism is dwarf fortress. It will be a long long time before AIs master that.

<comment><USER5> One of the limitations of GTA that I've found most frustrating is that the game is seemingly programmed to prefer red lights in the streets, I think this is to force you to slow down a bit and let the loading occur more seamlessly, but it's still annoying to just run into traffic jam after traffic jam.

<comment><USER6> You adhere to the traffic regulations in GTA? :)

<comment><USER7> No but the NPC cars do, creating little hard-to-pass blobs of traffic

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Ok, I am _so_ glad that we're not training AI to "win" at GTA V. That is not an android I want to meet on the street :-)

The transitive property of teaching an AI to drive given a world simulation, makes me wonder if you can train a network to recognize speech by feeding War and Peace through a text to speech program and then sending the audio data generated into an RNN. Where is the ground truth in such a system which you can calibrate your results by?

<comment> semi-extrinsic > Where is the ground truth in such a system which you can calibrate your > results by?

Answering standard reading comprehension test questions? - "Why did Pierre meet Natasha at the church?"

</comment></comment><comment><USER9> I'm not sure I would like to live in a world with autonomous vehicles that were trained in Grand Theft Auto.

<comment><USER10> I don't think anyone is thinking of training on GTA then reusing it on a real car. It's just a convenient way to work out the algorithms and architecture in a controlled way that might then work out with the real world data.

But I do think GTA's unrealistic physics engine[1] will hurt it for simulating the cases we care most about (accidents, lost of traction, etc)

[1] GTA is a regular submission on /r/GamePhysics which collects funny game physics glitches: [https://www.reddit.com/r/GamePhysics/search?q=GTA&sort=top&r...](https://www.reddit.com/r/GamePhysics/search?q=GTA&sort=top&restrict_sr=on&t=all) [https://www.reddit.com/r/GamePhysics/search?q=GTAV&sort=top&...](https://www.reddit.com/r/GamePhysics/search?q=GTAV&sort=top&restrict_sr=on&t=all) [https://www.youtube.com/channel/UCy332GE6lcMVP612IUs7ODw/vid...](https://www.youtube.com/channel/UCy332GE6lcMVP612IUs7ODw/videos?sort=p&view=0&flow=grid)

<comment><USER11> exactly, i saw a similar complaint as to the gp's in a thread about working with classical computing to simulate quantum computing

my argument then was 'imagine prototyping on a machine that needs to be kept at 17millikelvin'

here the argument is 'imagine prototyping on a two ton missile traveling at lethal speeds through highly populated areas'

</comment></comment></comment><comment><USER12> This is cool.

Next step is obviously teaching it how to kill people in the game and get away from police.

I'd like to see the AI go on a rampage and win :).

That should be enough to wipe off a lot of smiles off our idealistic faces :)

<comment><USER13> It's all fun and games until the Second Amendment is used to protect the ownership of armed drones that can be used to assassinate anyone anonymously simply by scratching the serial numbers off.

<comment><USER14> yeah that's definitely a scary possibility, though I would think explosives would be the more realistic threat

anyways, electronic actuation of a firearm is already illegal in the US. if you're firing a gun in any way besides actually pulling a trigger with a finger, chances are good it's illegal (with some small leeway such as turning a handcrank)

<comment><USER15> If you're going to murder someone, you're probably not too worried about arms- control laws. You're already planning on doing something _way_ more illegal.

But if you're going for some sort of obscure criminal achievement by only breaking _one_ law at a time, then just make your lethal device something that does not legally qualify as a firearm. A black-powder muzzle-loader or a Bulgarian ricin-umbrella would do the trick, depending on whether you care about noise or not.

Or, like the parent mentions, make a grenade, and deliver it via remote- piloted aircraft.

Realistically, though, very few people on this planet would bother with any of that when they can often just kick down the door of your home, kill you with a tool purchased from a hardware or sporting goods store, and just leave before police can respond. The clearance rate for violent crimes in the US is already low enough, even for people who give no thought whatsoever to covering their tracks.

I'll continue to assume that unmanned aircraft are carrying cameras, and assume that if someone wants to murder me, they need a compelling reason to have a more complicated plan than to just bash my skull in with a big rock when I'm not looking.

</comment></comment></comment></comment><comment><USER16> Who's ready for some "Deep Pimping"?

[http://gta.wikia.com/wiki/Pimping](http://gta.wikia.com/wiki/Pimping)

ABSTRACT: We explore the adage "pimpin' ain't easy" in the context of deep reinforcement learning in the OpenAI environment GTAV. We train a Deep Q-Network (DQN) to optimize player cash flows. Over the course of training, our network learns to keep these hoes in line, and discipline misbehaving customers.

</comment><comment><USER17> This is incredible! Huge fan of GTA V / Rockstar as well as Open.AI

Will be super interesting to see how all of the randomness in GTA V driving influences autonomous driving AIs.

I could see them being a bit more "cautious" in order to deal with all of the crazy drivers (and events) sharing the roads of Los Santos.

<comment><USER18> Thanks! Yes, the game is highly scriptable[1], so you can test the types of scenarios you'd want. We've disabled violence against the driver, but could enable[1] it on-demand if it's useful for training.

[1] [http://www.dev-c.com/nativedb/](http://www.dev-c.com/nativedb/) [2] [http://gtaforums.com/topic/822314-guide-driving- styles/](http://gtaforums.com/topic/822314-guide-driving-styles/)

<comment><USER19> What happens when a car hits a pedestrian at speed? Is it still a bloody mess as in the normal game?

</comment></comment></comment><comment><USER20> I wonder if AI is where games can make use of many cores. If the game logic can not be reasonibly parallelized anymore, the unused computing capacity can be used to add "intelligent" agents into the game. They don't need to do revolutionary things, just being more complex, more "reasonable" and less predictable than the usual agents would be enough. I think that has the potential to make some games more interesting.

Imagine a game like Warcraft 3 where the units actually seem to have a personality and follow "their guts" in battles rather than following simple patterns like attacking the nearest enemy.

Maybe I'm ignorant about the current state of the art and games already do that. Do you know examples?

<comment> squeaky-clean > Imagine a game like Warcraft 3 where the units actually seem to have a > personality and follow "their guts" in battles rather than following simple > patterns like attacking the nearest enemy.

> Maybe I'm ignorant about the current state of the art and games already do > that. Do you know examples?

It's a really tough trade-off between fun and realism. It's cool that your units have personality for a bit "Oh, look, those 2 dwarves are fleeing from the troll fight early!", but not fun once they disobey your intentions "That fucking troll has 3 hit points left. If my 2 dwarves didn't run away, it would be dead! Now it's destroying my base!"

One game I remember striking a good middle ground with this was the original Battle For Middle Earth. It changed animations, sounds and other aesthetic things based on AI, while the game remained the same mechanically. If your units ran up against a mountain troll, you would see the ones in front of it cowering and holding up shields, while units behind attack furiously, but every unit was really doing the same damage. Or if you left troops of Orcs just standing around too long, they'd break out into fights and kill other orcs, but the animation is done as a trick where as one orc model "dies", a new one spawns in the group. So you didn't really ever lose Orcs, even though there would be orc corpses strewn wherever you had been camping.

I think the main reason though is just that AI doesn't really sell. Graphics, physics and voice actors sell. You can blame a failure on AI, but very rarely is success attributed to good AI in a game.

<comment><USER21> >I think the main reason though is just that AI doesn't really sell. Graphics, physics and voice actors sell. You can blame a failure on AI, but very rarely is success attributed to good AI in a game.

The main reason that I avoided buying Civilization 6 is that the AI was notoriously bad. For any kind of turn-based strategy game, I want the AI to be superb.

<comment> shawn-furyan I'll note that this actually makes the original point, since despite your avoidance of the game, Civ VI is estimated to have been the top grossing game on Steam in 2016. So, even in this case, the game has sold gangbusters despite lackluster* AI.

*I didn't intentionally rhyme lackluster with gangbuster. Rather these are the words that sprang initially to mind, and it's a kind of satisfyingly weird combo, so I'm leaving it in.

Source: [https://galyonk.in/steam-sales- in-2016-def2a8ab15f2#.nxpnuko...](https://galyonk.in/steam-sales- in-2016-def2a8ab15f2#.nxpnuko8h)

</comment></comment></comment></comment><comment><USER22> It looks like it's not just using the camera/vnc images as input, but also uses depth data to simulate LIDAR input ([http://deepdrive.io/](http://deepdrive.io/)).

I wonder, what makes it hard to rely on solely the 2 cameras to calculate/guess the depth, similarly to the way us non-robotic drivers do it, when it comes to machine sensing.

EDIT: found this: [https://web.stanford.edu/class/ee368/Project_Autumn_1516/Rep...](https://web.stanford.edu/class/ee368/Project_Autumn_1516/Reports/Appiah_Bandaru.pdf) on the topic. Is there anything else I should read up on?

EDIT: found lots of articles: [https://www.google.com/webhp?sourceid=chrome- instant&ion=1&e...](https://www.google.com/webhp?sourceid=chrome- instant&ion=1&espv=2&ie=UTF-8#q=optical+flow+depth+estimation)

</comment><comment><USER23> Open world games seem like a great place to work with AI. What has traditionally prevented more intelligence, in my mind, was the ability of AI to understand and integrate with its environment. For example, if I say I threw a baseball, you can visualize that. In a virtual world, the AI can "see" and potentially interact with every object to increase understanding. Great experiment and I look forward to seeing how far you can take it!

</comment><comment><USER24> Off-topic, but does anyone know whether it's the GTA engine used to record these videos:

[https://www.youtube.com/watch?v=T5kooJeXGiI](https://www.youtube.com/watch?v=T5kooJeXGiI)

There's a ton of them on YouTube, but I can't figure out how they make them.

<comment><USER25> It's just a normal stock game or some mod added.

Some use the GTA V. director: [http://kotaku.com/on-youtube-millions-watch- shows-for-childr...](http://kotaku.com/on-youtube-millions-watch-shows-for- children-made-enti-1788282618)

The creepy thing is that this video is targeted at 0-3 year olds who randomly tap on mom's iPad. If you think like that now it's clear why the title is just keyword stuffing, the lame music and the spiderman outfit.

80 million views in a month is a lot of money: [https://www.youtube.com/watch?v=ipTJNNvW- Gw](https://www.youtube.com/watch?v=ipTJNNvW-Gw)

Just checked their channel: 4,316,098,805 views

<comment><USER24> "80 million views in a month is a lot of money"

What would 80 million views translate to in dollar terms?

<comment><USER26> I have seen $2,000 / million views, so $160,000 / month?

</comment></comment></comment></comment><comment><USER27> This is super cool. Are there any good resources/how-to modify GTA V for things like this? I'm looking in to GTA main website and wikies but info for developers looks thin.

Here's few links I found so far:

[https://en.wikipedia.org/wiki/Modding_in_Grand_Theft_Auto](https://en.wikipedia.org/wiki/Modding_in_Grand_Theft_Auto)

[http://www.dev-c.com/gtav/scripthookv/](http://www.dev-c.com/gtav/scripthookv/)

[https://www.youtube.com/watch?v=wS8oJTHqf8Q](https://www.youtube.com/watch?v=wS8oJTHqf8Q)

[https://www.reddit.com/r/GrandTheftAutoV_PC/comments/38wct0/...](https://www.reddit.com/r/GrandTheftAutoV_PC/comments/38wct0/guide_to_modding_gta_5_safely_and_effectively/)

[https://steamcommunity.com/sharedfiles/filedetails/?id=55807...](https://steamcommunity.com/sharedfiles/filedetails/?id=558079253)

[http://gtaforums.com/forum/109-tutorials/](http://gtaforums.com/forum/109-tutorials/)

Also I'm reading that GTA folks went out of their way to prevent modding. So may be everything you do with GTA is a hack and there is no supported APIs for developers?

<comment><USER28> Rockstar's policy is that you're free to mod the single-player game, but any modding whatsoever of the online game is forbidden, no matter how miniscule. They detect it and it's a potential ban.

That said, they don't publish anything about modding and don't have an official API available. Everything has been done third-party.

The modding has caused real problems in the online play - player in godmode, handing out in-game money, etc. No matter the efforts of R*, they seem to be unable to eradicate it for good. The group of players I previously played the game with, including myself, have all become extremely disenchanted with the game by the online modders.

<comment><USER29> It ultimately boils down to a cat and mouse game. What you end up with are extremely intrusive anticheat tools that do all sorts of nasty Trojan-like things; for example, injecting DLLs into a player's host machine to determine if they have modified the memory of the game.

The anticheat process itself will hide and run in kernelmode. I don't know about you, but I don't feel terribly comfortable having a closed source binary running in ring0.

<comment><USER28> I'm more comfortable with an anti-cheat blob running in ring0 than I am game- mod code running in any ring.

But, I run on a machine that was purpose-built for running games and if it all goes to hell, I just wipe and reload. All the games are from Steam, so they're easy to get back on there (and I only play a handful of those, so ...).

</comment></comment></comment></comment><comment><USER30> This feels like an important step in the democratization of AI.

We have algorithms, data, and computing power which are unevenly distributed. Providing easy-to-access training data for a self-driving car feels like a huge step in democratizing the data aspect of car AI.

<comment><USER31> With all respect I think "easy-to-access" is a bit of an understatement when you are comparing a 3D game to real-world sensor data. They are barely comparable.

<comment><USER32> If it is highly scriptable, we may see people simulating sensor data within the GTA V environment. Data streams from cameras, lidar, and other types of sensors could be simulated with as much noise as you want. This could be used as input to train an AI using data that looks much closer to what you'd see in the real world.

<comment><USER33> This is way easier said than done. Even with added noise, the sensor data from the real world is nothing like simulation.

</comment></comment></comment></comment><comment><USER34> Probably Uber should practice in GTAV, rather then on real roads:)

</comment><comment><USER35> It would be really great to see OpenAI sponsor AI contests, the # of people contributing / progressing on specific goals would be enormous.

This is really awesome, great work to those involved.

</comment><comment><USER36> So imagine I'd like to start doing cool stuff like this by myself. Where do I start?

</comment><comment><USER37> too bad GTA V is not open-source for researchers. I don't know how customisable it is, but e.g. experiencing snow/ice or a failure like spontaneous pressure loss in one of the tires would be interesting to simulate.

<comment><USER18> Yes, it would be great to have source! The modding community does make up for this in some ways. For example, mods provide access to things like finding memory addresses for things like steering angle that weren't available in the API for game scriptors working on GTAV:

[https://github.com/crosire/scripthookvdotnet/blob/dev_v3/sou...](https://github.com/crosire/scripthookvdotnet/blob/dev_v3/source/scripting/Vehicle.cs#L841-L856)

re snow: It is possible to lay fresh snow anywhere in addition to drive through this area with snow burms, etc..

[https://www.youtube.com/watch?v=rOo2mtuPkKQ&feature=youtu.be](https://www.youtube.com/watch?v=rOo2mtuPkKQ&feature=youtu.be)

Traction is definitely affected in snowy areas.

re tires bursting: it seems possible

[https://github.com/crosire/scripthookvdotnet/blob/2b14a22ad3...](https://github.com/crosire/scripthookvdotnet/blob/2b14a22ad3095e6147289af45c0bf49797a3c39e/source/scripting/Vehicle.cs#L1650)

</comment></comment><comment><USER11> > artificially slowed to 8FPS

i'm sure this is to offer relief to the nn but does this further simulate 'reality'?

are our autonomous vehicles seeing at 8 frames per second?

<comment><USER38> As the average human reaction time is 0.19 seconds, I don't see 8FPS too bad for a start. More FPS is always nice, of course.

<comment><USER39> Reaction times and flicker fusion rates[0] are orthogonal. 0 ms latency at 8 FPS is different than 190 ms latency at 120 FPS. E.g. If there is an event that takes less than 125 ms, at 8 FPS there is a probability that you would miss it entirely! At 190 ms latency with 120 FPS, you would be guaranteed of seeing a 125 ms event, but processing of that information would take 190 ms longer.

[0] [https://en.wikipedia.org/wiki/Flicker_fusion_threshold](https://en.wikipedia.org/wiki/Flicker_fusion_threshold)

</comment></comment></comment><comment><USER40> Just a heads up to anybody who tries to get this running, you may have to request permission from Amazon to get the required instance going (g2.2xlarge). Apparently they are making sure people aren't trying to set up botnets and bitcoin mining. Just sent in my request, we'll see what happens.

</comment><comment><USER41> The video shown is a bit underwhelming; the car moves at 1 mph with pretty much just one car behind it the whole time.

<comment><USER42> How much real-time is needed to get 1k (or 1M) iterations, to improve it so that it moves as fast as the speed limit allows while obeying all the laws?

<comment><USER18> Convergence on target speeds and steering is actually very quick (a matter of hours on a modern GPU / AlexNet / 227x227 RGB) once you have the data. Generating training data with the correct speed limit and providing speed limit based rewards is the missing step. The training car already obeys traffic laws.

<comment><USER42> Cool! You've proved my point :)

</comment></comment></comment></comment><comment><USER43> This would make an interesting plot for a movie. Someday hacks GTAV ai into Google self driving cars. Suddenly "do no evil" company becomes "fuck the world up" and man tries to fight cars.

Anything that has wheels is dangerous to humans!

</comment><comment><USER44> Imagine using this to train and to "unit" test a self driving car. With realistic graphics and the ability to control events, I wonder how useful this could be. (e.g. simulate near accident situations)

</comment><comment><USER45> It looks like it has driving support but I read the post and can I make a pedestrian agent? It doesn't appear that I could. This looks like something I would like to play with and an awesome project.

</comment><comment><USER46> Damn, OpenAI is a house

</comment><comment><USER47> Did anyone try to feed this AI with actual dashcam images and see if the driving learnt in GTA would be applicable in real life situations ?

</comment><comment><USER48> Is it possible to return observations that a well equipped driverless car would receive (e.g. lidar, sonar, multiple cameras, IMU, etc)?

<comment><USER48> It seems like it would almost be easier to hire some of rockstar's game designers and build a realistic self driving car environment from the ground up, given the amount of funding OpenAI has

<comment><USER49> Nah. This is second-system syndrome thinking. Definitely easier to modify the existing huge working system.

<comment><USER48> GTAV is closed source

</comment></comment></comment></comment><comment><USER50> Next Step: Build Street View map of GTA V.

</comment><comment><USER51> I know this is totally unrelated, but i'd love to see a ML framework have a go at NetHack!

</comment><comment><USER40> This is super intriguing! Didn't have GTA V for PC but now I do!

</comment><comment><USER52> I'm not sure how anti cheat methods will work now.

</comment><comment><USER53> looks like the blog post got deleted? why???? Sad face :(

</comment><comment><USER54> This is actually pretty exciting!!!!

</comment><comment><USER55> I live in the future.

</comment><comment><USER56> What could go wrong

</comment><comment><USER57> now the interesting part: put it in front of ethical problem (take one from [http://moralmachine.mit.edu/](http://moralmachine.mit.edu/)) and lets see what it does

</comment><comment><USER58> I'm probably one of the most critical people of the new AI-safety craze, but even I wonder if training AI on games like GTA is a good thing to do... if it is something I'd be hesitant to let my kids play, should we really be training our mechanical offspring on it?

<comment><USER59> It is not playing the game, it is using the game as a driving simulator.

<comment><USER58> It's using the game as a driving simulator by playing it.</comment></comment></comment>