The Code Documentation Fallacy - <AUTHOR> http://voices.canonical.com/jussi.pakkanen/2012/09/25/the-code-documentation-fallacy/ <comment><USER2> I think the article hits on the wrong conclusion. Don't write less documentation because you're going to assume it will all suck anyway. Insist on writing better documentation instead.

Having said that, I find it helpful to write documentation before writing the actual code. Specifically for more complex code pieces for which the behaviour is not immediately obvious.

For me, writing documentation serves as a form of 'rubber duck debugging'[1] before the actual bugs occur. Explicitly writing out the intention of a piece of code in plain English often makes the concept much clearer in my brain and immediately brings out possible problems with my initial design. Problems I can fix before wasting time iterating through code implementations.

This is also the reason I very much enjoy writing thorough READMEs for each library I produce. These explain in abstract concepts what the entire library API is intended to accomplish. Additionally, I try to include actual usage examples. As with code-level documentation, this brings up possible problems before they occur.

The fact that it makes it clear what the code does, months after I last worked on it, is entirely bonus.

[1]: [http://en.wikipedia.org/wiki/Rubber_duck_debugging](http://en.wikipedia.org/wiki/Rubber_duck_debugging)

<comment><USER3> The best kind of documentation is that which is checked by the compiler and guaranteed to be correct - the code itself. Code that is well written, in good style using sensible variable names, can be as descriptive as good comments. Using a good static type system allows you to encode properties into your code that are guaranteed to be valid.

I certainly think some comments have their uses - but these are generally at the level of how systems and modules work, and the concepts used therein, as discussed elsewhere in these comments. I agree with the article that only 5% (or less) of functions need individual comments attached.

<comment><USER4> I've been maintaining some code written with this philosophy, and I find it lacking.

Code is very good at answering "How" but often the reader needs to know "Why" or "Why not".

In fact, the maintainer of your code will rarely be reading it to figure out how it is working - almost always the next person looking at your code will want to know why it is not working, or will be attempting to change the behavior.

Comments can guide as to pitfalls that you've avoided in your implementation and can answer the all-too-frequent question, "What were they thinking?!"

<comment><USER5> Another problem: It seems like developers in the "clean code needs no comments" crowd are also the ones least likely to write clean code in the first place.

<comment><USER6> Clean code needs no internal comments, but that's very different from whether software contracts (which are inherently outside the code operation and therefore need comments) need comments.

I strongly dislike extraneous comments. But even some clear code needs some external communication alongside it.

</comment></comment></comment></comment></comment><comment><USER7> My problem with code documentation is that documentation is done at the function or class level. When I'm looking at new code I would prefer a "concept of operations" describing how the whole thing works together rather than piecemeal function documentation.

This is especially important with open source code. I'm not going to donate my time to working with an existing codebase if it is going to take hours to figure out how it all pieces together. Examples are fine but what I really need to know is the why. At least when I put up with this at work I'm getting paid by the hour.

<comment><USER8> The old classic literate programming paradigm helps with this. It allows one to write documentation that gives you an overview, provides you with whatever ordering and connections you feel appropriate and, with my implementation of it, even takes care of most of the tool chain. [https://npmjs.org/package/literate- programming](https://npmjs.org/package/literate-programming)

<comment><USER9> this brings a smile to Knuth's face

</comment></comment></comment><comment><USER10> _> When some coder changes the function, it is very easy to forget to update the comments_

It isn't.

All public APIs should have documentation, even if you believe it's obvious what they do. This documentation never goes out of date because once you release your API it tells you what you cannot change. If you changed the code so that your documentation is now wrong - this code change is a bug and you should fix it. Because _there 's other code in the wild that relies on the behavior that you promised_.

Of course in practice you do have to change that behavior every once in a while. But this should be a big deal (that usually includes bumping up version numbers, mentioning it in release notes, etc). If you're changing it so often that updating the damn comment is an issue you either document implementation details that don't belong in API documentation or your API is unstable crap and nobody should be using it.

PS. Complaining that API documentation gets out of sync with the code is like complaining that unit tests break when you change the code. Duh - that's what they are there for!

<comment><USER11> I don't disagree that public APIs should be well documented. However, I believe it's very easy to forget to update the comments. Especially in scripting languages without type safety. There is little to remind the programmer that arguments were added, types were changed, argument order was modified, behavior was changed, return value was changed, locking behavior was changed, memory allocations were changed, etc. Obviously unit tests should help catch many of these changes. However, it's still up to the coder to remember to change the comments. When you're behind on a deadline or you're juggling 50 different API changes because you're still alpha I'd say it's pretty easy to forget to update a comment.

See [http://api.jquery.com/jQuery.ajax/](http://api.jquery.com/jQuery.ajax/) if you want an example of on API that could easily have a few typos in it. Who is checking to make sure it's 100% in sync with the code 100% of the time? Not trying to say this example has bugs in the docs but there's a LOT of behavior described that could be out of date.

<comment><USER12> If you're forgetting to update the comments frequently enough to be a problem, you're probably forgetting to update the code that consumes the API too. That quality in general falls by the wayside when you're crunching is no reason to abandon quality altogether.

<comment><USER13> Forgetting to change the code that consumes the API means that things don't work, which is visible right away. Forgetting to change the comments will be visible gradually and cruft accumulates. Also, sometimes comments wind up being non-local and just aren't noticed. I agree that in some senses it's "no excuse" but that doesn't mean it won't happen. Documentation that can break visibly when things change is better than static documentation.

<comment><USER12> > Forgetting to change the code that consumes the API means that things don't > work, which is visible right away.

Oh how do I wish this were the case!

> Documentation that can break visibly when things change is better than > static documentation.

Agreed, but I would not call your average run-of-the-mill unit tests "documentation", nor can all documentation can be programatically tested.

<comment><USER13> > > Forgetting to change the code that consumes the API means that things > don't work, which is visible right away.

> Oh how do I wish this were the case!

It's certainly not always the case, but it's a whole lot more likely to be the case than for unchecked documentation.

> > Documentation that can break visibly when things change is better than > static documentation.

> Agreed, but I would not call your average run-of-the-mill unit tests > "documentation",

I think "is it documentation" is probably more of a spectrum than any particular threshold, and run-of-the-mill unit tests probably do fall on this spectrum though I'd probably agree that they're not particularly far along it (though that surely varies with the habits of those writing the tests).

> nor can all documentation can be programatically tested.

As a practical matter, that's certainly currently the case - tooling is not set up for testing documentation, and there are things we'd want to check that would be hard to check in any event. Theoretically also, there are certainly properties that can't be statically demonstrated. I'm not entirely convinced that there's nothing _we 're interested in_ that couldn't eventually be got at _for the programs we care about_ , though it's certainly a possibility. Regardless, it seems an ideal worth pushing towards, and if tested documentation is interwoven with untestable documentation such that some conceptual locality is preserved it's less likely (though absolutely still possible, to be sure) that you'll forget to update the other when you're forced to update the one.

</comment></comment></comment></comment></comment></comment><comment><USER6> First, I think more documentation always beats less documentation, assuming reasonable quality. What the article is getting at is the idea that documentation for the sake of documentation never results in any quality.

When I write a new module for LedgerSMB (I won't vouch for older code either by myself or others) I actually start writing the documentation. The reason here is that the documentation is written primarily to establish the contracts under which the code operates. This includes concept of operation documentation as well. It isn't just aimed at other programmers. It is aimed at documenting the code contracts so that it is clear what are acceptable operations at first.

So if there is a fallacy it is not that more code documentation is better (since that is often true, IMO), but rather that telling people to document for the sake of building documentation works.

<comment><USER14> "What the article is getting at is the idea that documentation for the sake of documentation never results in any quality."

X for its own sake is rarely good.

</comment></comment><comment><USER15> One thing to bear in mind is that documentation serves two different audiences.

One audience is the people who will _use_ your code. For them, every external method should be properly documented (sure, use a tool for this). And make sure it's good enough that (barring debugging situations, because you wrote perfect code) your "users" never have to look inside your code. (If I think about this in C++, I think you should be able to look at a properly commented header and never read the code)

And then there's the poor slob who's going to come in and debug/fix your code some day. He may not need to have every method doc'd but he damn sure needs to know what's tricky, what's interesting, where the gotchas are, etc.

</comment><comment><USER16> I've done plenty of dynamic language work where some documentation would go a long way. In Javascript, every library function should explain what all optional (i.e. undeclared or reinterpreted) parameters do, what gets returned, and likewise for all callbacks, plus `this` points to. It's very stylish for some insane reason not to do this, and it drives me insane.

<comment><USER17> Don't get me started on Javascript. It's a problem with dynamic languages in general - the signature does not document what interface the parameters should implement (eg, a parameter 'file' may be a file path or a file handle, but you'll only know that by looking at the implementation), but it's particularly egregious when nothing tells you at first glance which parameters are optional.

I've now started to document public functions. As my classes usually have few public functions, but many private ones, the code to comment ratio remains acceptable, though maintaining the documentation remains a challenge.

<comment><USER18> Agreed completely about how awful dynamic languages can be in this respect. I've found what helps a lot is consistent naming of parameters. For example, numberOfWidgets (an integer) versus widgets (a collection) or file (a file handle) versus path (a string).

<comment><USER17> Sure, consistent naming helps a lot (though I'm partial to short-yet-readable alternative like widgetCounts). Sadly, sometimes ambiguity is unavoidable. And other times, you're just dealing with bad code.

But consistent naming does not help with the "optionality" of parameters.

</comment></comment></comment></comment><comment><USER19> Documentation does not exist for your own benefit. It exists to help other people on the team, or the programmer that inherits your code, quickly understand your code. The key word here is quickly. Yes, a programmer can trace code & figure it out. On a large chunk of code, however, that is exceedingly inefficient. Whether or not documentation is obvious to the person that wrote it is a very poor test for the documentation's utility.

I used to hold a similar opinion -- "Document the non-obvious". The problem is that in a project of sufficient size, almost everything can slide towards non- obvious. Is price the base price, or unit price * quantity? Is the method name cancel_subscription_and_notify_customer really effective? Of course, I could use cancel_subscription, but then I'm not telling programmers about the email that goes out, or cancel_subscription_and_notify, but who am I notifying? The marketing department? Generally I find really descriptive method names to get unwieldy very fast. Further, if you say document the non-obvious, the tendency is towards zero documentation.

The value statement depends on how fast your team grows or changes, and the expected lifetime of the project.

If you are working on a project alone, and that will never change (e.g. it isn't something a business relies on), then you probably do not need documentation. This is also true if you are bringing on a dev a year, and the team size will always remain relatively small. Similarly, if the project is relatively short-lived (like a game), then dropping documentation could be a good idea. Maybe, I'd at least concede there are merits to doing so. Documentation isn't free, of course.

On the other hand, if you are working at a company that's trying to rapidly grow, needs to bring on devs quickly, or has developers moving from one project to another frequently, then I'd say documentation is very important. You are going to save your team a huge amount of time by taking a little time upfront to explain what you are doing, why, and the consequences of each method. Even simple methods deserve documentation for consistency sake.

If your documentation rots, then you handle it the same way as test rot. Make sure the team knows that docs are necessary, they need to spend the time on it, and if that means more time for features, so be it. I can say from experience that writing documentation after the fact is pretty gnarly.

</comment><comment><USER20> The less that is described by the signature, the more necessary it is to document a function. In C++/C#/Java, while the parameters and return values are often fairly obvious, exceptions are completely undocumented by the type system (unless you use Java checked exceptions, which you shouldn't), and so should be documented manually (and asserted in unit tests, to ensure the documentation doesn't become incorrect).

</comment><comment><USER21> This is an ever ongoing discussion between two kinds of programmers (documenting and non-documenting). Just decide for yourself or your team what works best. I myself like to even document other people's code after I see what the method does if the method isn't speaking for itself. If I am developing I want to see in my IDE when adding an existing method, a popup which tells me quickly about what the method does and the arguments it has and what it returns. Instead of (everytime) having to jump to the code to see what it does.

Just one simple (real life) example:

      public String convertText(String text) {
       return text.toUpperCase();
      }

This method is already named wrong in my opinion and should be refactored to something like convertTextToUpperCase to understand what it does without having to document. But if your methods get more complex I think a little comment on top of the method describing what's going on really cannot harm. Especially if the code is difficult to read for new people.

The point is in the end to keep the documentation in sync with the code and that takes indeed some effort. I myself always make documentation for a method in Java-doc style, so only above the method, if it's more complex than a simple getter/setter-method. I always tend to think in terms of the official Sun Java API-documentation, which I use(d) so often to know how all the classes/methods work, that it might also make my own code more readable/understandable when I or someone else has to work on my code if I have documented it. Inside the method code I try to comment little to not.

@snowwolf: I agree, but it's just an example to show that a method name should speak for itself

<comment><USER22> Just to comment on your example, that method shouldn't exist as it adds no value to the codebase - it is purely redundant code. Especially if you were to rename it to convertTextToUpperCase.

The only situation where the method would make sense is if you wanted to be able to change the implementation in the future (TitleCase, LowerCase etc.), in which case a better renaming would be covertTextForDisplayInTitles (i.e. Use the method name to comment why we need to convert the text). That has the added benefit of also telling you what the method does in your IDE just from its signature.

<comment><USER23> I was thinking about this recently, and wondered how other people think of this. Working in a high level language like Python, you can do quite a lot in one line, for example in a list comprehension.

Should a "productive" one liner be put in a separate method / function with meaningful name, or better a comment beside it?

I personally find short methods often decrease code readability, as you constantly have to jump around, and can't read anything from top to bottom.

<comment><USER24> My approach is frequently to "name" a complicated list comprehension by shoving it into a one or two-liner closure function in the current scope. Then you don't have to jump miles and the code which uses it becomes readable.

<comment><USER23> So what is the advantage of this approach over having a comment beside it? Sounds slightly more complex to me.

<comment><USER24> Because comments can lie.

Also, it can be harder to express the generator in a useful way in English than the code itself if the code is well written.

The purpose is to make the code as clearly self describing as possible. I would only do it if it made the code read more like a human language and remove too much complexity from one place.

</comment></comment></comment></comment></comment></comment><comment><USER25> Probably all of us agree that being forced to comment everything will lead to some bad/useless comments. But the examples you showed were simply bad comments. Just because there are bad coders out there who don't care about the quality of their comments for whatever reason, isn't a reason to avoid comments. I'd treat anyone who wrote that first sample comment in a similar way I'd treat a programmer who writes unreadable code; that is, probably take the time to teach them some good commenting practices.

Many of the reasons that speak against commenting apply to good variable names, too. Maybe someone will come later and change the way the variable is used but won't change the name. It doesn't mean we should avoid descriptive variable names, though.

Also, with comments, as with any form of communication, the audience is the key. Let's say I'm a senior programmer somewhere and I'm writing comments. Often the train of thought seems to be "well, using this variable name/adding this comment clears it up for me". But that's usually not nearly enough for junior coders who are new to the codebase, and who are often the target audience. They'll probably still go "wtf" after reading a comment aimed at a senior programmer with an understanding of the codebase and a programming experience to match.

In addition, the obvious point to make is also that code is good at answering how, not why.

This is a bit of a pet peeve of mine, I guess since I've met relatively many coders who claim that good code should comment itself and ditched commenting altogether. Their code has usually ranged from above average to downright awful, and has, on average, been rather unreadable.

</comment><comment><USER26> There's a principal/agent problem with documentation too. The writer of it rarely gets the benefit. Many times they will never see or meet anyone who does. But they have a lot of other competing priorities from highly visible requesters.

I am interested in examples of companies that get this right.

<comment><USER27> I agree with this assessment. Whenever I have the opportunity to get feedback from someone who has used my documentation, I make every effort to get it, and then update the documentation. It's gratifying to know that someone benefited from it.

</comment></comment><comment><USER28> "As a rough estimate, 95% of functions in any code base should be so simple and specific that their signature is all you need to use them."

Welcome to reality!

What is simple for one guy is really hard to understand for an another one.

What you expect is that every bigger function must be split into dozends of smaller functions, only to have a cleaner parameter part. But this make the code flow unreadable.

<comment><USER29> > What you expect is that every bigger function must be split into dozends of > smaller functions, only to have a cleaner parameter part. But this make the > code flow unreadable.

That's exactly what every good, experienced developer tries to do: Splitting complex stuff into more simpler functions that are easier to understand. Obviously the end result is not unreadable, on the contrary!

</comment></comment><comment><USER30> I'm surprised nobody has mentioned Master Foo and the Programming Prodigy. [http://catb.org/esr/writings/unix- koans/prodigy.html](http://catb.org/esr/writings/unix-koans/prodigy.html)

</comment><comment><USER31> In many languages, the 90% comments from this article are "doc strings", while the 10% comments are actually comments. They have different uses, and should be treated accordingly.

I also don't understand the fear that doc strings will go out of date. At least with dynamic languages that's part of the point: if the doc string is wrong, then either the contract has changed and not been updated or the code is fulfilling the wrong contract. Both are useful things to know.

More often, the doc string is correct, and serves both as a guide to the code "here is what you are about to read" and as a quick summary. if you're trying to decide, say, between iterate-dirs and walk-dirs, that summary is perfect, while reading the code would be an annoying digression.

</comment><comment><USER32> The "turing test" of comments.... If I can distinguish whether the comment was written by a human or generated by some auto-documenting software then the comment may be useful; if I cannot tell whether it was written by a human or auto-generated then it is useless.

</comment><comment><USER33> ...and for those who like concise quotes:

`If the code and the comments disagree, then both are probably wrong.' \-- Norm Schryer

`Don't get suckered in by the comments -- they can be terribly misleading. Debug only code.' \-- Dave Storer

</comment><comment><USER34> My rules for good docs:

1\. Begin with a usage synopsis in the form of sample code. It should hit the most important functions and show how they fit together. E.g. for a drawing library, show how to instantiate an image, draw a circle with arbitrary fill and stroke colors, and write the image to disk.

2\. For each function or method, open your comment with a straightforward description of what the function does, even if it's _absolutely, undeniably obvious._ If you're writing an math library, you should even say what the sqrt function does. It doesn't hurt, it costs you very little effort, and you might help someone who's just beginning to learn about the problem domain.

3\. For each parameter, document its possible types if your language doesn't encode that information in the function signature. Even if you're using something like Haskell where it does, you might need to comment on the type. E.g. if sin takes a float, say whether it's in degrees or radians.

4\. Provide sample code for functions that have to be used in tricky ways. E.g. if the function requires special setup or context.

</comment><comment><USER35> Ok, the problem is not the documentation. It's the specifications. This problem is exacerbed by management methods such as Agile/Scrum, where no specification document is built from the collection of task descriptions stored in Jira (and IF you are luck to have anything significant in the task descriptions, more often, from what I've observed, it's hand waving and shin pointing than anything precise written down).

And even if some specification document is written, it still remains the problem because when going thru all the phases of analysis design coding and debugging (whatever the period of the cycle you use), it is not updated!

Now we should probably distinguish API elements from internal implementation stuff (but the blog article mentions APIs).

When documenting internal stuff, unless you've developped internal APIs (which you should do!), the documentation can indeed be descriptive, to help maintainers orient themselves and avoid pitfals.

When documenting API, what you need mainly, is the specifications of the API. This will be the "contract" with the client code, and if there's a discrepancy between the API specification and the implementation, then it means there's a bug (somewhere, of course one could decide that the specifications where wrong, and need to update the specifications instead of the code). Most often it will be a bug in the code.

But the point is that either you have tools to track the specifications elements down to the line of code, so that when you create or modify a line of code, you have easy access to the specifications, or you put the specification in the docstrings (documentation comments) in the code, to get the same easy access. And note that this is a read/write access: specifications may need to be updated when the code is maintained.

So I would agree, write less documentation, write more specifications. Close to the code.

<comment><USER27> I tried a google search for "shin pointing" and only got pictures of people doing literally this.

I got it from context, obviously, but can you explain this turn of phrase to satisfy my curiosity?

<comment><USER35> i meant it quite literally. You know, in real life meetings, with body language, odors, hand waving and all that non verbal communication. If everybody knows what we're taliking about, then what's the point of writing it down? Yeah, right. Come back six month later and with half the team turned over, and see how useful thos old Jira tasks with no writte specifications will be useful...

<comment><USER27> Oh, I see. I've always known "hand waving" to be a Star Wars reference. i.e. "These are not the droids you're looking for."

</comment></comment></comment></comment><comment><USER36> So basically the author is making the argument that comment quality is correlated with quantity.

As far as I can see, this is backed exclusively with "but the ones that do probably...".

Let's just say I 'm not convinced yet.

<comment><USER37> I think the argument being made is that comment quality is more important than quantity. The author gave an example where the code with less documentation had more useful descriptions in order to illustrate this point; I don't think he necessarily meant that one caused the other.

<comment><USER36> That's something we can all agree on, but then why bring quantity into the discussion? I prefer lots of useful comments over few useful comments.

<comment><USER38> To paraphrase The Incredibles, if everyone is special, than no one is.

If 10% of the functions are commented, I will assume that those 10% are more important or have more error prone or dangerous usages. If every function has a boilerplate comment, I lose that information.

<comment><USER39> You can catch garbage comments in code reviews just as easily as you can catch other forms of programming garbage during review. The fact that someone might make a lot of bad comments is not a good argument against commenting, it's a good argument against bad commenting.

I also think that if people are in the habit of having to comment it makes them more likely to document things like expected values of the input, what the return could be expected to be, and if there are any caveats.

</comment></comment></comment></comment></comment><comment><USER40> At its best, documentation (or a subset thereof) can and does serve as a cross-reference -- like a cross-reference in an important calculation. If the two results don't correspond, you know you have a problem. (Even if you have to explore _both_ paths to learn what the problem really is and where it lies.)

Something to consider, the next time you find yourself inclined to complain about documentation. Is it the documentation, or the fact that it's not _useful_ documentation?

</comment><comment><USER41> I personally favour only commenting on the unusual. I also think that if you are doing TDD then the tests should do a good job of documenting the code. I can't count the number of times someone has asked me to explain some code I have written where my first response is "okay, let's look at the tests".

All that being said, it really depends on the scenario. My second last project was a shrink wrap product development that ran a million dollar plus accompanying piece of hardware. This project quite rightly required us to write thorough and consistent comments and docs. Expensive product, complex code, high risk = thorough docs. I am currently working for a small rapidly growing and changing business, where the software is mostly internal. Low risk, quick development required, constant change, low complexity = waste of time and money creating and maintaining good docs

</comment><comment><USER42> I don't get it, having a few comments scattered about in the source code is better than having a separate document that fully and succinctly explains the application API?

And what's to stop you from putting the comment about race conditions in the Doxygen-style comment?

<comment><USER43> Single-to-noise ratio.

You can put it in there, but I probably won't read it. I basically assume that all javadoc/doxygen-style summary/@param/@return-style docs are entirely noise.

I much prefer the "docstring" style comments, especially though in Clojure, or Python.

Examples...

Clojure:

        user=> (doc map)
        </comment></comment><comment></comment><comment></comment><comment></comment><comment>-
        clojure.core/map
        ([f coll] [f c1 c2] [f c1 c2 c3] [f c1 c2 c3 & colls])
          Returns a lazy sequence consisting of the result of applying f to the
          set of first items of each coll, followed by applying f to the set
          of second items in each coll, until any one of the colls is
          exhausted.  Any remaining items in other colls are ignored. Function
          f should accept number-of-colls arguments.

Python:

        Help on built-in function map in module __builtin__:

        map(...)
            map(function, sequence[, sequence, ...]) -> list

            Return a list of the results of applying the function to the items of
            the argument sequence(s).  If more than one sequence is given, the
            function is called with an argument list consisting of the corresponding
            item of each sequence, substituting None for missing values when not all
            sequences have the same length.  If the function is None, return a list of
            the items of the sequence (or a list of tuples if more than one sequence).

And on the far other end of the spectrum, here's some C# / MSDN docs:

        Syntax

          public static IEnumerable<TResult> Select<TSource, TResult>(
              this IEnumerable<TSource> source,
              Func<TSource, TResult> selector
          )

        Type Parameters

          TSource
            The type of the elements of source.

          TResult
            The type of the value returned by selector.

        Parameters

          source
            Type: System.Collections.Generic.IEnumerable<TSource>
            A sequence of values to invoke a transform function on.

          selector
            Type: System.Func<TSource, TResult>
            A transform function to apply to each element.

        Return Value
          Type: System.Collections.Generic.IEnumerable<TResult>
          An IEnumerable<T> whose elements are the result of invoking the transform
          function on each element of source.

        Usage Note
          In Visual Basic and C#, you can call this method as an instance method on any
          object of type IEnumerable<TSource>. When you use instance method syntax to
          call this method, omit the first parameter. For more information, see
          Extension Methods (Visual Basic) or Extension Methods (C# Programming Guide).

Entertainingly, this is only _one_ of many overloads in the C# version. The Clojure and Python functions are variadic with parallel traversal of collections. Those languages spend their precious docs space covering the corner cases.

<comment><USER44> What about this?

List.map : ('a -> 'b) -> 'a list -> 'b list

In languages like OCaml, or Haskell the signature provides a lot. In this particular case, if you think about it, you conclude that it's almost impossible to build any other implementation: You get a function from 'a to 'b and a list of 'a thingies. Now how on earth do I use these to get to a list of 'b thingies? No documentation needed IMNSHO ;)

<comment><USER45> Apply reverse, tail, etc. on the result of 'map' and you get something of the same type. Types don't tell all.

<comment><USER46> What if you use a vector whole length is indexed by a Natural in the type signature? It would still allow permutations and such (like reverse), but not tail.

Using a dependent type system, it would actually be possible to statically ensure that map can only map, and do nothing else whatsoever, but it does become rather unwieldy.

</comment></comment></comment></comment><comment><USER47> I personally think that leaving a function-level comment snippet empty is better than either filling it with useless information, or not having it at all. It serves as visual separator, and it's ready for me to fill when I have something to write.

</comment><comment><USER48> So true! This is what happens when comments are produced because some coverage goal.

It's sad how comments here are being (on purpose?) misrepresenting the point. Another methodology jihad?

Edit: LOL! inmediate downvote. I have my answer now.

</comment><comment><USER49> As an amateur coder I started commenting in my code only recently. There are two main reasons for this, to enable others to read my code more easily, and secondly to enable myself to read my code more easily.

printf("If you use C macros"); int _i=1729; while(_i--) printf(" that are defined in terms of other C macros"); printf(", you will be in a world of hurt " "unless you have good documentation written by a human.\n");

</comment><comment><USER50> TL;DR: The people may be stupid, or may do a messy work, therefore it must be the methods they use or the requirements that are faulty in the first place.

<comment><USER51> Better TL;DR: Would you rather have 10% of functions commented or 90%? Author argues that 10% is better, if all functions are documented then documentation is obvious, noise, and often wrong. Document the non-obvious cases.

<comment><USER50> Would you rather go to one extreme or to the other one?

Please downvote me again.

</comment></comment></comment><comment><USER52> My emacs+doxymacs doing it semi-automatic - I dont know how, but hope these tags @param will show flyhelp at mouse pointer when moving over params. Eclipse doing that but not emacs currently. Its all that I need. Often forgot parameters description and orders.

But agree often documentation older than code. Its very fast older and add many noise.

</comment><comment><USER53> Here is a box.

You can choose to have it filled 90% of the way or 10% of the way. Most would choose 90%, right? But guess what—

THE BOX IS FILLED WITH BEES!

<comment><USER13> If I'm a beekeeper, that might be what I want...

</comment></comment><comment><USER54> I think the effectiveness of guessing what the function does by its parameters' types and its return type depends on whether or not the function performs side effects. That said, if it doesn't, or can't (even better), then the name and types should be documentation enough.

</comment><comment><USER55> OP is a bit extreme in his conclusions, but not entirely incorrect either.

Insofar as I've experienced, there's a mostly finite amount of goodwill that will go into commenting. Extra comments once that limit is reached tend to be sloppy or useless.

Essentially the same for unit tests.

</comment><comment><USER14> i think this is still a half way house.

code is documentation. bad source code is hard to read, or in practical terms - slow to read. good source code has minimal comments and documentation.

writing documentation before you actually need it is often a waste of time as well. at this point surely you have a design to follow, or something other than a brainless application of 'documentation' to store that valuable information into?

as many commenters already point out what you almost always want is very high level documentation, and I think this should be present as a design (assuming you have one), if you don't have one then you need a wiki or something. Something that doesn't pollute your code with garbage...

</comment><comment><USER56> Saw a group pitch a product that was like RapGenius for code documentation, don't think it went anywhere? Both writers and users could add their commentary in a layer that was separate from code, seemed like a clever way to address documentation?

</comment><comment><USER57> Personally, I would rather a useless comment (which I will ignore if the endpoint signature is self explanatory) than no comment at all.</comment>


Telepresence Robot Crosses Atlantic Five Times Trying to Get Home - <AUTHOR> https://www.fedex.com/apps/fedextrack/?tracknumbers=806020716375&cntry_code=us

<comment><USER2> Is there anywhere we can read about the actual robot or the project?</comment>


Removing Array Duplicates - <AUTHOR> https://flak.tedunangst.com/post/removing-array-duplicates <comment><USER2> Usually a fine approach would be to construct a hash table as we walk the array, and if the element is not seen put it in the hash table, otherwise "remove" it by setting it to the empty string. If the hash table implementation is good, we can pretty much find out whether an entry has been seen before in O(1) time, making the whole process O(n).

That's what the author called "alt" and I don't understand why the author objects to it. The memory needed is usually proportional to the number of distinct elements in the original array. You don't generally have to copy the original values (usually you just copy a pointer).

<comment><USER3> Although the examples are all about strings, there's no reason to be so specific. One nice property of the algorithms described in the article is that they only require the elements to be comparable (even better, the first two implementations only require them to have an equivalence relation), while yours requires that they be hashable.

<comment><USER4> Is there an example of a type that has an equivalence operation but cannot have a hash operation? Seems to me that those two are identical. If you have a subset of elements within a type that defines identity, then you can hash that identity.

<comment><USER3> Yes, in a garbage-collected language a ref cell is typically unhashable since the only identifying data about it is the pointer location, which could change between calls to the hash function. However, you can still define equality pretty easily (just compare the pointer values). This is also my best example of something with an equivalence relation that is not comparable.

I don't have a ready example of something that is comparable but not hashable, but it isn't uncommon in my experience to come across an opaque type whose author simply didn't expose enough information to hash it, while they did expose a compare function.

</comment></comment></comment></comment><comment><USER5> The author uses Go in this example, and they're really just struggling against the language. This is a pretty good example of why choice of language matters: in C++ this could be a simple, linear pass through the array with an std::unordered_set storing visited elements, with an optional std::remove_if/std::erase at the end if they wanted to actually get rid of the elements. All done in linear time, space overhead, and likely with shorter and faster code.

<comment><USER6> Yup, you can implement the whole ordeal in C++ like this:

      int main(int argc, char** argv) {
        std::vector<std::string> strings{
          "apple", "cAt", "cat", "Dog", "apple", "dog", "Cat",
          "Apple", "dOg", "banana", "cat", "dog", "apple",
        };

        strings.erase(
          std::remove_if(
            std::begin(strings),
            std::end(strings),
            [seen = std::unordered_set<std::string>{}](std::string_view str) mutable {
              std::string lower = "";
              std::transform(std::begin(str), std::end(str), std::back_inserter(lower), ::tolower);
              if (seen.find(lower) == std::end(seen)) {
                seen.insert(lower);
                return false;
              }
              return true;
            }
          ),
          std::end(strings)
        );

        for (auto& str : strings) {
          std::cout << str << "\n";
        };
      }

<comment><USER7> Yeah, this is definitely the downside to Go. You can write more or less the same code as your C++ in Rust like this:

        use std::collections::HashSet;

        fn main () {
            let strings = vec![
              "apple", "cAt", "cat", "Dog", "apple", "dog", "Cat",
              "Apple", "dOg", "banana", "cat", "dog", "apple",
            ];

            let mut set : HashSet<&'static str> = HashSet::new();

            let strings : Vec<&str> = strings
                .into_iter()
                .filter(|string| set.insert(string))
                .collect();

            println!("{:?}", strings);
        }

(runnable snippet: [https://play.rust- lang.org/?version=stable&mode=debug&editio...](https://play.rust- lang.org/?version=stable&mode=debug&edition=2018&gist=7d19d63ac220173cc7a40d2695841b69))

EDIT: updated to make use of the fact that `set.insert` returns a boolean.

<comment><USER8> The output form this is a little different form the article, as it additionally requires:

* that the uniqueness check is done case-insensitively,

* that the casing from the first instance of the string in the input array should be used in the output, and

* that the output should be ordered based on the order of the first instance of the string in the input array

Possibly an unusual set of requirements (I'd generally expect that if removing duplicates from a list, that the output order wouldn't matter) but hey, different problems have different requirements.

The desired output is:

        apple
        cAt
        Dog
        banana

while that code snippet produces:

        apple
        cAt
        cat
        Dog
        dog
        Cat
        Apple
        dOg
        banana

<comment><USER9> I've tried to fix the Rust code, but I'm getting some lifetime errors. Can anyone spot the problem? (Note: I've used C++ for most of the time and am still uncomfortable with Rust)

      use std::collections::HashSet;

      fn main () {
        let strings = vec![
          "apple", "cAt", "cat", "Dog", "apple", "dog", "Cat",
          "Apple", "dOg", "banana", "cat", "dog", "apple",
        ];

        let mut set : HashSet<&'static str> = HashSet::new();

        let strings : Vec<&str> = strings
            .into_iter()
            .map(|string| (string, string.to_lowercase()))
            .filter(|(_, lower)| set.insert(lower))
            .map(|(string, _)| string)
            .collect();

        println!("{:?}", strings);
      }

<comment><USER10> The lower value is of type &String because the argument to filter is of type &(String, String) and the pattern match needs to propagate the &. The reference is only valid for the body of the filter closure (that is, it is &'short String, for some anonymous/unnamed lifetime 'short), but it is being put into a set that expects &'static str (the explicit type on the let line). The &String to &str type coercion is fine, but it's not correct to treat the &'short as a &'static (the short one is potentially invalid after filter's closure returns).

Changing the type annotations won't make this compile, because it is actually catching a real bug. The iterator is lazy and the full pipeline is executed for each element at once: lowercasing, inserting into the set, inserting into the Vec that's the result of the collect, and deallocating the lower string. This last step is the key/danger: if the HashSet held references/slices to the lower strings (instead of owning them), those references would become dangling immediately and future look-ups into the set won't work right/will trigger undefined behaviour.

The problem is a little clearer (and mostly fixed) if you simplify the code slightly by removing the two map calls, and instead call to_lowercase in the filter directly:

            .into_iter()
            .filter(|string| set.insert(string.to_lowercase()))
            .collect();

This form is a type error, that can be corrected by changing the type annotation to be HashSet<String>, or even removing it entirely and letting type inference handle it. The HashSet owning the strings is the key, so they only disappear after the entire iteration is complete, not after each element.

<comment><USER8> Even with the hash set solution, it would be nice to not copy the string contents, which unfortunately is needed if the hash set contains lowercased versions of the strings. On the other hand, converting all strings to lowercase then comparing them isn't a great way to case-insensitively compare strings anyway, as Unicode doesn't guarantee that will work.

The UniCase crate defines a wrapper around strings with a case-insensitive Eq implementation, so this works:

        use std::collections::HashSet;
        use unicase::UniCase;

        fn main () {
            let strings = vec![
                "apple", "cAt", "cat", "Dog", "apple", "dog", "Cat",
                "Apple", "dOg", "banana", "cat", "dog", "apple",
            ];

            let mut set = HashSet::new();

            let strings: Vec<_> = strings
                .iter()
                .filter(|&string| set.insert(UniCase::new(string)))
                .collect();

            println!("{:?}", strings);
        }

[https://play.rust- lang.org/?version=stable&mode=debug&editio...](https://play.rust- lang.org/?version=stable&mode=debug&edition=2018&gist=b8109c4f270266724b14d426ade5bcc8)

<comment><USER7> This is pretty cool. People might think it's cheating to pull in a crate. But IMO the fact that its super easy to pull in a crate in Rust is one of the best things about the language.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> This article is incomplete. The Haskell Library "discrimination" has a linear time, "productive" (in that it can output partial results before the full input has been seen) version of this:

[http://hackage.haskell.org/package/discrimination-0.3/docs/D...](http://hackage.haskell.org/package/discrimination-0.3/docs/Data- Discrimination.html#v:nub)

A simpler non-productive algorithm uses sorting, but you can do so in linear time and we have known about this for some time. Sadly, this information is not well distributed among software engineers in industry.

<comment><USER12> You can get partial results with really any enumerable style api pretty easily, e.g. python

        def distinct(l):
            seen = set()
            for s in l:
                if s.lower() not in seen:
                    yield s
                    seen.add(s.lower())

<comment><USER13> There's also the classic one-liner:

        unique_array = list(set(non_unique_array))

<comment><USER11> Which is fine. I'm fairly sure that's n log n, although sometimes those end up being n^2 if they're biased towards memory friendliness.

<comment><USER14> It's O(n). Python's set has O(1) insertion and lookup.

[https://wiki.python.org/moin/TimeComplexity#set](https://wiki.python.org/moin/TimeComplexity#set)

<comment><USER11> The page you link says it's O(n) for lookup.

<comment><USER14> It says O(1) for lookup. Average case is the one that matters here.

Worst case is for people who've intentionally written bad hash functions. ([https://xkcd.com/221/](https://xkcd.com/221/)) Python has put significant effort into good default hashes[1] so you won't be vulnerable to attackers feeding in maliciously colliding hashes, nor will you have performance degradation from biased bits in the hashes if you've written an insufficiently uniform hash function.[2]

[1] [https://python- security.readthedocs.io/vuln/cve-2012-1150_ha...](https://python- security.readthedocs.io/vuln/cve-2012-1150_hash_dos.html)

[2] For instance, pointers are usually aligned to cache line boundaries, and are biased toward page boundaries, and in C++, the hash function of an integer/pointer is the identity function. So in C++, a std::unordered_set of pointers will have ridiculously high collision rates. If you have 1024 pointers in a set of size 2048, you will only have only 32 buckets that have values in them, and each of those buckets will have roughly 32 elements, and the zero bucket will even more. Python doesn't let you shoot yourself in the foot, and whitens your hashes before going to the dictionary. C++ "solves" this problem by providing an API which allows you to override the hash function.

<comment><USER11> > It says O(1) for lookup. Average case is the one that matters here.

No, it's actually not. Everyone else and the article were using the worst case asymptotic, so you'reswitching to this vernacular and suggesting, "Oh we mean the asymptotic of the average."

Redefining the entire conversation to your notational convenience is not a very fair call, and doesn't speak well of your intentions.

And what's more, it's typical to use big-theta notation to discuss average performance anyways. So even if we WERE using that, we'd be using different notation to be more consistent with the literature and less confusing.

> Worst case is for people who've intentionally written bad hash functions.

No. It's for people who do not know what kind of input they're going to receive or what kind of hardware they're going to execute on.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> The code seems to "remove" items from the array by setting an empty string to them. But in all mainstream languages I know of, an empty string is also a valid string. So the item isn't "removed" it's simply replaced by the empty string. Am I completely misunderstanding what the author is trying to do?

Edit: Thanks: I missed the passage about this being the first step of two, with the compaction coming later (that step isn't in the article at all).

<comment><USER16> His goal is to get ["apple", "dog", "cat"] from ["apple", "dog", "apple", "cat", "apple"].

He does it in two steps[0]:

Step ONE: Replace any duplicate values with the empty string (the tombstone) in the original array. (Going from ["apple", "dog", "apple", "cat", "apple"] to ["apple", "dog", "", "cat"]).

Step TWO: Compact the original array by creating a new array out of all non- tombstone values.

He shows 4 different algorithms for performing Step ONE. He shows 0 algorithms for performing Step TWO because it's trivial.

[0]: >We'd like solutions that conserve time and space. All solutions here work in place, using the tombstone technique to create a sparse array, then compacting it later.

<comment><USER17> That means it removes empty strings from the input.

If this were a library function, I don't think that's acceptable (rationale: if you want to remove all empty strings, but the library function doesn't, you can easily correct that. If you want to keep the first empty string, but the library function removes it, calling the library function is as good as useless)

<comment><USER18> On the flip side, if we're going to be discussing libraries, remember that things like "Maybe<String>" aren't free. That "Maybe" wrapper in the general case has to expand the size of the underlying data structure to adjoin an additional element to it, and that's something else a library may want to think twice about.

I say "in the general case" because in the _specific_ case it is not always necessary. ISTR Rust implements a specialization on pointers where you can syntactically wrap an Option around something and it's smart enough to use the null pointer directly as the missing case under the hood. However, if you've got something like a plain ol' machine int, you have to expand it somehow to get an Option or Maybe, because the compiler is not entitled to remove even a single element out of those for its purposes.

How acceptable it is depends on your ability to declare an in-range (for the data type) element as the "impossible" element. Being able to use an in-range element is more efficient, but less general. In this case, simply by declaration Ted says empty strings are not valid. The next time he does this, they may be, and the technique would have to be adapted to that. A library that uses some equivalent to Option or Maybe is a good safe default for a library, of course. Another safe default is to create a new array and return that, too. It's not that hard to end up in a place where the safe default library is not suitable, although nowadays it takes enough data to choke a horse to get there since our computers are so darned fast.

<comment><USER8> > ISTR Rust implements a specialization on pointers

Yes, and NonZeroU8 for Option<NonZeroU8> and friends:

[https://doc.rust-lang.org/std/num/struct.NonZeroU8.html](https://doc.rust- lang.org/std/num/struct.NonZeroU8.html)

</comment></comment></comment></comment></comment><comment><USER19> At least he mentions the name of his technique: setting tombstones.

Of course it's highly problematic not to explain why he choose "" as tombstone. It looks like a legal array value to me, so offline compaction will have a hard time to separate a tombstone from an empty string. Without documentation and assertions that "" is not a legal value on insert, certainly a bug.

But he fails to list other common useful techniques, without tombstones:

1) if it's the first index to be removed, just advance the array pointer by one, and keep the original pointer separate for the final free call. very useful for strings, cutting off prefixes.

2) sparse arrays: he mentioned it, but he doesn't use it in his code. Just leave out the hole. very useful for matrixes.

3) temporary hashes. if the array is long (like >256), use a temp. hash to find duplicates.

4) copy to new array, leaving out the duplicates. mostly together with a temp. hash. but for short arrays a linear search is also doable. he showed the linear search, but only with tombstones. This is the functional approach, always copy instead of destructive modifications. This is done in offline compaction.

5) for large arrays use a bloom filter to find dups. can be tuned to the best percentage. (This is what I used in my file deduper, which turned out better than normal hashes)

<comment><USER20> > not to explain why he choose "" as tombstone

Because his "neutral pseudocode" (go) doesn't allow setting the string entries in an array to nil. You have to have some value, so commonly you'd use "".

<comment><USER21> But one can not add an empty string to the array which is an obvious bug.

There are slices in go which should be used in this case.

<comment><USER20> It's a toy example in a semi-funny (or trying) post. Did you really expect handling of all the edge cases?

<comment><USER19> No because it's Ted Unangst writing about "Removing Array Duplicates", the guy who added string arguments being comma-splitted at run-time instead of using safe and fast bitmasks to new OpenBSD API's. So we know already that he has no idea what he's doing, and shouldn't be let to any important API in serious projects.

But he writes like he has an idea what he's doing, so I had to explain the common cases for this problem. Using tombstones is a special case and you need to be lucky to have such a special illegal value available. Most cases don't allow it, esp. in strongly typed languages.

</comment></comment></comment></comment></comment><comment><USER22> Semisorting, and then running a filter to remove duplicates (which will be consecutive after the semisort) can solve this problem in O(n) expected work & space and O(log n) depth w.h.p. AFAIK the approach is reasonably practical as well. Note that the algorithm is not in-place.

Paper: [https://people.csail.mit.edu/jshun/semisort.pdf](https://people.csail.mit.edu/jshun/semisort.pdf)

</comment><comment><USER23> Because the 'neutral pseudocode' was actually valid Go code I just benchmarked it:

            BenchmarkDedupe1-4       3000000               521 ns/op
            BenchmarkDedupe2-4       5000000               275 ns/op
            BenchmarkDedupe3-4        500000              3337 ns/op
            BenchmarkAlt-4           1000000              3174 ns/op

Source: [https://gitlab.com/hwj/rad](https://gitlab.com/hwj/rad)

(The 'alt' implementation is my own and maybe not what the author had in mind with 'a searchable data structure to record seen entries'.)

<comment> dan-robertson Benchmark times are going to vary quite a lot with size of array and number of unique elements so recording times for one specific size of array doesn't say much.

</comment></comment><comment><USER17> Another alternative: only loop looking for duplicates when a Bloom filter ([https://en.wikipedia.org/wiki/Bloom_filter](https://en.wikipedia.org/wiki/Bloom_filter)) tells you you may have seen an item before.

O(n²) in theory, but typically a lot faster in practice, at small fixed (semi- fixed, if you size the filter based on the size of the input) overhead.

Also requires you to be able to normalize you strings (for strings, lowercasing or uppercasing may not be enough, depending on culture)

<comment><USER24> That's "The alternative which shall not be named" and chances are you'd just use a set as your secondary data structure, there's limited point to using a bloom filter unless the input sequence is absolutely humongous.

<comment><USER25> It depends entirely on what you need to guarantee. Using a hash tabel you guarantee with a very high probability that you only remove duplicated elements. If you need 100% guarantee, then you must do something else.

<comment><USER26> Hash tables don't work in a way that you assume here. Original keys don't get lost, so you can check whether a key is actually in the table or just makes a collision.

In its simplest case, HT is an array1 of array2s of key-value pairs, where array1 is indexed via [hash(key) % a1.length] and then array2 (aka bucket) is searched linearly for a given key by comparing contents.

<comment><USER25> A hash table does not need to have a collision resolution strategy. It's only definition is key/value pair. That you can have buckets function to avoid collisions is "extra" so to speak.

C# linq .Distinct() does not do that, it only checks on the key.

<comment><USER26> A hash table is a well-known term that covers multiple algorithms, but I never seen one that worked the way you describe. False positive lookups is a property of bloom filter-like struct, not of a hash table. Using false positives to remove duplicates would be wrong indeed.

Ed: I missed "It's only definition is key/value pair" part. No, that's called key-value list or table. _Hash_ table uses _hashes_ of keys to partition itself into quick lookup segments - that's the point.

</comment></comment></comment></comment></comment></comment><comment><USER27> I gave this a shot in Perl 6 as can be found at [https://gist.github.com/8a58c3af3520e127fe7852d834e5e0fa](https://gist.github.com/8a58c3af3520e127fe7852d834e5e0fa)

I build a very simple^1 non-balancing binary tree and skip any insertation that would reproduce a duplication. Since the tree is not balanced, the order of insertation is preserved. Search time goes up without a good balance but the tree will have the same amount of nodes than unique elements in the original array. Each element is 6 machine words plus some gc overhead. If it's faster or slower then a hash table depends on the ratio between elements in the array and the number of duplicates.

1) Perl 6 got a general compare operator that works well with a wide range of types. So I don't have to care about types at all. If I would have to care I could monkeytype the operator candidates for custom types into the language.

<comment><USER28> Default &transform function should be *.fc

The one-liner solution is: @data.unique(as => &fc)

</comment></comment><comment><USER29> My first thought was using a map to store what's been seen and run through the list keeping copying the values that haven't been seen to a new list in order or just marking them as dupes in place with the empty string.

This seems to be the "alt" case and is dismissed by the author but would like to hear a fuller explaination of why this is a problem?

<comment><USER30> Too slow, too much hashing, too many copies.

<comment><USER31> It will probably be faster than the code in the article.

<comment><USER30> How?

<comment><USER5> It requires a single linear pass instead of a convoluted sort.

<comment><USER30> ... what sort? Are talking about the same thing here?

<comment><USER5> The code in the article has a sort in it.

<comment><USER11> Why would that be "more copies" than building up a hash table is what I don't get.

<comment><USER31> It requires many passes through the list, and even O(LogN) is probably enough to make it slower than a hash based O(n) approach depending on list size. Hashing is fast. If space is an issue, just use an appropriately sized bloom filter.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER32> From TFA: "These examples are written in neutral pseudocode that should be adaptable to most any imperative language."

False. The examples are written in Go, a generally easy to read language. Maybe the author thought it is a funny joke. Low bar :)

<comment><USER24> > The examples are written in Go, a generally easy to read language.

But not necessarily so, #3 and #4 are just godawful.

edit: and as you point out below, Go's sorting interface is completely alien and doesn't easily translate to any other language.

<comment><USER32> Agreed, the sort interface in go is one of the worst parts, and doesn't translate to any other language I'm familiar with. The article was mildly entertaining but by the end left much to be desired.

Shoulda just gone with python, rust, c, crystal lang.. anything else for sorting!

</comment></comment></comment><comment><USER33> Why don't you just use a set?

<comment><USER11> Constructing sets has a running time, too. All you're really doing when you build a set is sorting.

<comment><USER34> That's not true for any reasonable set implementation. Sorting is unnecessarily time-intensive for maintaining distinctness. Hashing is sufficient.

<comment><USER11> Which set implementation are you referring to?

And also remember the context here: you're removing duplicate value's from an array. So you're inserting N items into a set. If the set insertion or enumeration involves even a O(log(n)) operation, you're at nlogn.

<comment><USER34> A hashset has O(1) membership testing and amortized O(1) element insertion.

<comment><USER11> But it requires a perfect hash. You need to guarantee over any potential input that you don't have collisions.

How do you do that over all inputs? If there is a generalized non- probabilistic perfect hash function I am unaware of it and I'd like to be aware of it.

<comment><USER34> No, it doesn't require a perfect hash. There are collision mitigations. In practice there several that can get to O(1). I'm not sure if there is a way to achieve O(1) in the worst case. But the worst case can be made arbitrarily unlikely, even against adversarial inputs.

<comment><USER11> > I'm not sure if there is a way to achieve O(1) in the worst case.

Then you're not at O(1). You cannot say "O(1) except in the worst case". That is like saying, "It is blue except when you look at it."

But you don't need these mitigations (most of which are O(log n)) because you never had the hash table. You should look at American Flag sort, because morally it's actually doing something that closely approximates what you're thinking about. That's why I brought it up elsewhere in the thread.

<comment><USER34> It's O(1) in the sense that git works. Git uses "commit hashes". If these collide, something would break, on par with the universe stopping working, I presume. But I'm sure, since this never actually happens in practice.

<comment><USER11> [https://arstechnica.com/information- technology/2017/02/water...](https://arstechnica.com/information- technology/2017/02/watershed-sha1-collision-just-broke-the-webkit-repository- others-may-follow/)

It was a bad design to rely on hashing algorithms to never collide. But it's worth noting that even SHA1 is much more robust than most hash table algorithms, which are meant to run even faster.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER35> The problem with deduplicating the data can be changed into another problem: ensuring we won't add the data if it would be duplicated.

So instead of trying to deduplicate the array from author's example, I'd change the 'add()' method so that it searches through the whole set and doesn't add the data if it's already added.

So now it doesn't require deduplication code at all.

<comment><USER36> That's what I though as well. In which case it's better to just keep it in a set-like structure initially, or just dump your entire array into a set afterwards. That's linear time, which is loads better than any of the solutions presented.

</comment></comment><comment><USER37> The typical way to solve this problem would be to keep the data stored in a normalized relational database. There would be a Customer table and a Visits table. Since the data is normalized, the Customer table would contain only one entry per customer. There would be no need to dedupe the data if it is kept in this format.

If moving the data into a normalized relational database is not an option, the next best option that I can think of is to use a hash table (as mentioned elsewhere in the HN comments). However, I can't imagine working on a system where it would make sense to write a custom solution for each type of data query (unless I was working on a database system).

</comment><comment><USER38> three and four might be close to the

[https://en.wikipedia.org/wiki/Schwartzian_transform](https://en.wikipedia.org/wiki/Schwartzian_transform)

but I like the Schwartzian better :-)

<comment><USER39> Learning about the Schwartzian transform was one of life's memorable "ah hah" moments for me. There's a whole set of higher-order layers to this programming thing!

As others have said, language choice matters.

</comment></comment><comment><USER40> An interesting question is how can you parallelize it? Suppose the data is huge, but you have many available threads, or machines..

Suppose there are N nodes, each of which owns part of the array. So one way is for each node to broadcast each entry to all of the other nodes. As each node receives a broadcast, it checks its array for duplicates, and deletes them.

</comment><comment><USER41> The only thing this post did for me was entrench my thought that I'd rather write Rust over Go.

As ugly as Rust can be to look at, I'd rather solve this in 5 lines of Rust that make sense than read through any of those Go solutions.

</comment><comment><USER42> Isn't a loop k from 1 to n, with an inner loop k to n, actually order n! (The article says O n^2) ?

<comment><USER43> Adding up 1+2+...+n equals to n(n-1)/2, which when converted to an order becomes O(n^2).</comment></comment>


Build an iOS App in 10 Minutes with Heroku - <AUTHOR> http://mobile.heroku.com/

<comment><USER2> Really, it's watch someone build an app in 10 minutes, moving so fast and with such preparation that the timescale doesn't matter.

I've watched so many instructional series on development recently; this was very weak on actual instruction, and heavy on the gimmick of being done in 10 minutes.

Also, it was already posted last week.

<comment><USER3> My immediate reaction: Ugh, all that boilerplate. I have no idea what is going on with all of that, and I'm not sure how I would work it out on my own.

<comment><USER4> Yeah, boilerplate isn't the greatest. Hopefully the instructions on this Dev Center article clear up a few things: [https://devcenter.heroku.com/articles/ios-core-data- buildpac...](https://devcenter.heroku.com/articles/ios-core-data-buildpack- app)

</comment></comment></comment><comment><USER5> I didn't watch the video but I'm pretty tired of this catch phrases, build an iOS app in 10 minutes coming from serious businesses.

Any app that was built in 10 minutes is not worth 1 minute of my time.

</comment><comment><USER6> Gross headline. I'd hope, at the very least, that a designer would spend more than 10 minutes designing the thing.

That said: I've dropped AFIncrementalStore into a Core Data project, and it was wonderfully easy to add to the project. I'm still getting started with it, but it's proven to be far easier than working with RestKit's setup.

</comment><comment><USER7> I would like to build an App in 10 minutes. Great catch. Still watched it and read it. Enjoyed.

</comment><comment><USER8> android coming?</comment>


Is 31 too old for startup? - <AUTHOR> http://www.entresting.com/blog/2012/11/13/im-31-entrepreneur-where-is-my-senior-home/ <comment><USER2> Don't let the press fool you: there are plenty of founders aged 30+ building innovative businesses. The average age of a startup founder is 40:

[http://www.thedailybeast.com/newsweek/2010/08/20/innovation-...](http://www.thedailybeast.com/newsweek/2010/08/20/innovation- grows-among-older-workers.html)

Also, YC has plenty of founders over 30: [http://blog.zencoder.com/2010/09/21/doing-y-combinator-in- yo...](http://blog.zencoder.com/2010/09/21/doing-y-combinator-in-your-30s/)

Personally, I founded my first company in YC when I was 30 (Parse).

</comment><comment><USER3> 31 is a perfect age. But so is 18, 24 and 42.

Side discussion - _"And I want to be the next Steve Jobs."_

These type of statements always irk me. I have no evidence to support my claim but I suspect that Bill Gates, Steve Jobs, Mark Zuckerberg, etc. rarely publicly stated they want to "be the next XYZ". Rather they had a vision to _do_ something bigger. For example, Gates' vision was to "put a computer in every home". These types of realities take decades, not just years. Most people can't think this far ahead.

<comment><USER4> That bugs me too, even more so when I hear it from VC or their brethren, because it seems like they should know better.

Steve Jobs was not the next Bill Hewlett; Elon Musk is not the next Steve Jobs; Google was not the next Apple; Facebook was not the next Google.

The only thing those all share in common is that they were the next big thing. Although you can use those examples as aliases for "the next big thing", they'll tend to change the traits that you're looking for. If you're looking for "the next Mark Zuckerberg" for instance, then it's easy to let yourself get trapped into looking for someone with Mark Zuckerberg-like traits -- and then you'll miss the next big thing, because that person isn't going to be "the next Mark Zuckerberg".

</comment></comment><comment><USER5> Yes, you're too old. You should just find some comfy corner, curl up, and die.

Now that the pity-party is over, let's focus on reality. One is NEVER too old to found a startup. And the fact of the matter is, there are many more startup founders (both technical and non-technical) over 30 than under.

Take a look at these links, and ask yourself again if you're too old:

<http://www.quora.com/What-is-the-average-age-of-founders#>

<http://techcrunch.com/2011/05/28/peak-age-entrepreneurship/>

TL;DR:

Ev Williams, Twitter: 35 in 2007

Jeff Bezos, Amazon: 30 in 1994

Reid Hoffman, LinkedIn: 35 in 2002

Elon Musk, SpaceX: 39

Quite moping around and build something.

<comment><USER6> Well off the top of my head at least two of the people on your list had earlier successes than the ones you listed.

Ev Williams did Twitter, but did Blogger before. Elon was 28 with PayPal

Probably not the best examples you could have chosen. It is like saying Bill Gates wasn't a success until he was in his fifties because it wasn't until then that he started the Gates Foundation.

<comment><USER5> I didn't choose the examples; just reposted them from the Techcrunch article.

As for Williams and Elon, true. But they did continue starting companies. Is there a difference in a first-time startup and follow-ups? Sure. Better networks; experience breeding efficiencies; reputation opening doors.

But the risk and the hard work (generally) are still there. Not too mention expectations of others and yourself from previous success can be the biggest hurdle faced. Been there, done that.

The point remains the same though. Startup founders are all ages, and there are many more 30+ founders out there than are trumpeted by the media. This is especially true in the non-technical startup realm.

</comment></comment></comment><comment><USER7> I have almost 10 years on you. I started my first business at 23, and sold my 5th 4 years ago before starting this one.

In my 20's, I was smart enough to know everything, and it cost me a lot of learning. Those entrepreneurs have advantages, mind you. A single minded purpose, a refusal to accept the status quo, and the notion (by investors) that you can be trained, molded, and focused.

In my 30's, I was smart, and chasing validation, victories, crushing competition of all shapes and sizes, and basking in the glory of success. I achieved some cool things, I spent a boat load of money on stupid stuff, and I started to learn how to be an adult.

I'm now at the beginning of my 40's. I understand far more. I piss off far fewer people, and I have a wisdom that propels me towards happiness and fulfilment, but I still have a fierce hunger for success, crushing competition, and tearing a hole in the status quo.

Now, however, I know exactly how to do it. I make far fewer mistakes, build bridges instead of walls, and have a rolodex filled with decision makers, allies, and mentors.

I work, constantly. Booze, women of questionable morals, and flashy toys don't have the allure they once had. I get shit done, and leave the chest thumping to the kids.

The spin on young founders comes from a different world, and it's not altogether wrong, but it requires a 500 Startups approach (invest in a bunch and hope a few make it).

Older founders have a much higher rate of success, if they can handle the psychological pressures that come with it.

If you're doing it to get rich, you'll most likely fail. If you're doing it for a lifestyle, the odds are stacked against you. If you're doing it to be cool, well, save yourself some money and buy a Mazerati.

If you're doing it because you have an obsession with the problem, and you can't sleep because you need to see it solved, and you can enlist others to be soldiers in your army to fight this obvious injustice/inefficiency, then you have a very strong chance of success.

Don't be a founder just to check it off your bucket list, do it because you have no choice but to follow/create your destiny.

Best of luck.

</comment><comment><USER8> +1 here as well.

I say never too old, but be prepared for the long haul. Many startup opportunities that would jumpstart your business favor the younger entrepreneurs with less to lose and/or less responsibilities.

There's also supposedly an overall bias against us geezers in startups but I can't speak (much) to that.

....and I'm sure ( being the wise old man that you are ;) ) you'll gladly take being a 35 year old millionaire. That "billionaire at 20" stuff is great, but it's also glitz.

</comment><comment><USER9> The desire to want to just "make it" seems deeply wrong-headed to me. Granted, I don't own a company so this may be one of the many times that I am wrong, but it seems like a pretty important characteristic would be having a particular product or vision you are passionate about.

And deeply wanting to be a "reluctant billionaire" seems contradictory. All that said, good luck to the guy. I sincerely hope I'll get the chance to eat crow on this one.

<comment><USER10> Others here express a dissimilar sentiment: the only durable reason to start a company, is because you want to start a company. If you're passionate about that, you have a better chance of getting the business side right.

</comment></comment><comment><USER11> I think it has less to do with age, and more to do with family status. Being 23 and a family is just as bad as 40 with a family. A 22 year old with no family is as good as a 45 year old without a family. Once kids and a wife/husband enters the picture, things change greatly. If you really want to take a crack at being a entrepreneur, do it before 'settling down'.

</comment><comment><USER12> It's never too late.

Ray Kroc bought McDonalds and grew it to a massive empire when he was 61. Read his book Grinding it Out.

Frank Lloyd Wright was broke several times in his life and considered through by the architectural establishment. He was 68 when he designed Fallingwater, one of the great homes every built.

</comment><comment><USER13> Being over 31 just means you didn't win the lottery.

The vast majority of us are simply going to make a living, a DAMN good living, but still something small where we own a house, have a family and maybe a vacation home in Arizona.

Startups aren't just for young people because they can (apparently) work 24-hour-days. That's a failed assumption and part of the Silicon Valley Echo Chamber.

You want to create a startup? Go for it. If you are an experienced professional then being in your 30's means you'll have more 'adults' willing to listen to you, you'll have more contacts in the industry willing to sign up to your product, and you'll have a larger spectrum of knowledge to know when to cut your losses or when to quit your dayjob and go all in.

</comment><comment><USER14> +1 as another 30-something entrepreneur...

</comment><comment><USER15> Ususally startups are for "younger people" because they can easily change and adapt themselves to a startup enviroment like go the extra mile, more work, frequent changes, learn new things, sometimes do things that can really go out you spectrum of work etc... instead someone older might have a family to maintain and therefore needs stability and safety for his job. But if this doesn't apply to you for whatever reason I do not really see any reason why you wouldn't fit.

<comment><USER16> Anyone who doesn't necessarily need to be paid for many months more than a pittance can do a startup.

Most families with 2 tech-type incomes currently who control their spending to something reasonable can have one of the two do a startup. You might have to move to a cheaper place if your rent/mortgage is too high, but other then that, you're probably in a BETTER place than most 20 somethings, who have no safety net.

</comment></comment><comment><USER17> "but it's just not as cool to be a 35 years old millionaire now days when there are 24 years old billionaires running around."

If you actually meant that, I'd say you're too immature. Honestly it's about making money, or changing something you care about in this world. if you want to be a founder because it's fashionable well ...

</comment><comment><USER18> I was a moderately successful founder at 27 ... Now I'm 39 and while I'm much more experienced, I have also become a little impatient, something I consider very dangerous for a startup. Are you impatient too?

</comment><comment><USER19> Why would you be old for something you want to do, and can do. If you asked me, "Am I too old to be a Wimbledon Champion" I'd say, yes. Don't get fooled by the media.

</comment><comment><USER20> Do a startup, see if it works. If you're doing anything else, or thinking of doing anything else, then no matter what age you are, it's probably wrong for you.

</comment><comment><USER21> If it is too old, too bad for me. NOT. Age is just a number. I am sure there are plenty of 31 year olds who are as successful as the 18 year olds.

</comment><comment><USER22> Wait... 31 is old? I thought 30 was the new 20

</comment><comment><USER23> I'm 32 working in tech and I feel like a dinosaur. :-(

</comment><comment><USER24> By even asking such a question I think the startup life is not for you.

<comment><USER25> Was going to say that myself. Downvotes be damned, this answer get to the heart of it.

</comment></comment><comment><USER26> Long answer: No.</comment>


Ruby 2.6 - <AUTHOR> https://anamaria.martinezgomez.name/2018/12/21/ruby2_6.html <comment><USER2> I am always happy to unwrap my Christmas present from the Ruby team! Some questions about this release:

\- Are there any benchmarks for the new JIT?

\- Why was `(Date.today..(Date.today + 1)) === DateTime.now #=> true` false in Ruby 2.5? It seems correct that right now is between today and tomorrow. Is it because one is a Date and the other a DateTime?

<comment><USER3> Been a while since I've used any Ruby but isn't the first expression a range where's the second a single value?

It'd be like [0,1] === 0

<comment><USER4> > isn't the first expression a range where's the second a single value?

`===` conventionally means 'matches', or 'includes', so an aggregate on the LHS and a scalar on the RHS is typical.

It isn't defined for `Array` beyond being the default of equality - I don't know why - but it is defined for `Range`.

</comment></comment></comment><comment><USER5> Fantastic Christmas present from the Ruby team.

Ruby is alive and well despite all the rumors of it's demise. I predict a resurgence in its use over the next few years.

Can't wait to examine and try out the new features.

<comment><USER6> What makes you think there'll be a resurgence in its use in the next few years?

</comment></comment><comment><USER7> Ruby used to be my most beloved language. I've since developed a strong appreciation for good type system. However, working with collections of any kind in Ruby is just such a joy. The methods around it are so well thought throug and make writing code a joy that in most other languages just would be tedious. On top of that the Ruby team is constantly working on making that aspect even better. I truly don't understand why it seems to me that no other language I use makes even an effort to provide something similar. Getting a map, a reduce and a filter function seems to be as far as most standard libraries are willing to go.

<comment><USER8> Can you comment on why a good type system became so important to you? I can understand the value in interfaces but not really in any other context.

<comment><USER9> Languages without ADTs and Optionals just feel impoverished.

<comment><USER10> Languages without full blown union types feel so impoverished. With good language typing support (including flow typing), optionals feel quite redundant.

<comment><USER11> But optionals are just a union type (T | None).

</comment></comment></comment></comment></comment><comment><USER12> This will be fun. I'm glad we now have some performance improvements to some common methods. For those considering playing with Ruby, you should! One thing I'm surprised not many has mentioned what makes ruby unique and awesome, BLOCKS!

<comment><USER13> Aren't they like arrow functions in JavaScript?

<comment><USER14> Just parameters that are functions, except that you can only have one per function. I still fail to see the advantage of blocks over being able to pass functions as arguments like in other languages.

<comment><USER15> One nice feature of blocks is that `return` will return from the enclosing function, rather than the block, so you can write:

        def find(x, l)
          l.each do |y|
            return y if y == x
          end
        end

Where in Lisp you'd use (return-from find x) and in other languages you might pass in a continuation or use a special return value protocol. It's a nice solution for higher order functions that are supposed to feel more language- level.

Also, you can pass functions as arguments like in other languages; lambdas behave like you would expect them to.

</comment></comment></comment></comment><comment><USER16> Really like the idea of using lazy ranges instead of magic negative indices! I.e `xs[6..]` instead of `xs[6..-1]`.

<comment><USER17> Negative indices aren't magic. They follow the obvious wraparound (modular) pattern: Negative `i` indexes the element in array `a` located at `i % len(a)`. If they worked any other way it'd be madness.

<comment><USER18> I don't think you got that index calculation right. -1 in array of 5 elements is located at 5-1 == 4, not at 1%5 == 1.

<comment><USER17> I don't follow your point. We're concerned with the value of -1 % 5, not 1 % 5. There's no definition of the modulus operator on which -1 % 5 = 1.

<comment><USER18> Phrasing issue, I think. I understood "Negative `i` indexes the element" as `i` itself being positive. Like you'd say "negative 1 indexes" (i is 1), not "negative -1 indexes" (i is -1)

</comment></comment></comment></comment></comment><comment><USER19> > Constant names can now start with non-ASCII capital letters. I am not sure > how useful this is

Very? If you want to code in non-Latin languages.

Also, π = 3.14159

<comment><USER20> I'm not sure about this being good for coding in non-latin. It might actually encourage behavior to mistreat ruby constants as regular variables. IIRC constants are not really constant, but only visual help for reading code and ruby just issues a warning on reassignment and these warnings are not shown by default.

I can totally imagine new devs just trying out non-ascii variables, see it not crashing and then actually writing all their code using constants.

<comment><USER21> > I can totally imagine new devs just trying out non-ascii variables, see it > not crashing and then actually writing all their code using constants.

Ruby _already_ supported non-ascii identifiers for variables; 2.6 added support for non-ascii first characters of constants, which must still be capital letters, as had always distinguished constants from variables. It doesn't make anything non-ascii a constant, it just makes constants consistent with other identifiers in terms of non-ascii support, while preserving the "initial capital means constant" rule.

And, as sibling comments note, reassigning a constant produces a warning, so devs making a mistake and using constants for variables (regardless of ascii- ness) will quickly be informed of their error.

<comment><USER20> I wasn't aware of non-ascii variables. I was thinking about UTF support for Japanese characters, when I read the change. There is no concept of lowercase/uppercase. 日本語 is this a constant or variable?

<comment><USER21> Since constants are the exception, and used to require an ASCII capital but now can start with ant capital, an identifiers starting with a Unicode character in a script that doesn't have upper/lower case distinction is a variable, not a constant.

</comment></comment></comment></comment></comment><comment><USER22> Is it a good time to learn Ruby? I mostly do React and Node + Java work professionally, but I've been looking into Ruby as a fun side project language. I would appreciate any anecdotes!

<comment><USER23> Ruby isn't perfect but it's a great utility language and can do a lot of powerful things with very little effort. Read through Eloquent Ruby

The only problems I wouldn't go to Ruby for are CPU intensive ones or ones where I need parallel threads (which you already have Java for)

<comment><USER24> You can also use JRuby. Powerful and can be very easy to the user.

<comment><USER25> Had a problem that I'd typically use Ruby for but needed to shed the GIL. JRuby worked beautifully. The only gotcha was I needed to use the JDBC driver for Sequel instead of the usual drivers. Obvious in hindsight but not so much with zero JVM development experience. I'd use it again.

</comment></comment></comment></comment><comment><USER26> The access to AST will allow for neat hacks like PonyORM on Python, which could be a way for me to improve Rebel::SQL. Tentative hacks by decompiling bytecodes or using Parser, while fun, proved practically unsatisfactory.

[https://ponyorm.com](https://ponyorm.com)

[https://gitlab.com/lloeki/rebel](https://gitlab.com/lloeki/rebel)

</comment><comment><USER27> What is the complexity of set operations on arrays? This seems a bit too much sugar to me, to the point where people will end up writing inefficient code because they will thing these methods are more performant than they really are. Usually, I like to see explicit conversions to sets to make the performance characteristics explicit.

<comment><USER19> In most real world applications, the cost of doing an in-memory data manipulation like this is negligible compared to I/O, waiting for the database etc. It's already the wrong language for someone who wants to squeeze out every CPU cycle, and imo it's a Good Thing for a language upgrade to embrace its strengths ie syntax sugar in the case of Ruby, as long as the perf hit isn't out of control.

I'd agree developers should pick the right data structure for the job and some APIs can obscure the cost to the point of misleading developers into woefully wrong decisions. I just don't think it's the case with array ops like union as it's easy enough to handle them efficiently. I too would be interested to see any benchmarks though.

<comment><USER27> I might be more receptive to this if this was added to all collections, then, rather than just arrays. (Maybe it was? I don't write Ruby, so I don't know for sure. Maybe Ruby doesn't have a generic "collection" protocol?)

<comment><USER19> Interesting point. Ruby has an Enumerable module, but these methods were added only to Array. Since the main native collections (Set/Hash/Array) all support them now, they could probably be added to Enumerable in the future, with class-specific optimizations if necessary.

</comment></comment></comment></comment><comment><USER28> one shot mode [1] for coverage will be handy in production as it has quite no overhead. Happy xmas Ruby devs!

[1] [https://bugs.ruby-lang.org/issues/15022](https://bugs.ruby- lang.org/issues/15022)

</comment><comment><USER29> I was trying to go download the latest Ruby or even read the latest release notes, but the site seems to be down. It's still ruby-lang.org, right?

<comment><USER29> Weird. I googled "ruby 2.6 download" and links from the results worked, but simply trying to browse to "ruby-lang.org" times out.

<comment><USER30> It just resolved for me. Maybe it'll clear up for you soon!

</comment></comment></comment><comment><USER31> Am I the only one to find -rcX and -previewX type of versioning confusing. like how much time is remaining etc.

</comment><comment><USER32> For when Ruby 3.x?

They should release soon, if we are to start seeing some decent adoption by 2028.

/pythonicsarcasm

</comment><comment><USER33> > non-ASCII constant names

This is actually a security risk, avoiding all Unicode security recommendations. See e.g. [http://websec.github.io/unicode-security- guide/visual-spoofi...](http://websec.github.io/unicode-security-guide/visual- spoofing/)

All they did was open the floodgates: [https://github.com/ruby/ruby/commit/f852af0e59899157ef695edc...](https://github.com/ruby/ruby/commit/f852af0e59899157ef695edccbe86d51fc04d23b)

no rtl checks, no spoofing, no mixed script checks, no normalization.

<comment><USER34> How is it it a security risk to name a constant in source code?

<comment><USER35> it can be used to create malicious pull requests that are visually indistinguishable from correct ones

<comment><USER21> > it can be used to create malicious pull requests that are visually > indistinguishable from correct ones

If your only check on the correctness of submitted code is visual inspection, you've got a much bigger problem than Unicode confusability.

<comment><USER27> How do you check your code? Presumably, you're not able to exercise the full range of inputs, so at some point you will have to look at the code and figure out what it is doing…

<comment><USER36> Which means you'd notice the value of a constant being exfiltrated, since a maliciously named constant doesn't magically expose secrets outside of the application.

<comment><USER37> I suggest you take a look at some of the submissions in e.g. the Underhanded C Contest. Where it can be excruciatingly hard to figure out where the backdoor is even though you _know_ that the code has one.</comment></comment></comment></comment></comment></comment></comment>


PewDiePie's Response - <AUTHOR> https://www.youtube.com/watch?v=lwk1DogcPmU <comment><USER2> The WSJ is burning their reputation with such sensational articles.

They are the place to find weird climate-change contrarian articles.

<comment><AUTHOR> I think the guy should sue them. He probably lost a lot of money over this nonsense.

</comment></comment><comment><USER3> WSJ Article -

[http://archive.is/iy1hR#selection-4695.71-4702.0](http://archive.is/iy1hR#selection-4695.71-4702.0)</comment>


There Are Only Integration Tests - <AUTHOR> https://dimes.github.io/blog/there-are-only-integration-tests.html <comment><USER2> This seems to miss a few points:

* Unit tests should test functional requirements within the code (i.e. for a given input, this code returns this output)

* Integration tests test the boundaries between systems (eg. libraries, code modules or distinct systems), which is why they are called "integration" tests

* Mocks are a only needed in imperfect world, which is our world usually: incidentally, they should be more common in integration tests, since it's harder to maintain perfect decoupling between distinct components, and you should never mock integration logic under test

* Unit and integration tests are both useful as documentation

* You might need system tests, which test the end-user functional requirements

The fact that unit tests needed dropping or changing as part of refactoring is a given: you are changing interfaces and their functional guarantees. Like the rest of the code, they need refactoring as well.

Considering the example given in the article, this seems to be mostly a misunderstanding of terminology and coupling of components in software.</comment>