Types and Tests - <AUTHOR> http://blog.cleancoder.com/uncle-bob/2019/06/08/TestsAndTypes.html <comment><USER2> > Is [not testing nil/invalid inputs] a risk? Sure, but only if some other > part of the system was written without tests. If you call this function from > a module that you wrote with tests, using something as effective as the TDD > discipline, you won't be passing nil or negative numbers, or any other form > of invalid arguments. So I'm not going to worry about that. It's on you.

The problem with this way of thinking is that it makes tracking down a bug much harder, and even if all software is correct at the end it will still make development slower.

Imagine a bunch of functions A, B, ... and Z where A depends on B and so on. If every one of those doesn't test what happens on incorrect inputs, then an incorrect input to A may cause an exception in Z. You would have to track all the way back up to find the cause of the issue.

For the given simple example, not testing negative of nil inputs may be fine because just looking at the code is enough to document it, but on a larger scale, failure conditions can be very obscure.

Simply put in a long chain of software, you will want to define on what values your function is valid and only accept those (so users don't think something works because only one case does). And to keep your code correct to that specification, you need to make sure it errors.

</comment><comment><USER3> > _My answer to that is that even using a dynamically typed language I don't > have to write the test for nil because I know that nil will never be passed. > I don't need to make states unrepresentable, if I know those states will > never be represented._

If only programming actually worked this way... You can never know what "will never be passed" unless you prove it, and types offer that proof.

Tons of bugs in languages that allow null to break their type system, for example, concern null pointer dereferencing...

</comment><comment><USER4> I think it is pretty easy to illustrate why fewer tests are needed in certain statically-typed languages.

JS

        function max(a, b) {
          if (a < b) {
            return b;
          }

          return a;
        }

        console.log(max(1, 2));
        // 2
        console.log(max(2, 1));
        // 2
        console.log(max([1,2,3]));
        // [1,2,3]

Rust

       fn max<T: PartialOrd>(a: T, b: T) -> T {
           if a < b {
               b
           } else {
               a
           }
       }

       println!("{}", max(1, 2));
       // 2
       println!("{}", max(2, 1));
       // 2
       println!("{}", max([1, 2, 3]));
       // Compiler error

In JS, it would be prudent to test for reasonable misuse of this kind (return undefined, throw Error, etc). The author claims _This is a dynamically typed language. You get exceptions when you mess up the types._ That isn't true in this case and it is not clear from the code what would/should happen.

In rust, you don't need any tests like this.

<comment><USER5> A common philosophy in JavaScript is to make it _just work_. For example if an array is passed, sort it and return the max value, which was probably what the programmer intended.

<comment><USER6> The max value of what? The array in one of the arguments? What if there are two arrays? Or an array and a number? Or a multi-dimensional array of numbers and any of the above?

I think this attempt to come up w/ reasonable interpretations for unanticipated usages of a function is kind of a rabbit hole. Beyond that I think it's also problematic to assume that all usages of a function in a language without a type system are intended usages, i.e. that programmers _mean_ to pass arrays whenever an array appears as one of the arguments.

<comment><USER5> Sometimes you want to call a function with something else, or are not using an IDE (no argument hinting), and just call the function intuitively. Instead of converting the input, or writing a middle-man, you edit the function to accept it as is. You don't even have to pass values in the right order, the function will figure it out.

<comment><USER6> There's nothing wrong w/ polymorphic functions, but there has to be a reasonable rule to apply for what kinds of shapes of data are permitted. Trying to aggressively handle unforeseen cases instead of indicating an error doesn't actually help anyone, it's akin to swallowing exceptions. It's why falsey values and automatic type conversion are some of the most problematic features in javascript.

Your example of how to handle max for array arguments is exactly the kind of thing I would discourage in js code, because it overloads the meaning of "maximum" to data that don't have a natural way of assigning an ordering if heterogeneous arguments are permitted (actually it sounds like you're saying something like max([1], [9]) should return "9", which doesn't correspond to an intuitive understanding of "maximum" at all given it returns something that isn't either of the passed arguments). Whereas the rust code can use the "PartialOrd" trait to ensure we only try to produce a maximum of like elements with a defined ordering relation. That's fine for a limited number of cases where its easy to show type equivalence in js, but I think it's problematic as a general design principle, and because you don't have ways to express the limits of polymorphism I've seen this kind of thing proliferate to unsafe uses of a function very quickly.

<comment><USER5> Only after you have tried to call the function with an array many times, that's when it would make sense. And there's another option too, to instead add a check and a friendly error. But it should be a fairly rare thing to do.

Naming things can be hard and maybe more so if you do not have type annotations. But being used to dynamic languages I always find type annotations a chore. And I think it's better to have a good name and no types, then a bad name and types. eg. max(...numbers) vs x(y:number, z:number) -> number . Even though x is more verbose it can be anything, but with max(...numbers) I can be fairly sure.

<comment><USER4> but then you have to test for this behavior, which was the counterpoint to the article.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> When I'm writing Haskell I still practice _Test Driven Development_ but I usually start with _Type Driven Development_.

The richer the type system, the more expressive the propositions I can encode, and the easier it is to write a proof — a term with a type that matches the proposition. Leaving me free to write only the tests that matter and eliminating the need to start with specifying degenerate cases.

You can express concepts like non-empty lists, constraints on kinds and types which encode the propositions you'd normally test by example using unit tests. It's simply not necessary in some cases to start designing programs with unit tests when you have a type system like Haskell's (or Idris', Agda's, or Lean's, etc)

<comment><USER8> The hidden thing about Haskell that makes it super-powerful for expressing safe programs (in my experience) is not only that you can express tight propositions in the types. It's that Haskell makes it very easy to write total functions over wider types.

When writing OOP code you often start with a number of base assumptions about the input and build the code around them. With Haskell, you make no such assumptions initially and either encode total functions (most of them!) or tighten the types until they match the true system requirements.

There are some Prelude functions which violate this understanding (`head` being a classic example). Writing code without these as much as possible makes your code almost guaranteed robust right out of the gate.

</comment></comment><comment><USER9> _The odds that this test will fail are on the order of a million to one. I could better those odds if I thought it necessary._

That seems... pretty bad? Am I missing something?

If you have 1000 tests like that in a large codebase with a continuous build, you're going to get an irritating number of random failures.

It's not really a concern until you have a really large codebase. But if you leave it until you have a really large codebase, it's a lot more effort to go back and fix the flaky tests.

<comment><USER10> 50 tests that fail 1% of the time will cause almost half of your builds to fail. And the odds of two failing in a row are about one in four. As the number of builds per day increases, the probability of a given day having multiple failed builds in a row (causing you to look closely at the build and commit logs) approaches 100%

<comment><USER11> Unit tests should not be flaky. Integration tests, acceptance test can be flaky by nature of networks but not unit tests.

</comment></comment></comment><comment><USER12> I've spent a lot of time now writing code in both Haskell and Elixir. Where I have found that static typing really shines is when you do large scale refactoring. Especially in a large code base it is much, much faster to fix compiler errors after refactoring, than to run the tests and figure out why each one is failing. In Haskell by the time my tests all compile after a refactor, they are already passing.

<comment><USER13> Yup. In a codebase without static types, large-scale refactors are probably just not practical, which may be why fans of dynamic types don't have this problem; they can't, and thus don't, do substantial refactors without rewriting.

<comment><USER5> There are also static analysis tools for JavaScript. For example Google closure compiler. Which can infer about 80% of all types.

</comment></comment></comment><comment><USER14> I would like to ask the question what we are actually trying to achieve with both types and with tests. The answer is to, in case of error, find the error as soon as possible instead of in production when the angry customer calls us. The sooner the error is found the smaller the problem is. If it is found in compilation it says 'you have an error on line 43'. If it is found in automated tested it says 'there is something fishy about functionality such- and-such'. Pushing forward finding the error from compile time to automated test time is a postponement and therefore not an improvement. Also, I really wish I could imagine encountering a production system with a test coverage approaching 100% but I have never been that lucky. In that case your type system is still protecting you but the tests that people failed to write are not doing very much.

</comment><comment><USER15> I highly recommend reading this tweet thread from Shriram Krishnamurthi on this subject: [https://twitter.com/ShriramKMurthi/status/113641175359047270...](https://twitter.com/ShriramKMurthi/status/1136411753590472707)

</comment><comment><USER16> I'd say there's also a big difference in the sort of correctness that static types guarantee vs the correctness that tests give you. Tests, to borrow some phrasing from Donald Rumsfeld, are only really effective against "known unknowns" not "unknown unknowns." Tests are only as thorough as the test writer and the "known unknown" cases they can come up with to test against. But as software developers we've all been in the situation where some case that you didn't think to test for is what winds up causing a bug (an "unknown unknown").

Static type systems on the other hand, can give you much greater guarantees of correctness (if they're sound). Due to the Curry-Howard isomorphism we know that programs in sound static type systems are the same as mathematical proofs. That's a much stronger guarantee than what you're given by testing alone. The problem of missing a case in your testing goes away if you have a mathematical proof that that case cannot occur. You've taken away some of the burden of thoroughness from the programmer and given it to the compiler instead.

In a perfect programming utopia we would all encode the desired properties of our programs in static types and have no need for testing because we'd have proofs of correctness. The problem of course is that writing those kinds of proofs into types is very time consuming and tedious and isn't realistic for most software development. So we still need tests to cover what is too expensive or complicated to type.

</comment><comment><USER17> I'm bewildered because every senior dev I've ever spoken to has been somewhat unanimous about the use of types and tests. Reading this I'm wondering if there's a special context where this proposed approach is more suitable? Possibly a single dev, tiny sealed program?

For every other system, be it with more than 1 developer or that will grow to more than 10s of functions, types and tests eliminate being anchored to the code you've written in the past.

The #1 be benefit to them is you can forget about what you've written. It's unlikely you'll remember in 6 months time that one function where you can't pass in negatives or Nil. Do you want to have to read the internals of every function you're calling? Isn't that a violation of some open/closed principal?

There is nothing sweeter than defining your intent in code contracts and confirming those contracts work before letting you or anyone else use them. If you don't want negatives being passed to your function, maybe unsigned integers aren't a good idea. Maybe make it explicit that you're sending in an 'age' value by defining it as a value object with its own validation. Make it blow up in the hands of the consumer as fast as possible. If you're waiting until runtime to tell you these things then you're not going to enjoy development.

And your peers will hate maintaining your junk code.

</comment><comment><USER18> The long-term benefit of typing doesn't show up in examples that fit in tweets: if the desired behavior is encoded in the types, the checks appear for free everywhere you use them, and you don't risk forgetting to test something in the n+1th function that you remembered to test in the nth.

</comment><comment><USER19> I really enjoyed this style of writing. It respectfully expressed why he disagreed with someone else, but acknowledged that both he and said person achieved their results and had tests which covered that fact. Nothing inflammatory or overly partisan, thanks :)

</comment><comment><USER20> Regarding the Haskell code, what about infinite lists?

1\. Why didn't QuickCheck test the infinite case?

2\. How do we write tests that fail (instead of hang) if an infinite list is mishandled?

It seems like infinite lists act somewhat analogously to nulls: rndSelect cannot handle them, and that fact is not obvious in its type.

<comment><USER21> > _How do we write tests that fail (instead of hang) if an infinite list is > mishandled?_

easy, just solve the halting problem!

<comment><USER20> Deciding type-safety of code also requires solving the halting problem, but haskell still tries!

I guess the question could be practical. Some functions are supposed to handle infinite lists: rfold, map, etc. How are these functions (or other, nontrivial ones) tested in practice, to ensure that they do not become accidentally strict?

<comment><USER21> > _Deciding type-safety of code also requires solving the halting problem, but > haskell still tries!_

does it? afaik typechecking is decidable for most type systems, and only becomes undecidable with something like dependent types (or c++ templates) but i could be wrong

</comment></comment></comment></comment><comment><USER22> I agree that you don't generally /need/ to test preconditions of internal code (although you should document them). However, if you're writing a library with an API that will be called by others, you should make sure your code fails fast, and you should assert that behavior in your tests.

Adding extra code to ensure you fail fast is often well worth it in future debugging time, even for internal code.

</comment><comment><USER23> This discussion is so played out.

[https://ro-che.info/ccc/17](https://ro-che.info/ccc/17)

<comment><USER24> I want to know what the people who are familiar with type theory but are not proponents of either static or dynamic type systems are up to. Did they move on to management? Switch to physics or painting? Retire?

<comment><USER25> Serious answer: type theory as a subject almost never discusses any idea like dynamic types (they wouldn't even be considered "types"). Presumably, if you only ever study it academically then you might actually have never even heard of dynamic types.

<comment><USER26> The founding act of denotational semantics was Scott's model of the untyped lambda calculus, which is surely the paradigmatic dynamically typed language. Moreover, its creation led to an immense body of work on type theory, because there are two two distinct, but related, intuitions underpinning what a type is.

On the one hand, you can think of starting with some given notion of computation (the lambda-calculus, say, or x86 machine code), and then you can think of types as picking out particular subsets of the set of valid programs. (In jargon, this is "types as retracts".)

In this view, a type is a property of a program. On the one hand, you can say that only things which pass type-checking are programs, and things which don't type check are simply not well-formed programs. In this view, a type is a kind of grammaticality constraint.

If you don't have a model of the untyped lambda calculus, you can't speak formally about the first view, which is very limiting. This is because being able to shift between the views is important for thinking flexibly about different kinds of problems.

Of course, you can also try to combine these two views. A few years back, Noam Zeilberger and Paul-Andre Mellies wrote a gorgeous paper, "Functors are Type Refinement Systems", which lays out the principles of how these two views interact.

[http://noamz.org/papers/funts.pdf](http://noamz.org/papers/funts.pdf)

<comment><USER25> I hear what you're saying---and appreciate the references---but I guess what I intended to discuss above was the "practice of dynamic types/tags" as is carried out in software development.

LC is clearly "untyped" in that it doesn't carry any type system, but that's a distinct idea from "dynamically typed introspection". That's an engineering system.

So maybe my error is to read into "dynamic types" as something more than the most utilitarian reading, but I also think that's usually what the "debate" ends up being about.

</comment></comment></comment></comment></comment><comment><USER27> If you're writing a standard library (or similar) then you do want tests to check that runtime errors are reported correctly with a nice error message, because you that is what the customers of your library see.

Within a single app of reasonable size, this isn't so important.

</comment><comment><USER28> >Static typing is an attempt to make software more mathematical. Type correctness is deductive and provable. However, type correctness does not imply behavioral correctness. Even when fully type correct the behavior must be demonstrated empirically.

I think this is the crux of the problem and my experience agrees on this statement.

At the end of the day, its Behaviour that matters the most.

</comment><comment><USER29> _What about invalid arguments? What if someone calls: (random-elements -23 nil)? Should I write tests for those cases?_

 _The function already handles negatives by returning an empty list for any count less than one. This isn't tested; but the code is pretty clear. In the case of the nil, an exception will be thrown. That's OK with me. This is a dynamically typed language. You get exceptions when you mess up the types._

 _Is that a risk? Sure, but only if some other part of the system was written without tests. If you call this function from a module that you wrote with tests, using something as effective as the TDD discipline, you won't be passing nil or negative numbers, or any other form of invalid arguments. So I'm not going to worry about that. It's on you._

This attitude coming from an experienced software developer is hard to fathom. Sure, if you're an experienced and meticulous developer that fully embraces TDD and you only work with other experienced and meticulous developers that full embrace TDD on the same type of software handling the same types of problems over and over, then sure, you might be able dodge a pretty high percentage of bugs a pretty high percentage of the time.

But most teams in the real world have a mix of inexperienced and experienced developers, meticulous and careless developers, developers that embrace TDD to varying degrees, and work on a variety of problems in a mix of domains with which the team members have varying degrees of experience.

So now your choices are:

1) code each and every function in a way that gives you high confidence that it works for the happy path and rely on the discipline of your team to ensure that no unhappy paths are hit

or, for approximately the same level of effort:

2) code each and every function in a way that gives you high confidence that it works for the happy path and which you have an extremely high degree of confidence that no unhappy paths even exist at all

It's just not a tough decision for me.

<comment><USER30> What stands out for me is the casual acceptance of this behavior for negative parameters. It returns an empty list so it's all good? No way! Asking for a list of negative length is a conceptual error. This should at least be an assert. That ensures nobody accidentally starts relying on bad behavior by prohibiting it. Of course, if you can use the type system to do this by using an unsigned type (and one without implicit non-failing conversions from signed types such as C has) then so much the better.

I think the author has the foundation of a good idea: don't write tests for things that aren't supposed to happen. (This is not the same as writing tests for error handling: errors are supposed to happen!) But he builds the wrong thing on top. You don't just ignore things that aren't supposed to happen and hope that they don't. You _make sure_ they don't happen, either by asserting at runtime or making them logically impossible at compile time.

Once you reach that point, then you see that a type is just an assert that's nicer to write, checked earlier, and is more comprehensive. The tradeoff is that they are less expressive, so you still need traditional asserts sometimes.

<comment><USER31> > You make sure they don't happen, either by asserting at runtime or making > them logically impossible at compile time.

One thing I find common with arguments from people like the author is that they don't consider types can be more powerful than the ones they're used to dealing with in the languages they use. It seems impossible to them that one could simply encode the parameter as being a type which _does not allow_ for negative values or `nil`. As you say, asserting at compile time that the program is correct.

<comment><USER30> I guess this is a specific example of the Blub Paradox.

I saw this a lot when Swift came out. Suddenly a lot of Objective-C programmers were learning a new language with a fairly different type system, and it was tough. The Optional type was an especially strong pain point.

What I found particularly interesting is that C (and Objective-C) _has_ optional types. All pointer types are optionals. What it lacks is optional non-pointer types, and _non-optional_ pointer types. Because optionality is an inherent part of pointer types in the language, you never learn to think about optionality separately. It's like a fish in water: it's so common you don't even see it.

I suppose it would show up even more strongly with someone used to a language like Ruby. If C programmers have a hard time really grasping the idea of optionality because the language doesn't make it explicit, then a Ruby programmer might have trouble with static types at all. (And, to be fair, the same no doubt applies in the other direction: someone coming from a strong statically typed language probably has some hurdles to understanding dynamic type systems.)

</comment></comment></comment></comment><comment><USER32> "My answer to that is that even using a dynamically typed language I don't have to write the test for nil because I know that nil will never be passed. I don't need to make states unrepresentable, if I know those states will never be represented."

You _don 't_ know that nil or whatever "won't ever be passed" in dynamically types language!

The fact that Bob Martin, a supposedly experienced software developer, genuinely, sincerely argues a variation of "I don't need to handle this case, since I _just know_ that this can't happen because... Obviously no one, certainly not me would do that" is inconceivable.

Does he not even consider what happens when someone else, i.e. _no him_ , passes "unrepresented" values to his function? WTH?

<comment><USER33> How a consultant could ever have this view baffles me.

</comment></comment><comment><USER34> There is a misguided notion on testing. Testing cannot establish correctness on a program unless you test every possible input and output. A test only proves a specific test case is correct, that's it, this is not actual program correctness.

Type checking is an actual proof that is light years ahead of a test in the fact that you need zero test cases to prove your program is correct in terms of types.

To achieve the same affect with testing you must test your function with every possible input for every possible type. An untyped addition function must test for what happens when you give that function a pointer to every possible permutation of a nested hashmap of strings and arrays otherwise you are not establishing correctness. What you are actually doing is taking a statistical sample of random inputs and guessing that the function is correct based off of that sample.

For greater predictability the test samples should be randomized following statistical procedures but this is rarely followed as most testers don't actually understand that they are taking a small statistical sample of a domain that covers an almost unlimited amount of possible inputs.

Tests are essentially selected with high programmer bias so a lot of bugs make it through.</comment>


SLAC results disagree with the Standard Model - <AUTHOR> http://arxiv.org/abs/1205.5442

<comment><USER2> To whomever changed the title to eliminate the phrase "new physics": that wasn't in any way sensational or editorial. This is a common phrase within the physics community. From the conclusions part of the paper:

>The results presented here disagree with the SM at the 3.4σ level. Together with the measurements by the Belle Collaboration, and the sizable difference between the measured and predicted branching fraction of B− → τ−ντ [35–39], _this could be an indication of new physics processes_ affecting decays with a τ lepton in the final state.

</comment><comment><USER3> 3.4 sigmas aren't enough. Call when you get to 5.

[http://scienceblogs.com/startswithabang/2011/12/13/the- large...](http://scienceblogs.com/startswithabang/2011/12/13/the-large-hadron- collider-the/)

and...

<http://trap.it/47HCqt>

<comment><USER4> 3.4 sigmas is the _within_ -model uncertainty - that is, given the assumptions about their instruments and methodologies, the result stands at 3.4 sigma, or about 1/1000.

I think that the uncertainty _about_ the model and assumptions is already larger than 1/1000, especially given the recent failed "faster-than-light neutrinos" LHC experiment. So while 5 sigmas would be nice, I would assume that the physicists over there are busy double-checking everything, and trying to reproduce the result, rather than improve their "sigma-score".

Can any physicist confirm? I'm not all that familiar with how these things work. Furthermore, if they perform once more the experiment and get the same 3-sigma result, wouldn't that compound into a >3-sigma total?

<comment><AUTHOR> The faster-than-light neutrinos probably got explained by a loose cable, sorry. :) [http://arstechnica.com/science/2012/02/faster-than-light- neu...](http://arstechnica.com/science/2012/02/faster-than-light-neutrino- result-apparently-a-mistake-due-to-loose-cable/)

As to the double-checking, the analysis behind this result used the whole data set collected by the BaBar experiment at the Stanford Linear Accelerator (SLAC). So that's years of data, and the experiment hasn't been running since 2008. The question of course is, can some other experiment produce the same results. As this is B-physics I would guess LHCb, one of the 4 main experiments at CERN's LHC, might be able to do this, but I really don't know for sure.

If you can repeat the same experiment and get another 3 sigma deviation from the Standard Model predictions then yes, you could combine the results to get a >3 sigma total (though they don't simply just add up to 6).

<comment><USER4> _The faster-than-light neutrinos probably got explained by a loose cable, sorry. :)_

Right:) That's exactly what I was referring to: the uncertainty about the model/experimental setup is larger than the N-sigmas uncertainty reported by the model and instruments.

<comment><USER5> Except the scientists behind the loose cable up front said "we don't understand these these results, we think they are false, please help us figure out why". At no point were they published for legitimate consideration.

</comment></comment></comment></comment></comment><comment><USER6> The idea of "New Physics" is sensational. There's only improvements to current models, and maybe new models for new situations. The Standard Model has worked out for too many situations for it to be just "wrong".

<comment><USER7> I would say that Relativity, and Quantum Mechanics, were 'new physics' over the old Newtonian model, although for many, many, many problems, Newtonian mechanics are still the best model. If this finally has found a major flaw in the Standard Model, I consider that extremely exciting news.

<comment><USER8> Relativity & Quantum Mechanics were massive paradigm shifts. In contrast, any flaw in the Standard Model, while exciting, will likely be orders of magnitude smaller in terms of impacting how we view the world.

Also we already know the Standard Model is not the complete picture (it cannot integrate QM & General Relativity).</comment></comment></comment>


A Recipe for Training Neural Networks - <AUTHOR> https://karpathy.github.io/2019/04/25/recipe/ <comment><USER2> I usually try a technique Andrej didn't mention here which helps me a lot in the debugging and modelling phase : Simulated data that encompass a single key difficulty of the problem.

For example in this line of thought in question answering, there is the BABI dataset which create auxiliary problems. So you know where the problems in modelisation are.

By pushing this problem to the extreme (for example in nlp you can have tasks that consist of repeating a sequence of character in reverse order to demonstrate that the architecture is indeed capable of memorizing like a parrot), you can often create trivial problems, which takes minutes to run on a single machine, and help discover most bugs.

You can also create hierarchies of such problems, so you know in which order you have to tackle them. And you can build sub-modules and reuse them.

Quite often the code you obtain then is very explainable and you know what situation will work and what will probably not work. But this network architecture is usually "verbose" and numerically optimize a little less well on large scale problems. The trick is then to simplify your network mathematically into something that is more linear and more general. You can reorder some operations like summing along a different dimension first. Semantically this will be different but will converge better. Because for a network to optimize well it needs to work well in both the forward direction and the backward direction so that the gradient flows well.

Once you have a set of simple problems that encompass your general problem, a good solution architecture is usually a more general mixture of the model architecture of the simple problems.

<comment><USER3> This is great, like TDD for ML!

</comment></comment><comment><USER4> This might be the most "Deep Learning" thing I've ever read:

> _One time I accidentally left a model training during the winter break and > when I got back in January it was SOTA ("state of the art")._

<comment><USER5> Hope they're not using a cloud instance - that sounds incredibly expensive!

</comment></comment><comment><USER6> >The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data. This step is critical.

This can't be overstated. I can't count the number of times I'm the first person to find a problem with the data. It's incredibly frustrating. Just look at your damned data to sanity check it and understand what's going on. Do the same with your model outputs. Don't just look at aggregates. Look at individual instances. Lots of them.

<comment><USER7> I was guilty of this once and would add some more specific advice; if your dataset consists of multiple possible labels for the same samples, do not just assume that the average of these labels will describe the best label. And don't assume that training with the unclean data will produce a net that magically learns to do the aggregation for you.

</comment></comment><comment><USER8> Well, this was unexpectedly excellent.

I don't think "stick with supervised learning" is very good advice, though. Unsupervised techniques sometimes work well for NLP and has worked well for other domains, such as medical records[1]. In particular, anytime you have access to much more unlabeled data than labeled data, it should be something you should at least consider.

[1]: [https://www.nature.com/articles/srep26094](https://www.nature.com/articles/srep26094)

<comment><USER9> Why "unexpectedly"? Karpathy has weapons grade knack for explaining complex subjects in plain terms. Case in point: [http://karpathy.github.io/2016/05/31/rl/](http://karpathy.github.io/2016/05/31/rl/) explains RL in a way even a non-practitioner will have little trouble understanding. Another prominent person with this skill is Chris Olah, one of the people behind Distill.

<comment><USER8> Not knocking on this author at all... It's just that nowadays if I see a title in the vein of "7 Tips to Train Deep Neural Nets for Complete Beginners On Rails With Keras and TensorFlow" I click on it more out of a sense of obligation than anything but I don't go in with very high expectations. So I was pleasantly surprised to find this article was substantive and high quality.

</comment></comment></comment><comment><USER10> As a practitioner, I found myself nodding in agreement again and again and again.

This blog post is full of the kind of real-world knowledge and how-to details that are not taught in books and often take endless hours to learn the hard way.

If you're interested in deep learning, do yourself a favor and go read this.

It is worth its weight in gold.

<comment><USER11> I agree. I have been using machine learning since the 1980s and deep learning for the last 4 years. This is great advice that I have both bookmarked and made into a PDF to store away in my searchable collection of research material.

Karpathy is amazing. I have had so much 'mileage' on two projects out of his unreasonable effectiveness of RNNs article.

</comment></comment><comment><USER12> Andrej is the guy doing Telsa's neural network for their FSD hardware. I truly appreciated his talk during the autonomy reveal the other day.

<comment><USER13> I have the opposite opinion. I dislike marketing wankery and investor bullshit, however entertaining it may be. Instead, I tremendously enjoyed Karpathy's Stanford class and lecture videos.

<comment><USER14> I thought Karpathy's contribution to the presentation was an excellent technical summary of what Tesla is trying to do. It clarified a lot of things for me what they're doing with Autopilot. Up until a few days ago my attempts at scrutinizing autopilot have been limited to little snippets of information here and there, rumours, and guesswork.

Bullshitting and wankery doesn't come naturally to Karpathy so the few spots where he was under pressure to do as much stood out like a sore thumb.

</comment></comment></comment><comment><USER15> >>> There is a large number of fancy bayesian hyper-parameter optimization toolboxes around and a few of my friends have also reported success with them, but my personal experience is that the state of the art approach to exploring a nice and wide space of models and hyperparameters is to use an intern :). Just kidding.

LOL. Human assisted training at scale is perfectly allowable for mission critical success. Especially if you enjoy an unlimited research budget!

You can follow these instructions to the letter. And the same problems around generalization will arise. It's foundational.

For 30fps camera images, handling new data in real time works fine for 99% of scenarios. But seeking usable convergence rates on petascale sized data problems such as NVidia's recent work on Deep Learning for fusion reaction container design requires a breakthrough. Not just in software. But computation architectures as well.

Deep Reinforcement Learning and the Deadly Triad

[https://arxiv.org/pdf/1812.02648.pdf](https://arxiv.org/pdf/1812.02648.pdf)

Identifying and Understanding Deep Learning Phenomena

[http://deep-phenomena.org/](http://deep-phenomena.org/)

</comment><comment><USER16> """If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization."""

Can someone translate this to PyTorch for me? Or give a simple example of how one would go about doing this?

It means, that if I have a 1:10 ratio in the data, an untrained net should predict positive in 10% of the cases, right?

</comment><comment><USER17> I am bookmarking this article, this is pure gold.

Also, it seems to me that most of what he says can be distilled into a boilerplate/template structure for any given deep learning framework, from which new projects can be forked - does this already exist?

<comment><USER18> Yes, it's called [http://fast.ai](http://fast.ai)

<comment><USER19> lol, when he wrote

>> model = SuperCrossValidator(SuperDuper.fit, your_data, ResNet50, SGDOptimizer)

under "Neural net training is a leaky abstraction" my first thought was, this IS fastai's API

</comment></comment></comment><comment><USER20> For anyone learning to build and train neural nets, this is a fantastic cheat sheet; Andrej is top-notch at explaining these kinds of things. The other posts on this blog are definitely worth a read as well!

<comment><USER21> A lot of this process goes beyond NN into generic ML. Especially understanding and diving into the data.

<comment><USER20> Yep agreed!

</comment></comment></comment><comment><USER22> "though NLP seems to be doing pretty well with BERT and friends these days, quite likely owing to the more deliberate nature of text, and a higher signal to noise ratio"

What is he talking about here? BERT, GPT etc are not unsupervised, they are pretrained on a task that has naturally supervised data (language modelling).

</comment><comment><USER23> In the blog, he refers to test losses at an early stage, like in "add significant digits to your eval". Does he actually refer to the test data or is he referring to validation data? I was under the idea that we were supposed to touch the test data only once at the end of all training and validation. What is the right way to handle the test data?

<comment><USER24> By "eval" you can also mean the training subset. As I understood is at the code to evaluate the network at a given point with a given dataset. For instance, after epoch epoch, the model is evaluated for both training and validation (you see both losses)

As you said, the test subset should only be used at the very last.

</comment></comment><comment><USER25> This recipe results in large amounts of time spent before any results occur (depending on the task you are trying to solve). Classification is an easy task to use this recipe, but when you venture into object detection or pose estimation, data collection, labeling, and setting up training and evaluation infrastructure is much more complex.

<comment><USER26> Can you expand a little bit? I often find if I skip one or more steps mentioned here, the later debugging is tremendously harder (and often involves go back to these steps again). Some of these advice like visualization are well supported in many frameworks usually through TensorBoard. Others really just good common-sense try-first-or-you-will-regret-later steps that don't require significant amount of time investment.

</comment></comment><comment><USER27> What a fantastic post, thank you for this.</comment>


What to do when a product that is a 'need' fails to pick up - <AUTHOR>

I am the co-founder of zonengage, a location based social network. You can find like minded people near you. Find Tennis, Soccer, Volleyball, Poker, Bowling, German, Italian and Mandarin Speaking members near you... So we launched in NYC and Chicago.
I have worked on this start up from 12 months now. Site is LIVE and we hit 1k users within 3 months. But we had 10k hits on our landing page. Which clearly meant UI UX is poor.
As we prepare to launch phase 2, which involves a complete UI change, I have a question -
Can a poor UI / UX be the reason for a website not picking up? I mean, we are connecting neighbors and solving a problem that exists in almost every neighborhood, yet, I am perplexed by not amassing many users. Any view points are welcome.. <comment><USER2> Getting 1k users from 10k visits is actually quite a good signup rate, but only getting 10k visits over 3 months is probably more likely the problem. This is the first I have heard of you, probably because I am in SF. This LBS space is crowded, so working on getting more attention and aware for what you are doing is crucial. At a 10% conversion rate don't worry about converting everyone who visits the site, worry about getting more visitors.

</comment><comment><USER3> Probably because there are dozens of other sites that do the same thing.

Here's a recent discussion about one of them. The feedback there could probably help you too. <http://news.ycombinator.com/item?id=5271266>

Good luck!

<comment><AUTHOR> Thanks. Yes there are several sites doing the same thing - but still we are providing the best package of all, well thats what we think. At the end of the day, except in metro cities, new meet ups are just not happening. Which means there is scope in this concept. Just not sure what we are missing! Anyway hopefully someday I return here with the answer :)</comment></comment>


Show HN: SSH Permit A38 – Central Managment and Deployment for SSH Keys - <AUTHOR> https://github.com/ierror/ssh-permit-a38 <comment><USER2> Very interesting project, I've been confronted with this kind of problem (managing SSH access for a large number of users) several times and don't think there's a perfect solution for it yet.

An alternative (but also imperfect) solution that I've relied on in the past is using certificate-based authentication:

[https://www.digitalocean.com/community/tutorials/how-to- crea...](https://www.digitalocean.com/community/tutorials/how-to-create-an- ssh-ca-to-validate-hosts-and-clients-with-ubuntu)

This still requires the generation of certificates for users but it doesn't require updating the key material on the servers themselves. Having short- lived user certificates then gives the admin an easy way to revoke access to a server without changing any keys there by simply not issuing a new certificate to the user (and in urgent cases to also revoke the certificate as well before it expires, which requires intervention on the server though).

I really hope that OpenSSH keeps improving their (still largely incomplete) PKI implementation, as I think it's a great feature for larger organizations.

<comment><USER3> FreeIPA - I wish more people knew about this. You can tie a public SSH key to a user (users can also self-register them) and it is automatically recognized on all hosts joined to the IPA domain, if you want to limit who has access to what the integrated RBAC facilities are there to handle that as well.

<comment><USER4> Cool! Thanks for the tip! :)

</comment></comment></comment><comment><USER5> Genuine question: what functionality does this provide that e.g. Ansible doesn't?

With Ansible, one can put the SSH keys into a .yml file in a format that is very similar to the examples in the OP, then the authorized_key module can be used to ensure that the key is present (or absent) on the remote servers. It's really-really trivial. Maybe the difference is that you can paste the SSH key into the CLI instead of a file... hm.

<comment><USER6> Can you post an example of this? Link works too.

<comment><USER7> Relevant documentation: [http://docs.ansible.com/ansible/latest/authorized_key_module...](http://docs.ansible.com/ansible/latest/authorized_key_module.html)

</comment></comment></comment><comment><USER8> What's the advantage to this over setting up an SSH CA?

[https://code.facebook.com/posts/365787980419535/scalable- and...](https://code.facebook.com/posts/365787980419535/scalable-and-secure- access-with-ssh/)

If you're in AWS, you can also look at Bless, which is Lambda-hosted and mints short-lived certificates with a command-line client:

[https://github.com/Netflix/bless](https://github.com/Netflix/bless)

<comment><USER9> If you're in AWS, wouldn't SSM agents on instances be preferable to SSH access? That provides for both access control (IAM for user access, SSM documents for constraining command execution authority) and auditing/logging of executed commands (CloudTrail).

This does not work for interactive terminal use cases, but does work (in my experience) if you're targeting immutable instances. It also has the lovely side effect that you can create scheduled tasks within the AWS control plane (if that's your cup of tea).

Example SSM client: [https://github.com/itsdalmo/ssm- sh](https://github.com/itsdalmo/ssm-sh)

Disclaimer: I'm implementing this in a large enterprise environment.

</comment></comment><comment><USER10> Slightly off-topic: I've seen a few references to the Asterix A38 scene in open source projects recently and it always seems to be a sure sign the developers are german. Is this actually a german-only thing?

<comment><USER11> It resonates highly with the kind of bureaucracy that Germans have to put up with.

Therefore it wouldn't surprise me if this scene is most popular with Germans and others nationalities only regard it as a funny, exaggerated sequence. Whereas for Germans it hits close to home.

</comment></comment><comment><USER12> All very nice but according to Circular B 65 you will also need Permit A39.

</comment><comment><USER13> Fun fact: there is a ship by this name: [https://www.a38.hu/en/](https://www.a38.hu/en/)

<comment><USER14> Not sure if the ship was named after it as well, but Permit A38 references a scene from an Asterix and Obelix animated film ([https://en.wiktionary.org/wiki/Passierschein_A38;](https://en.wiktionary.org/wiki/Passierschein_A38;) [https://www.youtube.com/watch?v=GI5kwSap9Ug](https://www.youtube.com/watch?v=GI5kwSap9Ug)). The comic series by René Goscinni and animated films were very popular in Europe, probably not so much in the US.

Permit A38 refers to a scene where the protagonists are referred multiple times within a overly beaurocratic Roman administrative office, so it has become sort of synonymous with a Sysiphean task in German language (at least in limited circles).</comment></comment>


How Green Is the Cloud? - <AUTHOR> http://www.wired.com/cloudline/2012/03/cloud-carbon/ How Green Is the Cloud? <comment><USER2> Ugh woefully inadequate article.

On a macro level the "cloud" represents a tremendous opportunity to reduce greenhouse emissions by consolidating many inefficient datacenters into fewer VERY efficient datacenters. The average 100+ person firm today probably runs their own datacenter, usually in a converted closet, floor, or other section of a commercial building. These locations cannot come anywhere close to matching the electrical and cooling efficiencies of large, professional datacenters. Secondly, by moving to "cloud" you can transfer electrical and cooling from high-cost areas to others. For example, a company located in the US Southwest would certainly reduce their carbon footprint by eliminating their datacenter and moving to a provider with a datacenter located in a cooler locale, preferably one powered by renewable resources.

Then you get additional environmental savings by co-locating many tenants on a set of shared or virtualized hardware vs each company running their own underutilized primary and hot backup servers. Add in storage arrays, networking gear, UPS, generators, etc. and it's quite a difference.</comment>


Viewpoint Toward a Computer for Visual Thinkers (1988) - <AUTHOR> http://www.scottkim.com.previewc40.carrierzone.com/viewpoint/index.html <comment><USER2> I was having trouble with the PDF link, here's an Internet Archive copy: [https://web.archive.org/web/20191014194206/http://www.scottk...](https://web.archive.org/web/20191014194206/http://www.scottkim.com.previewc40.carrierzone.com/viewpoint/viewpoint- dissertation.pdf)

The video is very good at getting the point across, though (naturally).

Also of some interest to this audience might be Raeder's 1984 dissertation, which shows a combination of visual programming, functional programming, and programming by demonstration. A big part of his model is naming things as rarely as possible: [https://archive.org/details/RaederProgrammingInPictures](https://archive.org/details/RaederProgrammingInPictures)

</comment><comment> dawg- Fascinating. I admit I only read the intro section but bookmarked for later. The page "on being interdisciplinary" with a kind of taxonomy of interdisciplinary work is really interesting.

I wonder what the author would have to say about the possibilities of "Spatial Computing" with AR that seems to be the next big buzz? It seems like spatial computing is all about manipulating data in a highly visual format.

</comment><comment><USER3> Well, that's time I'm not getting back.

edit: to be more specific, this is very bare. add "visual" representations for the concepts that ultimately need to be expressed--verbs, prepositions, etc-- then we're back to linguistics, but with ad-hoc pictographs.</comment>


Leveraging S3 as a CDN in a modern web application build - <AUTHOR> https://medium.com/@deanslamajr/a-webpack-built-elastic-beanstalk-deployment-leveraging-aws-s3-as-a-cdn-54d62b851263 <comment><USER2> This is a very confusing article. It begins:

        Web applications typically begin their life serving their
        own static assets(javascript, css, etc) to the browser. An
        alternative to this strategy would have these assets be
        hosted by a highly-available content delivery network or
        CDN (e.g. Amazon Simple Storage Service (S3)).

S3 isn't a CDN. It's a storage service. And it has egress charges (per request and bandwidth). Using it as a CDN seems like madness. Especially when Amazon has a CDN called CloudFront. Or you could stick everything in S3 and use a third party, free CDN like Cloudflare. By all means store your assets in something like S3 but don't use it as a CDN. It's not.

<comment><AUTHOR> Yeah, you right! Thanks, I've updated the post

</comment></comment><comment><USER3> We're just using "web-scale" unironically now to describe things meant to operate on the scale of the world wide web. I guess this is progress?</comment>


Laser Damage on Image Sensor from Tattoo Removal Laser - <AUTHOR> https://www.youtube.com/watch?v=B3MzObRgaA0 <comment><USER2> For that brief moment when the laser flashes, it is pretty bright. Makes sense to me that it could damage a camera sensor, since it can probably do that to your retinas, too.

There's a reason why you're supposed to wear eye protection whenever you're working with certain types and classes of laser.</comment>


An Eye Tracking Study on CamelCase and under_score Identifier Styles [pdf] - <AUTHOR> http://www.cs.kent.edu/~jmaletic/papers/ICPC2010-CamelCaseUnderScoreClouds.pdf <comment><USER2> I knew it! I've hated the camelCase style with a passion from the day I first saw it. I'd be very curious to know if there's _any_ developer who didn't grow up on windows actually likes camel case style.</comment>


Ask HN: Your opinion on GWT vs raw JS - <AUTHOR>

Im considering starting a new web app and want to know more about GWT vs just Javascript (GWT compiles Java to Javascript). Does GWT offer productivity gains over just using JS? Any tips on using GWT? <comment><USER2> You've left out a third option: using a mature pure-JavaScript library like JQuery, Closure, or YUI.

IMHO that's the best of the options, because you can fall back to pure JS if you need extra flexibility or speed, and yet still don't have to deal with cross-browser bugs.

<comment><AUTHOR> ok yes I know about those libraries...I guess what Im asking is why use GWT when such libraries exist?

<comment><USER2> If you have a piece of algorithmic code that really needs to exist on both the server or the client, and maintaining two versions of it will be too much of a pain. That's why Wave uses it (the Wave team would probably give you other reasons too, I've heard they like it, but that's the only technically defensible one IMNSHO).

If you want to have the same code run on both the server and client, your only options (that I know of) are GWT, server-side JS like Rhino or V8 or Node.js, and Pyjamas. GWT's the most mature of those.

Also, a _social_ reason to use GWT (and perhaps the real reason Wave uses it) is if you know Java and don't want to learn JavaScript. This doesn't hold much water with me - what sort of a programmer wouldn't want to learn a new language? and JS is a pretty nice one all things considered - but it's convincing for lots of other folks.

<comment><USER3> This.

One of my developers is crazy about GWT, and has shoe-horned it into a number of apps that he's primed, and it's proven to be problematic for a number of reasons.

The main problem we have is that a lot of our apps fit into a larger portal, so we have to worry a LOT more about JavaScript and CSS collisions, and generated JS/CSS just doesn't work well for that. Outside of a portalized environment, it might not be THAT bad, but for us, it's proven painful.

In the event that you find yourself duplicating your server side code and client side code, or writing excessive form validation to match server-side constraints, something like GWT is probably the way to go.

</comment></comment></comment></comment><comment><USER4> Disclaimer: I've never used GWT professionally before.

I think its a good idea if your web application is driven by the JS instead of just using it to add interactivity. As in, something like Gmail or wave which are basically "javascript applications" rather than "web applications", if you know what i mean.

JS can get a bit annoying when trying to impose the needed structure to write big applications. That structure is "built-in" when using something like Java.

Also, with a JS-heavy application, you'd need to take care of the different browser quirks(via library or otherwise). Apparently, GWT can take care of this for you.

Having said that, I don't typically like "generated" code. At least not code that I don't generate myself.

<comment><AUTHOR> thanks. what development tools do you use to develop JS applications?

<comment><USER4> Oh just the typical stuff. I'm old school so...

Your favourite text editor(I use vim) or you can be a bit more fancy and go with something like Netbeans or Eclipse(although I always go back to an editor). And firebug for debugging.

You can add a bunch of stuff to the mix, but that's all I need really.

(Oh, you might also want to throw in a mini-fier for when you're done.)

</comment></comment></comment><comment><USER5> If you use GWT integrate it nicely into your build process. Worked at a place where it was a bit klunky whenever you made a change to the GWT code and subsequently GWT has left a bad taste in my mouth but I think that's probably unfair.

If we'd taken the time, we could have ironed out the klunkiness.

Might be worth building a vertical slice of you app, first with jquery (or another JS library) and then with GWT and see which one suits it better.

</comment><comment><USER6> I love my JQuery</comment>


A list of PostgreSQL libraries, tools and resources - <AUTHOR> https://github.com/dhamaniasad/awesome-postgres <comment><USER2> What is the best way to ETL data from one Postgres cluster to another (by ETL I mean things like data sanitization and denormalization)?

a) I think most people end up doing some kind of batch synchronization, but I'm interested in streaming solutions.

b) A lot of folks use trigger based replication, but triggers have to be on the primary/master node, and not just on the replicas.

c) Another common solution is to force the database client to write to a message broker, but that opens the door to data discrepancies and synchronization issues.

d) In theory I think the best way is to do something like bottledwater-pg or pg_kafka [1] [2] [3], but I'm not sure how battle hardened these are. I think logical replication of the WAL is the right approach, but there is still not much tooling around this.

[1] [https://github.com/confluentinc/bottledwater- pg](https://github.com/confluentinc/bottledwater-pg)

[2] [https://github.com/xstevens/pg_kafka](https://github.com/xstevens/pg_kafka)

[3] [https://github.com/xstevens/decoderbufs](https://github.com/xstevens/decoderbufs)

PS: There are a bunch of interesting MySQL solutions out there, such as Zendesk's Maxwell:

[https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)

<comment><USER3> Google "Postgres BDR". It is new, but it allows you to replicate changes on master to another master. With some configuration it can even change replicated data.

<comment><USER2> Thank you. I believe that bidirectional replication will solve the ETL problem, if it can be configured to stream from a replica in the source/oltp cluster (to minimize network IO on the source cluster master db) to the master of the destination/olap cluster. The olap master can mirror the oltp data, but also use triggers to denormalize oltp tables.

One thing I'd like to see is the ability to sanitize/munge data from the source cluster before it is sent to the destination cluster, for masking sensitive data in the source cluster (PCI, HIPAA, etc).

</comment></comment></comment><comment><USER4> With regards to psql2csv: The default psql can already do this very nicely. Just use \copy.

\copy (select whatever from whatever) to 'yourlocalfile.csv' with (format 'csv')

and if you want column headers, add a "header true" inside of the with clause.

Generally, \copy works just like COPY[1] but it does so from the remote server to the local machine, whereas file names given to COPY are relative to the server.

Yes. A dedicated tool might feel easier initially, but once you know how \copy works, you can always get a CSV file from whatever database you're connected to and no matter what machine you're on.

[1]: [http://www.postgresql.org/docs/current/static/sql- copy.html](http://www.postgresql.org/docs/current/static/sql-copy.html)

<comment><USER5> psql2csv is just a wrapper around what you're describing. The point is that you don't have to write the "boilerplate" when all you need is a single query. Plus, it works nice with stdin and stdout, making it handy for queries stored in a file, or for piping the CSV to another program.

</comment></comment><comment><USER6> I found this talk by Christophe Pettus [1] very informative. The title is somewhat misleading as most of the talk has little to do with Python, but it's a good introduction to more advanced Postgres concepts. Also available in PDF form [2].

[1] [https://www.youtube.com/watch?v=0uCxLCmzaG4](https://www.youtube.com/watch?v=0uCxLCmzaG4)

[2] [http://thebuild.com/presentations/pycon-2014-pppp.pdf](http://thebuild.com/presentations/pycon-2014-pppp.pdf)

</comment><comment><USER7> Does anyone know of sample PostgreSQL databases? Something akin to Northwind ([https://northwinddatabase.codeplex.com/](https://northwinddatabase.codeplex.com/))?

<comment><USER8> Not sure if this is what you mean but PostgreSQL 9.5 introduced TABLESAMPLE. [https://wiki.postgresql.org/wiki/What's_new_in_PostgreSQL_9....](https://wiki.postgresql.org/wiki/What's_new_in_PostgreSQL_9.5#TABLESAMPLE)

<comment><USER7> Sorry, no, I'm meaning example databases with 'fake' or sample data.

</comment></comment></comment><comment><USER9> Can't it just be a 'list' of Postgres stuff? Do we really expect that it's generated by an algorithm if someone doesn't write 'curated'? If it's a good list, I'm going to find it interesting no matter how it was created.

<comment><USER10> We took 'awesome' and 'curated' out the title, as per usual.

<comment><USER9> Thanks for your awesome curation of this site :-)

</comment></comment></comment><comment><USER11> Good work. I am a big fan of PostgreSQL. It has made life easier for me.

</comment><comment><USER12> Amazon RDS doesn't make the list of "hosted Postgres"?

</comment><comment><USER13> Is there a PostgreSQL that anyone here can recommend?

<comment><USER14> You can try Postgres "zero install" with Postgres.app for OSX, otherwise docker. [http://www.win-vector.com/blog/2016/02/databases-in- containe...](http://www.win-vector.com/blog/2016/02/databases-in-containers/)

<comment><USER13> Thank you! But I meant book, a PostgreSQL book; but somehow I didn't type that :(

<comment><USER15> The PostgreSQL manual [1] is pretty good IMO. Have you looked at that?

[1] [http://www.postgresql.org/docs/manuals](http://www.postgresql.org/docs/manuals)

<comment><USER16> The PostresSQL manual & documentation are one of my favourite things about PostresSQL, especially when I was first starting out :).</comment></comment></comment></comment></comment>


Ask HN: I will pay you $2000 and code the MVP for your side-project - <AUTHOR> Premise 1: Investors/Incubators over-estimate their ability to pick good ideas/startups.
Premise 2: Software by a lone developer is not pragmatically different from software by a big A-team. It's the market fit and marketing / sales that makes or breaks the project.
Premise 3: Most freelancers will not build and/or follow-through with their ideas, because they are not sure it will sell.
Premise 4: HackerNews has a decent number of people who know how the world works, and how a little glue would make it better.
Based on these premises I present The Proposition 2.0 (following version 1.0 [1]):
I'll pay you $500 for your "real-life problem that needs a software fix" idea and market validation / research. I will build the MVP and give you another $1500 to bring us our first paying customer(s). We split the resulting product 80-20 as co-founders.
[1] https://news.ycombinator.com/item?id=5037694 [2] https://www.simonnouwens.com/hn <comment><USER2> This doesn't make much sense. A 20% royalty is an absurdly high price to pay for an idea, for which you'd do all the execution; meanwhile, if you expect them to do substantial work as a "co-founder", 20% is absurdly too low.

<comment><USER3> The idea is probably to get a lot of pitches and then select the idea, no matter what it is, from whichever applicant has the most wealthy accredited investors as family.

</comment></comment><comment><USER4> I love building. I'd do an MVP for much less equity if I'd actually get it. But I won't. One of my MVPs turned into a $500M acquisition. Got $0.

This is normal in Silicon Valley. So many unicorns hiding skeletons of enthusiastic side-project coders.

I'd say don't do it, but... to be honest, I never could have hyped my own prototype the way they did. I didn't have the rich kid VC connections that they did. And I needed my day job. So really I lost nothing. At least I got the story.

<comment><USER5> Are you able to mention the company, either publicly or privately?

<comment><USER6> Whatever happened was probably entirely legal and probably ethical as well.

You can be a cofounder, and have ownership even if there is a lack of contracts and paperwork, and that likely would have come up during that $500M acquisition. I can't find it, but there was a Ycombinator - backed startup whose sale was blocked by a founder who went AWOL early.

If he was an employee, or did otherwise hand over / accept compensation for his work, well, that's just business as usual: employees generating wealth for owners.

The lesson is to have clarity of what you want out of a certain work, and then have some kind of contract, before you start working.

</comment></comment></comment><comment><USER7> This seems like a doubly losing proposition. If you created my idea and I only get to keep 20%, I wouldn't assist you very much in maintaining the business. If I were to partner with someone, I would resent shelling out 20% to an absent partner. Ideas aren't worth a fifth, and a fifth isn't a suitable carrot/stick.

<comment><AUTHOR> Thanks for your comment. I think 20% is a fair price for idea/validation/first sell and direct access to a customer space. If it's going to be a $100m company in 2 years then yes, 20% is going to hurt, but honestly I don't expect that.

<comment><USER8> Maybe not hurt if it was not dilutable, the initial 20% holder could be issued more shares to keep their percentage at 20%. By the time 3 or 4 rounds are raised the initial 80% owner might only retain 20% after dilution. So in the end you have to look at the cap table after future rounds, the fact that during a period of history (probably before there is any real cash to distribute) the split was 80/20 won't matter.

<comment><USER8> But more importantly, if I build the product and get the first customer what do I need you for? 2k is chicken shit. Unless the idea is a throwaway, something I'm not passionate about, I'd just build it myself.

<comment><USER9> Suppose you have more seemingly good ideas then you could possibly build yourself.

</comment></comment></comment></comment></comment><comment><USER10> How did the first[1] offer went?

[1] [https://news.ycombinator.com/item?id=5037694](https://news.ycombinator.com/item?id=5037694)

<comment><USER11> If he/she is back with the same pitch, it probably went well enough to try again.

<comment><USER12> Or maybe it didn't go well enough, and that's the reason for a second attempt. You can't say either way which is what makes the question worth asking and a potential answer extremely interesting. Though an actual answer, not just a deference to speculation.

</comment></comment></comment><comment><USER13> Premise 5: My expectation is that you don't have anything close to what I would expect -- especially for an 80% share -- so why should I waste my time with you?

<comment><AUTHOR> What do you expect from me?

</comment></comment><comment><USER14> Sounds good. Though part of being a cofounder is more than MVP, is there any vesting provision? Also not 100% clear who gets the 80%

<comment><USER15> On the link from [2] it says:

> _We split the resulting product 80-20 (me-you) as co-founders._

I found it confusing that there was no [2] in the body of the post, only the footnote list.

</comment></comment><comment><USER16> There is a lot to running a business beyond the first MVP and customer. Being a cofounder is as intense if not more intense than being married. I would almost flip it around so you take a 20% passive stake with an priority to a dividend or a royalty on revenue so that you aren't expected to manage multiple companies as a "founder".

I am assuming that you are trying to get multiple ideas off the ground.

<comment><AUTHOR> Multiple ideas - maybe, totally depends on what comes up. Though it would change the intention of the deal, flipping around the deal wouldn't be something I'd say no to right away. It's a proposition after all, not a ridged set of rules we have to follow :)

Got anything in mind?

<comment><USER16> I have a list of 40 ideas that I don't have time to build. Ideas are easy. MVPs seem easy in theory, iteration to find product market fit; that is hard. Execution and building a team after product market fit is harder. That's where most the work is.

All of this is why a brand new founder seems like a novice when they are very secretive about an idea or want an NDA; ideas are easy. In reality, executing on an idea depends on domain expertise, which most developers, investors, random people you tell your don't have, and know they don't have. And most people know following through on an idea is a lot of work — they aren't that interested in the hassle.

Paul Graham used to give ideas to founders for free.

I would suggest spending more time developing an analytical framework to identify problems. Don't think in terms of "ideas". Think I terms of a problem faced by an actual customer or user.

My favorite framework comes from Aaron Harris. He says you should find a customer that has their hair on fire because a prototype fire extinguisher would solve a very serious problem for them. They wont care how basic, ugly, broken, or unreliable your fire extinguisher is and won't care what price youre asking; they want to put out the fire.

<comment><USER17> Maybe the OP is trying to find problems, ideas that fit his unique skills and expertise to execute, and wants a little bit of help to validate.

</comment></comment></comment></comment><comment><USER18> So you'll pay $500 for an idea you like, pay $1500 after first customer, and then pay a 20% royalty after that. Seems like a potentially good deal.

Can you elaborate a bit more about what type of projects you are looking for? Any specific markets?

<comment><USER19> The idea is that for 20% you do all of the marketing/business side of things, as I understand it.

<comment><USER20> For 80%.

</comment></comment></comment><comment><USER21> I like how some people are complaining that he'll take 80%... for what amounts to just giving you an idea and the first client.

Usually we hear the opposite side on HN. "Why would VCs own so much if all they offer is money when we (programmers) do all the work".

I think people should take him up on it.

<comment><USER11> Giving up 80% for "an idea" is fine.

Giving up 80% for the first customer? That's a big ask.

Someone noteworthy said that 50% of building a business is going from 0 to 1 customer. The other half is going from 1 to many customers.

That being said, this person is assuming a lot of risk that the partner is flaky and doesn't follow through, that financing won't be available, etc etc.

In reality, I suspect they're probably building out a 4-6 week MVP, getting a couple of customers, then selling the business as a "turnkey startup" for ~$50k (of which they get 80%/$40k). Not a bad proposition.

<comment><USER22> _Giving up 80% for the first customer? That 's a big ask._

The customer only exists if the idea is turned into an actual product. Until then he's just a person who wishes his problem could be solved.

</comment></comment></comment><comment><USER23> For a second I thought I was reading through my post history when this came up!

<comment><AUTHOR> Thanks again for inspiring this in the first place :)

<comment><USER24> I'm a big fan of both efforts. Ideas are a dime a dozen (if that) and I like the unconventional and challenging approach that you both took. I'm rifling through my idea notepad to pitch my hat into the ring :)

</comment></comment></comment><comment><USER25> Out of curiosity, what was the outcome of The HN Proposition 1.0?

Edit: [http://www.simonnouwens.com/hn/index.php/2019/03/31/init- com...](http://www.simonnouwens.com/hn/index.php/2019/03/31/init-commit/) answers the question

<comment><USER26> "The idea gained some traction, and hmexx received over 250 emails, which were filtered down to 5 picks. Out of those 5, three were discontinued (1 after market research, 2 after failure to deliver MVP). The 2 ideas that were put online appear to be [http://parametrek.com](http://parametrek.com) (an experimental product search) and [http://www.textrep.com](http://www.textrep.com) (an open-house notification system). Both projects as well as the experiment ended up floating somewhere on the software-graveyard, which is a shame, as the idea is great and the premises are (mostly) correct. Besides that the proposition generated a lot of community interest, which ended up being unused by the project." \-- [http://www.simonnouwens.com/hn/index.php/2019/03/31/init- com...](http://www.simonnouwens.com/hn/index.php/2019/03/31/init-commit/)

<comment><USER27> I point people to the flashlight search on paranetrek all the time. I had no clue about the backstory.

If you want a new flashlight check out [http://flashlights.parametrek.com/](http://flashlights.parametrek.com/)

<comment><USER28> Everything he wrote about that was wrong.

I was never accepted into the original Proposition. I did not get any money or advice. But my elevator pitch was good enough to convince me to go through with it anyway. And I'm hardly "floating in the software graveyard."

But I am glad you enjoy my site ^_^

<comment><AUTHOR> Rectified this on my website, and apologies to parametrek for getting it wrong (and suggesting the project failed). Only found some sparse information online, and failed to do my homework properly after that. It's a pity hmexx didn't share too much of the results, would have been an interesting piece of information.

Sorry parametrek, didn't intend to bash your work - you obviously did better in the side-project business then I did so far :)

</comment></comment></comment></comment></comment><comment><USER29> Shit I'll pay you $10k and only keep 10%. Finding real life problems that (truly) require a (tenable) software fix are getting harder and harder to find.

<comment><USER30> I will take it ;)

Are u a FE coder by any chance? Or a marketing/growth guy. I am in backend/algo so any mobile/browser-related dependency becomes a blocker for me. Sigh.

B2C, paid product, well differentiated, plenty of demand.

Drop me a line at firstname.lastname@example.com (disposable, 24hrs, messages may be read by others)

<comment><USER29> Full stack. I've forgotten more frameworks than most care to ever learn.

<comment><USER30> Hey. Sounds great. I didnt see any incoming messages, so maybe it was deleted.

Please re-send contact details to email@example.com (this is a secure alias. Other people will not be able to see messages. )

</comment></comment></comment></comment><comment><USER31> Just to clarify who gets the bigger share of equity?

</comment><comment><USER32> This is perfect idea. Here's usecase.

At our startup, our main cashflow is reselling on Amazon. The current business model we are using shares a lot of similarity with other Amazon sellers. There are a lot of workflows that are cookie cutter across other sellers/companies that can be automated/improved with custom tools.

There's demand for what we need (automation tools, api tools to request specific materials from Amazon), we will even pay and be the first customer ourselves.

If we can get equity for you to build this tool, this is a no brainer idea!

You can reach out to me in my profile, love to setup a call to discuss a bit more on this.

<comment><USER33> What do you resell on Amazon and what kind of automation tools are you looking for?

We run a startup that helps brands to automate their ecommerce and we are focusing on Amazon. The challenge here is that for automation/AI we need a sizeable data and it's hard to get revenue with bad forecasts.

We have been on this for 2 years and our forecasting is improving along with revenues. You can drop me an email if you would like to chat further.

<comment><USER32> Hi Ganesh, I couldn't find a direct email from your profile.

We resell name brands on Amazon, we work with brands to resell their products on Amazon. One of the most often request is MAP violator takedowns, we need a way to monitor brand's listings on Amazon to find people who violate MAP pricing.

Another request is sales opportunity. Like what listings could use improvement, and what listings can use more adspend. With Amazon, you can reverse engineer a lot of this stuff, it's presenting it to the brand in a seamless manner that's the hard part. I'd be happy to discuss more. You can get my email in my profile.

<comment><USER33> Indeed we can use "reverse engineer" or ML models to get more ideas. I wrote about using Deep Learning to predict sales here: [https://towardsdatascience.com/predicting-amazon-sales- using...](https://towardsdatascience.com/predicting-amazon-sales-using-deep- learning-10aa2b237af5)

We work with sellers in a slightly different way: We are more focused on best geography and time to sell a product. In essence we predict how much you will sell in future and in what locations. I will drop you an email and we can talk further.

</comment></comment></comment></comment><comment><USER34> Further information here: [http://www.simonnouwens.com/hn/index.php/2019/03/31/init- com...](http://www.simonnouwens.com/hn/index.php/2019/03/31/init-commit/)

After reading that, I'm interested. Emailed you.

Also, to the person who asked about who gets bigger share of equity: as mentioned in the link above, it looks like you get 80%, the OP gets 20%.

Nevermind: It's OP 80%, you 20%.

</comment><comment><USER35> Short version:

I'll buy 80% of your idea for $500.

In case first customer(s) will ever come - i'll give you another $1,500.

Not clear: if no customer(s) will ever come after X days - who owns the idea?

<comment><AUTHOR> Thanks for your point. There are a lot of details which need to be discussed but depend on the idea, clients, product type, implementation etc. Ownership and exact targets being one of them.

</comment></comment><comment><USER36> This proposition's assumption is that there's something _special_ about ideas. Reminds me of the kind of person that might make you sign an NDA before telling you a "cool idea I had last night." It's no secret that YC -- like basically all incubators -- invests in _people_ , not _ideas_. Teams will sometimes fumble for years before they find product-market fit.

There are no shortcuts.

<comment><USER37> Great point. I give the author of this post props for trying this and it has been an entertaining proposition. But to me premise 2 is admitting there will be massive shortcuts taken here. There is now way one person could build a product that would scale as well as a product made by an entire team. You might be able to get it about 90% there, but the last 10% is the hardest IMHO.

In my experience this kind of rationale is what leads to MVPs being spun out quickly using hybrid tools which have a hard time scaling. Sure it kinda looks like the real deal, but the product will limit the growth potential of your company.

<comment><USER17> There are plenty of products/services built by a single person that only hired more people once they were able to afford it and needed it. This is especially common when access to VC funding is pretty much non-existent and you can't just hire a bunch of people.

</comment></comment></comment><comment><USER38> I posted a similar proposition [https://webuild.tools](https://webuild.tools) (build the software for free and retain the client as the first customer) I have time/interest to build side projects and built many too. But nothing takes off as it is never backed a solid business need. And I am no where close to solving this problem.

<comment><AUTHOR> Similar indeed, though I'd hope to bump into someone that is willing/able to sell it to their direct environment. Doing a free build/first customer deal seems too risky to me indeed (over-optimizing on what this single person needs). I suspect "premise 2" has to do with this.

I'm interested in hearing a bit more about your experience if you have some time. What kind of projects have you done so far? Nothing that ended up moving the needle a little? Even after further marketing/promotional efforts?

<comment><USER38> Thanks for asking. It's usually web apps built around some kind of networking needs. I built one where people could ask for recommendations from friends. Another one was a hybrid app where users can set a single status for themselves. Now I am working on an amazon reseller site that has curated products. I built another one where users can post their SaaS apps (kind of app store for web products). It usually dies within a few weeks because I am lost on how to promote/market these apps. And this goes on a loop. So I was looking for people with a proper business case. While I learn a lot building these apps, I figured it will be more useful if I can build it with some purpose.

<comment><USER32> can you drop your contact info in your profile so that we can contact you? We do amazon reselling, interested in learning more about the reseller site you are working on. Also have tools that we are looking to build ourselves (we will pay for), that other sellers like us are interested in paying to use as well.

</comment></comment></comment></comment><comment><USER39> Premise 1: I agree, many investors have more money than sense.

Premise 2: I completely agree; in fact I think that solo side projects are coded more efficiently than commercial software.

Premise 3: I disagree. I code most of my side projects, because I have too much free time. I just don't follow through on the documentation/uploading part. This may be because of market fit, or copyright worries, or because I like coding more than writing up. Nevertheless I do still add projects to Show HN, especially when trying to apply for jobs.

Premise 4: Yes, I agree, Hacker News is a wonderful community.

I'm not a marketer, so I can't provide your market validation/research. Rather, I'm a maker like you! But I have no shortage of ideas that I'll talk about for free. For example, my most recent script (27 March) was a rewrite of VideoSubFinder that runs in 5 minutes instead of 2 hours. It's all part of my Pingtype Chinese learning project, which still hasn't taken off, but I keep building on it because it teaches me so much better than textbooks.

</comment><comment><USER40> I run SwiftMVP.com -- We once tried offering half-price if people give up the white label and credit us. Apparently, $5000 are not the issue.

<comment><AUTHOR> That's a cool project! Mind sharing how you get your customers?

</comment></comment><comment><USER41> My idea is to build a digital giving economy. You get to keep 100% of everything given to you through the site. If successful, you can retire comfortably with little money while being held by the giving community and go forward in a life of service, if you so choose.

As the site produces no money, you can keep 100% of all revenue.

</comment><comment><USER42> I'm currently pondering how strange it is that I would take either side of this deal.

I have half baked ideas I'd love someone to take and run with and me collect a bit of that.

And I'd love to hear profitable ideas in industries I'm not familiar with and put some code together.

Yet I don't have an idea I'd want to build into a business myself. Weird.

<comment><AUTHOR> That's a great point, and something I've pondered over the last 2 days (I got the same impression).

Any idea why? Perhaps the hidden expectation that a collaboration like this magically will eliminate all my/your own barriers?

<comment><USER42> I think the social proof aspect is a big part of it.

</comment></comment></comment><comment><USER43> Here's the math: If OP can speed up the rate of hitting acquisition and/or increase valuation by >= 5 times, it's a good deal. Say if it would take you 20 years to do something, and this cuts it down to 4 years, it's a good deal.

</comment><comment><USER44> This proposition probably needs further refinement, but I like it because it will explore an area where there's little historical data or real world examples.

I think that the first few dozens of "pitches" will give a much clearer sense of what kind of ideas you can attract, and also what's the friction in the interaction between the "producer" and the "idea person".

Keep going. And please, if you can, share more once you've been through this for a while. Very curious to see where this will go.

</comment><comment><USER45> half the comments here think you're offering to do the mvp, the market research, and find the first customer, all for 2k+80%.

when i read it it looks like you're offering to do the mvp but you're buying the idea, market validation and first customer for 20%.

which is it?? the first option makes sense, but its not how i read it. if thats what you mean you should clarify.

if in fact you are just offering the mvp, you are _terribly undervaluing_ the work involved in validating the idea. the 80-20 split is backwards.

</comment><comment><USER46> This is a bad great idea!

With a little tweaking of the numbers, I think I would gladly accept a similar offer if it came from a trusted friend.

</comment><comment><USER47> I like this idea; I had a company based on this premise (we took 20, owner took 80 and we did the dev and seed invest); I would like to go back to that now that I have a much larger network of investors and business development partners. I am looking for cofounding full stack devs to join me as I am now much more on the business side.

<comment><AUTHOR> Care to explain? :) What kind of business did you have? Did it work out? And what's the problem in doing it again?

<comment><USER47> The problem was that programming comes before marketing & business. I mean there is an idea and a plan ofcourse and you try to validate that as much as you can, but basically, after that, the programming/development phase kicks in, which was what we did for our %. When the business person/inventor/cofounder(s) after that said; 'oh we tried to market but failed, ah well, shame' then you are stuck with 20% of nothing while you did the work already. Now that I have far more business and legal sense, I don't think there is much of an issue doing it again.

</comment></comment></comment><comment><USER48> Anyone in the u.s. doing this? I'd be willing to give up 50% equity and no cash for what I think is an awesome idea.

</comment><comment><USER49> > HackerNews has a decent number of people who know how the world works, and > how a little glue would make it better.

Wait what

</comment><comment><USER50> ! I have a couple Ideas both a wedding subcontractor platform...and mindfullness reminder app.

As for the wedding platform , I even have the people who would be available to signup as sub-contractors.

Venues could signup we get a percentage for advertisement we handle all payment processing, hmu if you are interested working together.

<comment><AUTHOR> Sounds like something that might have potential! (the subcontractor idea, the mindfulness reminder is something you can't really monetize and most likely exists already)

Any experience in the wedding subcontractor field? I'd love to discuss some idea's if you like (email in profile)!

</comment></comment><comment><USER35> I had this idea to build social network where people could share photos with friends.

Where I can pick up my $500?

<comment><AUTHOR> Let's use ad's and stuff! It's going to be huuuuge!

<comment><USER51> Only pictures of people when drunk - we'll call it FacedLook.

<comment><USER52> Call it Sh*tFaced.

</comment></comment></comment></comment><comment><USER53> Let me propose a hypothetical? business arrangement where my startup selects and screens applicants and their ideas to you / your team. How would we arrange the split funding in this case with a similar spirit?

</comment><comment><USER54> We are working on an exchange and wallet solution that provides an insurance feature similar to US FDIC.

Evermint Exchange [https://evermint.com](https://evermint.com) firstname.lastname@example.com

</comment><comment><USER55> How do you 'split' a product if it's a SaaS?

Who pays/manages the platform?

</comment><comment><USER56> Hi Simon, I sent you an email with the subject the same as the title of this thread. I am concerned that it will be filtered into your spam box, so kindly check for an email from me. Thanks!

</comment><comment><USER57> I feel like these sort of mvp type ideas work great for software only, as soon as you involve a physical device how do you go about avoiding capital risk? I don't think you can!

</comment><comment><USER58> What language(s) do you prefer using? How about databases? I'd be interested in starting a dialogue with you. My email is jwheeler726 at googles email service.

</comment><comment><USER59> I sent you an email, hopefully we can talk. I find this intriguing and would like your opinion on whether or not my idea has legs. Thanks for posting.

</comment><comment><USER60> Idea: A platform for general discussion that takes what's best about HN, Reddit and even 4chan/5ch etc. while eliminating most of their flaws.

</comment><comment><USER61> We have a call center on the cloud , which we are planning to sell , if you are interested reach out to me harini at contactcentral dot io

</comment><comment><USER62> Why are you using a different account than in the version [1]?

Edit: Now I understand. Version 1 was made by another user.

</comment><comment><USER63> 1\. I like the alternative to the status quo for "starting a company". Even more so b/c you're addressing the unmet implicit desire to finish side projects.

2\. Premise 1 is agreeable. P2 doesn't hold past the initial VC demo, let alone MVP stage. P3: probably approximately correct. P4: I would agree.

3\. Skimming the comments here and seeing mixed/confused feedback, I assume those taking this more seriously are keeping silent.

</comment><comment><USER42> Ok I've got an idea for you involving vending machines. Email me if you want to hear more.

</comment><comment><USER18> How many projects are you looking to do? One at a time?

</comment><comment><USER64> How do we contact you for a proposal?

<comment><AUTHOR> hn at simonnouwens dot nl

</comment></comment><comment><USER65> Can you clarify how much equity you want. 80% sounds too high sorry but its not very clear.

<comment><USER47> Seems much confusion about this; 80% would be a weird deal so I assumed 20%.</comment></comment>


Ask HN: Enough of the reviews - WHY is Google launching +1? - <AUTHOR>

What do you think? <comment><USER2> +1 is just the latest iteration of Google's social networking attempts, so in a way they've always been working on it. As for why, well, Google is in the business of organizing data and having detail data about its users' social life is a goldmine. Although I'm not sure that it will improve the quality of their search engine, as many people have stated, it will most definitely tie users even more to Google and that's always a goal in itself.

<comment><USER3> I think it will help with some of the spam since users will generally not +1 a spam site. It will also add an extra attribute (social) to the search algorithm.</comment></comment>