Ask HN: For a startup, when should I begin writing tests? - <AUTHOR> Hello!
We're building our MVP and have got commitments from our first few customers. I'm still young and haven't managed production software before, only personal projects.
I'm hesitant to sacrifice developer velocity for tests. I believe product agility is a huge asset early on, and I think testing cuts into this.
I'm also afraid of digging an inescapable hole of untestable spaghetti code and tech debt.
I'd love to hear the thoughts of more experienced engineers and founders on this issue! How did you balance it, and what would you have done differently? When should I start testing?
My current idea is to skip testing, and instead put that effort into product development and a robust deployment pipeline with easy roll-backs. Ideally this would allow us to move quickly and revert any big mistakes, without the burden of a complex test suite.
If it's important, we're React+Go, but I'm not looking for stack-specific advice. <comment><USER3> We wrote tests from day 1, but I have mixed opinions about the value of it. It certainly has helped at times, but it also adds a huge tax to everything.

Pros

* Established the proper culture from day one. If you end up being successful, this is extremely valuable. Culture is very, very hard to change.

* Eliminated a lot of bugs and regressions.

* Gave us confidence working in a regulated space.

Cons

* Adds a lot of time to initial development

* Product complexity doesn't really require testing in the early stages.

* Slows down ability to pivot

\-----

If I were to do it again. I'd write fewer tests, but establish milestones for requiring changing/additions to be tested.

* My rule of thumb would be: If you can't easily test it in the UI, write a test. Or, if it's business critical, write a test.

* Verify authorization and authentication. It's very important.

* Test anything that makes/loses money (for example, Strip, Recurly, etc).

* Jest snapshots are quick and easy. Use them.

* Write tests for utility functions

Don't test:

* Database migrations

* UI interactions (unless critical). They take a lot of time to write and UI changes frequently.

* Vendor/3rd party API integrations beyond surface level. In my experience, quick implementations always involve a bunch of mocking/stubbing - making these types of tests less useful.

</comment><comment><USER4> Start writing tests the moment you have something to lose. Earlier if you like, but no later than that moment.

Tests are about system integrity. If you risk losing something due to lack of integrity, tests are worth your while.

Think of it like health insurance: When are you comfortable forgoing health insurance, and when are you not?

</comment><comment><USER5> I would highly, highly recommend tests. The problem with not writing tests is that you _think_ it will save you time, but instead, it will end up slowing your velocity greatly as your codebase grows and it becomes harder to discover what specific change broke feature X.

You don't have to write thousands of unit tests. But you should absolutely have a basic test suite with integration tests that give you confidence in a new release. And it will make it easier to hire developers in the future.

A good test suite doesn't slow you down at all, in fact, it speeds you up by helping you identify problems rapidly so you can fix code before it becomes a real problem.

</comment><comment><USER6> If you have no tests at all I would start with simple E2E tests to ensure your critical parts of app are working like sign up page and billing. You can add later other important sections of your app to ensure happy paths for your features are working fine.

When I started my own product for Ruby & JS developers [https://knapsackpro.com](https://knapsackpro.com) I was doing mostly unit testing and later relayed on E2E tests for testing user dashboard to ensure happy paths are covered.

It's very easy to introduce weird bugs and it's much faster to run CI build to test your app than manually verify if it works correctly in your browser when you make some changes.

</comment><comment><USER7> Working inwards from the edges of your product is a quick strategy that could provide immediate business value (compared to working from the bottom up watching metrics like coverage) and that means _end-to-end testing_.

Testing the product with end-to-end tests will stop you from shipping a completely broken product to your customer, where the most damage can occur. Add the test script at the end of the build and alert to a dashboard. "Alert" can mean email, and "dashboard" can mean "PC showing inbox for testalerts@mycompany". You can get that running immediately. Alerts won't tell you what broke of course, though you can use version control and bisecting to isolate breakages to commits, which is almost as good.

When you find bugs, then each bug is an opportunity to put some scaffolding around smaller subsystems as smaller tests to show the actual bug, as well as the fact that the bug has been fixed. Avoid committing the test without committing the fix. It's a bad habit. You can show how the test _used to fail_ in your commit message.

You'll get an idea as to what are the most problematic pieces of your app. If one part is much less reliable than the others, then you can select that subsystem for verification with bottom up unit tests. Don't spend too much time on each one though — it's highly likely that in exercising your APIs through unit testing, you're going to see ways in which they need to be refactored to make more sense and, importantly, become more reliable.

<comment><USER8> End to end tests also tend to be more fragile and require a more complex test environment, this means you might end up spending too much time fixing broken stuff (tests, environment) and getting a lot of false failures.

On the other hand if you start early you can take testability into the design and make at least some of the above problems go away, for example by having a modular architecture that allows easy mocking.

I do recommend to minimize the use of UI tests (Selenium for web for example) as they tend to be even more fragile, focus on the most obvious sunny day scenarios and never ignore a broken test as it get rotten over time

</comment></comment><comment><USER9> > I'm also afraid of digging an inescapable hole of untestable spaghetti code > and tech debt.

How many engineers do you have and how fast are they likely to write code? It depends more on the product that you're building rather than the stack - <USER2> the tech _is_ the product, spend more time on making sure it works as advertised. If the tech _enables_ the product, spend less. Any competent engineer can pick up thousands of lines of technical debt/spaghetti code and spend a few afternoons swearing at it and figure out how to modify it. Technical debt is a problem, but more of a problem when you hit millions of lines, hundreds of engineers, and you need to be sure you can change stuff without breaking it.

Debt is useful - as long as you can take it on in a calculated way, don't stress about it too much. You might need to pivot and rewrite everything anyway, or at least have your fundamental assumptions changed in such a way that requires a partial rewrite. Tests won't help you much in those scenarios.

</comment><comment><USER10> I was in the same boat as you a year ago (solo developer building reviewnb.com). I wrote API tests for the backend (assert API x with y input returns z) and completely forgo any unit tests. This has given me 80% benefit with 20% effort.

I wrote these tests after launching the product publicly and just before support requests / bug fixing period started with the simple aim of avoiding regressions as I work through the improvements.

If I were to give general suggestions on the topic,

\- In most cases having high level sanity tests is low effort high value decision.

\- To write tests or not shouldn't be a binary decision. E.g. If you think a specific aspect is going to need refinement work then tests for those modules can help you immensely.

\- Going for high % coverage very early on for the sake of mental satisfaction or bragging rights is usually not a good time investment

My only regret is not having any automated tests for frontend. Investing in some basic selenium tests after launch would have been worth it.

</comment><comment><USER11> I'd write high level tests, integration ones for sure, from the get go.

These don't take too much time to write and speed up your development significantly because you can make significant changes and be sure that certain flows still work without doing all the manual testing over and over.

</comment><comment><USER12> Try to use tests to save time? A good compromise might be that if something doesn't work the first time you try it, you can write a test to make debugging faster. If it turns out you made a bad design decision down the road, you can always delete irrelevant tests and reapply the rule about testing to save time.

Unfortunately, inexperience is going to make it take longer to write tests. That probably won't be what kills your startup though. Most companies die when they try to scale due to a variety of organizational issues, and having a few tests kicking around when you start bringing new developers in will probably help you a bit in this respect.

<comment><USER13> I'm curious - what kind of organisational issues relating to hiring new developers would be alleviated by having some tests? Are you referring to incoming developers being put off by the perception of a legacy codebase which is horrible to work with? Just curious!

<comment><USER14> Not perception, here's what happens in projects I've been on with poor test quality (low quality tests, too few tests, sometimes essentially no tests, etc.):

Experienced developers (the ones who made it) are generally fast, quick to identify the source of an issue or where to make a change. They have better than even odds of not mucking it up (discovered by users later because testing in-house is too poor to catch it).

New developers are either paralyzed by uncertainty (will this really do what I want, does it have an unintended effect elsewhere that I can't detect?) or make changes and have better than even odds ( _much_ better) of mucking it up and creating a mess. Hopefully it's discovered before release, but most likely it's discovered after.

One of the critical things for new developers to a project is the opportunity to make changes and see what happens, with good feedback. If you permit them to make the changes, but lack the ability to provide good feedback on the effect of those changes, then they cannot learn the system _from the perspective of a developer_. Along with exploratory testing, it should be possible for developers to make compilable, but arbitrary, changes and see tests fail. This lets them know the impact of a change and understand how that section of code relates to the system as a whole. That holistic view is impossible on larger projects without good testing, or more experience than you want to wait for.

The importance of this will vary based on the importance of the system and its size and scope. Smaller size and scope, then they can probably learn the system well enough (say, less than 100k lines of code, depending on language). Beyond that size, the system will become a rotten mess and your customers will leave you if they have any choice.

</comment></comment></comment><comment><USER15> Once you get experience in having a TDD approach, my view is that you don't lose time with testing. Writing tests is a way to formulate your thought and assumptions. You implement this in your code and you selectively run these using the tests. You reduce the size of the code that you think about to small building blocks. This makes you much faster-writing code. The amount of rework getting components to work together is reduced allowing you to implement features faster.

Only if your code is complex, will you need a complex test suite. If your code is complex you won't be fast implementing features anyways. TDD will help you keep your code simple.

<comment><USER16> Of course you loose time testing. Maybe not writing the tests themselves, but usually setting up data for tests is the most time consuming part.

</comment></comment><comment><USER17> Rather than skipping it as you are currently considering doing I'd add them as soon as possible for at least some end-to-end coverage of important features and authentication. If you have two or more developers I believe regression can take place shockingly quickly and it's kind of sad to be made aware of a major problem by a user, I know first hand.

One thing that has been touched upon is that you are setting the culture, if you say you're hesitant to sacrifice velocity for tests already that will probably become the long term culture and one day there will be no tests and everyone will be afraid to make any changes. Tests do take up front investment but they run multiple times a day and thousands or even tens of thousands of times over the lifetime of a product and pay it back.

I think striking a balance where you're not overly concerned about coverage but have key end-to-end flows covered can make things more efficient while saving your skin and help avoid unhappy situations. I say limited UI front-end tests (these are the greatest time sink) of key flows, but then as much coverage of apis as you can do.

Also I'm founder at Tesults ([https://www.tesults.com](https://www.tesults.com)). Check it out. There's a free plan but I'll go beyond that and give everyone who posted in this thread one full target free if you email me. So you can report your test results nicely too and keep track of failures without cost. Any direct feedback is appreciated.

</comment><comment><USER18> I can speak to this. I wrote the vast majority of code for two years for a startup. The stack, if it matters, was ruby on rails with jQuery.

I would write tests from the get go. I would have to look at the got history to be sure, but I believe I started adding to the test suite after about a month of adding code (I did use an existing open source project as the base of the product, and they had a lot of tests written). Your fears about spaghetti code are valid and tests will help with that. At the least you can refactor with less fear, though with golang that will be less of an issue than it was with ruby. In addition they will help prevent regressions. When I (or, far more likely, our users) found a bug in the system, I would write a test before fixing it. Therefore, we had very few bugs return.

However, a deployment pipeline is critical too, because otherwise your tests won't get run often enough (especially if they are slow. Due to the way I wrote the tests, and the number of them, it took approximately 20 minutes to run through all the tests in our ci environment).

Re: your fears of slowing things down. In my experience a test suite, even a bad one, will speed things up once you are in production. This is because it will prevent regressions and doing manual testing, both of which chew up additional time. I saw this at other companies.

</comment><comment><USER19> At the very minimum you should write a test each time you find a bug, so that it doesn't happen again. If you already know what the problem is writing a test case for it should be simple. The only time writing a test suite would be hard, is if you are doing something special like writing a VM guest or embedded, where automating tests take much more work. Even then, just knowing that the basics work all the time is very beneficial, even if the test suite is small.

</comment><comment><USER20> I think others have made the point well: tests give you the confidence to refactor and change and know if you broke something a customer expects to work. One caveat is that they should be good tests and not overly coupled. Don't mock as those are brittle. How so? Usually mocks assert more than contracted behavior. If you have a method under test, it doesn't matter what its guts are doing. You shouldn't assert its state. What matters is that its observable actions and errors behave as expected. You don't assert that internal functions of that method are called so many times in such and such order with the right arguments. Just test at the exposed edges.

You will want unit tests this way and to verify error handling and logic flow. You will want integration tests or acceptance tests to verify things work together and can perform the actions your users will take. These pay dividends and prevent the manual need to verify you did not break anything on a given change.

You also don't need to exhaustedly test everything. Start with critical parts. And, like others said, add to your test suite any bugs to prevent regressions. A paying customer who reports a bug, sees it fixed, and sees it come back is not a happy camper.

</comment><comment><USER21> \- Have only the most important use cases covered with automated integration tests.

\- Don't spend time in writing unit tests until

\- You have paying customers and a steady revenue

</comment><comment><USER22> _I believe product agility is a huge asset early on, and I think testing cuts into this._

I disagree that testing cuts into agility.

I believe that in this case "agility" means that you can change your product in significant ways quickly. Presumably, when you make these changes, you need it to still work. Having tests already in existence will inform you when you've broken something you didn't mean to change. If you don't have tests that can run frequently on your swiftly changing, rapidly pivoting codebase, you will break things and you won't even know until it takes off someone's foot.

If you don't have these tests, you can't rapidly change your product with confidence that it still works, and I suspect that you would value "working" above "not working" very very highly.

If you want agility, you need tests. Lots of automated tests. Testing doesn't cut into your agility; testing _gives_ you agility.

</comment><comment><USER23> We have automated tests for some parts of our app, depending on how much value tests add.

For example, code that parses data generally has lots of tests. It's usually easy to write unit tests for parsers, and it's hard to write parsing code correctly. These tests are invaluable and catch lots of errors, especially when you add features to the parser, or refactor the code, or optimize it. I would not touch any parsing code that doesn't have tests, because no matter how smart you think you are, you are going to break something everytime you make a change.

We also have tests for code that interfaces with 3rd party components. When those components change behavior, we can detect problems early.

We don't test most UI code, since tests are often difficult to write, need to be updated every time we make changes to the UI, and they catch few bugs.

</comment><comment><USER24> This is a good question. Everyone aspires to have a good test suite but if you are building an MVP with limited time and resources you can get by using a service like [https://sentry.io/](https://sentry.io/)

It will email you a stack trace whenever an error occurs and also keeps a log of them which you can order by frequency and so on. It's an extremely useful service.

When you get bigger and have regular users and need stability then tests start to become important so that you don't break existing things when rolling out new features.

Not having tests comes with a risk though. For example, I once broke our registration flow and didn't know about it for a week. Flipside is we hardly had any users registering at that point. So it's probably a good idea to cover a few of your core user journeys. Hope that helps!

</comment><comment><USER25> If the accepted wisdom was true, that you can trade quality for speed, then you shouldn't write tests.

I do not believe that to be the case. I find the biggest value of tests is being able to change the code more quickly. Running tests is MUCH faster than a regression script.

There is a scale where you'll never have to change the code enough for the tests to pay off. Personally, I think that scale is quite small, maybe a week of development work.

However, it takes some experience to get good enough at the tests that you feel that benefit. If you're not there, the answer is probably different. However, if your startup scale is non trivial, I'd bet that the payoff if learning that skill and building some testing framework from the get go would be worth it.

</comment><comment><USER8> Adding a bit to the great answers here. Are you doing Exploratory Testing [0] ? if not then you probably should.

You will probably find out that you need a bunch of tools to help you exploring the product, so why don't you spend some time turning those tools into crude test automation ?

Remember that the best test is the test that actually runs, an unwritten test doesn't really help you.

[0] [https://martinfowler.com/bliki/ExploratoryTesting.html](https://martinfowler.com/bliki/ExploratoryTesting.html)

</comment><comment><USER26> I think it depends on the nature of your application. In my experience, in a startup situation the product design can be iterating quickly so both the tests and the code itself can be out of date very quickly. Personally would wait until I was close to product/market fit before writing tests.

Obviously, if you have a clear understanding of the finished product like in many enterprise applications then testing makes sense.

I would however, strongly recommend it if you are doing nuclear reactor software ;-)

</comment><comment><USER27> I see software as composed of: data (what) -> control (when) -> alg (how). And software should be designed in this order.

Hence you should write tests only when you want to freeze your design in a specific area. I.e. make sure that you got the domain model right, and freeze it via tests, etc.

I found this book very valuable:

[https://www.manning.com/books/unit- testing](https://www.manning.com/books/unit-testing)

</comment><comment><USER28> I would try to make adding tests as easy as possible. That is for every piece of code you write think to yourself: How would I test this. If in many cases the code only makes sense in the context of the whole application running this might be a sign of danger. Basically you thinking testing might slow you down is a sign this might be the case. For some problems it is way easier to write tests than to solve them correctly.

</comment><comment><USER29> I stood up the product in 23 days, launched, will be 3 month young in couple of days. We have test cases written on our open to public python library, just one, nothing more than that. We purposefully asked a outsider, a freelancer, to write test cases for us looking in to the API documentation available on the website. This way we validated our documentation as well as the python library.

</comment><comment><USER30> Start writing test as soon as you start losing time due to regression bugs. But don't ignore the first signals :).

</comment><comment><USER31> If you don't do it from the beginning chances are you'll never do it. Testing isn't an after thought, I mean, it ends up being one for most companies and it shows...

</comment><comment><USER32> There are various levels of testing, at an early stage you can write only integration and functional tests, later on when you get time you can write unit tests as well

<comment><USER33> If you don't unit test from day 1 then there's a strong probability that you wont be writing your code in a way that you _can_ unit test later. e.g. mockable dependencies. I'm included to say go with TDD right from the start.

</comment></comment><comment><USER34> If you have hardly any experience, how can you be confident in your belief that "product agility is a huge asset early on"?

</comment><comment><USER35> If you're changing something that's working, but a bug could corrupt your data, you should probably test that :-)

</comment><comment><USER36> > I'm also afraid of digging an inescapable hole of untestable spaghetti code > and tech debt.

if you are worried about tech debt you should watch "The art of destroying software". It will be hard to understand at first.

Here is TL;DR. Writing small services that will take only one week of work to rewrite. That way you can burn your services and recreate when you truly understand your domain.

Recently I worked on a REST API using go. I think the go package system has good support for writing small and independent packages. Just make sure your microservices are micro.

You can do the same for your React Codebase.

</comment><comment><USER37> I like to share my experience: Writing tests from day 1 is essential. Think of tests as a road for your business logic, which is a car. It might seems expensive to write tests but bad quality code slows you down within weeks rather than months.

Unit tests are important tests and are very useful for engineers. As others mentioned, thinking about test-ability keeps code simple, which is a must when you want to move fast. Test-ability (have the TDD mindset and thinking, if you don't then apply TDD until you think tests while writing your code) usually translates to readable and maintainable code. When you write good unit tests, then most assumptions are documented in an automated way and tested at every execution of the unit tests. When a unit test fails, then the error message and the test name should tell you precisely the assumption and what is broken, that ultimately makes you really fast in maintaining and (with a good overall design) changing code.

When you want to be really fast then have fewer or no integration tests. That is because integration tests usually span over several parts of your software. So a change in any part will lead to probably several changes in the corresponding integration tests. That makes it more expensive to change things fast. On top of that, many integration tests are more complex to write because of the work in mocking adjacent parts of the software.

System/Black box tests. Those test are very important for evaluating business value. One tests common happy paths from your customers and validate that business value/ feature x still works. Again like with the system tests, the process of doing, deciding and prioritising will lead to clear business values and very important a better shared vision between the developers and business. The software serves the purpose to be used by a customer, the system tests are ultimately a distilled version of what you the business is selling and it's an automated process to ensure that this value is delivered.

Even the best software developers do mistakes and forget. I have not yet seen a team which profits from not writing tests. It takes some time to write good software -> testing and approaches like TDD are some of the ways to start climbing that hill.

About your questions:

A robust deployment pipeline is the right way to go and can be even better leveraged with tests. Example: you can connect your system tests to the roll- back functionality and re-deploy the last working state.

You are right about your worries about Spaghetti code and technical dept, from my experience they are always an issue from early on. But also keep in mind that no one is perfect so even if it's only you writing code, you will find code written from you, which gives you goose bumps. A nice approach to code quality is the in-place refactoring, that is you don't do re-factoring as a task. Each time you touch code and it's not very clear or the structure/ design is not ideal then you re-factor with each and every task (piece by piece). A rule of thumb: When a 1 day tasks adds 3 days of re-factoring (only fixing the touched code), then you know that you are sliding the slope of lower productivity and you need to act and going back up the slope usually takes a lot of discipline in the team (that's an extreme example, personally I act much earlier).

Your current idea has some risks associated with it. I like to expand with an example: Usually a code base changes and grows with new features, rarely it gets smaller. When your code base grows and your team grows -> productivity declines (people: communication overhead, technology: overall more complex/bigger system, probably many more factors). In the first weeks you might deliver very fast without tests but in the future that is not given. Let's assume that you and your team are unicorns and will produce perfect code all the time. Your productivity declines naturally by having a bigger system and more people. In the case of declining productivity it is very unusual to add workload like writing tests. It might be easier in your project/startup, but usually it's hard to add more workload. The future business will probably be under constant pressure to deliver fast and follow the market. Now back to the real world, you will probably face a more pronounced slowdown than in the unicorn case. Several risks are coming from your decision, if you think you can manage them then try and keep your eyes peeled and learn from every mistake along the way (btw that's always good advice for every part of your journey ;) ).

Tip use a tools like SonarQube ([https://www.sonarqube.org/](https://www.sonarqube.org/)) to asses and measure obvious code quality.

Another point I like to share: Keep in mind that good tests don't stop bugs, there will be enough bugs even with good tests. Tests help developers/engineers to understand code and assumptions, they help to reduce bugs from being introduced, they help you doing the right design decisions and much more. You can even write strongly unreadable code with a very good rating from SonarQube. Maintainability, readability and change-ability of your code are human related problems and only slightly technology problems.

Software Quality often relates to tests, if you replaces tests with software quality in your question this article has some good talking points: [https://martinfowler.com/articles/is-quality-worth- cost.html](https://martinfowler.com/articles/is-quality-worth-cost.html)</comment>


Ask HN: Did you ever have to push against founders/CEOs having database access? - <AUTHOR> <comment><USER2> No, but there seems to be more to this than a simple question about access.

Usually when a question about access comes up the root concern is more the accountability. Access used appropriately is rarely a problem, but access misused can be very damaging.

In small organizations is not uncommon for everyone to have access to everything. The required knowledge domain is small enough that everyone knows what's going on and everyone needs to be able to address issues without undue delays from process. The larger the organization grows this begins to break down and access needs to be segregated to groups that understand their areas. This prevents a person from making changes to things for which they do not understand the impact.

If your organization is at this point and you feel like you need to push on denying access to something, be it founders or others, you should consider the why of this. Do these people insist on access to the DB due to lack of trust of those responsible for the DB? Are you as committed to the organization as a founder or CEO? Are they causing disruptions/outages by having this access?

Ultimately founders, owners & c levels are usually the chief decision makers and therefore should have the ability to access nearly anything, and they'll view it as such. Their house, they should have keys to it kind of thing. So perhaps the argument to be made is, just because it's your house you shouldn't leave a mess because we all have to live here too.

</comment><comment><USER3> No, normally they will not ask for it unless they are actually doing software or system engineering work on the products. If they are the non-engineering type then they normally have an executive dashboard interface that they use to find the information they are looking for vs a DevOps dashboard.

For those that are not actually doing work on the product that request access or any managerial/executive types they are normally forbidden direct access to any production or testing systems unless they are actually doing engineering work. This access is denied via physical access controls ( no PMs, managers or executives in the datacenters, network closets or anywhere there are servers or networking gear ). Logical access controls, there systems cannot access the development network or databases, they go through the frontend or internally through a web application interface just like finance, accounting, marketing, etc. go through.

The only people that can directly access the servers or networking equipment are a small amount of top level networking, security and system engineers. As for most companies, especially those with industry certifications or that work with the government (RMF for example) want you to insure that least privilege access and access based on job role be used.

This means the PMs and executive level people do not get direct access to anything and have to go through the proper channels if they need engineering channels if they need more information than what is not available through the standard interface.</comment>


'sudo make me a sandwich' has become a reality - <AUTHOR> http://www.willowgarage.com/blog/2011/12/12/tum-rosie-and-pr2-james-add-popcorn-and-sandwiches-their-accomplishments

<comment><USER3> very cool.

Of course, this won't be how food is prepared by robots ultimately. Each step will more likely be performed by small robots optimized for the task.

Food will also probably be driven from place to place by robotic vehicles. You might have one company making noodles and sending them to other companies five minutes away that incorporate them into a dish or other product.

Virtual supply chains will eliminate a lot of capital cost of setting up a restaurant and reduce the delays that destroy freshness and necessitate preservatives. Like fabless semiconductor companies, there might be prepless restaurants.

Another company may just do all the selection and slicing of your vegetables for you. They may be picked overnight at their peak and delivered to you so you can throw them in your omelet. The slicing might even happen on the trip to you.

I can see you selecting a recipe and a web app that prices and source all the ingredients for you. If you love to cook but don't enjoy finding or prepping some or all of the ingredients, such arrangements could help you save money and get better results.

Quite a few people eat the cheapest and most convenient foods they can find. But in the future, healthy food will be cheaper, fresher and will find you, hopefully making crap truly uncompetitive.

<comment><USER4> Your assuming a vary high density food distribution network. Most restaurants in the US are in fairly low density areas. So while food is often prepared offsite, the shipping delays result in preservation issues.

<comment><USER5> He also assumes cheap transportation of goods. Economics & business is poised to change as we know it when oil production starts flagging, if comparable alternatives are not found.

"Buying local" may one day no longer be the luxury option at the grocery store ;)

<comment><USER4> It often costs more energy to buy local food than to transport it 1/2 way around the world. Boats are amazingly efficient, trucks are not.

<comment><USER3> Actually, robots growing things on the roof of your house will use even less fuel.

<comment><USER4> This might look like a good idea, but you need to include the energy costs from building all those tiny robots. Fixing them when they break down. Building a strong enough roof to hold them, 6 inches of topsoil and your crop. Irrigation and fertilizer and pesticides for that crop. Storage space for that crop for the better part of a year including some form of climate control to minimize spoilage etc etc.

Honestly, sectioning off the mid west and transporting the food to you by trains is actually a much better idea. You can still use robots but you get to use really big and cheap ones that do a simple job and get regular maintenance by people that know what their doing. You can also minimize the contamination from air pollution etc.

PS: Moving 2 weeks supply of food around the world by boat can easily take less energy than driving to a 5 miles to the super market to pick it up. If you want to be as environmentally friendly as possible, move to a big city and become a vegetarian. Living in the countryside feels more natural but it's much much worse for the planet.

<comment><USER3> Once robots are growing the food, a lot less of it will be imported. Even with the cost of maintenance, robots will be cheaper than human labor, eliminating a major reason for importing.

I guess if you think of robots as they are today, it may seem like they'd break down a lot. But agricultural robots will eventually be more like insects and, like many living things, cheaper to replace than repair.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> I wonder, how long until we get the first completely automated fast food restaurant?

You walk in and download the menu over the wifi, pick what you want and pay with your card. You are then issued with a long unique code.

A machine prepares your meal and your phone prompts you to approach one of the vending areas. You then enter the code or scan an image and your food appears through a hole.

How many employees would be needed for serving > 100 per hour?

<comment><USER7> It's going to take about 40-50 years to reproduce what humans can do in a restaurant today with robots. More likely, the food preparation process will be altered to allow for easier automation. With the exception of a supervisor/engineer (that position won't be automated for at least another 100 years) - I think we'll see a fully automated fast food restaurant in about 20-25 years - though some elements of the restaurant business (Cleaning, French Fries, Order taking) - will likely be automated sooner - there will be demonstration restaurants with 50% automation in about 10 years.

[note: And yes, I do realize that the above predictions will require radical advancements in automation that are not easy to predict - call me an optimist. :-) ]

<comment><USER8> I think that you're wrong and I will illustrate why. When I graduated from college I spent an evening with friends and we mused what we'd buy with our new paychecks. I wanted a brand new Mustang and indeed I purchased one. But one friend, the only computer major, said he wanted a computer.

This was pre-pc era and the one he wanted cost as much as a house. We teased him all night about how speedily he could balance his checkbook or play checkers. Finally he'd had enough, pounded the table and said with a red face in ten years you will all have a computer that sits on your desk.

Our only exposure at that time was to the IBM 360 mainframe the school had and that only made us laugh harder. Truth is he was right, every last one of us had a pc in ten years.

The computer industry moves faster than you think. A few years later I went to a local store to see one of the first laser printers for sale. This one was $10,000 and as a group of us looked at it with the proud sales guy. I said as soon as I could get one for under $1500 I'll buy it. Everyone looked at me like I was crazy and the salesmen said maybe in ten years it will happen. Actually I got an HP Laserjet in around three years for $1499 and still have it - somewhere.

I'd wager that most of us on HN will have a robot in our house in well under ten years. I can't wait;<).

<comment><USER9> Robots are harder than computers or printers.

<comment><USER10> It depends on the robot.

A Roomba can be had for the price of a few weeks' groceries.

Industrial assembly robots run about $60k, though deployment costs are over 3x that. Curiously, reprogramming a robot is considerably more expensive than the robot itself, according to one source.

If you have a well-defined, and reasonably attainable task, costs are likely predictable and possibly finite. If you've got a poorly defined and hard-to- reach goal, expenses will likely rise to meet resources.

<comment><USER11> One thing I'll comment on this is that I have a Roomba, love it, but it does require quite a bit of human care. About every three clean cycles (I have a one bedroom apartment) I have to take it apart, clean all the hair that gets stuck around the brushes + gears) and assemble it back together.

<comment><USER10> FWIW, you'd have to perform similar maintenance on a standard vacuum.

As well as, you know, vacuum with the damned thing.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> I think there is a somewhat unstoppable direction towards automation but it will probably be shaky and filled with potholes.

For example, its been interesting to see the consumer response to the self checkout stands in the big chain stores. Those self-checkout stations in their current form I would say are somewhat of a failure to encourage automation at least from a consumer's standpoint. I'm a techie and I actively avoid them. Why? I and the other people I'm waiting in line behind take forever because we are all slow in finding the bar codes on each product. Not only that, 1 out of 10 times something seemingly goes wrong with checking out a product - <USER2> product needs to be verified by a store manager, the product gets scanned twice, etc. which all usually end up taking longer than the regular checkout lines. Interestingly a few of the stores near me just recently removed their automated systems after having them for many years. (I'm actually hoping one day they slap RFIDs on everything and you can just push your basket through a scanner.)

But I think the risk with robots and automated systems is when things go wrong when interacting with an unpredictable human. Technically when things go wrong its usually user error but not through the consumer's eyes - to them its a stupid robot that keeps on throwing their sandwich on the ground and their response is usually to run back to the old reliable way.

<comment><USER13> I think that's really an implementation issue, rather then a problem with the idea. The current batch of self-checkout systems are very very fragile. It seems that instead of optimizing them for throughput/efficiency, they optimized them for detecting if you are trying to steal something. For example, something as simple as brushing the weight sensor will trigger an alert that takes 15-20s to go away. There's really no reason to do that unless you are worried about people stealing. In reality, checking the weight is a very poor way of detecting this (every system I've seen has an "I'm using my own bag" option that makes the alert go away.

These are actually very unoptimized from the UI point of view too. For example, most of the people have their own loyalty card they scan at the beginning. Why not track these, and adjust settings on the machine based on it? For example, I mute the volume whenever I start, that would be nice to keep track of. Defaulting to showing me a list food that I've purchased over my last few trips would make things go quicker instead of me having to dig through pages of items looking for common things.

</comment></comment><comment><USER14> I'm rather interested in the amazing amount of abstraction achieved between "get instructions from internet" and "grab this pot and do shit with it". Fantastic.

That said, when the bread didn't go in the toaster for a second time I was half hoping it would start beating it until the toaster was a pile of screwed up metal.

</comment><comment><USER15> Willowgarage are also the guys behind the Point Cloud Library, an awesome computer vision library for processing point clouds (e.g. what the Kinect produces):

<http://www.pointclouds.org/about.html>

The Kinect Fusion implementation by Anatoly Baksheev in their trunk is very impressive:

[http://www.pointclouds.org/news/kinectfusion-open- source.htm...](http://www.pointclouds.org/news/kinectfusion-open-source.html)

If anyone has a Kinect and Ubuntu and wants to try it, I've compiled an unpolished tutorial on how to do that:

<http://hobonaut.com/>

</comment><comment><USER16> It's been a reality for a long time in my household. My girlfriend says it to me a lot ever since I introduced her to Linux.

</comment><comment><USER17> Not to take away from the accomplishment, but NYC Resistor did this 2 years ago: [http://www.nycresistor.com/2009/02/27/sudo-make-me-a- sandwic...](http://www.nycresistor.com/2009/02/27/sudo-make-me-a-sandwich- robot/)

</comment><comment><USER18> Why would you need administrative privileges for that?

<comment><USER19> I'd say that any command that can make something physical happen in the real world has a whole different set of security concerns than we've faced before. So its a serious concern.

But on the off chance that anyone missed the inside joke: <http://xkcd.com/149/>

<comment><USER20> But it's not like it's a new problem. I mean printers make something physical happen in the real world and they aren't exactly new, and last time I checked I didn't have to use sudo to print a document on Linux.

I am not sure one should need root privileges to burn a CD either.

<comment><USER21> It is, however, smart to put resource limits on a shared printer so that no one user can cause it to burn through reams at a time. And even as expensive as ink is, the ingredients of a sandwich are more expensive than the ink and paper needed to print a typical document.

Additionally, we've been seeing more and more viruses attacking SCADA systems (most notably Stuxnet), which show that people have not been taking the security issues of robotics seriously enough.

<comment><USER20> There have been fun pieces of malware for printers too. People who do not learn from history....

In fact there was an old virus that infected C64 disk drives..... These are still not new problems.

</comment></comment></comment></comment></comment><comment><USER22> To think that xkcd can make stuff like this happen...

<comment><USER23> More xkcd real life fun: <http://xkcd.com/chesscoaster/>

</comment></comment><comment><USER24> It's actually 'Siri, make me a sandwich' now.

</comment><comment><USER25> I resent being too stupid to work there. :(

<comment><USER26> Me too. Every time I make popcorn, I burn it. Damn you, robots! They're taking our jobs, I tells you.

</comment></comment><comment><USER27> No rule to make target `me'. Stop.</comment>


Uncle Mining, an Ethereum Consensus Protocol Flaw - <AUTHOR> https://bitslog.wordpress.com/2016/04/28/uncle-mining-an-ethereum-consensus-protocol-flaw/ <comment><USER2> So does another Ethereum hard fork mean we have:

ETC - ethereum classic, the 1st pre-hard fork coin

ETH - ethereum as it is now

ET2 - the post 2nd hard fork version of ETH?</comment>


Mtime comparison considered harmful - <AUTHOR> https://apenwarr.ca/log/20181113 <comment><USER2> In Ninja I sorta stumbled through some of the same issues described here. I eventually realized that the interesting question is "does this output file reflect the state of all the inputs" and not anything in particular about mtimes, and that"inputs" includes not only the contents of the input files, but also the executables and command lines used to produce the output.

If you squint, mtime/inode etc. behave like a weak content signature of the input. And once you have that perspective, you say "if mtime != mtime I had last time, rebuild", without caring about their relative values, and that sidesteps a lot of clock skew related issues. It does "the wrong thing" if someone intentionally pushes timestamps to a point in the past (e.g. when switching branches to an older branch) as an attempt to game such a system, but playing games with mtime is not the right approach for such a thing, totally hermetic builds are.

One nice trick is that you can even capture all the "inputs" with a single checksum that combines all the files/command lines/etc., and that easily transitions between truly looking at file content or just file metadata. The one downside is that when the build system decides to rebuild something, it's hard to tell the user why -- you end up just saying "some input changed somewhere".

<comment><USER3> Is there a description of Ninja's algorithm anywhere? I looked at the manual [1] and didn't quite see it.

Does Ninja use a database like sqlite? It seems like it has to if it does something better than Make's use of mtimes. (e.g. the command line, which Make doesn't consider.)

I looked at redo (linked in the article) and it uses sqlite to store the extra metadata.

[1] [https://ninja-build.org/manual.html](https://ninja-build.org/manual.html)

<comment><USER2> No, sorry. And I also mixed what Ninja actually does with some random observations in that comment.

Ninja does use some database-like things, but they are just in a simple text/binary format. It's actually been long enough that I have forgotten the details.

[https://ninja-build.org/manual.html#ref_log](https://ninja- build.org/manual.html#ref_log) (contains a hash of the commands used) / [https://ninja-build.org/manual.html#_deps](https://ninja- build.org/manual.html#_deps) (database-like thing with some mtimes, see [https://github.com/ninja- build/ninja/blob/master/src/deps_lo...](https://github.com/ninja- build/ninja/blob/master/src/deps_log.h#L29) )

<comment><USER4> This reminds me, I should study ninja's binary format and maybe borrow it :)

The sqlite3 database used in redo was just something I threw together in the first few minutes. sqlite was always massive overkill for the problem space, but because it never caused any problems, it's been hard to justify working on it. I'd like to port redo from python to C, though, and then the relative size of depending on a whole database will matter a lot more.

<comment><USER2> Someone rewrote ninja from scratch in C at one point and it's shockingly tiny. (No tests = no extra abstractions to make testing possible.)

<comment><USER3> I found samurai and it is indeed tiny! ~3400 lines was less than I was expecting for *.[ch] !

[https://github.com/michaelforney/samurai](https://github.com/michaelforney/samurai)

Ninja isn't too big though. It looks like about 13K of non-test code, which is great for a mature and popular project. Punting build logic to a higher layer seems to have been a big win :)

</comment></comment></comment></comment></comment></comment><comment><USER5> The "popular misconceptions" section seems to have a couple of the author's own misconceptions.

* On precision, he notes "almost no filesystems provide that kind of precision" (nanoseconds), but I would honestly say the exact opposite statement. ext4, xfs, btrfs, ZFS are some of the very common file systems that support this. He cites that his ext4 system only has 10ms granularity, which is most certainly not the default, but likely a result of upgrading from ext2/ext3 to ext4. As an aside, NTFS only has a granularity of 100ns.

* It is unclear what he means by "If your system clock jumps from one time to another...". If this is talking about NTP, it's probably accurate. My first reading was "daylight saving" or time zone changes, in which case, everyone uses UTC internally and such changes don't affect the actual mtimes. (You might get strange cases where a file listing regards a file modified at 01:45 to be older than a file modified at 01:20, but if you display in UTC, you can see it's just DST nonsense)

<comment><USER6> You're looking at the 90th percentile, he's looking at the 99th percentile. Claiming things like "everyone uses UTC internally" is obviously wrong when many people just a few years ago were still setting up systems in localtime when dual booting with Windows. Upgrading to ext4 is also not a rare edge case.

<comment><USER5> Upgrading to ext4 is not extremely rare, but the recommended procedure involves mkfs and copying files over anyway: [https://ext4.wiki.kernel.org/index.php/UpgradeToExt4](https://ext4.wiki.kernel.org/index.php/UpgradeToExt4)

In-place upgrades do have the potential to leave some non-default options for the final ext4 file system, such as 128B inodes instead of the 256B ones, which is where certain features like reduced timestamp granularity comes in.

<comment><USER4> In any case, my system was installed from scratch quite recently using native ext4. As others have pointed out and as I linked in the article, it's likely a kernel issue. I assume many people have the same thing.

</comment></comment></comment></comment><comment><USER7> > the .git/index file, which uses mmap, is synced incorrectly by file sync > tools relying on mtime

This part implies that the index file is written via mmap, but that's not true. It is fully rewritten to a new tempfile/lockfile, and then atomically renamed into place.

Git does not ever mmap with anything but PROT_READ, because not all supported platforms can do writes (in particular, the compat fallback just pread()s into a heap buffer).

</comment><comment><USER8> From this article I learned that build systems don't have the fundamental choice: mtime or checksum. Instead a better solution is mtime plus a bunch of other things. The article explains the faults of mtime and checksum clearly.

This insight makes me want to try redo.

One thing I dislike about redo is that it probably does not work well on Windows. Has anybody ever tried?

Redo also makes me wonder: Is a build directory just a habit from using Make or is it a flaw of redo to not support that well? With "build directory" i mean the concept where the build process generates files in an extra directory which can simply be deleted and nothing does pollute the source directory.

<comment><USER9> > _One thing I dislike about redo is that it probably does not work well on > Windows. Has anybody ever tried?_

Yes, although only for a toy project. I'm sure it'd work fine with Cygwin or WSL, it might work with MSYS, but I've definitely had it working with busybox-w32[1].

> _Is a build directory just a habit from using Make or is it a flaw of redo > to not support that well?_

You can use redo in a Make-like fashion by putting a single `default.do` in the root of your project that decides what to do by examining the filename it's been asked to build. That does give up some of the benefits of redo, however (since a single file builds everything, when you edit that file redo wants to rebuild everything).

Having a separate build directory that can be easily wiped is a good idea, but I'm a lot less worried about it (or things like 'make clean') now that I have 'git clean -dxf'.

[1]: [https://frippery.org/busybox/](https://frippery.org/busybox/)

</comment></comment><comment><USER10> [http://jdebp.eu/FGA/introduction-to- redo.html](http://jdebp.eu/FGA/introduction-to-redo.html)

hoping jdebp will chime in..

<comment><USER8> I have never before seen this idea to put CXXFLAGS into a file and treat it as a file dependency. That would also work with Make. Clever idea.

Nevertheless, a build system which does this implicitly is better, imho.

<comment><USER4> Perhaps unsurprisingly, djb seems to have pioneered this with his Makefiles, which generally produce, then depend on and run, a 'compile' script that contains the flags.

</comment></comment></comment><comment><USER11> The article suggests writing explicit rules to check for changes in the toolchain.

These dependencies can be recorded automatically with LD_PRELOAD. LD_PRELOAD can redefine functions such as fopen and let you record what files are read when running e.g. cc.

This makes it feasible to record the entire relevant state of a system, files and environment variables, at build time.

An argument in favor of checksums is the use of build caches. Switching between branches on large codebases triggers a lot of rebuilds. With a build cache, that can be avoided. SCons is a build system that uses such a cache.

<comment><USER12> Depending on LD_PRELOAD is extremely fragile and finicky.

Not only can a process sidestep libc entirely by calling the `open`(2) syscall, but there are often many ways of combining function calls to achieve the same outcome. This method will also fail completely on systems that have new, previously unknown functions that are not monitored by the LD_PRELOAD solution.

Worst of all, a LD_PRELOAD solution would not cover operations that are done on the behalf of the target program by external programs via IPC (think system daemons and dbus), at least not without intercepting and interpreting all io that target does.

In short, it doesn't scale.

</comment></comment><comment><USER13> > Random side note: on MacOS, the kernel does know all the filenames of a > hardlink, because hardlinks are secretly implemented as fancy symlink-like > data structures. You normally don't see any symptoms of this except that > hardlinks are suspiciously slow on MacOS. But in exchange for the slowness, > the kernel actually can look up all filenames of a hardlink if it wants. I > think this has something to do with Aliases and finding .app files even if > they move around, or something.

I don't think this is true anymore with APFS.

</comment><comment><USER14> The checksum issue could be addressed by having the compiler generate the checksums in a "sidecar" file and have the build system depend on _those_.

Obviously this faces the "not every toolchain will support this" but you could have a switch to use checksums and continue to use the older approach by default.

You would have to call cksum instead of touch in your Makefile.

With a modules system (like C++20 is trying to embrace) you could in theory generate a vector of checksums (with subranges in the file) for each file and only recompile portions of the file that needed it. It's rather an accident of history that we use the granularity of a file at all.

</comment><comment><USER15> Why not just use a complete checksum of the file?

The average project is what, a few MB? Less? Most of which is going to be cached after the first compile anyway.

Even on an enormous codebase, as long as you have an SSD or a nontrivial amount of RAM I can't see this being an issue.

You don't care if a file is newer - you care if it's different!

<comment><USER16> This is discussed near the end of the article. It works best when the file system itself stores a checksum in its metadata so it does not have to be calculated for each file for each build. It's not appropriate when your build may include dependencies based on other side effects besides file content. For example, sometimes you depend on the timestamp of an empty file, or the success or failure of another step based on e.g. a log message, to trigger other actions

<comment><USER15> Is it common to build over NFS? I can't immediately see a use case - collaborative editing or something? Even in that case, wouldn't it be easier to build on the box?

The other case seems valid in a sort of 'if your build intentionally makes use of mtime, you'll need to look at mtime'. It seems like an odd thing to do in the first place - I guess Makefile as deployment rather than for building?

A 470K LoC project I have here with >1000 files takes 0.04 seconds to do a full sha256sum traversal on my box from the cache. That's single-threaded.

If I drop caches, it takes approximately 1 second (from spinning rust, not SSD).

<comment><USER17> NFS is getting so rare that some systems aren't even organized to accommodate "mount -o ro /usr" anymore.

<comment><USER16> NFS is widely used in HPC to mount user home directories on compute nodes.

</comment></comment></comment></comment></comment><comment><USER18> Those looks like some good ideas, because currently I do use just mtime base (for programs with multiple files; many of my programs are only one file and so don't need to deal with stuff like that).

</comment><comment><USER19> Ugh, these "* considered harmful" blog posts...

the only message I read is that someone wants to believe they're as smart as Dijkstra. /s</comment>


Google 2.0: Why MIT scientists are building a new search engine - <AUTHOR> https://www.youtube.com/watch?v=YmF7KvsldGU <comment><USER2> Danny Hillis worked on the original Google knowledge graph?

[https://underlay.mit.edu](https://underlay.mit.edu)

<comment><USER3> More specifically, he made FreeBase, which Google bought and then extended into the Knowledge Graph, before eventually shutting off the original, open/"free" database.

That being said, as the developer of the system that Google is built upon these days, if he's got a version 2, people really should be interested in that, as obviously his version 1 worked pretty well. And if anyone wants to strip the competitive advantage away from Google, supporting Underlay and it's place in the public domain would be extremely powerful.

</comment></comment><comment><USER4> This was eye opening. Great to see these kinds of foundational or plumbing work being down to tackle the explosion of information and fisinformation out there. If anyone else has more knowledge to share about this that'd be awesome.</comment>


Focusmate – virtual coworking and accountability partners to help productivity - <AUTHOR> https://www.focusmate.com/ <comment><AUTHOR> Really cool idea!

I'm surprised that it's free. Kind of wish it cost a nominal amount ($10/mo?) so that I knew I wouldn't be matched with a crazy person.

</comment><comment><USER2> I've been using focusmate for months, it's surprisingly effective (I figured since they can't see my screen I'd go on hn/twitter anyways, but I dont). Not only is it good for staying on task, I find the biggest benefit is committing to be in your chair at a certain time.

</comment><comment><USER3> tried it, it's sorta like crossfit for work - kinda awkward at first, but effective.

translation: it held my feet to the fire and got me paid (as opposed to laid, like crossfit lol)</comment>


The silent crowd that kills angels - <AUTHOR> http://maxkle.in/the-silent-crowd-that-kills-angels/ <comment><USER2> link is "404 page not found" error. Is this the one you intended? <http://maxkle.in/how-angel-investors-destroy-talent/></comment>


Ask HN: Node SaaS Frameworks? - <AUTHOR>

Hi,
I am about to start on building a SaaS application and was wondering if there were any node (or other) frameworks that are geared towards SaaS/multi-tenant solutions?  e.g. things like managing recurrent billing, hierarchical security models, etc.
Thanks! <comment><USER2> Hoist ([http://hoistapps.com/](http://hoistapps.com/)) seems like it might be something up your alley.

</comment><comment><USER3> I don't know if this is what you are looking for but I created Handyjs (handyjs.org) as a base framework for all my Node projects.</comment>


The Mathematics of Paul Graham's Bias Test - <AUTHOR> https://www.chrisstucchio.com/blog/2015/paul_grahams_bias_test.html <comment><USER2> > So rather than comparing _mean_ performance, we'll compare _minimum_ > performance.

If I'm understanding correctly, the new test is based on a single data point from each group, rather than an aggregate statistic (like mean). I'm no statistician, but it seems like this data would have _far_ too much variance and noise for this to be a useful test.

The minimum performer could be someone who had a sudden personal crisis. Or who had 10 competitors suddenly pop up. Or any number of other circumstances outside their control. The minimum performer is, almost by definition, an outlier. It doesn't seem rational to suppose that an outlier is representative of the group.

I can understand that statistically this test may be more rigorous. In practice I would expect it to be less rigorous. Because the assumption it makes (that a single outlier is representative of the group) seems even more dubious than the assumptions required for Paul's original idea.

<comment><USER3> The sample minimum (or maximum) is not an inherently unstable statistic. If there is sufficient density in the distribution near its minimum, the sample minimum can be quite robust. For example, consider that the maximum likelihood estimator for the upper bound of a uniform distribution is simply the sample maximum, and the minimum-variance unbiased estimator is also based on the sample maximum[1]. (This method was used by the Allies in World War 2 to estimate the total number of German tanks by sampling the serial numbers from destroyed tanks[2].)

Of course, a real thresholding process would not be perfect, so the lower bound of the distribution of accepted candidates would not be a perfect vertical cutoff as in the examples. Just like any process that adds additional variation to the data, this would reduce the statistical power. You could accept more bias in return for lower variance in your test by taking, say, the 5th percentile instead of the sample minimum as your test statistic. (You can think of the sample minimum as the zeroth percentile.)

[1] [https://en.wikipedia.org/wiki/Uniform_distribution_%28contin...](https://en.wikipedia.org/wiki/Uniform_distribution_%28continuous%29#Estimation_of_maximum)

[2] [https://en.wikipedia.org/wiki/German_tank_problem](https://en.wikipedia.org/wiki/German_tank_problem)

<comment><USER4> It's very interesting. And what are the most suitable formulas we can use to measure how robust it is?

<comment><USER5> If you are estimating a parameter with a sample statistic, two properties it should have are unbiasedness and low variance: [https://en.wikipedia.org/wiki/Minimum- variance_unbiased_esti...](https://en.wikipedia.org/wiki/Minimum- variance_unbiased_estimator)

</comment></comment></comment></comment><comment><USER6> One other thing which both this and PG's original theory get wrong:

Their basic premise is wrong, if bias continues to exist after the selection event in question.

For example, if YC had (hypothetically) a real bias against black or women entrepreneurs, it is _almost certain_ that future funding rounds, as well as all possible exit scenarios, would exhibit very much of the same bias.

In which case, the future "performance" of those candidates would be poor, and by PG's definition unbiased even though the only meaningful result is that YC is no _more_ biased than subsequent performance evaluations.

<comment><USER7> Let's assume that bias does persist past selection through the duration of the program. Does that change the interpretation when you look at First Round Capital's data that shows its female founders outperforming the males by 63%? I don't think it does.

The test may not be sufficient to prove that you have no bias, but it may be good enough to prove that you do. When it does indicate bias, it seems likely to be correct.

To put it another way, if it is 1948 and the only three black people in Major League Baseball are all superstars, then the distribution of baseball skill among black players is extremely unbalanced or there is a lot of bias keeping the average and moderately-better-than-average black players out.

<comment><USER8> The interpretation is still somewhat unreliable as in many cases there's still another hypothesis, which is that $UNDERREPRESENTED_MINORITY actually overperforms due to _favourable_ treatment after the selection process

Of course favourable treatment can't make people into superstar startup founders or baseball players (and I'm sure any special treatment afforded to black baseball players in the 1940s was the complete opposite of favourable). But more generally it can make an organisation with fair selection processes look like it sets a higher bar for $MINORITY because it addresses low numbers by being very keen to promote and very reluctant to fire/deselect members of said minority, so these kind of studies still have to be considered with care.

(Of course even if an organisation is proactively treating a minority group favourably after selection doesn't mean that conscious or unconscious biases don't exist in the selection process.)

</comment></comment></comment><comment><USER9> The fact that they have to exclude Uber for no good a priori reason should have been raising red flags all over the place.

"But Uber skews the results!" So what? You don't get to just throw out data points you don't like without good reason.

If your "test" is that sensitive to individual outliers, then perhaps it isn't really a good test after all.

<comment><USER10> Dropping outliers is common in statistical analysis.

<comment><USER11> Dropping outliers can be done when outliers cloud the analysis, but doing this in an analysis of startups is inane since startup investors' _entire goal_ is to find outliers.

<comment><USER12> Possible. In this case, we're not looking for outliers or measuring based on financial success, but trying to tell if the VC is systematically biased anti- woman.

It's not clear that dropping outliers is a bad idea there. It's also not clear it's a _good_ idea, granted.

<comment><USER13> Well, if you are trying to measure whether men founders or women founders you have funded on average make more, then you would have to include Uber. The real issue with the analysis is that results are unlikely to be statistically significant due to small samples and high variance, which means they are useless.

</comment></comment></comment></comment></comment><comment><USER14> Very nicely done Chris, but the basic problem with Paul's analysis is not the mathematics (this can be fixed as you have shown), but the underlying data. Any data set you could get to measure bias in the start-up world is too small and messy to tell you anything useful. No matter how sophisticated your analysis, if the data is garbage then all you will end up with is garbage.

This does even consider the problem of data dredging which First Round Capital engaged in.

<comment><USER15> Maybe First Round's data set is too small or messy to get meaningful results, but the entire startup world has plenty of data to potentially pull some meaningful conclusions about bias.

I wonder if all YC companies would be enough data points to learn something useful. Or maybe grab a large swath of VC-funded startups including First Round's investments and many other top firms.

Most of the raw stats in Chris's post was above my head, but I'd love to see this applied to a larger data set of fundings.

<comment><USER14> The basic problem is getting hold of good data. Most VC rightly consider their data in this area very valuable and they are not going to part with it easily. Not even Paul delved into YC's data.

More fundamentally even if you could get enough data, the data is just too messy to analyse and draw any valid conclusions.

</comment></comment></comment><comment><USER16> > The idea is generally correct - bias in a decision process will be visible > in post-decision distributions

I find what's wrong with the idea more fundamental, that it talks only about the 'selection process' but in fact bias that impacts success or failure can come at other points.

<comment><USER14> This is really important. Lets say the whole VC ecosystem is biased against redheads (just to pick a random group). What would happen is the redheads would under perform other groups as they were discriminated against at each stage of the VC lifecycle. They would not show up as a group that over performing later. The only bias you can detect using Paul's approach is bias that only applies at the initial stage and not later.

<comment><USER17> Just to cross the beams of pedantry here for a moment, a widespread and well known --- if less than serious or systemic --- cultural/social bias against red haired people, probably first coming into public consciousness in North America due to the infamous South Park 'Ginger' episode, has in fact primed you to select "redheads" as a non-contentious example of a plausibly ethnic group that might be discriminated against, something that every red-haired person knows, although you apparently do not. This means that the choice is statistically insensitive and the social methodology is poor.

Actually, choosing an identifiable group at random would be both socially and statistically unwise, as, following Patero distribution, there are vastly more minority/extreme minority distinguishable groups of people than there are majority/significant minority ones; this means, firstly, that any group randomly selected with equal biasing between all groups has a high probability of being subject to actual discrimination, mooting any social benefit of choosing a group at random; secondly, that the generalizable qualities of the group chosen would therefore have a distribution with very little deviation (if I'm using my terms correctly) and would be highly predictable, thereby obviating any possible statistical benefit of doing so.

<comment><USER18> I'd prefer to imagine that it's because the majority of the HN readership reads Nature for their regular dose of science fiction: [http://www.nature.com/nature/journal/v453/n7194/full/453562a...](http://www.nature.com/nature/journal/v453/n7194/full/453562a.html)

Alternatively, I'd be OK imagining that it was subconsciously chosen here not at random, but because I used this reference for an example in the previous thread.

Is South Park another journal worth reading? Are they open access?

</comment></comment></comment></comment><comment><USER19> I think this is the first post that's a DH5 on pg's _How to disagree_ scale ([http://paulgraham.com/disagree.html](http://paulgraham.com/disagree.html)). Not only that, the OP is charitable enough to explicit state why it's not a DH6:

 _Paul Graham wrote an article about an idea. The idea is generally correct - bias in a decision process will be visible in post-decision distributions, due to the existence of marginal candidates in one group but not the other. But the math was wrong. // That's ok! Very few ideas are perfect when they are first developed._

I'm not good enough at statistics to check that OP's math is sound, but _this_ is the mindset of a scientist. OP reasons rigorously, finds a way to salvage the core insight and improves on it. As readers can see, it took quite a lot of work and prior knowledge to do.

If I were pg I would consider putting a link to this post on both the _disagree.html_ and _bias.html_ as a note for posterity.

<comment><USER20> To be fair pg's scale isn't perfect. DH1-DH5 is something you can improve if you want to refute a given argument. DH6 is about choosing the "main" argument you want to refute. There is nothing to improve between DH5 and DH6, especially if you agree with the main argument but you disagree with some minor argument.

</comment></comment><comment><USER21> I think this is a really good start for the most common types of bias. A few counter-examples that might slip through the cracks of this test:

Only examining the sample without looking at the population of applicants has its limits. Especially as multiple interviews becomes the norm, filters that don't affect the distribution of outcomes will be missed. For example, the person screening resumes might weed out anyone with an ethnic-sounding name. A different person, who is not biased, interviews the candidates. The quality of the candidates accepted will be the same, but the number of minority applicants will be smaller than it should be.

Measuring outcomes allows for external biases to distort the results. Start with a company that is biased against women, so that the average female founder is better than the average male. However, that same level of sexism exists in the market, such that the company's performance is hampered due to prejudice against the founder. The VC's bias would be hidden by the counter- bias in the market.

</comment><comment><USER18> (comment reposted from the earlier submission that didn't catch: [https://news.ycombinator.com/item?id=10513574](https://news.ycombinator.com/item?id=10513574))

Hi Chris ---

In the earlier thread, it seemed like some people were reaching different conclusions because they were using different definitions of "bias". I think my working definition would be something like "there existed in the actual applicant pool a subset of unfunded female founders who should have been statistically expected (given the information information available to the VC's at the time of decision) to outperform an equal sized subset of male founders who did in fact receive funding".

Alternatively (and I don't think equivalently?) one could reasonably take bias to mean "Given their prejudices, if the same VC's had been blinded to the sex of the applicants, they would have made funding choices resulting in higher total returns than the sex-aware choices they actually made." I'm sure there are many other ways of defining "bias". Could you define what would need to be true for your test to show that "the VC process is biased against female founders"?

<comment><USER22> My definition is the same as yours - it's exactly about the existence of rejected women who are better than accepted men.

This particular test is terrible for VC since the min return in VC will always be zero. But if you build a noise-sensitive version for something like college admissions, what needs to be true is a) bias manifests as raising/lowering the bar for one group relative to another, and b) both groups have a significant number of members near the cutoff.

As an example of the type of bias this test would detect, consider U-Michigan's point system [1]. An extra +1.0 GPA was added to black applicants. I.e. an Asian person with 3.9 GPA and black person with 2.9 GPA were equivalent. This would result in Asian people having a higher min GPA than black people.

[1] They replaced the point system with vague human heuristics when the supreme court said point systems can't be racist, but vague heuristics can.

</comment></comment><comment><USER23> Maybe I'm missing something but this seems like a pretty perfect application for the bootstrap - a remarkably intuitive but powerful framework. Without loss of generality, imagine that you have two populations, A and B, and that you want to test some hypothesis about a statistic of A being different from a statistic of B (mean, in this case). Using the simplest form of the bootstrap you would do the following:

1\. Pool and randomly label the data from A and B 2\. Sample with replacement and form two partitions of the same cardinality as the original A and B groups 3\. Compute the differences in mean 4\. Rinse and repeat millions of times to form a distribution of mean differences 5\. Check if the observed difference in means (from the true A/B labels) is statistically significant relative to the distribution found in (4)

This has some problems with fat tailed distributions but tends to work great otherwise. It's so simple that it avoids a host of pitfalls that can arise with other resampling schemes (what's being proposed is a type of resampling), and I love that it makes basically zero assumptions on the underlying data.

</comment><comment><USER24> Sorry, I still believe that a better approach is in

[https://news.ycombinator.com/item?id=10484602](https://news.ycombinator.com/item?id=10484602)

That post shows that what PG is doing is a first-cut effort at a statistical hypothesis test but with being vague on assumptions and without any information on false alarm rate.

In particular, in my post, get to compare sample averages without making a distribution assumption. Indeed make no distribution assumptions at all.

Yes, distributions exist, but that does not mean that we have to consider their details in all applications!

Come on guys, this is distribution-free statistical hypothesis testing, and we should be able to use that.

</comment><comment><USER25> So the alleged flaw in pg's reasoning is his assumption that the best applicants from two large groups of humans should turn out to create equally successful startups on average, if the selection process is not biased.

Is this really such an unreasonable assumption, given that pg restricts the applicability of his bias test to groups of equal ability distribution and that we can assume that both groups have roughly the same amount of capital at their disposal?

The question is if the "equal ability" qualification is sufficient to make sure the distributions are roughly similar. But that is not a mathematical issue.

<comment><USER22> The point of the post is to relax the "equal ability distribution" assumption. If the distributions are identical, any disparity in outcomes must be caused by bias.

</comment></comment><comment><USER26> Honest question: suppose I pick a plausible sounding h(x), perform the proposed test, and get a vanishingly small p-value. So I feel pretty happy about rejecting the null hypothesis. But the null hypothesis is a conjunction: A group members accepted have cdf a(x), B group members accepted have cdf b(x), and h(x) satisfies various technical conditions related to a( ) and b( ). So when I reject the null, I'm saying the data were unlikely to be generated by the hypothesized process. Couldn't that simply be because I guessed the wrong form of h( )?

</comment><comment><USER27> Thanks for the write-up Chris. Now I understand why I couldn't follow the path of logic you were laying out in our original discussion in PG's article's comments.

The main problem I was having is that you are assuming our observation variable is the latent skill or potential value variable (which you're calling x here). However, the article by PG was talking solely about the average of returns (let's call it y).

So the reason I was confused is that, assuming that the outcome of a startup is dependent only on x, we are really observing y ~ f(x) = \int_0^1 g(x)h(x)dx, where h is your cut-off criteria for x, g(x) is some unknown payoff distribution for a given skill level, and I'm assuming our x is in [0,1] without loss of generality. So in essence, the real problem here, even if you could see all of the individual returns for a given portfolio, is that you have to perform a very, very difficult deconvolution problem. And I'm pretty sure it's non-identifiable without some other information or additional parametric assumptions.

Thinking out loud a bit, let's assume that y is actually log(return), where a return of 1 is breaking even and 0 is losing everything. Since log(0) is undefined, most startups return 0, and very few exit for less than 1, I would think we could model this as a point-inflated normal distribution: p(y) = c * \delta_0 + (1-c) * N(\mu, \sigma^2). Given this, we could then model our latent parameters (c, \mu, \sigma) as being functions of x. Since the model is separable, we can even just look at the zeros and non-zeros in isolation. Then we can come up with a test from there, but I'm not really sure what that test would be at this point. Anyway, that's a completely different line of thinking, but it seems much more tractable in practice.

</comment><comment><USER28> It's since been dressed up with some mathiness, but this idea was originally proposed in the comment threads of pg's original article. [0] See the responses there for a few reasons why it just won't work.

To be concrete, assuming "performance" is measured as return on investment, min(performance) will always go to to -100% (i.e., bankruptcy) with a large enough sample size.

[0] [https://news.ycombinator.com/item?id=10484200](https://news.ycombinator.com/item?id=10484200)

<comment><USER29> With a large enough sample size, someone might find a way to break the -100% barrier.

</comment></comment><comment><USER30> Interesting tests which can reveal your current implicit (read hidden) attitudes toward race, gender, color.

Worth to do and discover few facts about ourselves, even if uncomfortable.

[https://implicit.harvard.edu/implicit/](https://implicit.harvard.edu/implicit/)

</comment><comment><USER31> To use maths/statistics to reason that YC is not biased against certain groups of applicants is amusing. But to even consider that technical female founders are weak candidates is disappointing.

The sample population was chosen by specific type of groups of partners. There is no female technical partners in the group. As a female technical founder, I am not interested in building 'tea-making bot', sandwich making bot or selling organic condom. IMHO we have different views in looking at problems and solving them. Without having female technical founder as partner, YC would be perceived to be biased.

The algorithm of selecting promising candidates will vary once there is a variety of partners.

</comment><comment><USER32> I think the more fundamental flaw in PG's argument, which is just as present here, is that it assumes the populations are otherwise identical. That's obviously not the case -- there's no random assignment for bias -- so this sort of test can't tell you anything direct about casuation.

Any credible statistical test for bias should be framed in the language of causal inference, e.g., as described by Judea Pearl: [http://ftp.cs.ucla.edu/pub/stat_ser/r350.pdf](http://ftp.cs.ucla.edu/pub/stat_ser/r350.pdf)

<comment><USER22> This test explicitly does NOT assume the populations are otherwise identical. See the graph right after Theorem 1 - it shows two unequal distributions satisfying the assumptions of the test. That's the whole point.

</comment></comment><comment><USER33> This still seems like nonsense to me. What if you reject each black candidate (independently) with probability 0.5, and the proceed to perform a fair interview process with all remaining candidates?

Surely the distribution of minimums would then be the same between all skin colours, but you end up employing half the number of black applicants that you should be.

</comment><comment><USER34> Wow that's a really nicely rendered page.

TIL about [https://www.mathjax.org/](https://www.mathjax.org/).

</comment><comment><USER35> Posting this, since no one seems to have pointed it out:

> So rather than comparing mean performance, we'll compare minimum > performance.

 __1. __This is a useless metric for startup investors to use, since (almost surely) the minimum performance in _every_ group of reasonable size will 0 (the startup went out of business)... and this will be true even if the investor is biased.

 __2. __The maximum statistic was rightly avoided here because for power-law distributed values (which startups returns are), you 'd need to know the population sizes to estimate if the distribution of {A} was different than the distribution of {B}.

If you're willing to take on faith that both A and B have the same distribution, then the test is easy: is the acceptance rate for As significantly different than the acceptance rate for Bs? If you've invested in more than, say, 100 startups, you have a big enough sample to check this... this requires knowing the size of application pools, and who was accepted though.

 __3. __I believe that in general it 's not possible to determine a bias from the kind of aggregate statistic pg is discussing without at least some knowledge of the sample space.

For example, using OP's method, you will find that almost every selection process in the world is biased _for_ you if you divide the world as {you} vs {non-yous} (you're doing significantly worse than the best non-you). And find that almost every selection process in the world is heavily biased _against_ you if you use the minimum statistic (you're doing significantly better than the worst non-you). This is also true for smallish groups (eg {your friends} vs {not your friends}).

The same is true for PG's method -- it's highly unlikely that {you} fall exactly at the average value of {non-yous}, or that {your friends} fall exactly at the average of {not your friends}.

 __4. __I believe that the math here is distracting from the core question.

Core question 1: Do men and women on average make the same choices?

If you believe that, then determining bias is easy: we already know who the investor funded. Is the number of men the investor funded different from the number of women? Yes? Then the investor is biased. This is much more direct than the the kind of forensic accounting pg is proposing.

I suspect that pg didn't propose this test because pg doesn't believe that men and women on average make the same choices. He knows, for example, that the number of female applicants to YC is different than the number of male applicants (a gendered difference in behavior). Google "men and women career choices" or similar if you're interested in learning more, or better yet, read some first person accounts from FTM men about the cognitive effects of taking testosterone.

Since it's clear that there's a gendered difference _before_ applying to YC, it seems very difficult to justify an assumption there would be no gendered difference in behavior _after_ applying to YC (or any other investment firm, FirstRound in this case). Given that, the question we were asking becomes much more confusing... a simple bias towards ideas and plans you understand/agree with/are excited by is a gender bias in as much as your gender caused you to like the idea or plan. Removing that bias (supporting plans you understand less, agree with less, or are less excited by) seems like an obviously bad idea.

Returning to the problem: if we accept that this sort of "makes sense to me bias" can be observed when looking for gender biases, we are left in a really hard place. That bias seems to be both a good thing, and confounds the entire analysis. Unless you've controlled for the "makes sense" bias, such analysis will apply pressure for investors to waste money from their perspective. This seems obviously bad.

Core question 2: which biases do we want investors to have?

Investors who knowingly pass up good opportunities on the basis of the founder's gender are punishing themselves worse than any company they pass over -- their competitors who aren't gender biased will get higher returns, and so will have more money to invest in the future. This is to say that gender biases for startup investing are self-correcting. The investors already have their self-interest maximally aligned with not being sexist.

I don't pretend to know which companies are worth investing in more than any other smart technologist. I also don't pretend to know to what extent gender differences cause differences in returns, so my answer is: investors should be as biased (selective about investing) as they see fit. Startups are positive sum for society, and anyone who can find a way to fund more of them profitably is making the world better.

In large part, this is because I find it very unlikely that any modern investor is knowingly sexist -- I think it's much more likely that the sort of "makes sense" bias I discuss above is at play.

Of course, this is an early thought that came from first principals, so counter arguments solicited. Perhaps there is something deeply evil about passing over startups you don't feel comfortable investing in (assuming that comfort has any correlation with founder gender), or perhaps there's some easy fix which makes previously dicy-looking ideas from {other-gender} founders look like obviously good investments. (If you know what that idea is, I'd love to know it too).

5\. Thanks to both pg and Chris for the fun math/philosophy problem. :)

</comment><comment><USER36> This kind of thinking could be problematic. What would happen if someone compared the performance of whites and persons of African heritage at college?

<comment><USER35> They would detect if the acceptance criteria were biased. This is a good thing, since after you've measured something you know if a change is in order.

</comment></comment><comment><USER37> Lots of math in here premised on shaky foundations:

>Group A comes from a population where the chance of graduation is distributed uniformly from 0% to 100% and group B is from one where the chance is distributed uniformly from 10% to 90%

>The mean of group B is not lower because of bias (which would be reflected near x=80), but because the very best members of group B are simply not as good as the very best members of group A.

Yes, if we can assume some a-priori knowledge about certain "groups" of people, then we can make a more "informed" decision. That's pretty much the definition of bias, isn't it? Paul Graham's point, as I understood it, was that those assumptions are often invalid. Therefore, bias could cause the market to under value someone or some company. Your counterpoint seems to be, "let's suppose those biases are legitimate."

<comment><USER38> Read it again. He's talking about the counterexample there. It's a hypothetical.

<comment><USER37> Yes I get that. The crux of his argument is:

>Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C

His argument is based the proposition that different sexes/races have different market value profiles. He needs to demonstrate why that is the case before proceeding to heavy math.

<comment><USER39> > His argument is based the proposition that different sexes/races have > different market value profiles. He needs to demonstrate why that is the > case before proceeding to heavy math.

Not really, his argument is that "PG's mean-post selection test (the 'PMST') is only valid if different sexes have the same distribution of abilities". If you or PG believe that the PMST is a valid way of showing that bias exists, the burden is on you to show that different sexes have the same distribution of abilities.

<comment><USER37> From the Paul Graham article:

>You can use this technique whenever (a) you have at least a random sample of the applicants that were selected, (b) their subsequent performance is measured, and __(c) the groups of applicants you 're comparing have roughly equal distribution of ability. __

So yes, OP is ignoring the entire premise of PG 's argument.

<comment><USER39> OP acknowledges this... "Unfortunately, using the mean as a test statistic is flawed - it only works when the pre-selection distribution of A and B is identical, at least beyond C."

To me the rest of the article is asks the question, "requirement (c) is really strong, is there a way we can use post-selection statistics to determine bias while weakening (c)? what if we tried measuring the post-selection minimum instead of the mean?"

Also PG edited his essay to add that disclaimer only after WildUtah's comment, so it's possible that OP hasn't read the updated version.</comment></comment></comment></comment></comment></comment>