Protect Kids, Get F*cked - <AUTHOR> http://medium.com/what-i-learned-building/6972e9e7c15b

<comment><USER2> I can add a bit more color to this story as I was intimately involved in it.

I worked at the company that was bought by HP in Oregon and then later in Texas. It was called RLX technologies and created the first blade server. After it was purchased by HP I stayed on for a few months before moving on to a startup in Oregon. It seemed like a great idea as they moved me back to Oregon.

My best friend, who got me on at RLX after my first startup (NEI: high speed telecom startup) had crashed and burned, had taken a position at this startup in Oregon (we both were from there and had met after graduating from Oregon State). It seemed like a way to continue the work we had started at RLX and would get me back to Oregon. So win-win.

Long story short: the startup had issues and he moved on to found IMSafer. I would of liked to but needed to stay on for another 6 months, so I wouldn't have to pay back half of my moving expenses.

Once I finally moved on to IMSafer the company was still small with only 6 of us. It was a great time and we built something truly exciting and helpful. We had a wonderful time at the Techcrunch 40…, but that's a story for another time. :) I believe Paul knows this story too, as it was one of the big reasons we got into Ycombinator!

I will skip to the end as Jason supplied much of the interesting info and the color to that debacle. But, I will add, just as you don't count your chickens before they hatch you never, never invite your new 'partner' (board member, investors…whatever) to a board meeting until the money is in the bank.

Once the assets were sold only 2 of us stayed on with the new company that bought us. It was a trying time, but my best friend did get to move back to Oregon and it was just us, keeping a rather complicated site up and running while working on creating the 'new' product that was to make all the money.

While, at IMSafer we had realized that there could be big money in using this software in a more enterprise fashion. We realized that many virtual worlds for kids were popping up and they had use for software like this. He, I and another one of our friends from our old RLX days set out to create a real-time piece to the software that was much better at moderating less dangerous content like curse words, etc. It would quickly mark words they found objectionable and pass the text on to the more complicated software outside their enterprise to look for the truly dangerous things.

From the point that IMSafer crumbled to when he and I decided to leave or got fired…depends on who you are asking, we had grown the business to a million dollar run rate in 8-9 months. But, we had been bought by a startup and we had very different opinions on how to run it with the purchasing company…and we lost!

About that time we left/got fired we decided to do something new. Both of us had vary different experiences buying a new car and decided we wanted to fix that. After getting in a small fight with our former employees we got them to relinquish all rights to CarWoo! and proceeded to build it. At the time it seemed a bit crazy to fight for something that wasn't much more than a landing page and a bunch of crazy ideas, but in hind sight…seems like genius….I have decided to tell it as a stroke of genius!

After we got into Ycombinator we told Paul about this story and at the time he said it was the worst 'pulled funding' story he'd ever heard. It's possible he's heard worse since, but somehow I doubt it.

5 of the 6 of us were married at the time and 4 of us had children. As we were in the tussle over CarWoo!, which was almost nothing at the time my wife had our second child.

Tommy and I got our call that Ycombinator had accepted us while sitting in benchmark's parking lot in the spring of 2009. We had come full circle i guess you could say.

We moved down to Palo Alto, left our wife and kids behind for the time being and the rest is history.

<comment><USER3> Quite a story, one of the best and most revealing ones I've ever read here, and I've been here since the beginning. Thanks for sharing, and letting us see "behind the curtain" a little.

<comment><USER2> Thanks. It's pretty cathartic to talk about it. I haven't thought much about it in a long time. I skipped a lot of the details around CarWoo! too...at some point we will get around to posting that.

</comment></comment></comment><comment><USER4> When Jason sent this to me as a preview, I did not think it would end up on HN. Interesting.

I want to respond to a few of the comments, as I was the main protagonist in getting this idea off the ground, CEO of the company, and the business/product guy surrounded by the talented engineers who made this happen.

The decision to invite the board member on to the call the day before the money was in the bank was my call. My biggest mistake of my career. I have learned from it, but it was my call and I blew it. Nothing about the situation gave us any reason to believe that we would have a problem. By the way, if anyone wants a front door key to a tier 1 VC firm, I am sure one of us still has theirs.

On the topic of the business model. Let's first talk about the derivation of the idea. Having spent my career in tech and Wall St, I have seen the migration of high IQ talent into spaces where the marginal utility of IQ was trending to 0. We wanted to find an opportunity where we could be smart and do new things, and that would, theory lead to money. The base thesis here was that no one had made money in the kid protection space for a while. IQ follows money. Viola. We were right - there had been no innovation in this space in a long time, and everyone was focused on keeping porn off computers. We decided to try and focus on keeping bad guys from getting to the kids.

We can all have an intellectual discussion about whether this was an overblown social issue or not. I don't feel like rehashing that here. What we did know is that we had a novel approach to solving this language problem, and we built an engine. We did not solve it with Bayesian analysis or ML. We went a different route, and the engine was very effective.

Our premise was that we could sell the software to parents. We did all the things many of you would do. Built an MVP (before there was a phrase for it), talked to customers, etc. It turns out that the axiom from real estate is true here. People lie. What parents said they would pay for, and how they reacted when using the software were totally different. Many of them expected the software to come with a computer, or be paid for by someone else. They told us they would pay for it, but in reality they expected it to come from an AV provider. That never would have surfaced in talking to customers ahead of time.

We then pursued a deal with McAfee. I don't know if at this point in time I can talk about that deal or not, but suffice it to say, that wasn't the golden ticket.

After spending time refining the business model to get parents to pay, we (as many companies do) fell backward into our business model. It started with a fortuitous call from a large company making a massively multiplayer game for kids. It turns out many of these companies who cater to kids have rooms full of moderators for in game chat. People are very expensive. Some tech found out about our software, installed it, and found it very difficult to beat. They were intrigued enough that they asked us if they could install it inside their server farm. When we took the call, every answer from us was "yes of course" but in reality we had no idea. Credit to Tommy and Erik for figuring out in short order how to make the demo a reality. The first company figured they could reduce their people count from 600 to 60 with our software being the first pass filter.

Once we figured this market opp out, we started reaching out to other companies. Long story short, we were looking at deals of $250K per year for our software. Bad timing being what it is, this was all happening around the time of the financing closing. So while Benchmark (wow, that's the first time I have said that name out loud in connection with this story in public since 2003) was moving forward on what they thought was a consumer software deal with massive growth potential, it looked like we had a business opportunity in enterprise-y software for community moderation for companies catering to minors.

When Benchmark pulled out (worst kick in the stomach feeling of my life having to pass that news on to the team) we scrambled. The deals from these game companies were coming along but we weren't going to have enough runway to get them closed. The eventual buyers of the company were building similar software, and they had heard our name enough in their calls with their potential customers when being told why they were not being moved forward in the deal that they wanted to buy us. It was a little bit of good luck and timing in an otherwise crap sandwich state of play.

To sum up: 1) We knew who we thought our customers were: parents 2) We were wrong about how they would act: parents didn't want to pay for this type of service, despite what they told us 3) We were wrong about the virality of the service: despite saying otherwise, parents wouldn't tell friends they were running this software because of what they perceived as potential social stigma 4) You can sometimes fall backward into your business model 5) Benchmark did us wrong, but that's life. Move on and do great things. Jason, Erik, Tommy and David are some of the most talented people with whom I have ever worked, and I rate IMSafer as the time of my life. We had a lot of fun, and built some amazing stuff (patent filing in case you are interested: <http://www.google.com/patents/US20090089417>) 6) Survivorship bias is a bitch, and there are many stories of smart guys not getting over the hurdle which never get told. We weren't idiots, or morons, or whatever. Sometimes things don't make it. The real question is what did you learn and how to you do something constructive with what you learned.

I am happy to answer any additional questions.

<comment><USER5> Thanks for the candidness and honesty. It is really refreshing to hear the details--mistakes and all!

</comment></comment><comment><USER6> _We received email after email from parents thanking us for saving their children's lives, literally._

Yes, please tell me more about all these children whose lives your service saved. Just like _To Catch A Predator's_ ridiculous premise (that digusting, fat 50 year-old men who live in their parents' basement are hooking up with hot teenagers), yours reeks of trumping up a problem that only exists on the extreme fringe. Literally.

 _one of the partners installed our software. He tested it, he used it, it worked. He got it. He called us and said he wanted to invest as soon as possible._

Who is this investor? I want to know because anyone who would vote money on an anecdotal experience like this is an idiot.

Your story is interesting, but only when it sticks to the parts about you rushing into an investment without due consideration or self-protection.

If you wanted funding, you should've asked Congress for it. They love to exaggerate child sex exploitation as much as your service did.

<comment><USER7> Indeed, tell us more about these children's lives you saved. I've done extensive research on these occurrences when my first startup internet provider first encountered these claims in the nineties. We never found any evidence whatsoever. We did find an overwhelmingly large number of people ready to go on a witch hunt or lynch these so called perpetrators, but that's been going on for thousands of years now.

Has anyone claiming child pornography online ever actually found something? Again, please show us, they would be my first datapoints ever in our assisting of the police in investigating these.

The business model of the OP is itself predatory on the fears and underbelly feelings of uninformed parents. Better to explain to those parents how to raise children so that they are cautious about anything they encouter online.

<comment><USER8> I used to work for a content filtering company. Yes child porn exists, and it's one of your worst days when these sites start popping up in what you're reviewing, no, we can't send the links to others to "prove" it exists. I no longer work there and I'm glad I don't have days like that anymore.

</comment></comment></comment><comment><USER9> What a shitty situation. But why, oh why, would you sell your house and move from Houston to California before the check cleared? We started Matasano when my kids were pre-K and a a full-day every-day handful, and I had to spend a couple months away from my family while we worked out the move (mine from Ann Arbor to Chicago). I know I'm not the only founder to deal with that situation. Was there some additional circumstance involved here, some other pitfall people should be on the lookout for?

<comment><USER10> I know directly of one case in which one of the terms of the acquisition was that in case of the founder's death within the retention plan, his family wouldn't get the remaining part of the money (which would be given at the end of the retention plan only if he was still alive). This founder, rightly so, thought that it was an awful concept, since his kids needed to be protected exactly in the case of his death. The purchaser said they would have not changed the contract since that was a standard clause and that it was unconceivable to consider changing it. The startup called everything off the day before the deal (even after it was announced), because of this.

The clause that was impossible to make disappear suddenly disappeared overnight.

P.S. Congrats for Matasano, I am a fan of your company.

<comment><USER11> _was a standard clause and that it was unconceivable to consider changing it_

 _The clause that was impossible to make disappear suddenly disappeared overnight_

Oh, if I had a nickel.

Younger readers (and older ones, too) should be aware that this is one of the easiest lies to slip from someone's lips: "oh, that's standard, we can't get rid of it."

You need someone experienced on your team -- even if not working for you, a friend you can call -- to find out just how "standard" these things are.

 _EDIT_ : even terms that really and honestly are "standard" can be changed

</comment></comment></comment><comment><USER2> I think I need to clarify what IMSafer did. A lot of arm chair QBs here talking about how's it's evil or that abuse of children in a sexual way just doesn't happen...that could not be further from the truth.

No offense (well we know where this is going), but anyone saying this has no f*$%ing idea what they are talking about.

What we found in those years with IMSafer and later with the company that purchased our assets was mindblowing in a terrible way.

In the US we weren't allowed to really do anything with the data we found even in the most awful cases. When the patriot act was in affect there were opportunities to turn some of the data over to the gov agencies that deal with exploited children, but it was unclear.

In the UK they were allowed to more freely use what they found to put pedophiles behind bars...which they did on more then one occasion.

In some cases it appeared that children had installed the software as a desperate cry for help against their abusive family members.

In one particular egregious case the offender was involved in group chats where he would try to normalize the behavior to his child by having other sick individuals tell her it was OK. Then, they would setup times for her to be online with them while terrible acts were perpetrated on her.

No one is acting as if this was common. But, it did happen. And we weren't selling the chat, running ads against it or in anyway trying to monetize the 'fear' or grief of our users.

A lot of people here do not have kids and don't understand how hard it is for parent's to talk their kids about all sorts of issues around sex and drugs and rebellion etc.

In many cases IMSafer was their last hope and it was very helpful to a lot of people.

We never started it because we wanted to spy on kids or know people's gossip. We did it because we realized that parents were ill equipped to deal with the new online world.

Porn was a not a concern. To us that was your child viewing something. What we were concerned with was someone coming into your child's world and talking to them without your permission. If the child was having an issue with their parent's these kind of people would pick at that and try to show the child they were on their side to win trust.

Places where parent's thought their kids should be safe just weren't and still are not.

Also, some kids are susceptible to these kind of attacks by predators. In some cases parent's had kids who had been sexually assaulted in real life already and they were very afraid, because of their child's mental makeup now, that they would make easier targets.

We had adults buying plane tickets for kids and working out details to get them to fly across the country to meet under the guise of some camp the kid was going to, etc.

It's real whether you want to deal with it or not. We did our best to balance online privacy while helping people.

I feel like we did a phenomenal job and we didn't sell fear. It was never part of our marketing or selling.

<comment><USER4> To pile on a little bit to what Erik shared here. There were several use cases that we could not have imagined when we set out on our path.

First, there was a case where a high school girl was equipment manager of her school football team. She was gang raped by some of the players. She was talking about that in chat with her friend, talking about committing suicide, and talking about how she couldn't tell her parents. She was afraid of what they would do to her. To her! Our engine caught much of this, and flagged it for the parents. It worked out in the end. We found out about this after the fact when the parents contacted us to let us know how our software probably saved her life.

Then there was the situation where there was a father who was a community fire chief. His child installed the software on his father's computer because his father was abusing him, and was spending time in really really bad chat rooms. The kid was hoping we were monitoring the alerts. The child eventually reached out to us asking why we hadn't sent the police. It was heart breaking.

Lastly, there was a kid who was buying drugs from a contact online. Is this a sexual predator? No. But a predator nonetheless. Our software flagged it.

There are bad people out there. Some of them are the Chris Hanson predator types. They aren't all sexual predators, but they are still looking to gain access to children without parental permission.

Children and young people react with "this software violates my rights!" As a child, under the age of 18, in the US, has no rights. A parent is supposed to parent. They have no right to privacy.

The software was not meant to record chats. It worked as a sort of a sliding window, and only when there was a flagged issue would a parent get a notification. That notification would include some of the chat around the offending line to give context, but nothing more.

With regard to access to kids - parents can look into a living room and ask "who is that?" Well it's Johnny. He lives down the street. The parents can talk to Johnny's parents. When a child is in an MMO and has a "friend" list a mile long, parents can't really ask "who is Fragg3rKill3r225?" Worse, their kid may not really know who that person is either.

At the core, IMSafer was about helping parents manage the relationships their kids were building online. As a parent, it is your job to know who has access to your kids, and how they are influencing them. Most parental control software is billed as keeping porn of the computers. IMSafer was about keeping the bad guys out of the kids lives.

<comment><USER12> _Children and young people react with "this software violates my rights!" As a child, under the age of 18, in the US, has no rights. A parent is supposed to parent. They have no right to privacy._

It's absolutely false that children in the US have no rights or that parents can legally violate any of their rights arbitrarily. It's closer to the truth that children don't have a legal right to privacy from their parents in their parents' home using their parents' computer and internet connection.

This attitude is problematic though. Instead of encouraging a more open dialog between parents and their kids, parents using spyware creates an adversarial relationship. Under those circumstances, the kids will find ways around it, which you describe elsewhere as "unfortunate". I have no doubt there were situations where your software resulted in very good outcomes, but I'm inclined to suspect that in most cases where parents installed it, there was something broken in the parent/child relationship. Unfortunately "No mutual trust with your teenager? Our robot babysitter will let you know if she's talking about suicide online." probably isn't the world's best advertising slogan.

</comment></comment></comment><comment><AUTHOR> This sounds like the story Jessica Livingston mentioned at Startup School 2012 <<http://news.ycombinator.com/item?id=4699862> >, if I remember correctly -- business in Texas got a term sheet, moved to California, investors pulled out when their user acquisition metric changed. Anyone have more color on this?

<comment><USER13> Yes, this is the same story.

</comment></comment><comment><USER14> Let me just quote one bit of that:

 _Our service was free as we were working out the business model. We actually had a genius business model but hadn't implemented it yet. We wanted more users so we spent money on marketing._

Maybe the real problem wasn't the insurance?

<comment><USER9> No, the real problem wasn't insurance; it was that a firm-seeming commitment to fund the company fell through at the worst possible moment.

A lawsuit wouldn't have fixed anything for them; it'd have taken years to resolve.

<comment><USER15> The real problem was they had no business model; spending money to get users when you don't even know what your return is? That's ludicrous. That's a core business issue from day one.

<comment><USER16> twitter, google & facebook might disagree

<comment><USER17> Twitter/Facebook grew virally, not through paid marketing to acquire users.

</comment></comment></comment></comment></comment><comment><USER12> _Age? Sex? Location? That's how predators started their hunt._

It's also how teenagers started conversations with each other in chat rooms a decade ago. It may still be, but I haven't been a teenager in a while. I imagine predators open with the same question, but actually using it as a filter would produce an unacceptable amount of noise.

This is a useful story about some of the mistakes one can make dealing with VCs, but it sounds like the product itself was a little questionable.

<comment><USER18> In reading the article and thinking back to being a kid who'd create free- trial Juno accounts and dig through drawers to find the Post-It notes that my parents' AOL accounts' passwords were on just to spend hours talking to strangers on the internet because it was exciting, I have to agree with the notion that this software rubs me the wrong way.

This is an issue of parenting; you can't just turn on an app and expect it to do your job for you. Kids won't be dumb if you treat them with respect and show them what can happen if they don't keep their eyes peeled.

Like the ridiculousness of the Dateline conversations, it was always very obvious when someone was creeping on me and it only took one click to block them. Allowing my already-fearful and ignorant parents to read random snippets of messages I was sending and receiving - particularly because a lot of our ASL?? comments were jokes to begin with - is a violation of my privacy and trust. Growing up in an environment like that would have completely shifted not only the way I use the internet today (especially if they'd gone ahead with their plans to implement it in social sites for kids), but would have affected my relationship with my parents in ways that would have left me hesitant to be myself at all.

Worse, when my cousins reached those ages at around the time this company was starting, social mediums had progressed and their problem wasn't talking to people online; it was posting very promiscuous photos and embarrassing YouTube videos showing the inside and outside of their homes and not putting any privacy barriers up. Those are much bigger concerns to me than knowing how to deal with stranger danger in text communications.

<comment><USER4> You are absolutely correct about the photos. Explicit photos were going both ways on chats. We were working on a solution to that, and had licensed some pretty cool tech from a small firm in DC.

re: the notion that _you_ would have known about a creep online, that's awesome. It's great that you were so self aware. Not every child can say the same. A favorite quote of mine applies here. No matter how you cut the math, and no matter how much you don't like the answer, 50% of the population is below average intelligence.

All of the content on our site explicitly suggested to parents that they should talk to their children _before_ installing our software. Talk about why they were installing it, and show them how it works. Every kid hated it. Of course they did. I would have. Unfortunately, kids have plenty of time to think about ways to circumvent protections their parents put in place. Parents have jobs, multiple kids, schedules to maintain, etc, and in general were outgunned on the technology front. They are all screaming for help, or suffering in silence, throwing up their hands because they cannot solve the problems for themselves.

When we started IMSafer, we were all in our late 20s or very early 30s. We were all hacker types. We also had young kids. One of our advisors was a police officer (personal friend) who spent a ton of time with us talking about the realities of the bad people out there. It's not Chris Hanson crazy, but there are really, really bad people out there. More often than not, related to the harmed minor.

<comment><USER12> I don't think it's really unfortunate that kids can circumvent technical measures. It's unfortunate that parents want a robot babysitter.

The primary danger comes from breaking the barrier between online and in- person. The best solution seems to me to be parents being understanding and supportive. Want to meet someone you talk to online in person? OK, fine. The parents come along and meet in a public place. If, instead the rule is "no meeting people from the internet ever", teenagers will do it anyway in a much less safe manner.

</comment></comment></comment></comment><comment><USER19> I'm confused. Did they protect kids? I don't remember if he said that or not.

Anyway, a few comments:

\- They paid for free users with no business model in place

\- They failed to complete requirements of their _previous_ investment deals

\- Handshakes don't matter, term sheets don't matter; nothing matters until the check clears

<comment><USER20> Startups should print out those last three bullets, frame it, and put it on the wall. It would save a lot of heartache.

</comment></comment><comment><USER21> It's a good lesson on how not to make poor business decisions, but the number of times the writer repeated the 'protect kids' phrase turned this into a really tough read. This wasn't a noble mission thwarted by evil VCs - simply a cautionary tale of how not to make startup decisions.

</comment><comment><USER22> They (Benchmark, a top-tier VC) broke a term sheet? That is...not awesome.

</comment><comment><USER23> The title is a bit disingenuous, but the story is worth reading. Thanks for taking the time to write and share, Jason.

<comment><USER24> It's charitable to say that the title is a bit disingenuous. I agree that the story (as well as the ongoing discussion) is very much worth reading. The unnecessarily provocative title is only slightly distracting.

</comment></comment><comment><USER25> While you made a lot of mistakes here, it's also pretty frustrating that there's a good deal of VCs who are basically only looking for the next Twitter. Why would anyone think viral growth would be realistic for this product? How does one acquire that much capital while knowing so little about business?

<comment><USER26> A good deal? From what I've heard, _every_ single VC in Silicon Valley shares this unhealthy obsession over "the next Twitter" (Just read pg's definition of "startup").

And as long as Twitter, Google and Facebook keep paying 8 figure (or 10 figure) sums to acquire "startups" without a business model, VCs don't _need_ to know anything about real business.

<comment><USER25> Well, I was trying to be nice so the ones that read the forum can convince themselves they are in the good group.

</comment></comment></comment><comment><USER27> The real lesson is that the money isnt there until the check clears. Selling your house and moving on a promise is risky. Even trying to raise money is a huge drain.

</comment><comment><USER22> It looks like failure to get D&O insurance prevented them from suing a top- tier VC for breaking a term sheet, but it's unclear how wise it is to sue a VC over something like that, anyway.

BTW, if anyone needs commercial insurance for their (California based) business, I've had pretty good experience with <http://www.alliedbrokers.com/> Allied Brokers in Palo Alto, so far. Prof liability, D&O, etc. are all things which are probably worthwhile after a >$1mm seed, depending on how litigious your sector is (b2b, particularly).

</comment><comment><USER28> Somehow I feel that a more accurate title would have been, "Fail to meet agreed terms, screw yourself".

</comment><comment><USER29> Lol totally evil product based on making people lives awful through fake fear.

But I guess that's many successful products, make people feel bad or their lives are missing 'something' and give them the solution for the $

Myself and friends have often talked about it. Be unethical but make money. Individual vs Herd mentality. And in the first world is it really that evil to fool the people who fall for it? I find it hard personally but maybe that's a weakness I should train to get over, not sure. I do give a % of my wage to 3rd world charity so would I still be up positive ethics wise.

Interesting article to see into that world anyway, something to think about.

</comment><comment><USER30> A bit off of the main point, but

> Age? Sex? Location? That's how predators started their hunt.

And how non-predators start their hunt (the same way that predators in the 1800s started their letters with "Dear Timmy").

It's a completely nonspecific starter question for anyone who is chatting with a stranger, and I wish the author were more careful with this.

</comment><comment><USER31> I suspect people who really want to "protect kids" find a way to do it that doesn't involve greed, lawyers, stupidity, and liberally cursing. Maybe there wouldn't be as much need to protect kids were there fewer people who talked about "getting f*cked" or screwed when talking about kids. Jerk.

<comment><USER9> As a parent of two middle schoolers, I am a lot less worried about adults using adult language to talk about business than I am about the people who can't tell the difference between that and actual threats to my children.

<comment><USER31> Whatever. Just another guy who thinks he's owed something because he did something that he thinks everyone should be gushing over.

<comment><USER9> Yeah. Stupid guy talking about his stupid life experiences. He should grow up.

</comment></comment></comment></comment><comment><USER32> I really enjoyed reading this article. More articles like this need to be on HN, rather than the run of the mill startup success stories. It's refreshing and visceral to hear how a startup can fail, being so close to success.

</comment><comment><USER33> "For almost two years they had been personally liable for what me and my friends were up to."

I don't understand this part. Shouldn't the investors have seen the insurance through themselves? It seems mad to rely on other people for that kind of thing. Sure it would have been nice for the startuppers to organize it, but really, it's a no-brainer that for such things you have to organize it yourself (because the people tasked with it have no real stake in it).

</comment><comment><USER34> Perhaps there's a niche for software that alerts the children and not the parents when they're triggering certain flags.

"Do you really know this person AFK?"

"How would you feel if this photograph was sent to all of your teachers, and everyone you know in school?"

And before people mention MS Office's Clippy - yes, it was was awful, but part of the reason for that was kludged implementation.

(<http://robotzeitgeist.com/tag/bayesian-inference-engine>)

This new implementation could have a "spy on your children" feature, but allow that to be turned off for the parents that find such things troublesome, allowing the warnings to go direct to the children.

</comment><comment><USER35> _We went back to other investment firms that we had also received term sheets from prior to committing to Benchmark, but after Benchmark pulled out, the news spread and we had the scarlet letter, or maybe the scarlet zero._

This is why VC-istan is dead. (Not actually dead, but "Microsoft is dead" dead.)

</comment><comment><USER36> I know how you feel exactly. I had a business in the same position except in the payment industry: we were processing roughly $1m a day until one day we had a frightening call from the bank. Two weeks later we were in liquidation and I was facing bankruptcy. Like you are saying (in an essence): the rug was being pulled from under you.

Thank you for sharing your story - by sharing, it it helps the whole startup community :)

</comment><comment><USER37> Sensational headline. TL;DR startups fail sometimes.

</comment><comment><USER20> Talk about counting your chickens before they hatch ... and not just about the part where they moved or added a board member before the money hit the bank account, but even the little "celebration" in Vegas. How is raising money a reason to celebrate in Vegas... especially when it hasn't even hit the bank yet?

While I feel bad for the founders here and the VC behavior is deplorable, a little cautiousness and common sense would have really saved the day here. "Let's not move until the money is in the bank, ok? Hey board member -- just step back for a few minutes until the wire clears. Once it does, we're all yours."

... All that being said ..... BIG KUDOS to the team for getting the investors their money back:

"We sold the assets for what our original investors had put in to the company, about $1.5 million, so we were able to give them their money back."

That's great -- I know very few entrepreneurs who have gotten well past the point of raising capital who can claim to have returned investor money.

So, while you might have gotten the Big Screw by VCs, in part due to their shadyness and in part due to your own over-enthusiasm and lack of diligence, I think you saved the day for your investors. They may not invest with you again, but they can't say they lost money with you either.

</comment><comment><USER38> Doesn't sound like they had any idea how to make money off this thing. Since for whatever reason they decided they didn't want to charge any money for the service.

What could they have done to make money from children's internet chat logs that would not be in some way shady?

</comment><comment><USER39> As an old teacher of mine used to often say, "Check's in the mail. Love you in the morning."

</comment><comment><USER40> The primary lesson here is as common sense as it is timeless: "You don't have the money till you have cash in hand." Only a fool would sell a house and move hundreds of miles on little more than promise.

</comment><comment><USER41> Wow that really sucks. In hindsight, could you have hit up your current investors for the bridge round _before_ shopping around the Valley, or would that have put VCs off too?

<comment><USER4> As one of my favorite private equity friends likes to say, "a bridge to nowhere is a dock."

We were offered some additional capital but it wasn't enough and our conviction was smashed. Until you've been in this sort of situation, you can't know how you will feel or react. Experience is a great teacher.

</comment></comment><comment><USER42> How does that directors and officers insurance (D&O) work? What is the problem with not having it?

</comment><comment><USER43> He got lucky that he was able to bounce back.

<comment><USER4> Jason didn't get lucky. He's incredibly talented. We, as a team, failed. It happens. I would go to war with any of my co-founders again, no doubt.

</comment></comment><comment><USER44> The title is misleading, immature and in poor taste. If HN moderators had a clue, they would edit it.

<comment><USER45> If you had the power to edit it yourself,what would you change it to? And why?

<comment><USER44> "Misadventures in Kiddie World"

Anything is better than what's up there now.</comment></comment></comment>


Re-engineering headphones - <AUTHOR>

I'm not sure whether this is a valid topic, or if it resonates with millions of others. In listening to a few online talks, I realize again my headphones have lost hearing in the right ear.
It's generally kicking the cord I think when I get up for a coffee. That's my working theory anyway. Sometimes seems to break.
But my question is, would it be worth a kickstarter to re-engineer headphones? Every pair I've bought seem to self-destruct within 1-6 months, and that strikes me as a bit fragile.
Are there more durable headsets out there? Is it an engineering flaw? I'll chip in my over 5 seconds of engineering experience with: "The wire needs to be more robust in the face of aggressive, beer-driven behaviour".
Typing this with my left hand holding an important bend in the wire of my headphones, listening to the Portal 2 finale song. Yes, I just beat it. Yes, I'm behind the times :) <comment><USER2> You get what you pay for. Good quality headphones come with strain relief, replaceable cables etc. I don't break headphones at all, but if you're rough on yours then invest in a more robust pair.

Also, why you have the cord running down by your feet where you can kick it? Buy a $5 extension cable at Radioshack and plug your headphones into that, you silly person..

</comment><comment><USER3> How many pairs have you bought? which ones?

I've never broken a pair. Most of mine seem to have some natural protection, through either detachable cables, wrapping above the ear, shirt clips, or recables. You are kicking the cord? The cord shouldn't be on the floor. Why not put it on a table?

If a headphone has a detachable cable that screws in, that'd be the best by far. That and metal-on-metal framing. If that's not available cheaply, then a Kickstarter might be interesting.

</comment><comment><USER4> I agree with Anigbrowl. You get what you pay for. If you buy cheap headsets or use ones that come default with your devices, you should in all seriousness expect a pretty low life span even if it exceeds that expectation.

Pay $100-ish and get a quality pair. I've had the same one for about 5, maybe 6 years, still works flawlessly.

As for your second question, always worth trying Kickstarter if you think its worth competing.

</comment><comment><USER5> I bought some headphones with replaceable parts via this guys: [http://www.acousticforge.com/](http://www.acousticforge.com/) which started as a KickStarter project: [http://www.kickstarter.com/projects/legendary/earbuds- time-t...](http://www.kickstarter.com/projects/legendary/earbuds-time-to- change-the-status-quo)

</comment><comment><USER6> In my case it's the rolling chair which causes the cord to eventually break. Doesn't matter if it's 100 dollar headphones or 12 dollar ones.

What I've since done is use an extension and reroute my audio cables behind my desk and aligned them the the monitor. I then plug my headphones into the extension instead. The cord is messy on the table though

</comment><comment><USER7> There are headphones out there that sell you the wires, so you replace the wire instead of replacing the whole cans. I have a pair of sony mdr-v6 and have had them for about 2 years, they still work perfectly fine.

<comment><AUTHOR> The other problem I've had is around the joints or the volume control, very fine wires that seem to snap easily.

I wasn't sure if its just a matter of them being made with an expiry date or if there's better ones out there, but I'll check those out, cheers.

</comment></comment><comment><USER8> You haven't told us which headphones you are talking about. My Sennheiser HD-414s are over 25 years old and still going strong.</comment>


Raising Money in London Almost Killed Our Startup - <AUTHOR> https://medium.com/good-audience/raising-money-in-london-almost-killed-our-startup-6ed9fca1ba88 <comment><USER3> This is going to be rough sherm8n. Hold on to your hat. I'm sorry if this upsets you.

To be honest, I'm suprised you've received any funding. You're a company that increases Twitter and Instagram followers. This is a deeply dodgy, largely unproven and mostly saturated market and you're charging prices that'll leave you out of pocket. No way are you worth $600,000. There's no way you can get enough customers.

You can spin this as a story about how you should never leave the comfortable bubble of San Francisco, but to my mind it's yet another story of how insane things are over there.

<comment><USER4> _To be honest, I 'm suprised you've received any funding. You're a company that increases Twitter and Instagram followers. This is a deeply dodgy, largely unproven and mostly saturated market and you're charging prices that'll leave you out of pocket. No way are you worth $600,000. There's no way you can get enough customers._

Unless the followers are actually relevant, interested and verified-as-human- beings accounts. If that's the case then $600k is a fraction of what the company could be worth.

There's a lot of 'ponzi scheme' follower increasing apps and systems ($5 for 5,000 followers, 'team followback', etc), where people sign up to get more followers by automatically following other people who join the scheme, but the number of good applications that can find people who would genuinely be interested in hearing your message is, essentially, zero. There _is_ an opportunity in the space but the problem is a hell of lot harder to solve in a worthwhile way than just increasing the vanity metric of "# of followers".

<comment><AUTHOR> You clearly get it.

<comment><USER5> I think it's a more familiar pattern than people want to admit.

We (OP3Nvoice at the time, now Clarify) went through Techstars London 2013. As much as the team loved London and the program, they just couldn't raise there. Fast forward a couple months and they've moved to Austin, TX. (I joined then.)

Moving to Austin created some challenges but within a couple months of restructuring the UK company to a US, the dollars started flowing and within 2 months, we ended up raising about $1.4M.

All of that said, we have some _great_ advisors and a couple small investors from London and they've been insightful and asked some hard questions. But despite both groups speaking English, the cultures and risk tolerances are wildly different.

And yes, Kyle is a great guy. :)

</comment></comment></comment></comment><comment><USER6> The main problem here seems to have been a lack of understanding that it's essentially a requirement now in the UK to be set up for EIS/SEIS if you want to raise money. Investors expect it.

We've been through several rounds of fundraising, and as an American from the bay area over here, it's definitely not the same environment as you hear about from YC companies, etc. raising in the Valley. But the UK/European market is absolutely smaller and more fragmented (language barriers alone are massive) so opportunities are not nearly as large. You also don't have the same degree of FoMO[1] going on, as there just haven't been enough big exits. I wouldn't expect that climate to change any time soon, or with any expedience.

But definitely be registered as a company in the UK, and set up for SEIS[2]/EIS[3], if you want to raise money here.

[1] [http://en.wikipedia.org/wiki/Fear_of_missing_out](http://en.wikipedia.org/wiki/Fear_of_missing_out)

Edit: Links to relevant docs on gov.uk

[2] [https://www.gov.uk/seed-enterprise-investment-scheme- backgro...](https://www.gov.uk/seed-enterprise-investment-scheme-background)

[3] [https://www.gov.uk/government/publications/the-enterprise- in...](https://www.gov.uk/government/publications/the-enterprise-investment- scheme-introduction)

<comment><AUTHOR> We were set up for SEIS/EIS and had an accountant review every thing. They then drafted the advance assurance to send over to HMRC. Definitely worth it to get this done before you start fundraising.

</comment></comment><comment><USER7> I'm an American, running a startup in Scotland, funded by a Scottish angel syndicate, which functions in many ways like a "mini VC". There is intense, constant griping both in Scotland and in the broader UK about the funding climate vs. Silicon Valley. Having been involved in a South Florida startup previously, I can assure everyone here that there are much worse places to raise money and run a tech company than Europe.

Most of the horror stories around fund raising in Europe that I've heard involve B2C companies, or very low revenue-per-customer B2B companies that are constrained by many of the same challenges around customer acquisition (low average rev per customer, not very sticky, can't afford an inside sales team). This makes these B2B companies effectively the same as B2C in many ways.

The key thing to recognise, in my opinion, is that it's extremely difficult to raise money for a software company that's pre revenue or likely in search of product/market fit _in almost any market_ other than a few cities in the US (NYC, Valley, Boston, Others?). Sure, there are counterexamples, but this seems to be true in general. This isn't just a European "problem" and my not even be a problem at all.

When I was playing in rock bands, we quickly learned we needed to play venues that fit our style. Playing in front of an old folks home crowd wasn't going to work. This sounds amazingly stupidly simple, but there's plenty of people ignoring this lesson in startupland with regards to fund raising. If you're sitting anywhere but a few select markets, reading Valley blogs, you're going to be in for disappointment when it comes time to apply all of your Valley knowledge in a non-Valley environment.

B2B land is so much easier than B2C land, and there's plenty of investment out there. Most software and most money made from software is in B2B land. Why not try it out? You could be surprised.

</comment><comment><USER8> Another London startup story: [https://medium.com/@buro9/the-journey-of-a- london-startup-wh...](https://medium.com/@buro9/the-journey-of-a-london- startup-what-i-learned-when-my-company-failed-c67acd74b862)

We had 50k users, some revenue (meeting our targets for that point) and the growth we had projected for that moment in time. We failed to raise money to get us past the end of our runway though.

Annoyingly, whilst the startup failed the product is still growing as an open source thing. It now has over 75k active users and has multiple installed instances.

<comment><USER9> I can't say trying to monetise yet another forum not working doesn't surprise me.

<comment><USER8> We had revenue, and the revenue was 10x that for forums with adverts and our monetisation was far less intrusive (visually and on a privacy basis).

The instance I still host has over 300 sites on it now, and remains profitable. Though without the company, this is now going into a non-profit which owns the server instance. The profits being used for charitable purposes.

We did not fail to produce revenue from communities, we failed to make the argument to investors about the potential market and to grow to seize it.

We could have remained a lifestyle business on the revenue, but our target was growth and we failed to sell that to investors.

<comment><USER10> Your product itself seems very good, there's just one glaring inconsistency for me (perhaps I'm wrong): the product is ambitious but your goals are not -- and to get a huge growth you need to at least target a huge viral market. The good old forums are not a huge viral market, imo.

For example, you could have tried to replicate a reddit environment: anyone can create a forum with a single click on your domain (don't even mention pricing), you could aggregate all forums on a front page to attract viewers, allow site wide discovery, etc.

<comment><USER8> That was on our roadmap.

We felt that one of the strong differentiators was how much control we gave to forum owners to shape the structure and identity of their space.

What we had planned was to focus on one market such that we'd have a small critical mass within that niche... in our case cycling forums which tend to exist with geographic focus yet have shared needs we could fulfil... and then create a directory from that to answer a "Find your local cycle forum" question before later making that question much broader.

We knew from feedback that "community", "shared interests", "forum"... all these things are nebulous and vague, they say nothing to anyone. Offering a directory and creation method would fail because no-one searched for those things, and no-one created those things.

We chose one segment to do well, from which we could expand into the other segments nearby (all sports and recreation interests), before expanding into entirely unrelated spaces.

We had a plan, but it required time and growth to execute.

</comment></comment></comment></comment></comment><comment><USER11> The headline suggests that it was their choice of London as a fundraising location that almost killed their startup but I don't think that's true. While it's certainly more difficult to raise money in London than in San Francisco, the real problem seems to be that they ignored a critical piece of advice:

 _> The lead investor encouraged us to get it closed quickly and not try to do anything fancy._

Having said that, it's deeply disappointing that they still haven't heard back from HMRC on the advance assurance for SEIS.

<comment><USER12> Indeed, you can basically stop reading the article at that sentence.

 _Always_ get the round closed quickly and don't try anything fancy.

</comment></comment><comment><USER13> Welcome to Europe.

Comparison between Europe tech startup scene vs the Silicon Valley one is quite unfair. I am based in Switzerland, you cannot imagine how hard is for a tech (worser if service) company to raise money. The model is extremely broken because the mind of an average European investor is completely different respect to the average American investor. In US the normal startup funding process works around a working prototype and how fast you can ship, in Europe, mainly here in Switzerland is about the protection you can give to a product with trademarks, patents and other things. So when you ask for money the first question is: is it protected? Or, can we protect it? Are barriers good enough so we can skip short-term competitors?

The second huge problem is the exit. In US, the IPO is a kind of fallback alternative to the acquisition, that is the main goal in a reasonable amount of time, so this makes the IPO not the favorite exit-strategy at all. In Europe it's actually the same, but the acquisition model simply doesn't work. You don't have any big tech company here giving the real exit-strategy option to investors. So the only chance you have to get them, is to prove that you can literally make a certain amount of money even in the worst case. Nobody believes that Apple, Amazon, Google, Facebook or whatever can acquire a Swiss startup company in the first 5 years of its life, so the result is that tech companies can barely raise money.

IMO, the Silicon Valley model is very risky, the motto "fail fast, fail often" is certainly attractive, but has a lot of downsides for funders and people involved, on the other hand the European scene is too slow, too hard and is always complaining about the fact that we are unable to compete against the Silicon Valley. I think the right balance is required, but it's just an opinion.

<comment><USER14> I'm in CH as well. I was talking to a VC guy who moved here from the US, and he has a similar story. People here are very conservative. The whole IP protections thing sounds like a business school teaching.

Anyway, would be interesting to chat if you're near ZH.

<comment><USER13> I actually work in Zürich! Definitely!

<comment><USER14> Have you got an email? Mine has my name in it.

</comment></comment></comment></comment><comment><USER15> In a nutshell,

\- Had a funding plan \- Got there quick with angels \- Got too big for your britches \- Discounted adviser's advice and got in bed with a big VC firm, doubling target funding \- Got bit by the big firm legal and the money got cold.

I kinda feel like this is going to happen even in the bay area if you start playing with major industrial investors or get too big for your britches.

Way cool that y'all were able to recover from the experience and are much wiser for it. But, other than the (not small, for sure) difference between US and UK laws (which your firm should have figured out), is it not so much an issue of raising funds in London as it is business practice maturity?

<comment><USER16> Yes I think that's one lesson: the risk of pursuing additional funds was not allowed to discount the utility of the 600. Perhaps if risk (say 10% success of more involved raise) were accounted for the 600 would have been more like 60, making the smaller deal 4x better. If someone offers me 600 I'll try not to lose my shorts. At the end of the day it's not a lot of money -- esp in London!!

</comment></comment><comment><USER17> Funny how i just spend a few minutes reading something about a company, followed a few links and all I know is that the founders like sunglasses. No idea what they are doing.

<comment><USER18> Well there is a link on the word 'startup' right at the beginning pointing to: [http://www.goodaudience.com](http://www.goodaudience.com)

<comment><USER17> Yeah, right next to Waldo :D

</comment></comment></comment><comment><USER19> TLDR: We had our funding lined up. Our lead investor advised us to keep it simple. We made it complicated. It almost killed us.

Apologies if that comes across as snarky, but that's honestly what I got from this.

<comment><USER20> The other point is that UK investors expect investments to be tax efficient for them; this means EIS/SEIS. It would be equivalent of going to US investors and saying, we'd like you to invest in our company but you'll only get tax breaks in the UK so too bad.

</comment></comment><comment><USER21> > I know what you're thinking. Why the f*ck would you move out of the mecca > for startups?!?

Actually no, I can understand perfectly well why you would move out of a ridiculously expensive city - but not why you would move to one of the few cities on earth that is about equally ridiculously expensive.

</comment><comment><USER22> It is great that you survived this near-death experience. From my experience VC's and Angels from Europe and US are two completely different animals. For fun just watch Dragons Den UK vs. Shark Thank US.

Venture capital firms in Europe vs. America: The under performers

[http://iveybusinessjournal.com/publication/venture- capital-f...](http://iveybusinessjournal.com/publication/venture-capital- firms-in-europe-vs-america-the-under-performers/)

<comment><USER23> It's a very different kettle of fish. In the UK, investment in tech startups is pretty barren - investment tends to be placed more readily with traditional engineering ventures - weapons, aerospace, material products - but for software... not so much - or at any rate, not in the model of angel rounds, etc.

The scene in the UK is rather more "be bought" \- people aren't anywhere near as willing to pour capital into a business, and instead prefer to buy equity from founders. Dragons Den is a good take - large amounts of equity, low valuations, founders selling chunks of their company far more for the expertise and connections and clients of the investor than for the money.

I'm not really too sure where this difference in culture arises from - <USER2> suppose in some ways business in the UK is far more conservative, less ebullient, and still dominated by an almost Victorian attitude to business relationships and social status.

Edit: Worth mentioning, I suppose - run a (not so startup any more, we're nine this year) in the UK, sold a minority stake a few years ago to a well aligned investor-partner who's a few years "up the curve" from us. Happened in the way these things do when they actually happen - we weren't looking, they approached us, we bit their arm off, and haven't looked back.

<comment><USER24> _almost Victorian attitude to business relationships and social status_

I've heard a couple of rants on this subject from people who were in a position to know. The UK is much more connection-driven than it looks. Much of our past and current industrial problems I would lay firmly at the feet of managers provoking needless conflicts with staff and failing to be open to change. Look at the car industry: the successful firms are _either_ small enough to fit everyone in one shed _or_ foreign multinationals. While the local brands coalesced into MG Rover before finally sinking among allegations of fraud by the directors.

This is partly why the UK startup scene is coalescing in London despite the very high costs there; you have to get personally acquainted with the finance people in order to get the money in.

(If anyone has comments on the Scottish startup scene I'd be interested to read them)

<comment><USER25> El Reg had a good article on this recently, people like Martha Lane Fox don't become serial entrepreneurs, as soon as they have one successful company they run off to join quangos where grubby commerce won't disturb their new establishment social position.

</comment></comment></comment></comment><comment><USER26> As I was reading the article a voice in my head was shouting "TIME KILLS DEALS! GET IT CLOSED".

Clearly the difference in approach here in the UK versus our cousins in the US caused delays that even a large self-sustaining business would seriously consider before undertaking.

Sorry it didn't work out but very important lessons learned I'm sure.

<comment><AUTHOR> It does and I learned that the hard way. The legal process was the delay and I should have pushed it along faster.

</comment></comment><comment><USER27> I am the angel investor and I have helped many companies raised many times. The simple fact is that the UK has lots of money looking for a investment home ( £1 billion plus invested last year ). Three years ago tech investment in the UK was almost zero. This money was offered but the deal was not simple as can eb seen on my post:

[https://medium.com/@doug_scott/response-to-raising-money- in-...](https://medium.com/@doug_scott/response-to-raising-money-in-london- almost-killed-our-startup-71d83db6479b)

</comment><comment><USER16> I guess the risk is higher in London because the funding pool is smaller, so if it goes to pot there's less places to turn, tho well done for getting all those commitments. What is your relationship with the funders now? Is there anyway to expedite the legals? When Ron Conway speaks about "getting your round over quickly" are the legals conducted differently in SF? Ultimately I think xkcd could draw a nice graph showing how the delay between commitment and consummation is directly (exponentially?) proportional to the probability of the deal falling over. It's never closed Til the money's in the bank is one thing I've learned.

</comment><comment><USER28> Glad you guys made it through. An off-topic question here. Why not facebook? If the goal is to reach out to genuinely interested people which you guys seem to do quiet successfully using twitter and instagram, why neglect facebook?

<comment><USER29> Not OP but having worked on Facebook apps in the past I get the feeling that people honestly don't want to be 'fans' of brands on Facebook. There's so much irrelevant stuff appearing on your timeline spamming your friends.

Seeing the odd product from a brand you follow on Instagram (here's our new dress) and Twitter (new cocktail ingredients) is more relevant.

</comment></comment><comment><USER30> Glad you guys made it through. Fund-raising is hard. Doing it across an ocean is even harder. It definitely feels like trying to raise in another country is a massive burden that may not be worth it.

<comment><AUTHOR> In hindsight, that was easy to see. We went where the money was. Had we closed, it would have been worth it!

<comment><USER22> Just curiosity, the investment was in Delaware C-corp stock or on a new UK entity?

<comment><AUTHOR> Delaware C-corp

</comment></comment></comment></comment><comment><USER31> Techstars encouraged a US company to move from SFBA to London?

<comment><USER32> Not that I've ever heard. But considering they applied and were accepted to Techstars in London, Sherman and Karl had a ton of connections to London that they were using to fundraise. Techstars wants founders to be where founders want their company to be.

</comment></comment><comment><USER9> $225k? You don't need an angel, you need a bank loan.

<comment><USER31> For a startup, outside of some very rare circumstances (inventory financing or equipment financing, with pre-existing great credit), you're not going to be able to borrow $225k with corporate liability. It would be entirely on the founders.

Having financed 2 businesses with personal liability (credit cards) and lost about $200k each time -- there are reasons you might not want to do this. It still makes sense in some cases. I'd probably do it again. It's probably not a good option for most people most of the time, especially if they can raise investment.

External investment also provides other benefits (validation, connections) that debt financing on credit cards doesn't.

</comment></comment><comment><USER27> [https://medium.com/@doug_scott/response-to-raising-money- in-...](https://medium.com/@doug_scott/response-to-raising-money-in-london- almost-killed-our-startup-71d83db6479b)

</comment><comment><USER33> When the author says "committed" does that imply "term sheet in hand"?</comment>


Latency numbers every programmer should know - <AUTHOR> https://gist.github.com/2841832 <comment><USER2> Most of these latencies were measured and written up for a bunch of systems by Carl Staelin and I back in the 1990's. There is a usenix paper that describes how it was done and the benchmarks are open source, you can apt-get them.

<http://www.bitmover.com/lmbench/lmbench-usenix.pdf>

If you look at the memory latency results carefully, you can easily read off L1, L2, L3, main memory, memory + TLB miss latencies.

If you look at them harder, you can read off cache sizes and associativity, cache line sizes, and page size.

Here is a 3-D graph that Wayne Scott did at Intel from a tweaked version of the memory latency test.

<http://www.bitmover.com/lmbench/mem_lat3.pdf>

His standard interview question is to show the candidate that graph and say "tell me everything you can about this processor and memory system". It's usually a 2 hour conversation if the candidate is good.

<comment><USER3> Pedantic quip: I have a hard time believing you guys were measuring half nanosecond cache latencies on a machine with a 100MHz clock. :)

And actually the cache numbers seem optimistic, if anything. My memory is that a L1 cache hit on SNB is 5 cycles, which is 2-3x as long as that table shows.

<comment><USER2> We didn't believe it either until we put a logic analyzer on the bus and found that the numbers were spot with respect to the number of cycles. I don't remember how far off they were but it wasn't much, all the hardware dudes were amazed that software could get that close.

tl;dr: the numbers were accurate to the # of cycles, might have been as much as 1/2 of 1 cycle off.

Edit: I should add this was almost 20 years ago, I dunno how well it works today. Sec, lemme go test on a local machine.

OK, I ran on a Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (I think that's a Sandy Bridge) that is overclocked to 4289 MHz according to mhz, and it looks to me like that machine takes 4 cycles to do a L1 load. That sound right? lmbench says 4.05 cycles.

I poked a little more and I get

L1 4 cycles, ~48K L2 12 cycles, ~256K L3 16 cycles, ~6M

Off to google and see how far off I am. Whoops, work is calling, will check back later.

</comment></comment></comment><comment><USER4> Honestly, I'd rather programmers know how to _measure_ these numbers than just have them memorized.

I mean, if I told them that their machine had L3 cache now, what would they do find out how that changes things? (This comment is also a shameless plug for the fantastic CS:APP book out of CMU).

<comment><USER5> Honest question: how would you measure an L2 cache lookup? (What program would I need to write which ensures that a value is stored in L2, such that when I read it later, I would know the lookup time to indeed be the L2 time?)

At that, how would one measure this kind of thing at the nanoseconds level? Would using C's clock_gettime() functions be good enough? Is there any other facility to count cycles which have passed between two operations?

<comment><USER6> What you could do is write a program that uses increasing amounts of memory. Measure the time it takes to complete a few million iterations of some function that accesses that memory. As the total usage increases, you'll see steps in the timing. For example, you might see a jump at ~32K, then ~2MB, then ~8MB.

<comment><USER7> I think you'd just need to access memory in increasingly larger increments (eg, sequential reads of 32k, 64k, ...2x size of l2 cache). But there's more to it as the L2 cache is often shared amongst cores in a multi-cpu system (where as the L1 is per-core). Also, the organization of the cache (eg, n-way set associative) means that you'd want to vary the size of the jump (stride) for your reads/writes and see how that also affects throughput.

<comment><USER6> Yea. All this and more is covered in the fantastic paper by Ulrich Drepper: What Every Programmer Should Know About Memory[1]. Was very enlightening.

1: <http://www.akkadia.org/drepper/cpumemory.pdf>

</comment></comment></comment></comment></comment><comment><USER8> Anyone who hasn't heard Rear Admiral Grace Murray Hopper describe a nanosecond should check out her classic explanation:

<http://www.youtube.com/watch?v=JEpsKnWZrJ8>

</comment><comment><USER9> This reminds me of one of the pages that Google has internally that, very roughly, breaks down the cost of various things so you can calculate equivalencies.

As an example of what I mean (i.e., these numbers and equivalencies are completely pulled out of thin-air and I am not asserting these in any way):

      * 1 Engineer-year = $100,000
      * 25T of RAM = 1 Engineer-week
      * 1ms of display latency = 1 Engineer-year

This allows engineeers to calculate tradeoffs when they're building things and to optimize their time for business impact. E.g.: it's not worth optimizing memory usage by itself, Latency is king, Don't waste your time shaving yaks, etc etc.

<comment><USER10> Yet everyone uses much slower RAM in servers and will likely continue to do so, all the while caches swell, etc.

Optimizing memory usage is almost irrelevant today, until it starts being a bandwidth problem, and that's still solvable but only through complex scaling strategies that also cost several engineer-years.

</comment></comment><comment><USER11> These numbers by Jeff Dean are relatively true but need to be refreshed for modern DRAM modules & controllers. Specifically, the main memory latency numbers are more applicable to DDR2 RAM vs the now widely deployed DDR3/DDR4 RAM (more channels = more latency). This has been a industry trend for a while and theres no change on the horizon. Additionally, memory access becomes more expensive because of CPU cross chatter when validating data loads across caches.

A potential pitfall with these numbers is they give engineers a false sense of security. They serve as a great conceptual aid - network/disk I/O are expensive and memory access is _relatively_ cheap but engineers take that to an extreme, and get lackadaisical about memory access.

When utilizing a massive index (btree) our search engine failed to meet SLA because of memory access patterns. Our engineers tried things at the system (numa policy) and application level (different userspace memory managers, etc.)

Ultimately, it all came down to improving the efficiency around memory access. We used Low-Level Data Structure to get the 2x improvements in memory latency:

<https://github.com/johnj/llds>

</comment><comment><USER12> Scaling up to human timeframes, one billion to one:

Pull the trigger on a drill in your hand 0.5s Pick up a drill from where you put it down 5s Find the right bit in the case 7s Change bits 25s Go get the toolkit from the truck 100s Go to the store, buy a new tool 3000s Work from noon until 5:30 20000s Part won't be in for three days 250000s Part won't be in until next week 500000s Almost four months 10000000s 8 months 20000000s Five years. 150000000s

<comment><USER13> I forked the gist and did something similar. Would you mind if I took some of your suggestions into my fork?

<https://gist.github.com/2843375>

<comment><USER12> Fine by me.

</comment></comment></comment><comment><USER14> I believe that this originally comes from Norvig's "Teach Yourself to Program in Ten Years" article: <http://norvig.com/21-days.html>

<comment><USER15> That's from 2011. Amazon's James Hamilton wrote about it in 2009, and it comes from a presentation that year by Google's Jeff Dean:

[http://perspectives.mvdirona.com/2009/10/17/JeffDeanDesignLe...](http://perspectives.mvdirona.com/2009/10/17/JeffDeanDesignLessonsAndAdviceFromBuildingLargeScaleDistributedSystems.aspx)

EDIT: this is wrong, Norvig's page pre-dates Dean's presentation [http://wayback.archive.org/web/*/http://norvig.com/21-days.h...](http://wayback.archive.org/web/*/http://norvig.com/21-days.html)

<comment><USER16> I make no claim as to who first assembled that particular table of data, but Norvig's article is dated 2001.

<comment><USER15> EDIT: I stand corrected.

The numbers seem to have been evolving and the original source seems to be that page.

[http://wayback.archive.org/web/*/http://norvig.com/21-days.h...](http://wayback.archive.org/web/*/http://norvig.com/21-days.html)

<comment><USER2> This is shameless self promotion (well, me and Carl promotion) but we were measuring these sorts of things in the late 80's and wrote a paper about it that got best paper at Usenix in 95. I think we had most of those numbers, not in the same format.

Pissed off the BSD folks because it made them look bad. Oh, well.

Helped make Linux better, largely because while the BSD guys refused to engage, Linus did. He and I spent many many hours discussing what was the right thing to measure and what should not be measured. We both felt that lmbench would influence OS design (and it's influenced processor design, see all the cache prefetch stuff, I'm pretty convinced that's because all the processor people used lmbench). Linus was already on the "OS should be cheap path" but lmbench helped him make the case to other people who wanted to add overhead because of their pet project.

The cool part about working with Linus was he was never about making Linux look better, he was about measuring the right things. If Linux sucked, oh, well, he'd fix it or get someone else to fix it. Awesome attitude, I feel the same way.

The only published work that might predate lmbench for these sorts of numbers is Hennessy and Patterson computer architecture. They talked about memory latency but so far as I recall, didn't have a benchmark. That said, that book is friggin awesome and anyone who cares about this sort of thing and hasn't carefully read that book is missing out.

</comment></comment></comment></comment></comment><comment><USER17> Took me forever to find this, but:

[https://plus.google.com/112493031290529814667/posts/LvhVwngP...](https://plus.google.com/112493031290529814667/posts/LvhVwngPqSC)

<comment><USER13> From that link, this brilliant visualisation: <http://i.imgur.com/X1Hi1.gif>

</comment></comment><comment><USER18> Is a single-text-file-github-gist the best way to disseminate this piece of knowledge (originally by Peter Norvig, BTW)?

What about a comprehensive explanation as to why those numbers actually matter?

Meh.

<comment><USER19> <http://www.infoq.com/presentations/Lock-free-Algorithms> early-on gives good numbers on-machine.

I like this visualisation too: <http://news.ycombinator.com/item?id=702713>

<comment><USER20> For anyone thinking about watching: the presentation is incredibly good. The name is kinda misleading -- they talk more about modern x86 architecture and actual numbers of various algorithms than the lock free algorithms themselves. Both of those guys have a high emphasis on measuring and testing to improve performance.

</comment></comment></comment><comment><USER21> These are good rules of thumb, but need more context. Plugging an article I wrote about this & other things a couple of years ago for FB engineering: <https://www.facebook.com/note.php?note_id=461505383919>

The "DELETE FROM some_table" example is bogus, but the rest is still valid.

</comment><comment><USER22> John Carmack recently used a camera to measure that it takes longer to paint the screen in response to user input, than send a packet accross the Atlantic: [http://superuser.com/questions/419070/transatlantic-ping- fas...](http://superuser.com/questions/419070/transatlantic-ping-faster-than- sending-a-pixel-to-the-screen/419167#419167)

I came across the post when I was looking for USB HID latency (8ms).

</comment><comment><USER23> Considering that so many programmers are currently enthralled with JavaScript, Ruby, Python and other very very high level languages, the top half of this chart must look very mysterious and unattainable.

</comment><comment><USER24> One of my favorite writeups on this topic is Gustavo Duarte's "What Your Computer Does While You Wait"

[http://duartes.org/gustavo/blog/post/what-your-computer- does...](http://duartes.org/gustavo/blog/post/what-your-computer-does-while- you-wait)

</comment><comment><USER25> In practice any out of order processor worth its salt ought to be able to entirely hide L1 cache latencies.

<comment><USER20> Agreed, I think the real lesson is: reading is 10x cheaper than branching, so if you can do something with 10 non-branching ops it'll be just as fast as a single branch.

<comment><USER26> Conversely: if you can do something with a branch that's correctly predicted 90% of the time it'll be just as fast as 10 non-branching ops.

Branch prediction is a tool like any other - don't neglect it when it can help you.

<comment><USER27> As a coder, how can I predict the branch predictor?

<comment><USER28> There are definite rules, documented by the CPU vendor. A forward branch is assumed not taken, while a backward branch is assumed to come at the end of a loop that will probably iterate more than once. See [http://software.intel.com/en-us/articles/branch-and-loop- reo...](http://software.intel.com/en-us/articles/branch-and-loop- reorganization-to-prevent-mispredicts/) for example.

I'd assume the particulars will vary between CPU manufacturers and families, but the idea that backward branches will probably be taken seems fairly universal.

<comment><USER29> This is outdated information; Intel chips have not used static prediction for conditional branches since NetBurst (Pentium 4).

"Pentium M, Intel Core Solo and Intel Core Duo processors do not statically predict conditional branches according to the jump direction. All conditional branches are dynamically predicted, even at first appearance."

\--[http://www.intel.com/content/dam/doc/manual/64-ia-32-archite...](http://www.intel.com/content/dam/doc/manual/64-ia-32-architectures- optimization-manual.pdf)

</comment></comment></comment></comment></comment></comment><comment><USER30> The thing here that is eye opening to me, and relevant to any web programmer, is that accessing something from memory from another box in the same datacenter is about 25x times as fast as accessing something from disk locally. I would not have guessed that!

<comment><USER31> That's why tools like memcached work like they do.

</comment></comment><comment><USER32> Such items are important to web developers and they can use them to justify looking at one or other technology. Or perhaps attempt at least benchmarking and have them as one of the guides in configuring and setting up services. Comes to mind why is Redis can be better then mongodb and in what configuration.

As well in discussion about this and that these can be of help too.

Adding misaligned memory penalties such as on word boundary and page boundary can enhance such document. This might be a good cheatsheet if one inclined to research and make one.

</comment><comment><USER33> I present to you Grace Hopper handing people nanoseconds out:

<http://www.youtube.com/watch?v=JEpsKnWZrJ8>

:)

</comment><comment><USER34> Also, these numbers don't mean much on their own. E.g. L2 Cache is faster than main memory, but that doesn't help you if you don't know how big your L2 Cache is. Same for main memory vs. disc.

E.g. I optimized a computer vision algorithm for using L2 and L3 caches properly (trying to reuse images or parts of images still in the caches). Started off with an Intel Xeon: 256KB L2 Cache, 12MB L3 Cache. Moved on to an AMD Opteron: 512KB L2 Cache (yay), 6MB L3 Cache (damn).

Also, the concept of the L2 Cache has changed. Before multi-cores it was bigger and the last-level-cache. Now it has become smaller and the L3 Cache is the last-level-cache, but has some extra issues due to the sharing with other cores.

The important concepts every programmer should know are memory hierarchy and network latency. The individual numbers can be looked up on a case-by-case basis.

</comment><comment><USER35> If this is your cup of tea, have a look at Agner Fog's resources: <http://agner.org/optimize/>

Also, I'd have a look at Intel's VTune or the 'perf' tool that ships with the linux kernel.

</comment><comment><USER36> How is a mutex lock less expensive than a memory access? Are such things done only in registers nowadays? This doesn't sound right.

<comment><USER29> Less expensive than a _main_ memory access. It must be that an uncontended lock/unlock can happen in cache.

<comment><USER37> No definitely not. A full memory fence surrounds lock/unlock.

<comment><USER38> I actually don't think that's true. My understanding is that on x86, atomic instructions have implicit lock instructions before them. (Or you can make some instructions atomic by putting a lock instruction before them.) Such instructions lock the bus and prevent other cores or SMT threads from accessing memory. In that way, you can safely perform an atomic operation on a value in the cache.

Note that this implies that atomic operations slow down _others_ cores and SMT threads.

<comment><USER37> Locks are often implemented using an xchg instruction, which is implicitely locked.

All processor's caches are committed/flushed for the affected cache line. So its correct to say other processors are slowed down. But it also in that sense IS a main memory operation, just not yours.

<comment><USER38> To be clear, then we agree that haberman was correct, and the value can be changed in cache.

<comment><USER37> Sure. It just isn't useful in cache. If it is a real lock, it has to be shared. Either the caches have to reconcile, or it has to go to main memory.

<comment><USER29> Just because a lock is shared does not mean that it's contended.

For example, some multi-threading techniques attempt to access only CPU-local data but use locks purely to guard against the case where a process is moved across CPUs in the middle of an operation (thus defeating the best-effort CPU- locality).

<comment><USER37> But it has to be available to those multiple cpus, right? So it has to go down to memory and back up to cache. Contended or not.

<comment><USER29> Maybe I'm missing something, but if the cache line is only in use by one CPU, I don't see why the value would need to be immediately propagated to main memory or to any other CPU's cache until it is written as part of the normal cache write-back.

<comment><USER39> Correct. Typically the cache snoops the main memory bus. If a remote CPU starts a read on a cached memory location, the caching CPU sends a "stall" or "retry" signal to the reader, does a cache flush to main memory, and then lets the remote CPU proceed with the (now correct) main memory read.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER40> What stands out here is how long a disk read takes (especially compared to network latency). Indeed, disk is the new tape.

</comment><comment><USER41> Does anyone feel like this is sort of an apples to oranges table? It compares reading one item from L1 cache to reading 1M byte from memory, without adjusting for the amount of data being read (10^6 more). It looks like the data was chosen to minimize the number of digits in the right column.

</comment><comment><USER34> What about L3 Cache?

What about Memory Access on another NUMA Node?

What about SSD?

Does a mobile phone programmer need to know the access time for disks?

Does an embedded system programmer need to know anything of these numbers?

Every programmer should know what memory hierarchy and network latency is. (If you learn it by looking at these numbers, fine...)

<comment><USER42> I'm not an expert, but:

L3 is generally on the order of the same time as main memory - it's main purpose is to reduce the total amount of requests in order to conserve bandwidth

SSDs are on the order of 0.1ms, so 100,000 ns, give or take a factor of 10.

Someone smarter than me will have to answer the NUMA node question.

<comment><USER6> I think L3 is much faster; perhaps 1/3rd of main memory, assuming the line is available and not in another core. Here are some numbers for L3 cache, from Intel (probably specific to the 5500 series)[1]:

      L3 CACHE hit, line unshared ~40 cycles
      L3 CACHE hit, shared line in another core ~65 cycles
      L3 CACHE hit, modified in another core ~75 cycles
      remote L3 CACHE ~100-300 cycles
      Local Dram ~60 ns
      Remote Dram ~100 ns

60ns at 2.4GHz is ~144 cycles, right?

1: [http://software.intel.com/sites/products/collateral/hpc/vtun...](http://software.intel.com/sites/products/collateral/hpc/vtune/performance_analysis_guide.pdf)

</comment></comment></comment><comment><USER43> I find myself thinking of figures like these every time I see results for benchmarks that barely touch the main memory brought up in debates about the relative merits of various programming languages.

</comment><comment><USER44> When working on low latency distributed systems I more than once had to remind a client that it's a minimum of 19 milliseconds from New York to London, no matter how fast our software might be.

</comment><comment><USER45> Interesting but unnecessary for most programmers today. I'd rather my programmers know the latency of redis vs memcached vs mysql and data type ranges.

<comment><USER46> Every programmer that understands (rather than "knows") the number mentioned in the link would know the numbers you are looking for (or at least, the relationships between those.)

I'm not sure this relation holds for the opposite direction.

</comment></comment><comment><USER37> Network transmit time is almost irrelevant. It takes orders of magnitude more time to call the kernel, copy data, and reschedule after the operation completes than the wiretime.

This paradox was the impetus behind Infiniband, virtual adapters, and a host of other paradigm changes that never caught on.

<comment><USER2> Huh. Data please.

Part of the reason I wrote lmbench was to make sure that what you are saying is not true. And it is not in Linux, kernel entry and exit is well under 50 nanoseconds. Passing a token back and forth, round trip, in an AF_UNIX socket is 30 usecs. A ping over gig ether is 120 usecs.

Unless I'm completely misunderstanding, you are saying that the OS overhead should be "orders of magnitude" more than the network time, that's not at all what I'm seeing on Linux.

I guess what you are saying is that given an infinitely fast network, the overhead of actually doing something with the data is going to be the dominating term. Yeah, true, but when do we care about the infinitely fast network in a vacuum? We always want that data to do something so we have to pay something to actually deliver it to a user process. Linux is hands down the best at doing so, it's not free but it is way closer to free than any other OS I've seen.

<comment><USER47> This may be ignorant and please correct if I am off base but given the same physical medium isn't sending 2k across the network the same cost whether you are at fastE or gigE? Given the network is not saturated?

<comment><USER2> fastE is 100Mbits/sec, gigE is 1000Mbits/sec, so given the same size packet, gigE is in theory 10x faster.

However, to make things work over copper I believe that gigE has a larger minimum packet size so it's not quite apples to apples on pings (latency).

For bandwidth, the max size (w/o non-standard jumbo grams), is the same, around 1500 bytes, and gigE is pretty much linear, you can do 120MB/sec over gigE (and I have many times) but only 12MB/sec over fastE.

<comment><USER47> If I have 1000Mbits to send then gigE is 10 times faster. But in the article we are transferring only 2k across the network. Mixing latency and bandwidth here. The latency to send 2k across an empty network isn't 10 times greater on a fastE versus gigE network, right?

<comment><USER2> 2K is going to be 2 packets, a full size and and a short packet, roughly 1.5K and .5K.

For any transfer there is the per packet overhead (running it through the software stack) plus the time to transfer the packet.

The first packet will, in practice, transfer very close to 10x faster, unless your software stack really sucks.

The second packet is a 1/3rd size packet so the overhead of the stack will be proportionally larger.

And it matters _a lot_ if you are counting the connect time for a TCP socket. If this is a hot potato type of test then the TCP connection is hot. If it is connect, send the data, disconnect, that's going to be a very different answer.

Not sure if I'm helping or not here, ask again if I'm not.

</comment></comment></comment></comment></comment></comment><comment><USER48> Lz4 is faster than zippy and much easier to use. It's a single .h .c file.

</comment><comment><USER49> How is mutex lock/unlock different from any other memory access?

<comment><USER50> It requires more cache coherence.

</comment></comment><comment><USER10> Someone should add basic numbers like ns count for 63 cycles modulo and that type of stuff - That'll help bad devs realize why putting another useless cmp inside a loop is dumb, and why alt rows in a table should NEVER be implemented by use of a modulo, for example.

Yes I know that's not latency per se but in the end it is too.

<comment><USER51> I think if you're worried about whether or not you use modulo to calculate alternating table rows (and you _don't_ work for Facebook), then you're almost certainly optimizing prematurely.

<comment><USER10> IT DOES NOT COST MORE TIME TO CODE CORRECTLY

Some approaches are NOT acceptable, it's not about optimizing prematurely, it's about coding obvious crap.

While you may be used to the usual "code crap, fix later" and "waste cycles, there are too many of it" , it doesn't mean you're right.

Everyone says it but you're still running on C (linux, unix), you're still going nuts over scaling issues (lol nosql for everyone) and you're still paying your amazon cloud bill.

<comment><USER51> I know you're ranting to the world at large, but I am _not_ "going nuts over scaling issues". All my websites are static HTML files. I regenerate them as needed using custom Python code and my "databases", which are text files in JSON.

I have several sites running on a single smallest Linode, and the CPU utilization virtually never cracks 1%.

Also, note that I am not advocating "coding crap". I'm talking about not berating coworkers over the nanosecond cost of an extra modulo inside a loop.

<comment><USER10> If said coworkers are actually trying to improve and can take the advice peacefully, I will deliver it peacefully.

The others I will be pleased not to work with.

</comment></comment></comment></comment></comment><comment><USER52> 2kB over 1Gbps is actually 16ns (i guess they round up to 20)</comment>


Lessons from Italy's Response to Coronavirus - <AUTHOR> https://hbr.org/2020/03/lessons-from-italys-response-to-coronavirus <comment><USER2> > It is worth emphasizing that these obstacles emerged even after Covid-19 had > already fully impacted in China and some alternative models for the > containment of the virus (in China and elsewhere) had already been > successfully implemented. What this suggests is a systematic failure to > absorb and act upon existing information rapidly and effectively rather than > a complete lack of knowledge of what ought to be done.

Yeh. All the bilions going to think-tanks, intelligence, crisis response centers, various experts etc etc. seems like completely wasted money.</comment>


Henry Kissinger: 'We are in a very, very grave period' - <AUTHOR> https://www.ft.com/content/926a66b0-8b49-11e8-bf9e-8771d5404543 <comment><USER2> Quoting Bernie Sanders "I will not take advice from Henry Kissinger" and "I happen to believe that Henry Kissinger was one of the most destructive secretaries of state in the modern history of this country". (You can verify the quotes from [https://www.theguardian.com/commentisfree/2016/feb/13/hillar...](https://www.theguardian.com/commentisfree/2016/feb/13/hillary- clinton-henry-kissinger-harms-her-campaign) .)</comment>