"I, Robot" – the 3 laws considered harmful - <AUTHOR> https://shkspr.mobi/blog/2019/01/i-robot-the-3-laws-considered-harmful/ <comment> one-more-minute I think Asimov knew all this – his stories are about how the laws can be twisted into having surprising consequences. It's meant as a Sci-Fi version of the "literal genie" [0].

Has anyone actually taken them as a serious suggestion in AI ethics?

[0] [https://tvtropes.org/pmwiki/pmwiki.php/Main/LiteralGenie](https://tvtropes.org/pmwiki/pmwiki.php/Main/LiteralGenie)

<comment><USER2> Seconded. Somehow, the fact that the Three Laws _are_ broken and Asimov's works are all about showing how and why they're broken is something that most people missed. The popular culture ran off with the Laws, forgetting the context.

<comment><USER3> I think most people haven't read Asimov. They only know the laws because of their hype, but don't know their origin or meaning.

Thus, we need more Asimov in TV.

<comment><USER4> Oh I would die for a R. Daneel Olivaw TV procedural!

(Hm, apparently BBC had made adaptations of The Caves of Steel and The Naked Sun in the sixties... [https://en.wikipedia.org/wiki/R._Daneel_Olivaw#Appearances_i...](https://en.wikipedia.org/wiki/R._Daneel_Olivaw#Appearances_in_other_media))

</comment></comment></comment></comment><comment><USER5> This is silly.

The Three Laws of Robotics are not harmful because _they can't actually be implemented_.

Asimov never details how the three laws are baked into positronic brains in so fundamental a way that they can't be disabled without destroying the robot. Heck, we don't even know how to build a positronic brain -- because it's a fictional technology!

So even if you _wanted_ to build some sort of AI that is guided by the Three Laws, you couldn't.

You might be able to build AI that is guided by certain ethical principles, but right now, in the current state of technology, it's all rules-based stuff -- like a self-driving car with a rule about slowing down to avoid an impact that could injure someone.

Nobody knows how to give AI an ethical intuition, much less tweak that intuition. (I'm an objectivist, so I actually prefer saying, "Nobody knows how to give AI the ability to perceive metaphysical principles.")

This whole post is like saying that Star Trek's Warp Drive is harmful because we shouldn't be trying to travel faster than light.

<comment><USER6> Correct. By the author's logic, the article should _also_ be considered harmful because it is quite vague. In fact, it does not even go remotely into the detail about vagueness. Moreover, it omits the fact that Asimov actually plays with these laws in all his books (I think this has been mentioned already in another comment), showing exactly the interesting consequences of the wording...

While I like AI policy discussions a lot, this article does not help unfortunately. :-/

</comment></comment><comment><USER7> The point of "I, Robot" was to show how these laws can lead to dangerous results if followed to the letter, and their inherent contradictions. I agree with the author: applying the Three Laws to actual AI research and engineering is dangerous. Thankfully, I haven't seen too many people in AI research that actually advocate for their use.

</comment><comment><USER8> I get the impression that most people arguing in favor of the 3 laws have not actually read I, Robot. There are many places in pop culture that the laws appear, so it's not impossible that someone might quote them devoid of context.

To catch people up: I, Robot is a collection of short stories largely follow either (a) robopsychologist Susin Calvin or (b) robotics engineers Powell and Donovan as they attempt to diagnose anomalous behavior by examination of the 3 laws of robotics, highlighting the logical flaws and traps of these simple laws. A secondary theme is the use of robots as a mirror or lens through which particular human behaviors can be blown up and examined, largely through the absence or magnification of particular traits.

In universe, the 3 laws are considered to be practical engineering safeguards (and eventually, over hundreds of years, a fundamental building block to the function of the positronic brain)-- even as they are shown to the reader to be the origin of many conflicts.

</comment><comment><USER9> "Considered harmful" has become a lazy writing device. There's nothing valid here that even says why these are harmful except possibly that they are "vague". The only thing that looks even remotely like a real concern (that people, even scientists, believe these are somehow magically hard-wired laws) is not cited and appears to be entirely manufactured.

<comment><USER10> I think you may have missed the joke. The article explicitly calls out the word "harm", as it's used in the three laws, as being a source of vagueness, which the article then goes on to say becomes fodder for some of the conflicts in the stories.

</comment></comment><comment><USER10> The article specifically calls out the word "harm" in the three laws as a source of ambiguity. This, as the author also points out, becomes a source of conflict in the I, Robot stories.

The author is making a joke using the "considered harmful" trope/meme in conjunction with the assertion that the word "harm" is ambiguous, thereby rendering the phrase "considered harmful" relatively meaningless.

Unfortunately, this joke seems to be lost on many of the commenters here. Technically minded folks seem to see the phrase "considered harmful" and proceed to lose their minds. Never mind that the phrase, in my experience, is almost exclusively used in jest. But based on people's reactions to it, "considered harmful" in a title might as well be flame bait. In fact, I seem to remember a serious article a while back called "'Considered Harmful' Considered Harmful" that should have put this whole thing to bed.

</comment><comment><USER11> Leaving aside the literary purpose for the moment - when I go back and look at them now, some multiple decades after I first read them, I see a massive human-centric focus that I never did before. WHY is a human life more valuable than a robotic one? WHY is human "harm" so bad? WHY are human commands so powerful?

I mean, for story purposes it all makes sense, but I'm fascinated that I never raised these questions myself before. I accepted human divine right unquestioningly.

<comment><USER9> > _WHY is a human life more valuable than a robotic one?_

This "but why humans" reaction is bizarre to me every time I encounter it. We care about humans because we are humans. There's nothing deeper to uncover. We care about ourselves.

Humans do not care about the value of a robot "life", because we generally feel it has none.

<comment><USER11> > There's nothing deeper to uncover.

For me, at least, you're coming at it from the other direction.

I'm not really asking "why" in the "how did this come to be" \- the answer to that is rather self obvious. I'm asking "why" in the "is this objectively actually correct? Is this how I want things to be?" \- which have no answer, but the effort of trying to find one can uncover plenty of deeper ideas.

<comment><USER9> There is no meaningful answer to your question. For religious individuals, the answer is dogmatic (and generally _yes_ , humans have value above all else except God(s)). For atheists, the answer is logically a vacuous _no_ , because all of existence is without meaning.

<comment><USER11> >> which have no answer

> There is no meaningful answer to your question

Didn't I say that? Didn't I point out that the ATTEMPT to answer was valuable?

>> Is this how I want things to be?

> because all of existence is without meaning

Atheists don't have be nihilists.

<comment><USER9> I'd love to see a meaningful argument to the contrary. Near as I can tell atheism implies nihilism and the extent an atheist disagrees is decided entirely by how convincingly they can lie to themselves.

<comment><USER12> Atheism only implies nihilism if you subscribe to the concept that values are objective and morality is universal, which begs the question of who determines values and what is morally "good" and how.

<comment><USER9> So you assert that an atheist can find meaning by following their _subjective_ values and morality, correct? This seems entirely circular and equivalent to saying that atheists can find meaning by valuing the things that they value.

How does an atheist hold subjective morals and values without those morals and values essentially being arbitrary?

<comment><USER12> What's wrong with them being arbitrary?

<comment><USER9> In what way are arbitrary values _meaningful_?

<comment><USER12> Define "meaningful".

You seem to have a really big chicken-and-egg problem here... you want morality to be universal. The problem is, "ethics", "morality", and "values" are all human constructs. We created them, we define them, we discuss them ad nauseum. They are as varied as we are. I don't think that makes them "meaningless".

<comment><USER9> I don't have a chicken and egg problem. The chicken and egg problem arises from an attempt to pull meaning out of an arbitrary universe.

You say in different words that things are meaningful only because we believe they are. That's fine if it helps you but it's also circular and rather solipsistic. Your stance is also literally nihilistic: "a doctrine that denies any objective ground of truth and especially of moral truth".

<comment><USER12> See: [https://news.ycombinator.com/item?id=19048897](https://news.ycombinator.com/item?id=19048897)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> The whole point of the 3 laws is not to define the exact laws that need to be followed, instead it is a comment on the fact that we must have some laws governing the operation of devices that have the capability for autonomous actions.

The fact that it is still discussed in regards to Robotics means the 3 laws are not harmful, in actuality they gave sparked discussion and thought. That is powerful, not harmful.

</comment><comment><USER14> From the day I learned about them, I considered them an unreasonable bar for robots; to implement them requires effectively far-beyond human capabilities. I figure, if you have the tech to implement the 3 laws, you have the tech to go sublime.

<comment><USER15> And they are computationally impossible to any degree of certainty, based on only a cursory examination of quantum physics, chaos, theory of computation, and complexity theory.

</comment></comment><comment><USER16> This actually comes up a lot - people writing about problems inherent in Asimov's laws.

What I never see is what you would replace them with, and the problems inherent in that. That omission to me is scarier than these 3 laws.

<comment><USER17> "An it hurts no one, do what thou wilt shall be there whole of the law."</comment></comment>


First fully functional 64-bit OS for the Raspberry Pi 4 - <AUTHOR> https://www.balena.io/blog/balena-releases-first-fully-functional-64-bit-os-for-the-raspberry-pi-4/ <comment><USER2> This is good news. I had assumed that raspbian was 64 bit by now. It seems weird, since the Raspberry Pi v2.1 (second revision of Raspberry pi 2) launched with a 64-bit processor. I can't find a date for that launch, but the pi 3 launched in February 2016. The Pi foundation was forthcoming about Raspbian being 32 bit in the pi3 announcement article, but the timescale they laid out for making it 64 bit was "the next few months". I guess the reason anyone bothered porting to 64 bit is the fact that the pi4 has a 4GB of RAM option (you can't address the full 4GB with 32 bits).

<comment><USER3> Debian arm64 works fine on the Raspberry Pi 3 ([https://wiki.debian.org/RaspberryPi](https://wiki.debian.org/RaspberryPi)). The Pi 4 changed the boot sequence so that Debian doesn't work any more, making this the first 64-bit OS for the Pi 4.

<comment><USER4> They didn't say it is the first 64bit OS that runs on the Pi 4. The key phrase is "fully functional" which has no definitive definition. It's just some clickbait, marketing jargon.

</comment></comment></comment><comment><USER5> The first? Plan 9 (9front to be specific) had support for it for a few weeks now.

</comment><comment><USER6> > All these fixes were pushed upstream and are available in the main Raspberry > Pi Linux repository.

Well that's just highly misleading. "upstream" in this case refers to the outdated downstream kernel provided by the RPi foundation. Mainline does not have support for brcmstb at all current.y

<comment><USER7> That's _an_ upstream, and it is defined right afterwards:

> and are available in the _main Raspberry Pi Linux repository_

Hardly misleading. It clearly says "Raspberry Pi" and it doesn't say "main Linux repository". They could've added the word 'kernel' but then again I suppose it is a general Raspberry Pi git repo. Merging the Raspberry Pi Linux kernel repo with Torvalds' mainline isn't their responsibility.

That's not to say this whole thing isn't an advertisement though. This whole balenaOS and balenaCloud thing is some kind of frontend for Docker and the like, but have you seen the Pricing?

I'll just patiently wait for Raspbian Aarch64/ARM64 version instead :)

<comment><USER8> It already is a huge improvement. So when it goes full 64 bit and all the memory is accessible, it'll give a second bump to that experience of improvement.

</comment></comment></comment><comment><USER9> There are already builds of Ubuntu 64bit that are very solid... [https://jamesachambers.com/raspberry-pi-4-ubuntu-server- desk...](https://jamesachambers.com/raspberry-pi-4-ubuntu-server- desktop-18-04-3-image-unofficial/)</comment>


Which photo sharing sites do you use? - <AUTHOR>

<comment><USER2> I have about 2000 photos on ourdoings, plus a handful on flickr, yahoo photos and shutterfly.

</comment><comment><USER3> Facebook and Flickr.

</comment><comment><AUTHOR> Flickr + BubbleShare

</comment><comment><USER4> Flickr & Facebook</comment>


David Cameron makes veiled threat to media over NSA and GCHQ leaks - <AUTHOR> http://www.theguardian.com/world/2013/oct/28/david-cameron-nsa-threat-newspapers-guardian-snowden/print <comment><USER3> He is only pissed because the NSA could not be bothered to bug his phone - <USER2> thought Angela Merkel worthy of attention as a "world leader".

He seems like a good hearted chap but he does let his mouth start running well before wiser heads are able to brief him on the facts (so to speak). In may ways his "off the cuff" honesty has been good for British politics but we have all got rather used to skipping his first statement on any subject - to await the inevitable reversal/adjustment.

</comment><comment><USER4> I really hope that one of the next Snowden revelations is another 'Merkel moment' for Dave, whe he finds out the the USG (who he considers to be his best friend), have been fucking his (proverbial) wife behind his back.

Oh how I shall laugh at that one.</comment>


Interview your family-story with this android-app. - <AUTHOR> https://play.google.com/store/apps/details?id=org.apache.cordova.interview&hl=en

<comment><AUTHOR> This app was created with the idea in mind that in most cases only the grandchildren are interested in the life story of the grandparents. Sometimes it is too late by then or the options are limited to get the desired information.</comment>


Ask HN: How does a foreign company get a Federal Tax ID? - <AUTHOR>

I am trying to setup a bank account and such, in the US, but my company is registered in Jamaica. How do I get a federal tax ID, EIN or SSN (or any combination of the above) ? Do I have to register the company in the US to be able to get those or can I do it some other way? <comment><USER2> File an IRS W-9 form, you can do it over the phone by calling the 800 number. Tell them you expect to have a U.S. tax liability and that's why you need the number.

<comment><AUTHOR> Looking at the W-9 form now, it says that I need a TIN (or SSN).

Won't they ask me for a SSN over the phone? I don't have one that allows me to work in the US.

</comment></comment><comment><USER3> Are you sure you need a US tax ID? When I set up a bank account with Harris Bank, they sent me a US tax form and I just crossed out the relevant sections and wrote "Canadian resident", and they were satisfied with that.

Of course, many banks, like many Americans, are probably not aware that there is a world outside of the US -- but the solution there is to find a better bank. :-)

<comment><AUTHOR> It's not so much just the banks. So far SVB sounds promising...am also investigating other banks.

But I just reached out to Braintree and they plainly told me I had to have a US Bank account + Federal Tax ID.

So I am trying to do a little research.

Plus...businesses in general tend to ask for that...so I would rather have it and not need it than need it and not have it.

<comment><USER4> Beware that you don't get classed as having a US business presence, that could cost you dearly.

<comment><AUTHOR> I was thinking about that.

If I file a W-9, I will have to pay taxes in the US...won't I? I want to pay taxes in Jamaica, not the US.

How do I achieve that?</comment></comment></comment></comment>


US Army Wants Biodegradable Bullets That Sprout Plants - <AUTHOR> http://www.livescience.com/57461-army-wants-biodegradable-bullets.html <comment><USER2> Problem 1: you want your forces to train with the same stuff we would use in combat.

Problem 2: using bullets that are not solid metal ball in combat violates the Hague Convention.</comment>


FBI couldn't access nearly 7K devices because of encryption - <AUTHOR> http://www.foxbusiness.com/features/2017/10/22/fbi-couldnt-access-nearly-7k-devices-because-encryption.html <comment><USER2> The FBI also couldn't access 2,456,789 million websites that used encryption around the world...And?!

</comment><comment><AUTHOR> Working as intended</comment>


Podcasting Community Faces Patent Troll Threat; EFF Wants to Help  - <AUTHOR> https://www.eff.org/deeplinks/2013/02/podcasting-community-faces-patent-troll-threat-eff-wants-help

<comment><USER2> The patent was filed in 2009. I wrote a podcast fetcher for an obscure Archos device in 2005. It was hardly the first app like that. But just based on my own personal experience, the things that this patent covers were in wide use at least 4 years before it was filed.

I can't say I've read any of this in detail, but on the surface it might be the most ridiculous one of these patent stories I've seen in a long time.

My thing was a fetcher, not a distribution system, but if there hadn't already been a well established standard for distribution (RSS), I wouldn't have had any reason to write it.

</comment><comment><USER3> I would really like to see the EFF try to get a precedent which solves this problem in the general case, rather than invalidating just this one patent. And I think I know what the solution has to be: class-action defense.

What's going on is this. They have a pretext to sue, which is not good enough to win, but is good enough to keep the case from being dismissed on summary judgment and to keep them from being ruled vexatious litigants. They use this pretext to start hundreds or thousands of copies of the same case, against different defendants, achieving economy of scale. If the case proceeds, the defense has to spend much more (even if they win), because they have no such economy of scale. So they're forced to settle instead.

But suppose one defendant could force the patent troll, in a pre-summary- judgment motion, to reveal the identities of everyone else they'd threatened with the same patent. That would allow the defendants to pool legal costs, merge all the cases into one courtroom, and have a fair trial, rather than just being extorted.

<comment><USER4> At which point _this_ NPE dissolves and reorganizes around a different patent to troll a different crowd. And the pool is still out a non-zero sum.

We need real reform. This is a job for Congress, not the courts.

<comment><USER5> Disagree, the courts have to fix this. Congress (and the Obama Administration) are ignorant and corrupt, they just passed fake "Patent Reform" and won't act again for some time.

</comment></comment></comment><comment><USER6> One of the issues here is the Apple settled with 'Personal Audio' and handed them $8 million -

[http://www.engadget.com/2011/07/09/apple-coughing- up-8-milli...](http://www.engadget.com/2011/07/09/apple-coughing-up-8-million- to-personal-audio-in-ipod-playlist/)

Small change for Apple, but could now establish a precendent for judges who don't really understand technology and have to go by patent law.</comment>


Bye bye spotify and co, hello YouTube music - <AUTHOR> http://youtube-global.blogspot.com/2015/11/a-youtube-built-just-for-music.html <comment><USER2> Thanks for editorializing the title...?</comment>


Tech companies ignore pleas on rail safety - <AUTHOR> https://www.politico.com/story/2019/08/10/tech-rail-safety-1412324 <comment><USER2> I'm so confused. How on earth is a maps app having railroad crossing markings going to save lives?

A driver needs to stop at a railroad crossing just like they do at a red light or stop sign. If a driver is ignoring _any_ of these, I really don't see how an app is going to help.

Or conversely, if a crossing is somehow not well-marked on the road then that's deep negligence on the part of the transportation department and it needs to be fixed -- it has nothing to do with apps. (All the crossings I know have a closed gate and flashing lights, although I understand this might vary locally.)

<comment> why-el I think there is a middle ground. Apps could announce the upcoming railroad crossing just like they do with a left/right turn. Somewhat of a silly example, but if there is a river at the end of a road and your only option is a left or a right, the app will not help stop you for driving the car into the river, but the instruction to go right or left does (albeit accidentally) help you avoid that. I don't see why it won't do the same with rail crossings.

Although you should focus on road signs more than you do the app, there could be a dangerous situation in which the app says "make a left turn at the light", but before reaching the light there is a railroad crossing. _Of course_ you need to stop at that, but the app can enhance the situation but treating it like another instruction. I see no problem here but an overall safety enhancement.

<comment><USER3> Ok, but why do that for railroad crossings and not any other type of dangerous crossing? Should we have these apps announce every stop sign, traffic light, and uncontrolled intersection you travel through for safety reasons?

<comment><USER4> The list is much longer. You're driving past a kindergarten; watch out for children. You're driving past a school. You're approaching an area with many users logged into Google so watch out for pedestrians. You're crossing a bridge. You're passing a point with an unusually large number of accidents in the past.

If that isn't to become some sort of text-to-speech database dump about the area, there has to be a good theory of the conditions for what's worthwhile. And I sense that the natural conditions are the same as the conditions for putting up warning signs, reducing the speed limit, etc.

</comment></comment></comment></comment><comment><USER5> Many of these got added to OpenStreetMap at that time. OSMAnd will announce them, the last time (years ago) I used it while driving over some the implementation left a bit to be desired, it announced them about 10 times.

Here's a quick query visualizing just how many there are:

[http://overpass-turbo.eu/s/Ls7](http://overpass-turbo.eu/s/Ls7)

(move the map and click "Run" to look at other areas)

I think the FRA data includes information about what type of control each crossing has. It wouldn't be a huge project to highlight the less controlled crossings that are on higher speed and higher traffic routes.

<comment><USER5> Here's the FRA's interactive map:

[https://fragis.fra.dot.gov/GISFRASafety/](https://fragis.fra.dot.gov/GISFRASafety/)

They have extended information about the crossings.

</comment></comment><comment><USER6> Strange article. It assumes that somehow having rail crossings marked on navigation systems would reduce motor vehicle to train accidents, if only the tech companies would implement it.

Do drivers really watch their maps to see upcoming hazards? I just glance once in a while to see how far until my next turn and which direction it will be.

<comment><USER7> I could certainly imagine that if e.g. Waze warned about railroad crossings that it could prevent some small percentage of almost sleeping drivers/truckers getting hit.

It's a numbers game. 999/1000 will be watching just fine without a warning from their navigation app; that warning might bring awareness to 999.8/1000 or so. Or something like that. That could result in a pretty dramatic decrease of accidents.

This seems like a no-brainer to me - they should add a (visual) warning about this.

<comment><USER4> Is there a visual warning already? Like this for example (from Germany)?

[https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Bl...](https://upload.wikimedia.org/wikipedia/commons/thumb/4/43/Blinklicht_mit_Halbschranke%2C_BO_1961.svg/800px- Blinklicht_mit_Halbschranke%2C_BO_1961.svg.png)

<comment><USER7> Sure, there are obviously visual warnings in real life, like that. Pretty sure that kind of thing is universal.

I'm talking about something that would pop on the Waze display, like 20 seconds ahead of reaching the crossing.

<comment><USER4> Why would this be an improvement?

It encourages taking drivers' eyes off the road. Can you offer an argument that its value is such that that's a net improvement?

</comment></comment></comment></comment></comment><comment><USER8> > Adding information on railroad crossings to mobile apps would "provide road > users with additional safety cues and ... reduce the likelihood of crashes," > the NTSB wrote...

I know this is going to sound like victim blaming, but are RR crossings really _that_ hard to see? Surely the apps are not the problem, but maybe from the people racing the train and losing? And I get that _some_ crossings might be hard to see until you're right there, but how is this tech's problem and not an infrastructure problem? IOW, if I wasn't using a maps app with added information, I would be in _more_ danger, so why doesn't infrastructure fix it?

<comment><USER9> There is no shortage of videos of people in cars pulling in front of visible trains. Drivers stop over rail tracks all of the time. Frequently pedestrians hit by trains, excluding suicides, have earphones in.

</comment></comment><comment><USER10> For reference, this is the crash referenced by the article: [https://en.wikipedia.org/wiki/2015_Oxnard_train_derailment](https://en.wikipedia.org/wiki/2015_Oxnard_train_derailment)

Apparently a driver drove onto the railroad tracks after confusing them with a road at night.

<comment><USER11> > Funding, however, remained unavailable in Ventura County for the estimated > $35 million grade separation project.

That's the key line in that article. Adding it to Google Maps doesn't fix this. Putting up a sign that says "No idiot, don't turn here" doesn't fix this. Better driver training doesn't fix this. Better working time rules (the driver was "severely fatigued") doesn't fix this.

Grade separation fixes this.

<comment><USER12> According to the Bureau of Transportation, there are 211,631 at grade crossings in the US [1]. At $35 million a crossing, that's $7.4 trillion.

[1] [https://www.bts.gov/content/highway-rail-grade-crossings- typ...](https://www.bts.gov/content/highway-rail-grade-crossings-type)

</comment></comment></comment><comment><USER13> I understand the motivation behind the NTSB recommendation, but think I net out siding with the app developers here. The last time I checked, the overwhelming majority of active at-grade rail crossings are well-marked (and usually controlled by lights/gates in populated areas). Adding potentially confusing clutter to apps feels like it may in fact _compound_ the problem if people mistake the tracks for roads/intersections.

Obviously, if apps are routing people down rail tracks, that's a big problem, but it doesn't seem to have been what happened here. Expecting developers to try to solve for every possible dumb human scenario doesn't end well.

</comment><comment><USER14> Do US railway crossings not have clear markings and raisable barriers? I find it hard to have sympathy for people who drive across railway crossing without actually looking at the road while driving a dangerous vehicle like a car.

<comment><AUTHOR> Some of it is people driving down the railway track. It may be confusing at night (all the incidents I could find mentioned in the bottom happened at night), particularly if there is a road junction immediately following the railway crossing, that you were supposed to turn on. Then an indicator to 'cross the railway, and then turn right' would be nice.

Although, I wonder whether US railway crossings can be hard to see at night.

<comment><USER15> Old-style railway crossings often included gates that closed across the tracks to prevent this happening. [http://i.ytimg.com/vi/3VF4hWvjC3g/maxresdefault.jpg](http://i.ytimg.com/vi/3VF4hWvjC3g/maxresdefault.jpg)

Is it really that hard for the railways to fix this problem with better modern automated gates?

<comment><AUTHOR> I've only seen those kinds of gates in Europe. Mind, I have not seen all US railway crossings, but I have seen a few, and none of them had gates like that. Plus, I think there is too little money in the railway network's budget to implement those gates, so asking navigation apps to do it for them is the cheap option.

<comment><USER16> Most of the ones I have seen have gates

(I'm in the US)

<comment><USER17> US rail crossing gates tend to move vertically, and there's never a barrier that crosses the tracks, only the road. Sometimes, they only block the road in the direction of travel which leaves an easy path for drivers to go around the gates if they're feeling impatient.

</comment></comment></comment></comment></comment></comment><comment><USER18> How often does this happen? Not all 270 railroad crossing deaths are similar to the specific accident mentioned caused by confusing a railroad crossing with a road.

At the same time, there must be billions of railroad crossings by vehicles every year. If map apps do anything to announce/mention railroad crossings, I would expect there to be some number of people that make this mistake _because_ of confusion about the announcement/indication instead of preventing it.

</comment><comment><USER19> How about people just pay more attention when they drive their 3000 lbs hunk of murder metal?

</comment><comment><USER20> Putting info about railroad crossing into maps is _bad_ , period.

People must be forced to look on the damn road instead of putting everything including speed limits in their navigation system - I would actually _outlaw_ displaying everything that can be seen with your eyes in navigation apps.

Otherwise such stories or "tin can opener bridges" are not ever going to disappear. If you're wondering about the bridge: [https://youtu.be/u_9rWH0p43A](https://youtu.be/u_9rWH0p43A)

There are literally flashing lights and a direction "OVERHEIGHT" blinking, and people are too focused on their navs/phones to notice it.

<comment><USER21> In all fairness, in Boston/Cambridge, students have been ripping open the top of rental trucks on Storrow and Memorial Drives since long before there were navs/phones. (Admittedly, the warnings didn't used to be as prominent as they are today.)</comment></comment>


Mounting poisonings, blindness, deaths as toxic hand sanitizers flood market - <AUTHOR> https://arstechnica.com/science/2020/07/mounting-poisonings-blindness-deaths-as-toxic-hand-sanitizers-flood-market/ <comment><USER2> I've had to walk into grocery stores where they force you to use a squirt of their unlabeled bottle. I've seen a place use an "anti-microbial" liquid that contained no alcohol at all (I was intrigued by it and found that it used benzalkonium chloride ...and I'm not sure it's at all effective against the covid-19 virus). It's like we're at the point of hand-sanitizer-theatre. I think an earlier shortage meant that businesses were just purchasing whatever that could get their hands on.

</comment><comment> xkcd-sucks Are these people drinking it? Everyone in my grad school chem lab washed hands with methanol to no ill effect

<comment><AUTHOR> > Are these people drinking it?

Apparently, yes [1]: "The agency is aware of adults and children ingesting hand sanitizer products contaminated with methanol that has led to recent adverse events including blindness, hospitalizations and death."

[1] [https://www.fda.gov/drugs/drug-safety-and- availability/fda-u...](https://www.fda.gov/drugs/drug-safety-and- availability/fda-updates-hand-sanitizers-methanol)

<comment><USER3> I'm a huge supporter of Darwin.</comment></comment></comment>


China to flatten 700 mountains for new metropolis in the desert - <AUTHOR> http://www.guardian.co.uk/world/2012/dec/06/china-flatten-mountain-lanzhou-new-area <comment><USER2> Another grandiose city project like the Russian ones from the soviet era, just more Chinese.

I wonder what these guys are smoking...</comment>


Deep Reinforcement Learning: From Board Games to the Boardroom - <AUTHOR> https://insidebigdata.com/2018/06/21/deep-reinforcement-learning-board-games-boardroom/ <comment><AUTHOR> Important to note: this post follows the announcement yesterday that Bonsai will be acquired by Microsoft: [https://blogs.microsoft.com/blog/2018/06/20/microsoft-to- acq...](https://blogs.microsoft.com/blog/2018/06/20/microsoft-to-acquire- bonsai-in-move-to-build-brains-for-autonomous-systems/)</comment>


Powa Technologies, once a UK fintech Unicorn, is in administration - <AUTHOR> http://uk.businessinsider.com/powa-technologies-lines-up-deloitte-as-administrators-2016-2 <comment><AUTHOR> From a day _before_ the announcement of administrators, a press release indicating an investment firm stepping forward to 'rescue' the firm: [http://www.prnewswire.com/news-releases/thompson- investments...](http://www.prnewswire.com/news-releases/thompson-investments- steps-forward-to-fund-powa-569459102.html)</comment>


Grepping logs is still terrible - <AUTHOR> http://asylum.madhouse-project.org/blog/2015/05/07/grepping-logs-is-still-terrible/ <comment><USER3> It's beyond me how he doesn't understand that text logs are a universal format, easily accessible, that can be instantly turned into whatever binary format you desire with a highly efficient insertion process (Splunk is just one of those that does a great job).

Here is the thing he doesn't seem to understand - all of us who are sysadmins absolutely understand the value of placing complex and large log files into database so that we can query them efficiently. We also understand why having multi-terabyte text log files is not useful.

But what we find totally unacceptable is log files being shoved into binary repositories as the primary storage location. Because you know what _everyone_ has their own idea of what that primary storage location should be, and they are mostly incompatible with each other.

The nice thing about text - for the last 40 years it's been universally readable, and will be for the next 40 years. Many of these binary repositories will be unreadable within a short period, and will be immediately unreadable to those people who don't know the magic tool to open them.

<comment><USER4> > text logs are a universal format

Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison.

Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope.

Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with three different types of your "universal format" in there. I'm not giving this example because it's contrived, but because I was dealing with it the very moment I read your comment.

<comment><USER3> But at least you have a fighting chance. What if that exact same data was dumped into a binary file, that you did not know how to decode?

Originally, you had a problem - the data wasn't formatted in a manner that you could parse cleanly.

Now, you have a new problem - not only is the data not formatted properly, it's now in some opaque binary file.

Saying that there are poorly formatted text files isn't a hit against text files, it's a hit against poor formatting. The _exact same problem exists_ if the file is in binary form, and not formatted properly.

<comment><USER5> Sure, an opaque binary file is pointless.

But that's not what most logging systems that log to binary files offer. They give you specs (example: [http://www.freedesktop.org/wiki/Software/systemd/journal- fil...](http://www.freedesktop.org/wiki/Software/systemd/journal-files/)) and tools.

Binary doesn't have to mean closed/opaque.

<comment><USER6> And the specs will be gone in 40 years. While ASCII will stick around.

<comment><USER7>
        And the specs will be gone in 40 years. While ASCII will stick around.

Why would they be gone? You realize ASCII is a 'spec' too?

If a binary format has an open specification, it's as future proof as ASCII. ASCII's durability is due to a clear and open specification that's easily implemented. Not some magic sauce that makes it instantly human readable.

That text you see? It's not what's actually in the file. That's just 1's and 0's like every other format. There's literally no difference between ASCII and any other "binary" format.

<comment><USER8> Text encodings have come and gone before, too. We don't use the Baudot code on modern computers still, and EBCDIC is confined to IBM mainframes.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Cool, so which standard binary log storage format should we all switch to?

Should I submit patches to jawstats so that it'll support google-log-format 1.0 beta, or the newer Amazon Cloud Storage 5 format? Or both? Or just go with the older Microsoft Log Storage Format? Or wait until Gruber releases Fireball Format? Has he decided yet whether to store dates as little-endian Unix 64 bit int timestamps, or is he still thinking about going with the Visual FoxPro date format, y'know, where the first 4 bytes are a 32-bit little-endian integer representation of the Julian date (so Oct. 15, 1582 = 2299161) and the last 4 bytes are the little-endian integer time of day represented as milliseconds since midnight? (True story, I had to figure that one out once. Without documentation.)

Should I write a new plugin for Sublime Text to handle the binary log formats? Or write something that will read the binary storage format and spit out text? Or is that too inefficient? Or should I give up on reading logs in a text form at all and write a GUI for it (maybe in Visual Basic)?

Do you know when I should expect suexec to start writing the same binary log format as Apache, or should I give up waiting on that and just write a daemon to read the suexec binary logs and translate them to the Apache binary logs?

Should I take the time to write a natural language parsing search engine for my custom binary log format? Do you think that's worth the time investment? I would really like to be able to search for common misspellings when users ask about a missing email, you know, like "/[^\s]+@domain.com/" does now.

I look forward to your guidance. I've been eagerly awaiting the day that I can have an urgent situation on my hands and I can dig through server logs with all of the ease and convenience of the Windows system logs.

<comment><USER10> The system should provide a standard API for writing and reading logs. The precise format of the underlying log files is thus rather unimportant at this level of abstraction. Other than the logging subsystem and recovery tools, there's no need for any software to be accessing such log files directly (outside of the API functions). This is how Windows has done it for years.

<comment><USER11> You missed the joke at the end where he correctly pointed out that Windows' logging is a total joke, and that discovering information from Windows logs is essentially impossible unless the tool writer specifically predicted your use case.

And that's the nub of it: text logs are for when you may have many varied, complex reader use-cases, and you don't understand all those cases well enough yet to lock them down forever, and you have a thousand excellent tools at your disposal that you would like to be able to continue to use.

Recent log spelunking for me included 'cat _log.? | grep fail | sed 's/^._worker_id$//g' | awk '{ print $5, $4 }' | sort -n -r | sed 30q'.

There's no analogue in any binary logging system I've ever found.

<comment><USER12> It seems to me that a simple transitional tool for a binary logging system would be for the implementer of the binary logging system to also include a tool that consumed a binary log file on stdin and produced a stream on stdout in one (or more, selecting which by command line arguments) common text log formats.

That lets you develop an ecosystem of supporting tools that take advantage of any strengths of the binary format, while still allowing the freedom of using the (initially, at least, probably far more capable) set of tools available for the text formats.

<comment><USER11> what is the point of such a 'transition' if there never arrives any point at which there is net added value to a binary format?

<comment><USER12> If there is some (not initially necessarily _net_ for _all_ users -- benefit being, after all, something that varies from user to users, but significant for some subset of users) benefit, the point is to mitigate the cost of moving out of a native text format, and increase the number of users for whom there is initially a net benefit, which also increases the initial use of the binary format and the effort likely devoted to building auxiliary tools which leverage it to some advantage, increasing the speed at which the net benefit of the format for a wider range of users is increased.

This may or may not ever make it a net benefit for _every_ user, but that's okay. There's a whole lot of space between "this technology is the best choice for everyone" and "this technology is the best choice for no one".

</comment></comment></comment></comment></comment></comment><comment><USER13> _Binary logs may be fine for you, but don 't force it on us!_

This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of incredibly bad _social_ "change management" by its proponents.

They are therefore deploying pre-emptive rhetorical covering fire against the day when greppable logs will be removed from the popular Linux distributions. Plain text is the lingua franca; binary formats bind you to their tools with a particular set of design choices, bugs and disadvantages. My adhoc log grepping workflow has a different set of bugs and disadvantages, but they're _mine_.

<comment><USER14> "The number of people administering small systems is much greater than the number of people administering large systems"

Do you have any evidence for this statement? Because it sounds all kinds of wrong.

<comment><USER15> [http://www.internetlivestats.com/total-number-of- websites/](http://www.internetlivestats.com/total-number-of-websites/)

If I read it correctly there are about 250 million active sites (roughly). It seems unlikely that they are all massive corporate sites.

As an aside, the idea that systemd is a good thing is hilarious to me at the least because it is so brash about making an important change to a huge chunk of the system. Yes the bugs will eventually get ironed out, but in the meantime? Count me out! I have work to do and am not interested in being a free tester for Redhat on my live systems.

<comment><USER16> I'm pretty sure that counts (eg) each wordpress.com subdomain as a separate website. [1] counts like that and gives a roughly comparable number.

That gives a lot of economy scales.

[1] [http://news.netcraft.com/archives/category/web-server- survey...](http://news.netcraft.com/archives/category/web-server-survey/)

<comment><USER15> The link I included states that those are unique hostnames. Perhaps they are including subdomains on the same ip address, but you might note that rather than quoting the 1 billion sites, I reduced that by their estimated 25% being actually active. Additionally they state that there are on average 3 users per site in 2014. Maybe that doesn't mean anything, but as a rough estimate that all implies far more small sites than large ones.

</comment></comment></comment></comment></comment><comment><USER17> Take this philosophy to an extreme and you end up with a dedicated data format and tooling/APIs to access the data for every subsystem, not just logging. Essentially, this is Windows.

The downside to this is that now you don't have a set of global tools which can easily operate across these separate datasets without writing code against an API. I hear PowerShell tackles this; I don't know how well. The general principle though harms velocity at just getting something simple done, to the benefit of being able to do extremely complex things more easily. See Event Viewer for a good example of this.

Logs don't exist in isolation. I want to use generally global tooling to access and manipulate everything. I don't want to have to write (non-shell) code, recall a logging-specific API or to have to take the extra step of converting my logs back to the text domain in order to manipulate data from them against text files I have exported from elsewhere for a one-off job. An example might be if I have a bunch of mbox files and need to process them against log files that have message IDs in them. I could have an API to read the emails, and an API to read the logs, or I could just use textutils because I know an exact, validating regexp is not necessary and log format injection would have no consequence in this particular task.

I do see the benefits of having logs be better structured data, but I also see downsides of taking plain text logs away. Claiming that there are _no_ downsides, and therefore no trade-off to be made, is futile. It's like playing whack-a-mole, because nobody is capable of covering every single use case.

<comment><USER18> > Essentially, this is Windows.

Actually any non-UNIX OS clone out there, including mainframe and embedded OSes.

</comment></comment><comment><USER19> Honestly - I agree about the ELK stack side - piping all your logs into ES / Logstash is a great idea. (Or Splunk / Greylog / Logentries)

If you run any sort of distributed system, this is vital. And while that counts as binary logs, I would argue that on the local boxes it should stay text.

I would agree, if you are running any sort of complex queries on your data - go to logstash, and do it there - it much nicer than regexes.

If on the other hand, you just want to see how a development environment is getting on, or to troubleshoot a known bad component tail'ing to | grep (or just tail'ing depending on the verbosity of your logs) is fine.

I don't have to remember some weird incantation to see the local logs, worry about corruption etc.

One problem I will point out with the setup described is syslon-ng can be blocking. If the user is disconnected from the central logstash, and their local one dies, as soon as the FIFO queue in syslog-ng fills, good luck writing to /dev/log , which means things like 'sudo' and 'login' have .... issues.

Instead, if you have text files being written out, and something like beaver collecting them and sending them to logstash, you have the best of both worlds.

</comment><comment><USER20> Windows has had binary logging forever. Is windows administration some wonderland of awesome capability for getting intelligence out of logs? Hell no.

For administering Unix like systems, the ability to use a variety of tools to process streams of text is an advantage and valuable capability.

That said, your needs do change when you're talking about managing 10 vs 10,000 vs 100,000 hosts. I think what you're really seeing here is a movement to "industrialize" the operations of these systems and push capabilities from paid management tools into the OS.

<comment><USER21> I think that largest problem with Event Log is overreliance on structure. Often you have one particular log record that you know is the problem, but no idea what it means because you have some generic event code and bunch of meaningless structured data.

Freeform text logs usually contain more detail as to what exactly happened.

<comment><USER10> That's not a limitation of the Event Log system itself, as you can easily write freeform text in the message rather than building it up with localised strings and structured data, e.g. [https://msdn.microsoft.com/en- us/library/6w20x90k(v=vs.110)....](https://msdn.microsoft.com/en- us/library/6w20x90k\(v=vs.110\).aspx)

</comment></comment></comment><comment><USER22> Grepping logs is terrible. Reverse engineering a binary format so you can diagnose why you are down/crashing/losing data is far worse. Logs should be handled as text until they reach their long term storage... then whatever helps analyze and query is fine...

</comment><comment><USER23> Yeah,in the presence of adequate tooling you don't need to grep logs. But how much more effort is required to use those tool-friendly loggings? Where is your god when the tool fails?

For me the main reason to access plaintext logs is they seldom fail, and they are simple. They are a bore to analyse, they CAN be analysed.

Anyway, this discussion only makes sense if the task at hand involves _heavy_ log analysis, don't complicate what is simple when it isn't needed.

As for the razor analogy, you're right, however I wouldn't change my beard to be "razor compatible only". In the software world I'd say it is still not uncommon to find yourself "stranded in a desert island".

</comment><comment><USER24> Oh jeez. Yes there are better and more performant tools for parsing optimised binary databases; nobody disputes that. And yes, tools like Splunk are more user friendly than grep; nobody disputes that either. But to advocate a binary only system for logs is short sighted because logs are the goto when everything else fails and thus need to be readable when every other tool dies. There's quite a few scenarios that could cause this too:

      * log file corruption - text parsing would still work,

      * tooling gets deleted - there's a million ways you
        can still render plain text even when you've lost
        half your POSIX/GNU userland,

      * network connection problems, breaking push to a
        centralised database - local text copies would still
        be readable.

In his previous blog post he commented that there's no point running both a local text version and a binary version, but since the entirety of his rant is really about tooling rather than log file format, I'm yet to see a convincing argument against running the two paradigms in parallel.

<comment><USER25> If 'tooling gets deleted' is a problem you probably have much bigger concerns than log files.

<comment><USER26> 'tooling gets deleted' could easily happen after changing logging systems... while it would be shortsighted to uninstall your old logging system entirely (if you have logs laying around in that format) it's not unheard of.

The more likely situation would be that the logs are stored on a shared storage server, and the machine you are using to look at the logs doesn't have the logging system installed.

<comment><USER25> > The more likely situation would be that the logs are stored on a shared > storage server, and the machine you are using to look at the logs doesn't > have the logging system installed.

So expose the shared storage to a system running any current mainstream Linux distribution. I understand what you're saying, but this still doesn't seems like a huge concern.

<comment><USER26> ... We were talking about logging systems with proprietary tools for manipulating logs. Ergo, 'any current mainstream Linux distribution' wouldn't have them installed by default.

</comment></comment></comment></comment></comment><comment><USER27> This is a discussion for a sake of discussion. The way I see it is that author has a niche situation on his hands and therefore should use a product designed for that particular niche, instead of complaining how everyone's wrong and trying to shove his perspective down peoples' throats.

</comment><comment><USER15> Sounds like somebody in the systemd camp. I really dislike added complexity when it is totally unnecessary. If people want to transform their logs into a different storage format, that is up to them. Text files, however, are a fantastically simple way of storing... (drumroll please) text. Surprising /s

</comment><comment><USER28> > For example: find all logs between 2013-12-24 and 2015-04-11, valid dates > only.

That's a straw man. If you're grepping logs, you don't need a regular expression that matches only valid dates because you can assume that the timestamps on the log records are valid dates. But I suppose

        2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01]))

doesn't look so bad.

The whole thing is similarly exaggerated.

<comment><USER3> Not to mention 99.9% of the searches one does of a log file isn't really that complex. Heck, I'm willing to wager that 90% + of my searches over the last 20 years have been in log files from a particular _day_.

That's the thing about having simple text log files - the cognitive load required to pull data out of them, often into a format that can then be manipulated by another tool (awk, being one of the more well known), is so low that you can perform them without a context switch.

If you have a problem, you can reach into the log files, pull out the data you need, possibly massage/sum/count particular records with awk, all without missing a beat.

This is particularly important for sysadmins who may be managing dozens of different applications and subsystems. Text files pull _all_ of them together.

But, and here is the most important thing that people need to realize - <USER2> scenarios in which complex searching is required, _by all means_ move it into a binary format - that just makes sense if you really need to do so.

The argument isn't _all text_ instead of binary, it is _at least text_ and _then_ use binary where it makes sense.

</comment></comment><comment><USER29> After reading the article I wonder if there are lots of tools that do all the binary advantages in indexes but leave the logs as text files, why that is not fine. To get the binary advantage the log does not have to be binary.

The example with the timestamps is also strange. No matter how you store the timestamps, parsing a humanly reasonable query like "give me 10 hours starting from last Friday 2am" to an actual filter is a complex problem. The problem is complex no matter how you store your timestamp. You can choose to do the complexity before and create complex index structures. You can choose to have complex algorithms to parse simple timestamps in binary or text form, you can build complex regexes. But something needs to be complex, because the problem space is. Just being binary doesn't help you.

And that's really the point here, isn't it? Just being binary in itself is not an advantage. It doesn't even mean by itself that it will save disk space. But text in itself is an advantage, always, because text can be read by humans without help (and in some instances without any training or IT education), binary not.

Yesterday I was thinking there might be something about binary logs. Now I'm convinced there isn't. The only disadvantage seems to be that you also lose disk space if you store it in clear text. But disk space isn't an issue in most situations (and in many situations where it is an issue you might have resources and tools at hand to handle that as well) It is added complexity for no real advantage. Thanks for clearing that up.

<comment><USER10> Another advantage of using structured data rather than free-form text is that you can more precisely encode the essence of the event, with fields for timestamp, event source, type of event, its severity, any important parameters, and so on. This permits logging to be independent of the language of the system operator. Rather than grepping for what is almost always English text, one can query a language-independent set of fields, and then, if a suitable translation has been done, see the event in one's native language.

When applied widely throughout a system, this leads to the internationalisation of log messages. Thus lessening the anglocentric bias in systems software. Windows has done this for years, at least with its own system logging (other applications can still put free-form text into the event logs if they wish.)

<comment><USER29> About your first point: Independence. You are less independent of English and more dependent on the binary format and the tools who can handle it. It's a trade-off. And it might be just an opinion, but for me it's not a good trade- off. Learning English was a one-time endeavour for me. But binary formats and tools have to be learned separately.

About what you put in the log message: You can also put different fields in a line of text. Not getting the advantage or trade-off here.

About the internationalisations: As non-English developers we force all our systems who have logging internationalisation to English system language so we have a common ground for the messages. Understanding the English message is nearly no burden. Log Messages are Event triggers, either in code or in a developer's/admin's mind. If I get a log message in my native language I don't know which event that triggers, which makes it actually harder.

Really. I don't know any non-English person who considers log internationalisation a good thing. Fighting anglocentricism is a very anglocentric topic. Outside of UK/US that's a non topic. We (non-English people) are happy that there is a language we can use to talk to each other and we don't really care how it came to be that widely known.

And even if you don't speak English, I don't see the advantage of parsing \x03 instead of "Error:.*". Both are strings that have a meaning which is rather independent of its encoding.

<comment><USER10> This is also just anecdotal, but I used to work with a Chinese sysadmin (I was in the UK, he was in China) who found it much more preferable to work on the localised Windows servers we had installed over there, as all the UI and messages were in his native tongue. I'm sure it was easier for him to gain expertise in the tools he needed for his work, than to become proficient enough in English to understand every obscure log or error message that the system might throw at him.

</comment></comment></comment></comment><comment><USER22> Grepping logs is terrible. Reverse engineering a binary format so you can diagnose why you are is worse. Logs should be handled as text until they reach their long term storage... then whatever helps analyze and query is fine...

</comment><comment><USER15> My main problem with this is that ascii is not something that will ever change over time. The data format is wonderfully static. Forever. Introduce a binary format? You get versioning. It is a major downside.

</comment><comment><USER30> What you lose when you move away from text logs is not any real benefit; what you lose is the illusion of control you have with text logs.

Text logs can be corrupted, text logs can be made unusable, you need a ton of domain-specific knowledge to even begin to make sense of text logs, etc.

But there's always a sense that, if you had the time, you could still personally extract meaning from them. With binary logs, you couldn't personally sit there and read them out line by line.

The issue is psychology, not pragmatism, and that's why text logs have been so sticky for so long.

<comment><USER15> A substring of text may or may not be a date and based on the excellent tools available in linux you can decide how to extract that "data point". If binary logging is little more than a stream of text, then that is fine, but I seriously doubt that is the push happening. Personally I prefer having a raw stream of data that I have to work with as best as I can rather than having to use some flag defined by somebody else to range across dates. That is the fundamental difference it seems: do you want a collection of tools that can be applied in a variety of ways or do you want the "one way" (with potential versioning... have fun!).

Again if the binary log is simply better compressed data, well we have ways of compressing text already as an afterthought. This really, fundamentally, seems to be a conflict in how people want to administer their systems and, for the most part, this seems to be about creating a "tool" that people then have to pay money for to better understand.

</comment></comment><comment><USER31> > Does database store the data in text files? No? That's my point.

This guy is a first class idiot who knows enough to reformulate a decided issue into yet another troll article. "a database (which then goes and stores the data in a binary format)". How about a text file IS a database. It's encoded 1s and 0s in a universal format instead of the binary DB format which can be corrupted with the slightest modification or hardware failure.

</comment><comment><USER32> I think there are a number of issues that are getting mushed into one.

* Journal is just terrible.

* some text logs are perfectly fine.

* when you are in rescue mode, you want text logs

* some people use text logs as a way to compile metrics

I think the most annoying thing for me about journald is that it forces you to do something their way. However its optional, and in centos7 its turned off, or its beaten into such a way that I haven't noticed its there.... (if that is the case, I've not really bothered to look, I poked about to see if logs still live in /var/log/ they did, and that was the end of it. Yes, I know that if this is the case, I've just undermined my case. Shhhhh.)

/var/log/messages for kernel oopes, auth for login, and all the traditional systemy type things are good for text logs. Mainly because 99.9% of the time you get less than 10 lines a minute.

being able to sed, grep, tee and pipe text files are brilliant on a slow connection with limited time/mental capacity. ie. a rescue situation. I'm sure there will be a multitude of stable tools that'll popup to deal with a standardised binary log format, in about ten years.

The last point is the big kicker here. This is where, quite correctly its time to question the use of grep. Regex is terrible. Its a force/problem amplfier. If you get it correct, well done. Wrong? you might not even know.

Unless you don't have a choice, you need to make sure that your app kicks out metrics directly. Or as close to directly as possible. Failing that you need to use something like elastic search. However because you're getting the metrics as an afterthought, you have to do much more work to make sure that they are correct. (although forcing metrics into an app is often non trivial)

If you're starting from scratch, writing custom software, and think that log diving is a great way to collect metrics, you've failed.

if you are using off the shelf parts, its worth Spending the time and interrogating the API to gather stats directly. you never know, collectd might have already done the hard work for you.

The basic argument he puts forth is this: text logs are a terrible way to interchange and store metrics. And yes, he is correct.

<comment><USER33> Not that I think it matters all that much but journald will be running on your centos box, but it'll be configured to spew text into /var/log.

Just type journalctl and you should see the data there.

<comment><USER32> I suspected as much, so long as I don't have to use its tools I couldn't care less. Unless it eats resources.

<comment><USER34> > Unless it eats resources

I have no idea on its effect in practice, but theoretically it should have negative effect because it soaks in all the logs, and _then_ it forwards them to the logging daemon, even when journal storage is turned off.

</comment></comment></comment></comment><comment><USER35> Of course you need to log some data in textual format for emergencies, but if you had a tool that indexes events on timestamps, servers, monitorees, severity and event type, while severely reducing the storage required, you would be able to log much more data, and find problems faster. Arguing binary vs text logs is like arguing serial port vs USB on some industrial systems.

</comment><comment><USER36> Great to see some effort in this area. I've been using New Relic and it's pretty great for errors because we've setup Slack/email notifications. However, there's nothing for general log (e.g.: access log) parsing. I'm installing an ELK stack on my machine right now and hope that it's enough

</comment><comment><USER37> Doesn't this just mean that we should have a more "intelligent" version of grep? For example, this "supergrep" could periodically index the files it is used on, so searching becomes faster.

</comment><comment><USER29> *edit: I'm wrong, this was not the link posted yesterday. [https://news.ycombinator.com/item?id=9496850](https://news.ycombinator.com/item?id=9496850)

</comment><comment><USER38> Isn't everything will be solved by a some kind of grep that's date/timespan aware?

</comment><comment><USER39> But then how will I watch the log files go by in real time?

</comment><comment><USER40> It seems to me that most of the worry about a binary log file being "opaque" could be solved with a single utility:

        log-cat <binary-log-file>

… that just outputs it in text. Then you can attack the problem with whatever text-based tools you want.

But to me, having a utility that I could do things like, get a range of log lines — in sorted order —, or, grep on just the message, would be amazing. These are all things that proponents of grep I'm sure will say "you can!" do with grep… but you can't.

The dates example was a good one. I'd much rather:

        log-cat <bin-log> --from 2014-12-14 --to 2015-01-27

Also, my log files are not "sorted". They are, but they're sorted _per- process_, and I might have multiple instances of some daemon running (perhaps on this VM, perhaps across many VMs), and it's really useful to see their logs merged together[2]. For this, you need to understand the notion of where a record starts and ends, because you need to re-order whole records. (And log records' messages are _going_ to contain newlines. I'm not logging a backtrace on one line.) grep doesn't sort. |sort doesn't know enough about a text log to adequately sort, but

        $ log-cat logs/*.log --from 2014-12-14 --to 2015-01-27
        <sorted output!>

Binary files offer the opportunity for structured data. It's really annoying to try to find all 5xx's in a log, and your grep matches the process ID, the line number, the time of day…

I've seen some well-meaning attempts at trying to do JSON logs, s.t. each line is a JSON object[1]. (I've also seen it attempted were all that is available is a rudimentary format string, and the first " breaks everything.)

Lastly, log files sometimes go into metrics (I don't really think this is a good idea, personally, but we need better libraries here too…). Is your log format even parseable? I've yet to run across one that had an unambiguous grammar: a newline in the middle of a log message, with the right text on the second line, can easily get picked up as a date, and suddenly, it's a new record. Every log file "parser" I've seen was a heuristic matcher, and I've seem most all of them make mistakes. With the simple "log-cat" above, you can instantly turn a binary log into a text one. The reverse — if possible — is likely to be a "best-effort" transformation.

[1]: the log writer is forbidden to output a newline inside the object. This doesn't diminish what you can output in JSON, and allows newline to be the record separator.

[2]: I get requests from mobile developers tell me that the server isn't acting correctly all the time. In order to debug the situation, I first need to _find_ their request in the log. I don't know what process on what VM handled their request, but I often have a _very_ narrow time-range that it occurred in.

<comment><USER4> What you're describing is journalctl.

<comment><USER40> I've not yet had the opportunity to try systemd. :-) Someday.

</comment></comment></comment><comment><USER41> Logstash, Kibana, Splunk

</comment><comment><USER10> Windows systems have had better log querying tools than grep for years now, with a well structured log file format to match. It's good to see Linux distributions finally catching up in this regard.

Not that the log files on Linux are all entirely text-based anyway. The wtmpx and btmpx files are of a binary format, with specialised tools for querying. I don't see anyone complaining about these and insisting that they be converted to a text-only format.</comment>


Argument: Deleting A.I. is murder. Copying an A.I. is as abhorrent as cloning - <AUTHOR> http://en.arguman.org/deleting-ai-is-murder-copying-an-ai-is-as-abhorrent-as-cloning <comment><USER2> Why is cloning abhorrent? I never understood this.</comment>


Ask HN: What's your personal finance system? - <AUTHOR> Everyone's solution from using Mint to multi-page excel spreadsheets is probably a little bit different.
I've been doing some research for a personal finance software product (likely an app) and would be interested to hear your personally tailored systems & solutions.
Could be around budgeting, saving, investing, etc. <comment><AUTHOR> Personally I use copilot, robin hood and an excel spreadsheet for tracking long term savings goals & giving amounts.

\- Copilot for an aesthetically pleasing heads up of how much $$ I've got as well as insights into where it's going,

\- Robin hood for stocks & trading

\- Excel for long term goals, targeted amounts for tithing & saving per month, and a rundown of my monthly expenses.</comment>


Synthetic Data: A bridge over the data moat - <AUTHOR> https://heartbeat.fritz.ai/synthetic-data-a-bridge-over-the-data-moat-29f392a52f27 <comment><USER2> Great piece!</comment>


Show HN: Stampery. Send notaries on vacation using the blockchain - <AUTHOR> https://stampery.co

<comment><USER2> It is a nice service, maybe useful, and I think it is also cheaper than the others that do the same thing. However the blockchain has no legal value under the government courts, so you cannot "send notaries on vacation".

Anyway, nice job.

<comment><AUTHOR> Thanks! Well, depending on what kind of service you wanna replace it's actually legally binding. For example, private documents can be easily stamped using our service. And anything creative (music, art, etc.) as well.

<comment><USER2> Yeah, I didn't know that, I'm commenting as if I know anything about it, but I don't.

Maybe you should explain that on the site.</comment></comment></comment>


Vote for new Java name - <AUTHOR> https://docs.google.com/forms/d/e/1FAIpQLSerWX0GwYZqxWXOnmM8D8shvzrw8PY0NH92TWwg6puCrttxsA/viewform <comment><USER2> "Our process whittled this list down to two names"

This reminds me of the Family Guy episode where Bonnie, the wife of quadriplegic Joe Swanson goes to Paris to get away from Joe and chooses a new partner - quadriplegic as well

<comment><USER3> It's an apt, funny analogy, but two off-topic nitpicks, if I may:

1) Joe is paraplegic. He has control of his arms.

2) I know you didn't mean it this way, but your comment makes it sound like Bonnie is trying to get away from her husband who's terrible because of his disability, and then she chooses another person who's terrible. That's what Family Guy was implying, for all I know.

I just want to point out that paraplegic people can (and many do) have full, normal relationships. A paraplegic friend of mine is happily married, and his wife doesn't have to do anything for him.

<comment><USER2> Wrt to 1) you are absolutely right - Joe is paraplegic.

As to 2) , not to belittle your own experience, but Joe's condition is exactly why Bonnie goes to Paris in that episode. For whatever it's worth Joe and Bonnie patch things up in the end.

<comment><USER3> Thanks for the respectful response. I know it can be really annoying when people nitpick these things.

It sounds like the original message is Family Guy's, though, so perhaps I shouldn't have brought it up here.</comment></comment></comment></comment>


Löve 0.7.1 (lua game framework) is released - <AUTHOR> http://love2d.org/forums/viewtopic.php?f=3&t=2503 <comment><USER2> For those who prefer Python, PyGame (<http://www.pygame.org/>) is a very good framework for making games.

Having said that, this looks _very_ interesting and I'll definitely keep an eye out for it in future.

<comment><USER3> LOVE2D is a joy to work in. It really is fun, it makes particle effects and physics and sound easy. The community is awesome.

I have been working, since September, on my senior project in Pygame. It isn't fun to work in. The documentation is great, but the community seems dead. Things are not easy in Pygame. Distribution is especially painful, with py2exe and py2app and setuptools. I've had a lot of trouble getting certain sounds to play. I would not recommend it.

Really, go with LOVE2D or Flixel or even Flash if you want to make games in a high level language. Not only that, but if you want to have _fun_ making a game, and not get bogged down in low-level details.

I would not recommend Pygame.

<comment><USER4> Do you know, would it work with LuaJIT?

I am always amazed at the speed of LuaJIT, at least in these benchmarks:

[http://shootout.alioth.debian.org/u32/code-used-time-used- sh...](http://shootout.alioth.debian.org/u32/code-used-time-used-shapes.php)

It holds its own against scala and java -server with only a fraction of the code size (but only on 32 bit machines).

<comment><USER5> Here's a thread from the forums regarding love2d+luajit:

[http://love2d.org/forums/viewtopic.php?f=4&t=2409](http://love2d.org/forums/viewtopic.php?f=4&t=2409)

Seems to be _somehow_ possible to get it working. It is not officially supported, though. I guess as luajit matures there is a decent possibility that it will be a supported option since games (even 2d) would benefit from the increased speed. OTOH portability of love2d would probably be decreased, too.

</comment></comment></comment></comment><comment><USER6> At this year's Global Game Jam (a rather fun event where participants create a game in 48 hours), most of the people from our site used Love, despite having no experience with either it or Lua.

You can view what my 5-person team created on GitHub: <https://github.com/icco/RainbowDeathSwarm> . It's rather horrid, but I suppose it's decent considering our lack of knowledge of either the language or making games.

</comment><comment><USER7> I showed Love to my kids over the school holidays, it was a nice sweet package. Single exe, runs on Windows, and there were enough tutorials to get one started. The only thing I had a problem with was with the names of some of the libraries, which had overt sexual overtones.

<comment><USER8> Lua is a great language to learn, too. Simple yet powerful. Plus it is used seriously in the games' industry.

</comment></comment><comment><USER9> None of these features or fixes have an impact on my current work, but I'm glad to be using something that's being actively maintained.</comment>


Startup employees don't earn more - <AUTHOR> https://80000hours.org/2015/10/startup-salaries-and-equity-compensation/ <comment><USER3> I'm gonna be straight with y'all, I earn more as a post office mail carrier than I ever did as a programmer for a silicon valley startup with $1 million dollars seed funding. I also surprisingly have more freedom even though I work basically like a slave. Nobody wants to admit it but we are in a tech bubble. Sure, if you are in the right startup things might be different. I had a great valley experience - but it was all a façade, the next recession will really highlight the failures of silicon valley startup culture. Wall Street got pounded last time, and I'm sure I'll get downvoted night and day for saying this here - but I'm sorry guys, you are the new problem. Silicon Valley is the new Wall Street. There, I said it. Ironically I worked in both places before joining the postal service. So I do have perspective. My claims aren't baseless either, your startup that might even be funded at a million dollars like my employer's was is just a tax write-off for your investor when it fails. They are playing slots with your dreams, time, and skill - the only person who truly loses is you.

<comment><USER4> Wow, someone gives a company $1million and you claim that the person is playing with dreams, time and skills?

Give us a break. No startup has to accept investor money. If you don't want their million, don't take it. Bootstrap with your own cash.

The reality is that going into startup for money is a big risk, it's a big gamble. Go in for the fun, go in to learn new things, play with new tech, go in to avoid ( big company politics, enterprise style software development, meetings, locked down environments).

Do it for the for the fun and if you make money great, if you don't, you come out with new experience. Long before fullstack became a buzz word, I earned my badges due to startup. I coded, I did design, I touched the DB, I setup the servers, wired the network, setup the firewalls. It was hella fun, I made $0. But my experiences from then still serves me till this very day.

Cheer up, the world is not such a terrible place.

<comment><USER5> > Do it for the for the fun and if you make money great, if you don't, you > come out with new experience. Long before fullstack became a buzz word, I > earned my badges due to startup. I coded, I did design, I touched the DB, I > setup the servers, wired the network, setup the firewalls. It was hella fun, > I made $0. But my experiences from then still serves me till this very day.

You can have these experiences at a large tech company too, while also being paid well and while collaborating with experienced people who can teach you a lot.

<comment><USER6> More likely you will be pidgin-holed into a very narrow area of responsibility with no autonomy or opportunity to grow.

<comment><USER7> Not if you keep moving.

</comment></comment></comment></comment></comment><comment><USER8> If your #1 job criteria is compensation, do not work at a startup. You'll be paid more at big companies.

If you don't mind being paid a little less, and want the opportunity to be a part of building a company from the ground up, consider working at a startup. There will be a lot more ups and downs than at a company like Google. But at smaller companies, you'll have the chance to have a material impact on the success of the organization instead of being just a cog in the wheel.

By no means am I saying you should accept a below market offer from a sketchy startup that offers no benefits and expects 12 hour work days (those unfortunately exist).

Disclaimer: I run a 6-person seed-stage company. To help keep our employees happy (and to be competitive with bigger companies when recruiting), we try to get as close as possible to big-company salaries, we don't expect anything beyond 40 hrs / week, 100% paid health benefits, 401k, 25 paid days off per year, snacks, food, full financial transparency, etc. (Not trying to self- promote here, mainly trying to communicate that not all small companies are bad just because they're small).

Also consider that the day-to-day work at startup is dramatically different than the day-to-day work at a big company. For people who don't mind being paid low $100k's instead of mid $100k's, the day-to-day work should be the main deciding factor when considering startup vs. large co. In my opinion, startups are more fun, you get to wear more hats since your role is likely less specialized, (ideally) there's less politics, you can have a real impact on the success of the business while (hopefully) watching it grow, and learn a lot really fast along the way.

That said, the big company route definitely offers higher compensation and more stability, and makes a lot of sense for people who are a little more risk-averse.

Also +1 on viewing equity as a lottery ticket. When evaluating offers from a startup, try to maximize your equity in negotiations (ie. more lottery tickets), but the fact is the majority of startups will eventually shut down, so don't use equity as a major factor when comparing offers between large and small companies.

<comment><USER9> I'm going to have to disagree that you are "a paid a little less" at startups. In my experience working for a startup is a 20-30% drop in compensation.

When startups say they are near market, they mean base salaries. They neglect bonuses, stock grants, signing bonuses, stock refreshes, 401K matching. These things add up real quickly. Even worse, I've found that startup salaries usually don't grow as large companies (very few startups do equity refresh, raises, COL increases, etc.)

Compound this 30% difference over a 4 year stint and things start to look bad, esp. when you think of $1m+ home prices left in startup havens like San Francisco & New York. If the majority of startups were in affordable cities, I'd agree that the compensation difference isn't that much, but the bulk of startups are located in places where that 30% drop means a significant change in quality of life.

<comment><USER10> My career has involved working at an e-healthcare startup (like 25 people), then NetSuite pre and post IPO (probably no longer a startup), then a tiny startup that folded after 6 months, then a 20 person startup, then a 60 person startup.

So 4 definitely honest-to-god startups, and one post-startup. The only one I didn't get raises and equity refreshes at was the tiny startup that folded after six months -- because it didn't exist long enough to have a compensation cycle.

I don't think this was exceptional. I very much take exception to "very few" startups doing equity refreshes, raises, etc. Some may not. Most do.

</comment></comment></comment><comment><USER11> This will surprise nobody on Hacker News.

The allure of startups isn't a guaranteed 20% more than a regular company's salary; it's the possibility of 1000% more.

<comment><USER12> It's like the lottery, except you can influence your odds a bit more.

<comment><USER13> It would be interesting to see what the expected outcome's distribution is with buying lottery tickets for $X/month vs. doing a startup (and potentially earn less by $X/month than one could have had at a larger corporation).

<comment><USER14> Interesting. If I spent 20% of my salary on lottery tickets, how good would my chances be of winning vs cashing in my stock at a startup that goes public in which I made 20% less salary? And what would the payout differences be?

<comment><USER15> The startup deal is a much better deal than lottery tickets. If you can get a 1% ownership stake and we assume a 10% chance of a 100 million dollar startup, the expected value of that grant is 100k. The expected value of buying lottery tickets with 20% of your salary is close to zero You should also consider the fact that if you are an early employee in a startup, your future expected payout is at least in part a function of your efforts

<comment><USER16> Is 10% a reasonable value for that chance? Is 1% a reasonable value for the ownership stake? What about dilution and payout rights?

A reasonable expected value of a lottery ticket is directly computable: jackpot times probability of a winning set. I'm not so sure the same is true for the value of startup employment.

<comment><USER17> Lottery tickets' jackpot is usually determined in part by sales of tickets for that drawing, and a winning combination wins a divided pot in the case of multiple winners (the probability of which depends again on sales of tickets for that drawing.)

So, no, you can't, at the time of buying a ticket, actually calculate more than a guess of they parameters you cite for determining its expected value.

<comment><USER16> The published jackpot is a very accurate value to use, and the probability of being the sole winner is a very reasonable probability to use. The actual expected value is not going to differ from that product by enough to matter most of the time.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> As others have posted, people don't go to small startups to earn more on average.

They're not looking at the average or median, they're looking at the 99%ile numbers, because they think the startup they picked has an angle or advantage, giving it a shot at a 99%ile exit.

80000hours' audience is clever enough to understand cumulative distribution functions or even just correlated sets of [25%ile, median, 75%ile, 90%ile, 99%ile] numbers.

<comment><USER19> I've always thought that historically engineers cared more about working on things they had more emotional investment in rather than working on things they had no passion for that makes money as an established business like ad tech or dog dating social networks.

<comment><USER20> Your comment made me think of the episode of the show Silicon Valley (in the current season), where the engineers want to build a scalable platform rather than a box (appliance). The negotiate with mgmt to let them work on the platform after they have worked on the box :)

As an aside, I thought this episode was very well done. The engineers believe their algorithm will get better over time as it ingests more data (hence, a scalable platform is the way to go). A box won't learn, and benefit from these improvements. However, large companies want boxes (servers/appliances in their data center) and aren't in a position to have their data leak out (for statutory or competitive reasons in many cases).

</comment></comment></comment><comment><USER21> This should surprise nobody.

If I had my copy of Cryptonomicon at hand, I'd quote the bit about everybody being impressed Avi could let them sell out for roughly what they'd have made at a regular job.

Yes, even in the '90s, this was pretty well known.

</comment><comment><USER22> It's easier to get hired at an early stage startup because there are fewer hoops to jump through. You can then use that startup to build industry connections and get your next job via inside track referral through someone you know at a bigger / more established company.

<comment><USER23> That's what I did. I was fresh out of a college nobody's heard of, joined a startup as their first employee and eventually networked my way into a big tech company.

Nearly doubled my salary when I eventually jumped.

</comment></comment><comment><USER24> This is just a minor annoyance, but the header image of this article, which has been used in other articles [0], is CC-BY 2.0 licensed [1], and yet is not attributed to the original author at all.

(Perhaps even more interesting is that it looks like it's being used to depict a startup company environment, even though it's actually a hostel in Amsterdam during the Wikimedia Foundation's annual international hackathon.)

[0] [http://silicon.nyc/code-academies-new-york- coding/](http://silicon.nyc/code-academies-new-york-coding/) [1] [https://commons.wikimedia.org/wiki/File:Wikimedia_Hackathon_...](https://commons.wikimedia.org/wiki/File:Wikimedia_Hackathon_2013,_Amsterdam_- _Flickr_-_Sebastiaan_ter_Burg_\(28\).jpg)

EDIT: I misread the license. It's CC-BY 2.0, not CC-BY-SA 2.0.

<comment><USER24> Follow-up: I contacted them, and they added the attribution, so all is well!

</comment></comment><comment><USER25> To be honest, I don't think startup employees (and by startups I don't mean Dropbox, I mean companies who have to count the cash they spend) are low paid. Obviously a startup can't pay like a company with tons of cash can. In tech and everywhere else. I like working at small companies because you get to do more things, because of the atmosphere, small team, agility, lack of bureaucracy, etc. Where I do think startup employees are getting absolutely screwed, specially in the US, is in the working hours. As an employee, you don't have founder shares. You should just be able to work your 9-5 hours and call it a day. If the founders think there's more work to be done, they can either prioritize better, hire more people, do it themselves, or just accept that not everything they want to do can be done with the resources they have. Period.

</comment><comment><USER26> I'm working at my third startup, and money has never been close to the top of the list when deciding. First, I'm old enough and experienced enough to be skeptical about the pot of gold at the end of whatever. Second, my time is far more valuable than money, and how I spend it, who I spend it with, and what I am doing are much, much more relevant to me.

</comment><comment><USER27> Obviously has been referenced a lot but still relevant: [https://danluu.com/startup-tradeoffs/](https://danluu.com/startup-tradeoffs/)

</comment><comment><USER28> This talks about aggregate earning rather than earning per unit of time. I think the latter is what people are/should be interested in, and from what I understand, startup employees put in a lot more hours.

<comment><USER29> Well, it depends on what you're trying to achieve. There are legitimate reasons to try to achieve the highest possible pay per year, no matter how many hours in the year you have to work in order to get it. In the case of 80000 hours, the goal is to donate as much money to charity as possible, and, if I may put it somewhat crudely, they would prefer you work as much as possible, only stopping work in order to eat and sleep, if that's the way to achieve the highest salary, since the noblest calling in their view is to dump as much money as possible into (the most optimal) third world charities, and 80000 hours has already taken the time to determine which charities maximize third world utility. (Technically, you need to deduct the negative utility to you from working increased hours, but your utility should be the same as a poor person in the third world, so that negative effect is minimal.)

<comment><USER30> Completely disagree. Even if you're maximizing for donations, any reasonable analysis would still have to privilege your own well-being as a giver. Lowering your burnout/regression rate from 25% to 5% by giving yourself a higher quality of life seems like one of the best investments you can make.

You don't need to expend wads of cash. But you don't need to be destitute as a software engineer to give effectively.

<comment><USER29> I actually agree with you. I was being a bit facetious - 80000 hours is somewhat infamous for a rigidly utilitarian view of effective charity and for optimizing for monetary means of increasing the effectiveness of said charities. But even they recognize the risk of burnout.

</comment></comment></comment></comment><comment><USER31> The thing I wish someone spelled out to me when I started my startup employment journey over ten years ago is that you need to pick the right company. Most startup equity ends up being worthless, so you need to think of yourself as a venture capitalist. Will this company really be the next big thing? Do I believe in them or am I just excited that they want to hire me?

Looking back I wasted far too much time working for companies that I knew deep down were never going to make me money on options and working on projects that I knew were going to fail. I wasted my time and I let these failure companies give me valueless options in place of pay that I would have received working for an established company. Job interviews are bi-directional and everyone tells you that, but it took me wasting many years to understand what that meant.

<comment><USER32> I think you've nailed it. If you're an employee at a startup, you should be doing the same amount of research as a VC, if not much more. A VC is only investing a bunch of cash that they can afford to lose, and probably only a little bit of their own money. As an employee, you're investing maybe 5 years of your life that you can never get back.

</comment></comment><comment><USER33> 80k is my favorite EA organization. I love reading their articles!

</comment><comment><USER34> I was looking for a job last year, and had a number of startups contact me as my 14 years of experience were interesting for them.

"We can't afford to pay much because we are a startup" came up on a number of occasions. Sorry, but I already get paid poorly relative to other engineers because I am in Spain, and I am not a charity.

</comment><comment><USER35> Uh, taking the average equity granted and multiplying by the average value of equity is flawed on so many levels.

If there's any correlation between equity granted and value, that analysis fails. Also, the average company value needs to be weighted by number of employees. Chances are if you take that average you'll get a higher number.

</comment><comment><USER36> This varies greatly from company to company. I would say in startups the pay related issues can have much greater variance than in established companies. The startups that have lots of cash flowing in but have urgent customers often pay well. The startups that haven't yet found a business model probably don't.

</comment><comment><USER37> Are they _happier_ ?

</comment><comment><USER38> In other news the sun rose this morning. How do these stories get upvote?

</comment><comment><USER39> On average? Of course not. It's a power-law! Averages mean nothing in power- law situations.

<comment><USER40> It's not a true power-law (assuming it's not a lognormal). It's truncated by the fact that anyone considering a startup would never have earned more than the world GDP, or more specifically, the peak tech market cap to date (Apple's $775b). Once it's truncated at a finite value, the moments become meaningful.

<comment><USER41> It is probably something similar to a negative binomial distribution. (Polya distribution)

You play until your VC decides not to fund you or you actually win. This can be done over either money or time.

This only works on startups as a group, for any specific one you would use a more complex method as trials are not independent. Discrete please - <USER2> distribution, derived by a Markov chain.</comment></comment></comment>