Visualizing Bayes' theorem - <AUTHOR> http://blog.oscarbonilla.com/2009/05/visualizing-bayes-theorem/ <comment><USER2> This is a very well done presentation - I am genuinely impressed. We could use more of these types of explanations. The more people who understand, the more help we have.

</comment><comment><USER3> I have always visualized probability and conditional probabilities in terms of Venn diagrams, it is easier to understand stuff this way. I think this is _the_ best way to teach probability to a beginner. What do you think ?!

<comment><USER4> I am reminded of something I read in Isaac Asimov's book on astronomy, where he talked about his great idea for how to visualize the size of distant objects and how lamentable it was that people don't use his obviously superior method.

I stared at the pages unable to comprehend how he thought the size of a penny held a mile away was easier to visualise than the alternative.

Likewise Eliezer's "Intuitive" explanation of Bayesianity - I've read through it twice (lightly) and it's thoroughly not intuitive. I'd need to study it not just read it.

Presumably there is some distance from my current mental state vector to any with a comfortable grasp of Bayesian probability, and some explanations will take a quick route and some a less direct one. Hence, I think there are different 'best' explanations for different people depending on what they already know, what they want to know and what they want to touch upon along the way.

I'm skeptical that there is one "_the_ best way" to teach probability, or anything else.

<comment><USER5> I think, for a majority of people, it _would_ be an easier way to study probability. Visualization makes things much more understandable than algebraic formula derivation.

<comment><USER4> _Visualization makes things much more understandable_

Does it? Are you <http://lesswrong.com/lw/dr/generalizing_from_one_example/> ?

<comment><USER3> Yea maybe I am bit biased about my opinion because I understood probability easier using visual examples, but people who I have interacted with mostly also understand things easier in the visual form. So I think it maybe _the_ best method to teach probability because it will reach out to a majority of people, ofcourse, there might be people who might understand it better if explained in a different way, but that maybe a minority, so all the different methods can be tried out in school.

One more thing I would like to add is that everyone has a biased opinion of things based on their perception of the world and that's why I think there should be debates. Generalizing from examples and experiences is a natural method developed during evolution.

If a kid touches a hot plate and experiences pain, I think it is perfectly valid that it should generalize that to all other hot plates. :)

</comment></comment></comment></comment></comment><comment><USER6> I think of it as the joint variable event space relative to (or normalized by) the condition variable event space. In a few words what is shown in the article diagrams.

</comment><comment><USER7> An explanation similar to this was what made Bayes' theorem really click for me when I took probability. Thanks Prof. Terpstra!

[http://www.ndsu.nodak.edu/ndsu/normann/statistics/faculty/te...](http://www.ndsu.nodak.edu/ndsu/normann/statistics/faculty/terpstra/index.html)

</comment><comment><USER8> I don't actually remember the formula for Bayes' theorem. I just visualize it as shown in this article, and write down equations from there. For me, anyway, that is actually a more reliable way of working.

</comment><comment><USER9> I just skimmed the article, having received treatments of Bayes' theorem many times over the course of my almost-finished college career; however, I think this is neat, because making use of visualization/imagery is a good approach to teaching and understanding statistics, which the human brain is (typically) not capable of handling well.</comment>


Inflation In One Page - <AUTHOR> http://www.scribd.com/vacuum?url=http://www.fee.org/pdf/the-freeman/hazlitt1104e.pdf

<comment><USER2> Ouch.

Explain my hypothetical case to me using the constructs of this "article".

1\. I sell widgets. My factory can make 1000 a month.

2\. On Tuesday, I sell 100 widgets at $1 each.

3\. On Wednesday, my widgets are featured on the news.

4\. On Thursday, I sell 800.

5\. I realize that I'm going to need more widgets! I decide to start selling them for $2 to cut demand a little, and to profit a little.

This is a demonstration of inflation, yet it has absolutely nothing to do with the money supply, but everything to do with the demand for my widget. For macro-sizing, see: oil.

Here's a better one-page explanation of inflation: <http://en.wikipedia.org/wiki/Inflation>

<comment><AUTHOR> From the link you kindly provided, your hypothetical case is not inflation.

 _Although "inflation" is sometimes used to refer to a rise in the prices of a specific set of goods or services, a rise in prices of one set (such as food) without a rise in others (such as wages) is not included in the original meaning of the word. Inflation can be thought of as a decrease in the value of the unit of currency._

<comment><USER2> >"Measuring inflation requires finding objective ways of separating out changes in nominal prices from other influences related to real activity. In the simplest possible case, if the price of a 10 oz. can of corn changes from $0.90 to $1.00 over the course of a year, with no change in quality, then this price change represents inflation. But we are usually more interested in knowing how the overall cost of living changes, and therefore instead of looking at the change in price of one good, we want to know how the price of a large 'basket' of goods and services changes."

Basically, I'm trying to point out that inflation is a hotly contested complex term, which we are currently demonstrating by linking to different sections of the same entry that somewhat contradict.

It is far to large a topic to be understood in a one page primer that states: "Government prints too much money."

</comment></comment></comment><comment><AUTHOR> pdf : <http://www.fee.org/pdf/the-freeman/hazlitt1104e.pdf></comment>


Facebook gets the expedited discovery it sought in Ceglia case - <AUTHOR> http://paidcontent.org/article/419-ceglia-v.-facebook-exchanging-the-evidence/

<comment><USER2> One would have to believe this is one of the items standing between Facebook and IPO.</comment>


New AWS Directory Service - <AUTHOR> http://aws.amazon.com/blogs/aws/new-aws-directory-service/ <comment><USER2> does this mean I can create LDAP-backed ssh login for my EC2 boxes without the hassle of actually running OpenLDAP?

<comment><USER3> That was what I wanted to know, too. Ever since IAM launched, I've been wanting to authenticate ssh against it. Sadly, the docs all seem to specifically say EC2 instances running windows.

<comment><USER4> I'm not able to spin up an instance to test with this week .. but if my linux hosts -inside- my network can auth to my corporate AD (and they can!) I don't know why an AWS EC2 instance would be different.

Add the software, configure, and presto?

</comment></comment></comment><comment><USER5> Amazon is systematically adding service upon service to their offering. I'll be interested to see where this ends up once they're all tightly integrated and working together.

<comment><USER6> Well, when we looked at moving stuff to the 'cloud', Amazon offers so so much. In effect Amazon can provide a vendor lock-in through the range of services (and their quality) it provides. That is in no way a bad thing, but just shows how other 'cloud' providers need to keep playing catch-up.

It's also an architecture consideration for the way we develop. Yes you want to use this Amazon feature, but if you do, moving to another provider becomes harder (e.g. we're looking to use SQS, but know if we moved provider I'd need to implement my own solution for that, probably mq rabbit).

To rehash a classic phrase: "Nobody ever got fired for buying Amazon."

<comment><USER7> SQS works fine when used from other hosting providers (or even from client- side JavaScript for that matter - with the right IAM token).

<comment><USER8> Anyone think a remote directory service would work well if a lot of your infrastructure is hosted elsewhere? I wonder if latency would become an issue.

Also, how hard is it to set up/care/feed a basic directory service? I've toyed with FreeIPA which was easy ([http://www.freeipa.org/page/Main_Page](http://www.freeipa.org/page/Main_Page), based on 389 DS). I actually did not realize Samba now includes a directory service and might check that out.

<comment><USER4> > Also, how hard is it to set up/care/feed a basic directory service?

Setting one up is pretty easy. The problem comes in when you find yourself depending on it, and then it gets out of whack, or Just Grows and you've got OUs and CNs all over the place.

At my last employer - a mid-sized manufacturer - we had several Tier II apps. These must be in working order or production is shut down. The shop floor application. Core ERP. Email. Above II is I which are applications that must exist or the company cannot operate.

There was only one Tier I application: Active Directory.

</comment></comment></comment></comment></comment><comment><USER9> This is really great news for AWS environments where compliance and user account management are an issue.

<comment><USER10> This is precisely the struggle that I've been having to deal with lately. Though, does anyone know if this works with OpenDirectory?

</comment></comment><comment><USER11> I didn't see any reference to it in the documentation, but I wonder if they'll provide an HTTP API for authenticating against these directories (or providing delegated auth services when connected to an on-premise AD instance).

</comment><comment><USER12> It's looking to me like you add users either through their "Simple AD" exporter, or you have to use "Active Directory Administration Tools"... there's no way to add my users etc. directly into their directory? [http://docs.aws.amazon.com/directoryservice/latest/adminguid...](http://docs.aws.amazon.com/directoryservice/latest/adminguide/directory_management.html)

</comment><comment><USER13> Playing catch-up with Microsoft Azure's existing services. Nice to see the competition. [http://azure.microsoft.com/en-us/services/active- directory/](http://azure.microsoft.com/en-us/services/active-directory/)

<comment><USER14> I can't tell if you are joking about the fact that Microsoft had support for their proprietary solution, Active Directory, in their cloud before Amazon...

<comment><USER15> AD is certainly proprietary but it's also LDAPv3 compliant.

</comment></comment></comment><comment><USER16> This article does not mention handling directory services between short lived EC2 instances and SQL Server. Right now the guidance is to use SQL Server Auth; but Windows Auth would be far better to use, especially in these circumstances.

</comment><comment><USER17> The question I have is can I use this as a central auth system the other way. AD is expensive as crap, but if this is self-healing and self-backup, then its a good idea for us. Frankly anything is better than Open directory

<comment><USER12> To me it looks like the directory is ONLY exposed within the VPC. So that would mean you can't auth from servers that live in other clouds or on prem.

<comment><USER18> How about via VPN into the VPC?

</comment></comment></comment><comment><USER19> I'm amazed they haven't done this already.

</comment><comment><USER20> What would be the rational for choosing this over say, Google Apps authentication?

<comment><USER21> Primarily to manage Windows accounts on Windows EC2 instances. Google Apps authentication covers websites via OAuth and OpenID, but not local user accounts on systems like Windows, Linux, and OS X.

</comment></comment><comment><USER22> Nice. Basically RDS but for Samba.

Box = cloud msft file sharing?

Now amazon going after the directory?

</comment><comment><USER23> Somewhere there is no progress. Stagnation.

<comment><USER4> Explain?</comment></comment>


Project management and how we use Trello at CPUsage - <AUTHOR> http://www.cpusage.com/blog/knowledge/project-management-and-trello-at-cpusage/

<comment><USER2> I'm trying to get our QC Lab to start using Trello. They get anywhere from 10 to 50 chemicals per day that they need to sample, test, and release. And 3 different departments send them these samples, some on schedule, some as need be. I've thought of putting put a big screen with their Trello todos and showing everyone in the company how they're backlogged or performing on schedule. Anyone have experience with a public read-only Trello board in their offices? What has the reaction been?

</comment><comment><USER3> How is Trello different from a task list with multiple status values?

e.g.

A shared task list can correspond to a Trello board.

Each task has a status field that can have multiple values (Pending, In Development, Done, In Test, In Beta, In Prod etc). If the list can be grouped by this status field, we have Trello "lists"

Cards are the tasks themselves.

Even Outlook + Sharepoint task list can do this.

<comment><AUTHOR> I agree that it is similar to a basic task list. For us, the differences are that Trello makes things like values and task assignments easy and clean. It is also a very visual tool, glancing at it can tell you a lot!

</comment></comment><comment><USER4> Trello is one of those tools that's really powerful but really open ended. It's hard to get started with so it's awesome to see how others use it!

<comment><AUTHOR> You know, I think you might have been the person that told us about it.....so, thanks!

</comment></comment><comment><USER5> Nice article. I'd add a board for tracking serious customer service issues that come up, including bugs and feature requests.

</comment><comment><USER6> Great article Jeff. Couldn't do it without Trello.</comment>


Ask HN: Why my app marketing approach is not working? [videos included] - <AUTHOR> Hi Hackernews,
As I was struggling with phone addiction, I decided to create an app to help me control my addiction while enjoying the benefits of social media and smart-phones.
 What does "No Notification Hour" do? It disables notifications for an hour and at the end of the hour will let you receive all the notifications that you have missed.
 what have I done up to this point? I have created 4 sets of ads for Facebook up to this point, spending nearly 20 GBP, and getting 4 installs through them. The ads contained 2 videos, one for young students in the UK and USA, and the other for young persons who live in Latin America and like meditating.
I don't know what another group of people I should promote it to that would give me a better result than the ones I have received.
I have created the following videos and used them for my facebook ads.
https://www.youtube.com/watch?v=s5OR2K6JgZY
https://www.youtube.com/watch?v=bj9mJNV49G0
https://www.youtube.com/watch?v=XFu41abtPIA
None  of my approaches to bring more people to use my app  does not seem to  be working. Can anyone help me find out what's wrong  with bringing more  people to use it.
Is there any way for me to improve on the marketing aspects? or is my product just not fit for the market?
https://play.google.com/store/apps/details?id=com.asadiltd.notificationmanager
Many thanks, George <comment><USER2> A few thoughts:

1\. The sample is likely too small to draw strong conclusions.

2\. What happened after those 4 people installed your app is a good place to start looking for answers. Did they open the app? Did they look around the app's screens? Did the enable it?

3\. From reading the play store listing, it's unclear why I would want to use this app. How is it different from my phone's built-in 'Do Not Disturb' feature, which can be enabled in just a couple of taps?

4\. Your UI seems a bit complex. For example, your second screenshot on Play shows an entirely blank screen, with _all_ functionality hidden in a menu activated after tapping on three dots.

5\. It looks like a pre-release beta, which might put some people off. e.g. in your first screenshot on Play, the header says NoNotificationHour. It's very unusual for UI text to use camel case, so I guess it wasn't a deliberate choice.

6\. Have a friend try out the app. Uninstall the app from your phone, open your app's page in the Play app, and hand the phone to your friend. Tell them you made the app and want them to install and try it. But _don 't_ say anything else. Just observe them and listen to what they say without responding.

</comment><comment><USER3> I think you are underestimating the scale of the work required.

What you are paying per app install is typical. In any real business, 5 pounds to acquire a customer is OK.

<comment><AUTHOR> thanks for your comment. What else can I work on?

My problem is that it would be alright to be paying 5 pounds if I was selling something, but right now I have put the app for free on the play store to reach some users initially before selling it. Should I try out making a paid version of the app for 5 pounds and advertise it?

<comment><USER3> Back when I was a physics graduate student there was an old guy who ran the stockroom who sold candy bars for half of what they cost in the vending machine. This meant he was buying them for 40 cents and selling them for 30 cents on the honor system.

One day he saw there was less money in the till than he expected and I told him that he should have expected to lose money and he said "I expected to lose money, I didn't expect to lose that much money!"

Unless you have a profitable business plan there is no "cost to acquire a customer" which is low enough. This is particularly true when it comes to internet advertising where you are competing against app installs where people expect to make money (either in terms of the cost of the app or in-app purchases) and also where it is scalable. (You could spend 200, 2000, 20000 or more pounds advertising it.)

Here is my crazy idea, which the app store probably won't let you do.

One way to make people stick to a commitment (say not smoking) would be that if you screw up you have to make a donation to some organization you hate. How about some app that charges you for breaking your commitments?</comment></comment></comment>


Writing Kernel Exploits [pdf] - <AUTHOR> http://ugcs.net/~keegan/talks/kernel-exploit/talk.pdf <comment><USER2> Linux kernel to be more specific.

The Mac OS X/iOS kernel (xnu) doesn't share the address space with the user process, so many of those techniques (I haven't read the whole paper) do not apply.

<comment><USER3> That's interesting. What are the reasons some kernels choose to share the address space, while others don't (and/or how is it even possible to avoid that?)

<comment><USER4> Mostly performance. Sharing the address space avoids switching the address space mappings (page tables, etc) and flushing the TLB every time the kernel runs (e.g. on syscalls).</comment></comment></comment>


You Won't Believe How Delhi Police Found the Uber Office - <AUTHOR> http://www.huffingtonpost.in/2014/12/07/uber-delhi-office-address_n_6283190.html?utm_hp_ref=india

<comment><USER2> I am not a big fan of uber. It seems to me that they have some sketchy business practices. In my city uber drivers are required to have a state and city business licences, but so far no one has even applied.</comment>


Why Flash Doesn't Suck - <AUTHOR> http://pixelborne.com/blog/why-flash-doesnt-suck/ <comment><USER3> "some obnoxious people who dare dub themselves members of the "web elite" [citation needed]"

Anyway, I clicked the examples the author gave of good all-Flash sites. As expected, they act weird, they pop up "loading" indicators all the time, information density seems tiny, I can't copy/paste text that I should be able to, scrolling doesn't work the way it should (argh that's _so_ frustrating -- that's a fundamental interaction!)... oh and I didn't even try hitting the Back button.

Sure, all those presumably "could" be fixed with enough work. And an MFC app _could_ be written that rivals the latest .NET or Cocoa app. But in practice, the vast majority of apps are going to "feel" like the platform they're on. And the Flash platform feels super-sucky for information-heavy websites, if you ask me.

<comment><USER4> I think many of your criticisms apply to current Javascript UIs also. I have seen so many UI quirks, even in the stuff you'd think would work right like YUI/JQuery UI. However, its saving grace is its composable nature.

The biggest problem with Flash in my mind, and what will seal its doom, is that monolithic mentality that you should have a whole site that is only a flash program. That's wrong and throws away a large part of makes the web great. If you aren't building stuff out of the dhtml/dom substrate, if you're not basing architectural decisions on REST, you are doing it wrong.

<comment><USER5> > The biggest problem with Flash in my mind, and what will seal its doom, is > that monolithic mentality that you should have a whole site that is only a > flash program

That what RIA is all about. If you really hates Flash only, write your text in HTML and transfer it to Flash using javascript. It's best for both search engines and non-Flash browsers.

</comment></comment></comment><comment><USER6> So many things are wrong with this rant.

1\. "You don't have to follow convention anymore." Generally, convention makes it easier for visitors to figure out the website and get how to interact with it. The prototypical example is having links in blue with an underline. Form elements that look like other forms' elements are also important.

2\. Innovation: Flash is a tool that has created a lot of innovation, yes, but so has Javascript. I'd argue projects like jQuery are promoting innovation much better: it's Javascript that runs on pretty much any browser without the need of a CPU-sucking plugin.

3\. Rain City bikes: the author needs to brush up on SEO. Firstly, the page has quite a bit of content that is NOT flash, as cached by Google ( see [http://184.108.40.206/search?q=cache:ggNvi1Rjd3UJ:www.raincity...](http://18.104.22.168/search?q=cache:ggNvi1Rjd3UJ:www.raincitybikes.com/+Dutch+Bikes&cd=15&hl=en&ct=clnk) ). Secondly, the anchor text of a page will make it rank higher for queries that match the anchor text; this is regardless of the contents of the page. The extreme examples of this are called Google Bombs: <http://en.wikipedia.org/wiki/Google_bomb> . For Rain City, this query shows you how much the anchor text is helping them: [http://www.google.com/#hl=en&q=allinanchor%3ADutch+Bikes](http://www.google.com/#hl=en&q=allinanchor%3ADutch+Bikes) .

In short: nothing in the post is really arguing why Flash doesn't suck.

</comment><comment><USER7> There is one area where Flash doesn't suck: it provides a great toolset for creating rich interactive audio visual web experiences. If you're making games or animations, it's a great choice.

However it's always (with a few rare exceptions) the wrong tool for making web sites.

</comment><comment><USER8> My biggest complaint about Flash is that it isn't truly part of the web. If you have a completely flash based site, all browser conventions go out the window: bookmarks, the back button, and deep linking. Sometimes even non- browser conventions are gone like cut'n'paste, printing, or saving to a file. If you're going to do that, you should have a good reason beyond "it's easy" and "you can have music"

Flash is most successful when it's used properly: for small regions of video or cool interactivity. It's least successful when it's used to replace a properly coded website.

I don't think the 'web elite' want Flash to die necessarily; they just want all the same cool toys but with something that plays nice with existing web conventions. And they also don't want to sharecrop on someone's proprietary platform. I'd argue the author has taken a very shallow look at the issues involved here.

<comment><USER5> > all browser conventions go out the window: bookmarks, the back button, and > deep linking.

Most of the ajax in the wild is no better than that. Instead of saying Flash breaks everything, why don't we think about the good n' old browsing behavior model. Does it suit our Web 2.0 world well?

<comment><USER8> Actually, a lot of Ajax handles the back button very well. The Extjs framework, as one example, lets you back through almost all the navigation including flipping between dynamic tabs. I'm not even sure how they've implemented that.

There was a Ajax chat application that was posted here recently. And this was my comment: <http://news.ycombinator.com/item?id=1128206>

The back button is an extremely useful feature and should probably be in every application. Whenever a user gets confused or lost they just click that button -- it's a universal undo for navigation.

</comment></comment></comment><comment><USER9> As a Flash dev myself it's exactly this idea of the _possibility_ that Flash provides that made the platform appealing to begin with and continues to do so. Sure, you can make some horrible things, but the sheer number of things that you can make is so huge. Is it a good idea to make a 3D visualization of live sound data from the user's microphone and run facial detection on their webcam to overlay the visualization over their mouth while they talk? No, probably not. But I _can_. And because I can I'm going to experiment a lot more. And out of a million experiments that are only possible with Flash, maybe one is going to produce something truly innovative and unique. That's the power of Flash.

New capabilities of HTML are going to open similar experimental doors, and we'll all be better off in many ways because of it. I believe anything that expands what we can do is inherently a good thing, even if it allows things that should never be done.

<comment><USER10> Precisely. This is the point I try to make. Flash gives life to the experimental web.

Anyone care to tell me why web devs should want Flash dead? No, not a dumb blanket statement regarding open source. Think.

Flash sets the stage (no pun intended) for innovation on the web.

What if a developer wanted to introduced a new approach to video conferencing / VOIP and allowed developers to play with the experience among multiple audiences?

Why is Flash the spooky village witch, to which we must burn at the stake?

We'll wait on the W3C to "standardize" innovative processes into browsers. Good luck with that.

<comment><USER11> _Anyone care to tell me why web devs should want Flash dead?_

Because it's broken and therefore a pain for users. Should something like Flash exist? Yes. But Flash itself is a bad implementation of a good idea.

<comment><USER10> Flash has issues and anything the W3S and WHATWG do is without fault. Down with plugin architecture! All ideas and possible innovations that live outside of our dying, simple-minded, slow-moving, "One or None" religion should be killed! _chant_ Kill Flash! _chant_ Kill Flash! _chant_ Kill Flash! _chant_

/sarcasm

<comment><USER11> Give us Flash's source code. That's the problem. Nobody cares about specs, they care that they can't use their computer when a website using Flash is open.

If you were a programmer, you'd know that bugs can be fixed. But not by sitting around and complaining, but by writing code. Since Adobe won't let anyone fix their code, so the only solution left is to stop using Flash (and start using something else). The "something else" is improving browser support for fast Javascript, Canvas, video, 3d acceleration, etc.

<comment><USER10> _Give us Flash's source code. That's the problem._ It is a problem, but I am having a hard time believing that perspective.

The OOS community _can_ create a player. Since the SWF format is inherently open there is nothing really stopping anyone from making an _awesome_ Flash plugin, yet no one has. Must me a reason for that.

Oh wait, there is a reason. Money. Time. Adobe invests millions into R&D every year. Money that is used to do test cases and audience dynamics. OSS community does not.

The problem is software patents. Not Adobe. Not OOS. And of course, not Flash. A root problem exists and Flash, like much software, is the result.

</comment></comment></comment></comment></comment></comment><comment><USER12> _deep breath_

The most fun I've ever had in programming was programming Flash. There, I said it.

Flash is an absolute blast. In my first experience with it, I was able to slap together a simple sprite-based game with basic physics, sound, and a video background. The whole project took about 10 hours, and most of that was learning the toolkit and ActionScript. ActionScript is still one of my favorite languages.

Yes, it's terrible for standards, has awful usability, etc. We should be pushing web standards hard. However, we shouldn't forget that Flash makes programming fun and accessible. That's pretty neat, and so I hope it never dies entirely.

</comment><comment><USER13> I think the article makes a pretty poor case for Flash with it's examples. Blizzard's marketing sites are the de facto example I give of Flash used right - incredible and unintrusive layering of audio and animation on top of a website. Flash makes the difference between good and great.

I'll leave these here:

<http://starcraft2.com/>

<http://www.worldofwarcraft.com/wrath/>

<http://us.blizzard.com/diablo3/?rhtml=y>

<comment><USER14> Just for the record, all of that could be done better with HTML5.

<comment><USER10> The record of what? Talk is cheap. Let's see some examples.

<comment><USER14> I meant that there is nothing in those examples that couldn't be achieved using HTML5. Basically, the effects could be achieved using HTML5 video for the background, canvas for the particle effects, and either your choice of canvas or CSS transformations / animations for the movement of the character pieces.

It certainly wouldn't work for the vast majority of visitors, as most people are still using Internet Explorer (hence why I can't produce any examples of AAA HTML5 sites), but there is nothing shown there that is inherently Flash only, technologically speaking.

<comment><USER10> I need to see HTML5+JS perform as well as Flash doing multiple things. Sorry, sliding PNGs across the page with jQuery isn't really the same.

If we're gonna go into debate about Flash vs. HTML5, I am gonna need to see more than simple equalizers and canvas bar charts.

Show me <http://mycanvas.landsend.com> or <http://www.myspace.com/fanvideo> in HTML5. I wanna see this level of complexity without a hindrance on development time, performance or scope. Enough with the double talk.

<comment><USER14> jQuery and PNGs are actually not HTML5. I'd recommend checking out the CSS animation / transitions modules at the W3C to get familiarized. Safari's implementation is actually hardware accelerated. Also, check out the SVG animation module. See the ADC for a pretty nice selection of examples which do actually rival Flash websites, often without even using JavaScript.

<comment><USER10> _jQuery and PNGs are actually not HTML5._ I understand they aren't the same. jQuery is often cited as a "Flash killer" of sorts, just covering the bases.

 _See the ADC for a pretty nice selection of examples which do actually rival Flash websites, often without even using JavaScript._

Right. Except that they don't always.

I'm not saying that Flash _should_ be what we use for creating innovative content. Standards are slow. Video tag? We're just starting to see this implemented? After how many years has video been a demand on the web?

Flash will always do what the browser can't handle native. New demands for VOIP, Web Conferencing, h/w accelerated 3D will keep Flash alive as well an IDE (with mind crushing bugs, mind you) that brings everything together.

And if it doesn't live as a file format, that would be alright, too. There is a different experience with creating content within the Flash IDE. If Flash eventually outputted in native standards, that would be a great thing.

<comment><USER14> If that was your point, you could have just said "Microsoft doesn't support HTML5, end of story."

I wasn't trying to argue that HTML5 has the same adoption as Flash, simply that HTML5 is technically capable of creating those "state-of-the-art" websites cited in the first post.

FYI, HTML5 also addresses hardware accelerated 3D in the WebGL module. ;)

<comment><USER10> How I love the "The Flash-hater side-step".

If you read, I wasn't talking about adoption rates (although that is also an issue worth noting).

You might have noticed that I mentioned that Flash brought about new innovations to the web. Most notably, how video is handled.

Try to look forward and see beyond HTML5, just as you seem to have no problem doing with HTML5 adoption rates.

<comment><USER14> To be honest, I don't know what you're saying.

All I wanted to say was that those three sites could be technically created using HTML5.

You then seemed to say that HTML5 doesn't actually let you create the effects exhibited in those websites. I corrected you and referred you to the modules.

Sorry for any misunderstandings. If we are clear on the above, and you would like to bring up a new point, I'd be happy to engage in a separate debate.

<comment><USER10> _You then seemed to say that HTML5 doesn't actually let you create the effects exhibited in those websites. I corrected you and referred you to the modules._

The side-step again. The _examples_ that you gave (a vague reference to ADC) did not support your argument. You might remember: "Show me <http://mycanvas.landsend.com> or <http://www.myspace.com/fanvideo> in HTML5." It was apparent to me that we had moved away from the Blizzard sites.

The very simplistic Blizzard sites (which I agree _could_ be done in HTML5 to some extent) are not what I would consider heavy or complex.

I asked: "I wanna see this level of complexity without a hindrance on development time, performance or scope."

You replied with the same double talk I expect in a Flash vs. HTML5 thread: "I'd recommend checking out the CSS animation / transitions modules at the W3C to get familiarized. " You mean the fully implemented CSS transitions?

If you can't give examples, then just say that. Don't come back with a vague blanket statement.

<comment><USER14> Both of those websites, like the Blizzard sites could be completely done in HTML5. Is there a specific element that you believe could not?

I already explained why I cannot give you an example of a AAA HTML5 website (Internet Explorer doesn't support it).

If you'd like to discuss the technical aspects of HTML5, feel free to email me at firstname.lastname@example.com - I am going to stop checking this thread.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> Consider, in the context of _Two big reasons why Flash won't disappear any time soon. The first is porn._ that there was porn well before the invention of .gif or the img tag, and well before the invention of flash.

Also consider that an important property of the web is that its content is searchable.

Finally, I think that there is a desire to have a more open video standard.

<comment><USER16> I'm pretty sure porn can switch to HTML5 <video> tags anyways. Not quite sure why Flash is a requirement there at all.

<comment><USER17> I think that he was speculating that since popular porn sites are already using flash that there is no need to switch.

Or that there is some unmentioned barrier to switching video formats that porn sites don't have the means or ways to overcome. Either way, his point is unclear and I agree; porn can switch to HTML5 <video> tags when/if there is sufficient browser support.

<comment><USER14> Or more likely, they will roll out HTML5 video support and use Flash as a seamless fallback, like YouTube, DailyMotion, and Vimeo are publicly beta testing.

</comment></comment></comment></comment><comment><USER18> Probably 90% of my time on the web is spent working with information - searching for it, reading it, manipulating it, submitting it. Sometimes I'm using Firefox and reading and typing with my own eyes and fingers, sometimes I'm copy/pasting or even writing scripts to submit forms and screen-scrape. After all, that's the sort of thing the Web was designed for - free exchange of information (modulo HTTP's 402 error code).

Flash sucks because it interferes with my usage of the web. Almost every use of Flash puts a barrier between me and the information I want to get or give, a barrier that can be very annoying (though never impossible) to overcome.

About the only thing on the Web that isn't "information" is "experience" - <USER2> can't think of any example besides games, but there may be some. You know, where the valuable thing is the interaction, the ping and pong. That's not a thing I can paste into a spreadsheet, process with a script or bookmark. For such things, I can accept that the powerful Flash plugin is a better platform than the documents-and-tables world of HTML.

Still, most Flash usage that I see on the web is just to add glitz and glamour to an otherwise informative page - swoopy page transitions, animations, sound- effects. To the extent that your website prevents your visitors from obtaining and making use of the information they came to your website to obtain, Flash sucks.

</comment><comment><USER19> "What if I told you that you can do anything you want. You don't have to follow convention anymore. You don't have to do things the way they told you to in school. You can do anything."

What a lie. I can't do squat because flash either is either a POS on my platform (OS X & 64 bit linux) or nonexistant (blackberry and iphone).

<comment><USER20> ..."Turns out that's not totally true, but within the realm of Actionscript there are, without a doubt, more possibilities, and more opportunities to innovate. This is why I learned Actionscript."

</comment></comment><comment><USER17> As I was reading this article I kind of felt the urge to go learn flash (again, just for fun).

</comment><comment><USER21> As to video: Flash pretty much replaced Real Player and HTML5 will replace Flash.

<comment><USER10> You're not very well informed, it appears.</comment></comment>


Healthcare individual mandate found unconstitutional - <AUTHOR> http://volokh.com/2010/12/13/initial-thoughts-on-the-virginia-health-care-ruling/ <comment><AUTHOR> The actual memorandum opinion is available in PDF form here: [http://www.vaag.com/PRESS_RELEASES/Cuccinelli/Health%20Care%...](http://www.vaag.com/PRESS_RELEASES/Cuccinelli/Health%20Care%20Memorandum%20Opinion.pdf)</comment>


Machine learning in UK financial services - <AUTHOR> https://www.bankofengland.co.uk/-/media/boe/files/report/2019/machine-learning-in-uk-financial-services.pdf?la=en&hash=F8CA6EE7A5A9E0CB182F5D568E033F0EB2D21246 <comment><USER2> Thanks for sharing. Was literally just reading it a few minutes ago with great intrigue. Would be interesting to see what is classified as real ML and how much of it is being used for actual decision making - I lead an ML team at a bank and everyone tries to sell it more than it is.

Fraud detection is one of the areas I've come across that are actually doing something interesting within retail banking. But agreed - plenty of opportunities for ML to make things more efficient.

<comment><USER3> A a paraphrase of a kind of self-deprecating but insightful comment from a ML prof I heard: There's a lot of hype around machine learning, but it can't really do that much. If you want to see what it's capable of, check your Amazon recommendations. Sort of works some of the time, and other times offers you complete nonsense. Amazon has very smart engineers and the best algorithms and tons of data. This is about the limit of what you can currently do.

</comment></comment><comment><USER4> Watching the UK try to adopt ML is like watching a toddler try to drive a Ferrari.

The BoE prides itself on being "up" with the latest talking points amongst the chattering classes. But take a look at how the BoE is actually run and you will get an idea why British society struggles with change.

Incompetence and knowledge of irrelevant information is pride of place in British society. Example: Carney has been the most accident prone BoE governor since...the last one, who was trying to raise rates in 2008...the Chief Economist was head of Financial Stability in 2006...yes, seriously (incidentally, he recovered his reputation by being able to hit headlines by talking about new trendy topics like ML).

Where ML has been implemented in the UK it is often by managers who have almost no competence in any real-world skills. They have heard about ML, think it is magic, and want to look like they know what they are doing. This has resulted in very bad outcomes (a recent story - [https://www.theguardian.com/society/2019/oct/15/councils- usi...](https://www.theguardian.com/society/2019/oct/15/councils-using- algorithms-make-welfare-decisions-benefits)).

I don't think this effect is at all well understood (economists, for example, tend to assume that the person in charge actually knows what they are doing...despite all the research suggesting the contrary in the UK). Insurance has done well because there is a good level of statistical knowledge amongst managers. But I suspect the UK will continue to lag the world (as it does in almost everything else) as the hype disappears, and the typical British suspicion of new technology and change returns.

<comment><USER5> I work in a corporate and am constantly amazed how much currency buzz words seem to have. We got a new CEO who is into "tech" and different directors are fighting to deploy poorly though out AI and VR projects. It has little benefit to the business but it helps people sell themselves within the company.

<comment><USER3> After working on a completely failed project in a non-software department of a large organization, I've realized that (depending where you are) the main deliverable for a lot of things is what your manager can put in his power point presentation to his manager.

Of course if the project lasts long enough the software being reasonably good starts to matter as you start finding out things you promised people don't work anymore, your huge rats nest of code makes it too hard to add features like, etc. But until it hits the "power point bottom line" a non technically educated manager will not care at all.

</comment></comment></comment><comment><USER6> Interesting to see that insurance is ahead in adoption, compared to capital markets, where I work. Suspect that the tree based models reported as popular are used for underwriting decision making. We sales & trading capital markets people tend to see insurance as the sleepy backwater of wholesale capital; just cash cow real money funds to be gouged by sharp market makers to put it bluntly.

<comment><USER7> Why is this? Insurance providers actually provide a valuable service (not necessarily implying that much of the enormous UK financial industry does not...) and is profitable and sustainable; I am surprised to hear it described as something to be "gouged". How would such a gouging occur?

<comment><USER8> It's worth noting that traditionally, actual insurance runs at a loss or break-even, most of the profits tend to come from investment.

This has changed recently, as investment returns have dropped, and may also be driving the adoption of better statistical techniques for modelling risk (as they can't rely on investment income to make up for insurance losses).

Also, having read this report, I'm very very sceptical about whether or not companies are using "ML". I suspect that most of them are just doing linear/logistic regression on larger data sizes, which isn't really the same.

<comment><USER9> If the linear / logistic regression stuff is improved through Bayesian methods and hierarchical modeling, and if it grows large enough that there is a need to use MCMC sampling techniques for approximate posterior inference, then even though the model specification itself would be very simple, I'd absolutely say this is "ML".

<comment><USER8> I really, really wouldn't. That's Andrew Gelman's gig, and he's been doing that since the 80's, when ML/AI was all about expert systems. I think of that as a statistical technique, not an ML one (and for some weird reason, many quantitative people in insurance don't like Bayesian techniques).

But then, our disagreement here just highlights the difficulties involved in getting consensus on what ml vs statistics actually are.

<comment><USER10> I think I have settled for ML = gets better with more examples ("learns"). So I'd say Bayesian models of all flavours are definitely in the ML umbrella. AI is super fuzzy though. I think of that as, does something you'd think a computer shouldn't be able to do but a human can, and is thus a forever moving goalpost.

<comment><USER8> I probably wouldn't use that definition, as all statistical techniques will improve as the number of examples increase (i.e. the estimate will be more precise). If you mean that ML normally estimates more parameters, and as such improves more with more examples, I would agree, but it's very difficult to draw a dividing line then (what about splines - loads of parameters, very flexible, but not normally classified as an ML technique).

<comment><USER11> Depending on how you look at "improvement", that's not strictly true. Often, large data sources are slightly biased relative to the population we want to generalize to (say, a single regional company's customers, trying to generalize to a national population). Working with larger and larger such data sets, that bias can become large relative to standard errors/credible interval width.

Xiao-Li Meng has had some interesting talks/papers about this and related ideas: [https://dash.harvard.edu/handle/1/10886849](https://dash.harvard.edu/handle/1/10886849)

<comment><USER8> I was never convinced by that paper, to be honest. I almost always use MLE so that's not really an issue. It just struck me as a pedantic distinction without a difference.

But if you use GEE, it's probably great to know.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> The key takeaway from this for me is that there is no shortage of machine learning practitioners in the UK despite popular opinion that it is a good career move

<comment><USER12> Where I am at the moment, a UK bank's retail data analysis service, any ML projects are set up on a "learn as you go" basis whilst still doing your main job. No specialised practitioners are deemed required.

Whether or not this is a good idea is moot: this is in the bowels of the bank and not the bleeding edge of commerce.

<comment><USER13> We don't really need specific ML practitioners for most jobs in the same way we don't need Genetic Algorithm practitioners in most jobs either, unless you're a researcher you can just read a book about it and start trying it out.

<comment><AUTHOR> Indeed. If you already understand the data and the business and you already know Python or R then you can download Keras in the morning and have something useful to the business running in the afternoon. Then you can add ML to your CV.

It's not clear where an ML expert who isn't already familiar with the data and the business fits in or adds value, or even what an expert really means in industry.

<comment><USER14> Domain knowledge is really important, you can't overstate that.

But you've also succinctly described why a huge percentage of ML as practiced in industry under performs or flat out fails. To a first approximation the person who reads a few tutorials and plays with example data sets, then sets out to apply it to their own domain, has no real idea what they are doing - and it shows.

</comment></comment></comment></comment></comment><comment><USER15> It's hard not to be cynical reading these discussions. Avoiding plastic or recycling it does near zero for the environment. Reducing travelling, reducing goods consumption, reducing construction, reusing things and protecting land are the real effective solutions but require sacrifice. Yet people refuse to go beyond performing ridiculous plastic straw theatre.

<comment><USER16> Is this comment supposed to be attached to "machine learning in uk financial services"?

<comment><USER15> No. I'm sorry. I don't know what happened.

</comment></comment></comment><comment><USER3> My frank opinion as someone ~2 years into grad school in something to do with computers and statistical inference (taken a bunch of numerical classes, ML courses, maybe ~1 year of grad level stats coursework): if you actually want to do statistical analysis for a living, you need get a statistics degree. CS people are irrelevant except for computational issues. As a CS guy probably you can be useful for scaling a system to do automated inference, or as an advanced database guy, or writing shell scripts or something.

If you go to grad school in CS and work hard in the right subfield you might develop the understanding of someone with part of a stats BS. Again, the part you would actually know something about is computation, which as far as I can tell isn't a real bottleneck for anything except very large automated systems. Otherwise some guy can just use R and his laptop.

I mentioned my status above because this is solely based on me evaluating things in an academic setting and not job experience, but this seems very clear to me once laid out like above.

ML is "hot" but it seems to me like an extraordinarily specialized area of expertise with little general applicability.

<comment><USER17> agree but -- previous eras of statistics used important and non-obvious ways to infer from subsets of data.. now, whole sets are used, and the subset part is more like a representation problem. Second, the things that are made to be important in school, are not always important outside of school. You do acknowledge that tacitly there..

The point of view in the post here is from an individual, regarding the work of another (hypothetical) individual, while the paper linked spends quite a lot of effort to characterize in the real world, real activity by large and very large, organizations, mainly business, and mainly the business of money.

You conclude that the field has "little general applicability" and that might be true in some sense.. but large organizations, and particularly large finance organizations, are not general at all, but rather composed of a very large number of specialized and repetitive operations.. exactly what ML and AI perform, no?

</comment></comment><comment><USER18> My understanding of the history of the stock market is that it was nothing short of a revolution in the sense that "normal people" suddenly could invest in large companies, and that these new types of publicly traded companies through the power of what we today would call "crowdsourcing" managed to bypass the old giants through their accountability to their shareholders and their ability to raise capital from the masses. So far, so (pretty) good.

Next, I'm reminded of when Elon Musk talked about integrated AI, and how we may one day end up in a situation where any type of intellectual or physical competition between humans, in the labor market or elsewhere, may just come down to how much money they have to spend on cybernetic, biological enhancement products.

My synthesis here is that from my perspective, the stock market has since the advent of digital trading tools, now vastly accelerated by expanded usage of machine learning in financial markets, completely undermined the original benefit of the stock market, where a pretty true democratization of the economy, almost an incredibly true one considering it was born in a capitalist society, has now been rolled back and undermined by "big capital" finding a tool that allows them to rig the game to their favor.

So if the assumption is correct that machine learning in financial markets essentially is just an interface for the already-rich to beat the never-rich by throwing more money at the problem, doesn't it stand to reason that these types of tools should be heavily regulated, if we want a stock market that has any resemblance of fairness and democratic influence on the economy?

I understand very well we are far away from this today, and I understand that rich people will not allow regulations that punish them, but if we go beyond the knee-jerk reaction of "cash rules baby", is this really the way things should be?

<comment><USER19> There are two big assumptions in your hypothesis that are incorrect in my opinion:

1\. The prevalence if machine learning in trading. 2\. That the game wasn't rigged against retail at a point in the past.

First, there is very little (almost none) sophisticated ML in trading. Sure, if you call linear regression or simple classification tools ML, then yes, there's plenty of ML in finance, and has been for decades. If you think that there are super sophisticated DNNs running the majority of trading today, you are very far from the truth. The vast majority of trading is based on fairly simple models. It happens mostly based on human insight and exploiting specific market flows. Sure, almost every firm has some token ML overlay for marketing purposes because it was the fad of the recent years. They all want the public to think that they have some super secret sophisticated AI that gives then an edge. It's all a smoke screen. Almost none of them mean it in earnest. In truth, it is incredibly difficult to build anything resembling a scalable and robust strategy with complex ML tools. The reasons are too numerous to describe here. a very small handful of firms have the capability and mandate to actually use these tools with success, but their market participation is minute, basically negligible.

Second, the game was always rigged in a sense. Big money could always buy privileged access to liquidity, information and tools. Nothing has changed here. If anything, the commoditization of all three of those things has lowered the barrier of entry. You can run a successful quant fund with a couple of friends now. That was much more difficult before.

</comment></comment><comment><USER20> Thanks for sharing ;)

Of interest as well may be London Fintech week July 4 2020

[https://www.fintechweek.com/](https://www.fintechweek.com/)

</comment><comment><USER21> Maybe before we rush to adopt machine learning in financial services we should consider the consequences of blithely giving this technology such a central position in our lives.</comment>


Setting up an L2TP over IPSec VPN server on Ubuntu, also compatible with mobiles - <AUTHOR> http://vitobotta.com/l2tp-ipsec-vpn-server/ <comment><USER2> And why isn't there a single command line script to do all this for you?

The point of course is that if we want security to be taken seriously by users - even sophisticated users who run their own Linux servers - then we must make this kind of stuff brain dead simple to setup and maintain.

And yes, of course most of us can actually write the script to do this, but the point is that we should not have to.</comment>


BleachBit "stifles investigation" of Hillary Clinton - <AUTHOR> https://www.bleachbit.org/news/bleachbit-stifles-investigation-hillary-clinton <comment><USER2> I don't this story will last long on hn.

It is very funny though.

Even if destruction of evidence in an FBI investigation should be a serious criminal offense.

<comment><USER3> Absolutely not.

I would love to see how this would be interpreted if the cronyism was not in place within the media and the executive branch.

Imagine how much uproar there would be within the SEC if you admitted to deleting all of the evidence, you would get Madoff type cover-up coupled with Mylan EpiPen disgust.</comment></comment>


Amazon receives FCC approval for Project Kuiper satellite constellation - <AUTHOR> https://blog.aboutamazon.com/company-news/amazon-receives-fcc-approval-for-project-kuiper-satellite-constellation <comment><AUTHOR> Boom. Direct competition with SpaceX's Starlink. I'd bet Google will do something about it really, really soon.</comment>


Estonia will block the certificates of 760 000 ID cards - <AUTHOR> http://id.ee/index.php?id=30519&read=38341 <comment><USER2> They tell people to update their certificates, but their servers have been constantly overloaded.

I've tried dozens of times over the last three days, zo no avail.</comment>


Should I buy ECC memory? (2015) - <AUTHOR> https://danluu.com/why-ecc/ <comment><USER5> While I was at Google, someone asked one of the very early Googlers (I think it was Craig Silverstein, but it may've been Jeff Dean) what was the biggest mistake in their Google career, and they said "Not using ECC memory on early servers." If you look through the source code & postmortems from that era of Google, there are all sorts of nasty hacks and system design constraints that arose from the fact that you couldn't trust the bits that your RAM gave back to you.

It saved a few bucks in a time period where Google's hardware costs were rising rapidly, but the ripple-on effects on system design cost much more than that in lost engineer time. Data integrity is one engineering constraint that should be pushed as low down in the stack as is reasonably possible, because as you get higher up the stack, the potential causes of corrupted data multiple exponentially.

<comment><USER6> Google had done extensive studies[1]. There is roughly 3% chance of error in RAM per DIMM per year. That doesn't justify buying ECC if you have just one personal computer to worry about. However if you are in data center with 100K machines each with 8 DIMM, you are looking at about 6K machines experiencing RAM errors _each day_. Now if data is being replicated then these errors can propogate corrupted data in unpredictable unexplainable way even when there are no bugs in your code! For example, you might encounter your logs containing bad line items which gets aggregated in to report showing bizarre numbers because 0x1 turned in to 0x10000001. You can imagine that debugging this happening every day would be huge nightmare and developers would end up eventually inserting lot of asserts for data consistency all over the places. So ECC becomes important if you have distributed large scale system.

1: [http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf](http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf)

<comment><USER7> > There is roughly 3% chance of error in RAM per DIMM per year. That doesn't > justify buying ECC if you have just one personal computer to worry about.

How do you make that leap?

<comment><USER8> It's an inappropriate leap. Consumers should have ECC memory too.

However the consumer market has long decided to settle for ECC nowhere and cheap everywhere.

ECC hardware comes at premium option that can easily be +100%. You need support in the memory, the motherboard and the CPU.

Given the price difference, personal computers will have to live with the memory errors. People will not pay double for their computers. Manufacturers will not sacrifice their margin while they can segment the market and make a ton of money off ECC.

<comment><USER9> Amd has modestly priced hardware that supports ecc

<comment><USER10> Was that the case before Ryzen? I know their new CPUs support ECC, but I'm not sure for earlier generations.

<comment><USER11> I think it was common for AM3 for example too.

<comment><USER12> ECC is officially supported by all AM2/3(+) CPUs and AFAIK all corresponding motherboards from ASUS. As in, you have it guaranteed on the spec sheet.

There are also reports of BIOS support in some boards which don't have ECC advertised. And you can try to enable it in the OS even without BIOS support, though some level of _hardware_ support is still necessary. As Linux documentation puts it: "may cause unknown side effects" :)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Can people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actually one of the few that can still be safer if you don't run with ECC memory. Source: Matthew Ahrens himself: [https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...](https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271)

<comment><USER14> Indeed. It's _true_ that the data _may_ be corrupted before hitting any disk[1], but once it _has_ hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from.

The main point of e.g. ZFS or Btrfs checksumming is that a) _at least it isn 't getting worse_, and b) I can _tell_ if it's getting worse.

[1] ... but if the bits are not generated by the machine that actually saving to disk, how do you know they weren't corrupted along the way? The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.

<comment><USER15> > The number of people who religiously check PGP signatures/SHA256sums or > whatever is miniscule.

• If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed.

• Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property.

• Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.)

• Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting _them_ (though not _the page itself_ ) from errors.

• ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on _installer_ ISOs embed their own checksums and often signatures.

• git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread.

Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.

<comment><USER16> Both macOS/iOS and Windows use code signing for executables, which should guard against most types of corruption.

Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage.

PNG has built-in checksums, though other image formats don't (JPEG). Not sure about video.

<comment><USER15> > Web pages (and anything else) transmitted over HTTPS are protected from > corruption in transit by TLS's hashing (which is vastly stronger than the > checksums at lower levels of the network stack), though that doesn't help if > the server has faulty memory or storage.

I didn't bring this one (or any other transport-level checksums) up, because we were talking about whether you can trust something "across the whole process"—from its origin developer's disk (where it might get an initial explicit checksum generated), to origin memory, across the network to a server's memory, to that server's disk, over the network again to a CDN reverse-proxy's memory, maybe its disk, then the network again to you, then _your_ memory, _your_ disk, and finally your memory again as you verify it. Oh, and a bunch of routers and switches in between, of course.

Static checksums that are baked into file formats or manifest files protect the file across that _whole_ chain. Transport-level checksums only ensure that the one part they're involved in happened correctly.

</comment></comment></comment></comment></comment><comment><USER17> Altitude also plays a factor in random memory corruption.

From the wikipedia article on ECC Ram, "Hence, the error rates increase rapidly with rising altitude; for example, compared to the sea level, the rate of neutron flux is 3.5 times higher at 1.5 km and 300 times higher at 10–12 km (the cruising altitude of commercial airplanes).[3] As a result, systems operating at high altitudes require special provision for reliability."

<comment><USER18> I wonder if this has anything to do with Microsoft's plans to build an underwater data center.

<comment><USER19> If I remember correctly, that research venture was mostly due to the potential of easy heat exchange and "free" energy via geothermal/tidal. Now that you mention this, though, it's clear that such a datacenter would also be naturally shielded from many things!

</comment></comment></comment><comment><USER20> I reproduced this by bit-squatting cloudfront.net after reading about it. So many memory errors!

[http://dinaburg.org/bitsquatting.html](http://dinaburg.org/bitsquatting.html)

Loved the variety as well. Sometimes though requests came to me the Host header was correct!

<comment><USER21> Wait so when someone typoes cnn.com as con.com, that is ipso facto a memory error? I guess I could see that if the characters are radically far apart on the keyboard? But doesn't a simpler explanation like "one person out of billions with Internet access typed the wrong thing" seem a lot more likely?

<comment><USER22> Domains that are only used as CDNs, like cloudfront.com, are almost never typed into an address bar. Errors in the domain name are more frequently the result of a bit-flip error.

<comment><USER20> Also, these typos are typically not easy ones since flipping a bit changes the letter in ways that are unlikely typos. With cloudfront.net an negligible number of people would be typing them at all. Close to 100% of the errors that I saw were loading either images, css or javascript files that some other page depended on.

</comment></comment></comment></comment><comment><USER23> Yes. Everybody reading this should use ECC RAM, and non-ECC RAM should be called "error-propagating RAM".

Random bit flips aren't cool, and they happen regularly. Most computers that have ECC RAM can report whether errors happen. I see them at least once a year or so. For instance, here are 2 ECC-correctable memory errors that occurred just last month.

Cosmic rays? Fukushima phantom? Who knows. You'll never know why they happen (unless it's like a bad RAM module and they happen a lot), but if you don't rock ECC you will never know they happened at all. You'll be left guessing when, years later, some encrypted file can no longer decrypt, and all the backups show the same corruption...

[1]: [https://www.dropbox.com/s/zndvy3nkv1jipri/2017-03-20%20FUCK%...](https://www.dropbox.com/s/zndvy3nkv1jipri/2017-03-20%20FUCK%20memory%20errors.png?dl=0)

[2]: [https://www.dropbox.com/s/6yeoedc7ajzq4u9/2017-03-20%20FUCK%...](https://www.dropbox.com/s/6yeoedc7ajzq4u9/2017-03-20%20FUCK%20memory%20errors%20detail.png?dl=0)

<comment><USER24> I remember the one time I bought ECC memory, for a PII-400. It was only 512MB or so I think, but in the 12 years that server ran I saw a grand total of 1 corrected error in the logs. Given how much of a premium that ECC memory was it felt like a waste.

</comment></comment><comment><USER25> An old article from DJB worth perusal: [http://cr.yp.to/hardware/ecc.html](http://cr.yp.to/hardware/ecc.html)

It's also worth noting that not all ECC (SECDED) is created equal: ChipKill™ and similar might not survive physical damage because of likely shorts of the data bus but a single malfunctioning chip producing/experiencing higher hard error rate is possible from which to recover.

Also, it'd be really cool if some shop a-la BackBlaze blogged about large- scale monitoring for soft and hard RAM errors across chip/module modules (+ motherboards & CPUs). Without collecting and revealing years data from real use, conversation devolves into opinion and conjecture.

Finally, not all use-cases can benefit from ECC (ie Angry Birds) however there are some obvious/nonobvious ones that can (ie router non-ECC DNS bitsquatting or processing bank transactions).

<comment><USER25> PS: Random-crazy thought.. it's curious with reduction of costs via Moore's law improvements that there aren't yet formally-verified, zero-knowlege systems which can end-to-end prove they performed computation/real-world side- effects and/or continue to safely store data. Why blindly trust anyone or any company with data that can be seized, lost or misused when distributed computation, communication and storage can be A2E with only limited participants knowing operations / plaintext? Perhaps: homomorphic encryption, blockchain-similar ledger or proof-of-work and periodic, authenticated hash challenge queries. Mix in relaying and other idle phony traffic to make triangulation more difficult. I think in order to assure sufficient distributed system resources are made available, μpayments a-la AWS but just covering costs would make it possible to have a persistent, anonymous computation and storage collective that would survive outages, FBI raids, single nodes going offline, etc.

<comment><USER26> Storage, [yes]([https://storj.io/](https://storj.io/)). Computation ... sure if you don't mind the server viewing the contents of your computation and can verify the results. Sadly, fully homomorphic systems incur waaaay too much overhead so you are constrained in what you can do (i.e. specialized DBs, zkSNARKs, etc).

Then, of course, there is the problem of network latency and bandwidth costs vs just keeping it all on one datacenter.

</comment></comment></comment><comment><USER27> A better question is why /shouldn't/ you use ECC memory?

Generally the answer to this is any context where you legitimately do NOT care about your data at all, but you still care about costs. This predominately devolves in to consumption only gaming systems.

In all other cases everyone would be better served (in the long run) by buying ECC RAM.

<comment><USER17> a common network topology is to have a load balancer distribute load to a number of cheap Http servers which internally connect to a centralized and powerful database server. In this case only the database server really needs ECC ram. The system is designed to be fault tolerant for any individual HTTP server node so the increased cost vs the problem it solves doesn't make sense.

I guess you could argue that a random bit flip could somehow make the HTTP server vulnerable and able to compromise the network however that risk is impossibly small. If we take IBMs estimation that a bit flip occurs at an approximate rate of (3.7 × 10-9) bytes/month and then divide it by the number of bytes in the system you can see that the odds of randomly corrupting a byte in memory that triggers a vulnerability is too small.

<comment><USER28> What about memory-error corrupted application data (or application logic) where the corruption occurred on load balancers or web application servers? There's more to data integrity than security holes.

<comment><USER17> If you write code that detects stack smashing and illegal dereferences then you can terminate the webservice and either have a watchdog restart it or if it crashes multiple times have it taken out of service by the load balancer. There are plenty of ways to handle hardware errors without throwing out the hardware and getting "better" hardware. Technically, You could have a faulty component somewhere between the Ram and CPU and then what is your expensive ram going to do? What if the CPU cache has errors? For many small businesses often the difference between success and failure is their ability to make things work without throwing cash at the problem.

<comment><USER28> Even with your proposed checks there remains a high probability to just get silent application data corruption, not crashes.

Regarding faulty components, that is one part of ECC's job, but the other part is correcting the regular bit flips that happen with nominally operating DRAM.

Flagging faulty components is more useful than you propose. There are not that many places where this corruption can occur, so being able to rule out RAM is very useful. The example you used, CPU caches, is actually already covered by ECC in most CPUs, including reasonably recent x86/amd64.

The tradeoff would be more worthy of thought if ECC was much more expensive

<comment><USER17> ECC ram is not a raid. If corrosion on a trace causes a bit flip from an adjacent line then the ram will recieve rhe corrupt data as valid. There is no parity ram stick to recover from. I never said ECC ram doesnt have a purpose. Im saying you are wasting your money if you think it's essential to running a web sever. Lets be real here, like 80% of computers on the internet stream porn. They dont need ecc ram

</comment></comment></comment></comment></comment></comment><comment><USER29> This article is gold in so many ways. It contains interesting bits of information on ECC, company history that I didn't know (Sun's and Google's namely), filesystem reliability (I never knew!), the physics of RAM (50 electrons per capacitor)...

It's a must read, even if only to get you thinking about some of these things.

</comment><comment><USER30> Depends on what you are doing. ZFS storage servers: Hell yes High-value data in my DB? Hell yes email server: Nope super cool gaming rig: Nope * Cluster: Hell yes

General office workstation: maybe.

I don't have the budget for 20 redundant copies. I do have the budget for slightly more expensive RAM. Especially on my ZFS storage arrays.

ECC memory is like Insurance. You hope you never need it. One real downside that I have found, is finding out _when_ that memory correction has saved your ass. RAID arrays can alert you when a disk is dead. SMART mostly tells you when disks are failing. I haven't found a reliable tool to notify me when I am getting ECC errors/corrections.

<comment><USER31> There is a hidden cost of ECC with regards to the chipset. None of the cheap chipsets support it, so on any home build, it's going to be expensive.

<comment><USER32> Not true with Ryzen, as long as you find unregistered ECC acceptable.

Somewhat not true with Intel, as some of the lower end Xeons now support it.

<comment><USER33> Ryzen ECC support is a mess, no AM4 motherboard currently on the market has implemented ECC support fully and properly (not even Asrock). It's better than nothing but you would be a fool to rely on it.

[http://www.hardwarecanucks.com/forum/hardware-canucks- review...](http://www.hardwarecanucks.com/forum/hardware-canucks- reviews/75030-ecc-memory-amds-ryzen-deep-dive-5.html)

"Kinda sorta works but the manufacturer won't stand behind it" is bunch of bullshit. If your data is worth using ECC in the first place - it's worth using a platform that has fully-implemented support, that has passed validation, that you know is going to work properly when you need it.

Until that happens - this is an application where Ryzen is simply not appropriate.

All of the modern i3s and Pentiums support ECC, but you do need the server chipset instead of the cheap consumer stuff. Good news though - <USER3> "expensive server boards" are roughly the same price as say, an AM4 motherboard with an X370 chipset.

Heck, you can buy a basic off-lease ThinkServer TS140 for only about $300. You'll only have about 4 GB of RAM but it's a shell to start building out (which is cheaper than having an OEM assemble it for you anyway).

<comment><USER32> _Ryzen ECC support is a mess, no AM4 motherboard currently on the market has implemented ECC support fully and properly (not even Asrock). It 's better than nothing but you would be a fool to rely on it._

Ryzen motherboard support is what is agreeably a "mess", not the processor itself, but at least it's functional on ASRock and select Gigabyte boards. As for "a fool to rely on it", not sure what you mean by that. The error correction itself is done by the hardware. Other than calling the initialization routines and providing logging/halt, the BIOS/UEFI isn't responsible for anything afaik.

I'm well aware that this isn't the full grade of ECC support offered by higher-end Xeons and chipset combos, but it's better than nothing and it's affordable.

Also, no offense, but I'm not going to rely on hardwarecanucks as an authority on this subject.

 _All of the modern i3s and Pentiums support ECC, but you do need the server chipset instead of the cheap consumer stuff. Good news though - <USER3> "expensive server boards" are roughly the same price as say, an AM4 motherboard with an X370 chipset._

The goal isn't ECC alone, at least not for me, the goal is an 8-core system with good single-threaded performance and ECC at a reasonable price. As far as I know, only Ryzen offers that.

So for me, I'm looking at the possibility of getting a single system that can give me decent gaming performance, good development performance, ECC support, and more, all at a price that leaves me with money for other components.

<comment><USER33> > Also, no offense, but I'm not going to rely on hardwarecanucks as an > authority on this subject.

Fine then. AMD says it's unvalidated and unsupported, is that good enough for you?

> I'm well aware that this isn't the full grade of ECC support offered by > higher-end Xeons and chipset combos, but it's better than nothing and it's > affordable.

So would you be OK with running Xeon engineering samples then? After all - they certainly pass the same "best effort" test. Personally since these are _server_ ES hardware - I'd tend to trust it more than consumer hardware like Ryzen, especially given their comparative age/maturity.

I just picked up a 10-core Haswell Xeon engineering sample for $140 last week. 40% more multi-threaded performance than a Ryzen 1700. The X99 mobo I picked up from Microcenter for $60 doesn't have ECC support but a bunch of them do.

Or if you want something that's official and you know works, there are surplus Sandy Bridge Xeons very cheap nowadays. A decent bit more multithreaded performance than a Ryzen 1700 - but you'll be giving up single-threaded performance. [http://natex.us/intel-s2600cp2j-motherboard- dual-e5-2670-sr0...](http://natex.us/intel-s2600cp2j-motherboard- dual-e5-2670-sr0kx/)

Or really - a full retail E5-2630 v3 is under $500 now on eBay. That's not really that bad if you just have to have everything in one box.

> So for me, I'm looking at the possibility of getting a single system that > can give me decent gaming performance, good development performance, ECC > support, and more, all at a price that leaves me with money for other > components.

What it comes down to: if you want everything in one box then be prepared to shell out. Everyone has this market segmented out, _including AMD_ (after all they won't stand behind Ryzen's ECC either). If you feel you need ECC, that's really not a valid solution.

If a Xeon doesn't cut it for you - sounds like you might be in the market for two boxes here. A server/workstation with ECC and good multi-thread performance, and a gaming machine that you can overclock and get the best single-thread performance out of.

(Also - in general, overclocking also seems kind of counterproductive to the aims to running ECC RAM - although I guess I haven't looked into that.)

<comment><USER32> _Fine then. AMD says it 's unvalidated and unsupported, is that good enough for you?_

No, that's not what AMD said, they said it isn't validated by motherboard partners. The functionality is there, it's up to their partners to use it.

 _So would you be OK with running Xeon engineering samples then? After all - they certainly pass the same "best effort" test. Personally since these are server ES hardware - I'd tend to trust it more than consumer hardware like Ryzen, especially given their comparative age/maturity._

That's not even a remotely accurate comparison.

 _What it comes down to: if you want everything in one box then be prepared to shell out. Everyone has this market segmented out, including AMD (after all they won 't stand behind Ryzen's ECC either). If you feel you need ECC, that's really not a valid solution._

Sorry, but so far all of your proposed "solutions" are summed up as: "If you give up significant performance, functionality, buy second-hand, or completely ignore official support statements, X competitor is the better deal!"

 _If a Xeon doesn 't cut it for you - sounds like you might be in the market for two boxes here. A server/workstation with ECC and good multi-thread performance, and a gaming machine that you can overclock and get the best single-thread performance out of._

No, the goal is to have one system, and at this point, Ryzen looks like the best option. If a competitor decides to release something equivalent, I'll consider them too.

</comment></comment></comment></comment></comment></comment></comment><comment><USER34> Yes.

Bit errors are uncommon and range from benign to crash.

Your storage has them, memory has them, network has them.

Non error correcting memory very significantly increases risk.

And this is the kind of risk you don't notice, until you do and when you do, it's often subtle, insidious, impossible to track down.

Servers absolutely. It's debatable on desktop, but we have huge RAM now. Might as well error correct. The bit error risk is small. Bigger RAM only adds to that possibility.

<comment><USER35> I won't bother on a desktop. I've been using 4 machines for the last 17 years with storage varying from 10GB to 2TB and RAM varying from 128MB to 16GB and haven't personally seen any kind of data corruption in motion (or at rest for that matter). Only had 2 mechanical drives fail (though predictably).

ECC is costly. The memory modules itself and the board required to support it properly.

<comment><USER36> The only reason ECC is costly is because Intel has a monopoly on the desktop/server chip market and they refuse to deploy ECC to consumer chips. The hardware is there, it's just fused disabled.

If ECC were only the cost premium and we assumed a linear relationship then it should cost about 1/8 more than non-ECC DRAM. Unfortunately Intel's decisions have knock-on effects that ripple through the rest of the market.

IIRC I saw somewhere that JDEC expects a future standard will require ECC to get acceptable error rates for all memory. At that point Intel won't have any choice.

</comment></comment></comment><comment><USER37> The article makes no mention of single event upsets (SEUs). These occur randomly when cosmic rays can cause a bit flip anywhere in the chip. ECC is a good way to mitigate SEU effects.

<comment><USER38> Sorry for nitpicking, but it's not the cosmic rays, it's the cosmic rays secondaries cascade shower (produced high up in the atmosphere when a cosmic ray interacts with a particle there).

</comment></comment><comment><USER39> I am typing this (finally!) on my new desktop build. I did mull over the decision for a while but finally went with Xeon and ECC. So the memory cost more - perhaps even twice as much - so what? I use my computer pretty heavily for my work - with several VMs running at a time. If ECC saves me a headache once a year, it will have paid for itself. If it never provides ANY benefit I will still not regret the peace of mind.

<comment><USER40> The parameters of the desktop ECC decision have changed massively with today's glacial replacement cycles. Today you make a one time payment for many years of avoided headaches and peace of mind, whereas back then any sign of unreliability would have been a welcome excuse for a cheap upgrade.

</comment></comment><comment><USER41> No-one's mentioned it yet, but we're in a post-Rowhammer world and ISTM this is relevant to the discussion: while not all non-ECC DIMMs are susceptible, the cheaper ranges generally are, and if your purchasing decisions are driven by hardware cost, that's probably what you'll end up with. Corruption due to malice is a rather different beast to corruption due to random cosmic rays...

</comment><comment><USER42> Rehashing an old comment:

IEC 61508 documents an estimate of 700 to 1200 fit/MBit (fit = "failure in time"; per 10e-9 hours of operation) and gives the following sources:

a) Altitude SEE Test European Platform (ASTEP) and First Results in CMOS 130 nm SRAM. J-L. Autran, P. Roche, C. Sudre et al. Nuclear Science, IEEE Transactions on Volume 54, Issue 4, Aug. 2007 Page(s):1002 - <USER4>

b) Radiation-Induced Soft Errors in Advanced Semiconductor Technologies, Robert C. Baumann, Fellow, IEEE, IEEE TRANSACTIONS ON DEVICE AND MATERIALS RELIABILITY, VOL. 5, NO. 3, SEPTEMBER 2005

c) Soft errors' impact on system reliability, Ritesh Mastipuram and Edwin C Wee, Cypress Semiconductor, 2004

d) Trends And Challenges In VLSI Circuit Reliability, C. Costantinescu, Intel, 2003, IEEE Computer Society

e) Basic mechanisms and modeling of single-event upset in digital microelectronics, P. E. Dodd and L. W. Massengill, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 583–602, Jun. 2003.

f) Destructive single-event effects in semiconductor devices and ICs, F. W. Sexton, IEEE Trans. Nucl. Sci., vol. 50, no. 3, pp. 603–621, Jun. 2003.

g) Coming Challenges in Microarchitecture and Architecture, Ronen, Mendelson, Proceedings of the IEEE, Volume 89, Issue 3, Mar 2001 Page(s):325 – 340

h) Scaling and Technology Issues for Soft Error Rates, A Johnston, 4th Annual Research Conference on Reliability Stanford University, October 2000

i) International Technology Roadmap for Semiconductors (ITRS), several papers.

If that's correct, the math is simple: you have bit flips in your PC about once a day.

It's just that (a) you often won't notice those transient errors (one pixel in your multi-megapixel photo is one bit off) and (b) a lot of your RAM is probably unused.

</comment><comment><USER43> Same topic, same conclusion, even more hard facts.

[http://perspectives.mvdirona.com/2009/10/you-really-do- need-...](http://perspectives.mvdirona.com/2009/10/you-really-do-need-ecc- memory/)

</comment><comment><USER44> In the late nineties, the Intel desktop chipsets such as 440LX and 440BX offered ECC functionality, all you had to do was spend ten or fifteen bucks extra on the memory. Great hardware.

I'm unhappy that Intel made things more expensive and complicated with their market differentiation, but from their POV it was logical. PC users were screwing up the reliability of their systems in so many ways via overclocking, and were habituated to accept crappy reliability via pre-NT Windows. PC users could have demanded ECC and they didn't. I'm sure that even when the chipsets made it easy, only a tiny fraction bothered to use ECC.

</comment><comment><USER45> For servers this is more or less a no brainer: it's not a huge extra cost and a failure will cost you more than the extra cost.

For a regular desktop system for personal use it's not so easy. The data volumes are much smaller, the temperature environments are usually better, they aren't running (other than maybe idling) 24/7, most of the stuff that is in ram isn't going to be mission critical (i.e. you don't have 32Gb of RAM filled with customer database records, you have it filled with read only FPS textures, compiler caches etc).

Unlike a business that has tons of data that is mutated, my data is mostly immutable such as photos etc. It's not a continuously changing dataset where a bit flip in memory is likely to find its way into my data and then into my backups which would be the case e.g. for databases or big creative work (movie editing etc).

</comment><comment><USER46> I've spent the last two weeks looking at Memtest86+ trying to figure out if either one of my memory modules is damaged, or if it is the motherboard. These tests take a long time, and yield different results from day to day.

I've decided to never ever again buy non-ECC memory, at least not on 24/7 servers as well as on workstations.

In a gaming machine / visual typewriter? Sure, non-ECC memory is ok.

</comment><comment><USER47> I think that, given the personal importance of computing devices and storage, no filesystem should exist w/o checksum of metadata+data, and no RAM should be without ECC. The slight increase in cost does not justify the risk.

</comment><comment><USER48> I searched for a good Linux laptop recently with ecc but didn't find much so settled on a kaby lake i5. Does anyone make them?

<comment><USER49> For example Lenovo P51 seems to support ECC (if equipped with Xeon processor). About the Linux support I don't know, but I've understood at least some other Lenovo models work ok with Linux.

[http://psref.lenovo.com/Product/ThinkPad_P51](http://psref.lenovo.com/Product/ThinkPad_P51)

</comment></comment><comment><USER50> If you can afford it, sure. That's one reason why I'm so happy Ryzen supports it on consumer processors: It makes ECC cheap.

</comment><comment><USER51> What are the odds of memory errors causing hard disk corruption / boot failure?

</comment><comment><USER52> Yes. Everyone does.

</comment><comment><USER53> some1 already mentioned row hammer so ecc yes :)

</comment><comment><USER54> Yes. Are we done :)

</comment><comment><USER55> "<pubDate>Fri, 27 Nov 2015 00:00:00 +0000</pubDate>"

Needs (2015) added to the Title I think.

<comment><USER56> Thanks! Updated.

</comment></comment><comment><USER57> I want to thank Jeff for assisting Dan in writing this article.

</comment><comment><USER58> Relevant post by Jeff Atwood: [https://blog.codinghorror.com/to-ecc-or-not-to- ecc/](https://blog.codinghorror.com/to-ecc-or-not-to-ecc/)

<comment><USER59> That post is linked in the _first sentence_ of the submission.

</comment></comment><comment><USER60> Do you use ZFS? If yes then you should use ECC memory.

Do you have a use case where you would want your computer to alert you when the ram is failing? If yes then you should use ECC memory.

Otherwise it's a nicitey and probably not worth the money.

<comment><USER61> Do you use ZFS? If no then you should use ECC memory.

Now the half truth becomes full-truth.

<comment><USER62> I don't get this association between ZFS and ECC. The recommendation to use ECC with ZFS basically comes down to "all that fancy data integrity checking that ZFS does won't protect you from memory errors, so you'll effectively lose that feature."

Are you OK with silent data corruption? If so, don't bother with ECC. If not, use it.

<comment><USER63> History. The ZFS folks, back when, were the only folks making much noise about the association between non-ECC RAM and corrupt data landing on disk.

The truth is, if you care about the notion that your disk should return the same data that software thought it was writing, you should use ECC with any file system. But The ZFS folks made noise about the issue, I think lots of people assumed the reason was that there was something special about ZFS that needed it, and now you have something sort of like an urban legend.

<comment><USER64> > should return the same data that software thought it was writing

Hint: In an OS using a page cache (= _every OS_ ) I/O errors are not reliably propagated to applications unless they explicitly sync their dirty pages.

<comment><USER63> I'm aware of that, but I'm not sure what I'm supposed to take away from it in this context.

<comment><USER64> That it's difficult to accurately define "what the application thought it wrote" when considering corruption at various abstraction layers; somewhat similar to calculating checksums over already corrupted data.</comment></comment></comment></comment></comment></comment></comment>


Raspberry Pi enigmatic announcement - <AUTHOR> https://twitter.com/Raspberry_Pi/status/908691250802565121 <comment><AUTHOR> The gif doesn't give much information, I analyse each frame, and it is this image, distorted (no hidden content) [http://hpics.li/17e2301](http://hpics.li/17e2301)

I tried to do an image search and it seems to be Original Content.

So what do you think , it is going to be a new Raspberry Pi?</comment>


The Himalayan tea train that's running out of steam - <AUTHOR> http://www.bbc.com/news/magazine-32995131 <comment><USER3> Railroads of this era (of any gauge), worked because at the time labor was so cheap, every 50-100 miles you could have people unload the cars and put them in another freight car, even a different gauge.

Then people around the 1860's had the great idea that you could leave the items inside the freight car, and just hook it onto the other train if the gauge, couplers, and brakes all followed the same rules.

Steam Locomotives have an odd attraction in that they were near the start of the industrial revolution, and have a visceral external motion that can be followed.

How many keep operating on tourist lines as the number of people remember them remains to be seen.

Even in the United States, while ridership remains high of tourist lines, fewer and fewer people have the skills to maintain a steam engine. Engines were once ordered out of a catalog, and parts were only a telegraph order away, but today it takes peculiar machinist skill to forge what were once mass produced castings.

<comment><USER4> Narrow gauge wasn't just gratuitous incompatibility, it was for areas where there wasn't room for the normal gauge, like mines and mountains.

Steam trains have a visceral appeal which has worked well as a tourist attraction for years. The number of people who remember them in operation has to be small; the last UK scheduled steam service was 47 years ago. So they've found a new audience.

I'm optimistic about the maintenance given the emergence of "maker" culture. While it's a very different prospect to desktop CNC and does require more skill, it's not a magic lost art. There has even been a crowdfunded new locomotive built, "Tornado": [http://www.a1steam.com/index.php?option=com_content&view=art...](http://www.a1steam.com/index.php?option=com_content&view=article&id=45&Itemid=54)

Edit: this has led me to finding that a large fraction of the UK's preserved steam locomotives were kept more or less by accident, as they were less profitable to scrap than wagons and were therefore put at the back of the yard: [http://en.wikipedia.org/wiki/Woodham_Brothers](http://en.wikipedia.org/wiki/Woodham_Brothers)

</comment></comment><comment><USER5> ::shakes Magic 8-ball:: Outlook not so good.

A _nearly identical_ situation has been unfolding over the past 10 years for a certain Taiwanese tea train.[1]

Constructed in 1906, the Alishan Forest Railway has so many similarities it looks like a joke. Narrow ~2 ft gauge, ~87km in length, 2,000+ meters in vertical elevation gain serving high mountain tea farms, 6%+ grades, ~5 Z switchbacks, Class B Steam for special occasions, Diesels for regular days, in the running for World Heritage sites, and in terrible financial straights after a long fight to make it go private[2].

The line was government run for most of it's life, privatized in 2008, and returned to government control in 2011. It makes $2 million USD in revenue each year but costs $6.5 million to run and needs another $30 million in repairs to keep it going[3] (due to a landslide as well). That said, if you get a chance, go on a Wednesday. Shay's are fantastic and we're down to 5 of these kinds of lines throughout the world.

I am really routing for Mr. Banerjee here but the odds seem stacked against him. Ultimately it was the installation of a two-lane highway with a plethora of shuttle services that made the Taiwan high mountain rail obsolete. If he can command a monopoly on the road as well, of course he will come out ahead. But from the sound of the taxi business in Ghum it looks like that battle is already lost.

[1] [http://en.wikipedia.org/wiki/Alishan_Forest_Railway](http://en.wikipedia.org/wiki/Alishan_Forest_Railway)

[2] [http://www.taipeitimes.com/News/taiwan/archives/2011/05/01/2...](http://www.taipeitimes.com/News/taiwan/archives/2011/05/01/2003502140)

[3] [http://www.taipeitimes.com/News/taiwan/archives/2011/05/01/2...](http://www.taipeitimes.com/News/taiwan/archives/2011/05/01/2003502140)

<comment><USER6> Gauge is less than 1 meter.

<comment><USER5> Thank you, edited to say "ft" not meters. Alishan is 2 feet, 6 inches. Darjeeling is 2 feet.

Both will rock you side to side like an armored car at a G7 protest.

</comment></comment></comment><comment><USER7> The article fails to mention that the DHR is a UNESCO World Heritage Site - only one of two railways so honored.

The DHR has actually tried to replace the Class Bs a few times since they were introduced. The original Class As were too flimsy for the job - and the Garret they ordered after the Class Bs was far too much of a locomotive for them - <USER2> the Class Bs remain, 100 years after being introduced.

I wonder what if anything I build tech wise will survive 10 years - let alone 10 times that.

</comment><comment><USER8> I have spent a lot of time in Darjeeling on tea and the said train. Indian Railways is completely different beast. People might have hear a lot of e-commerce boom in India but Indian Railways online ticket booking system remains THE largest e-commerce system in India. Also their website is probably the shittiest e-commerce platform every built with less than 1% of transactions actually become successful.

Check out what ails indian railways in this 10 part series by one of the latest consultant hired to turn around IR.[1]

[1] [http://swarajyamag.com/columns/railways-choice-and- competiti...](http://swarajyamag.com/columns/railways-choice-and- competition-i/)

<comment><USER9> _Also their website is probably the shittiest e-commerce platform every built with less than 1% of transactions actually become successful._

Is there a source for that percentage? I've found [http://www.medianama.com/2011/09/223-irctc- reports-13-2m-tra...](http://www.medianama.com/2011/09/223-irctc- reports-13-2m-transactions-for-august-2011-1-73-down/) that estimates 75% success on the banking side, but presumably that doesn't count failures in UX, validation, etc. I'd be really interested in reading more.

</comment></comment><comment><USER10> @OP Why is this post on HN? Please stop posting just to promote your agenda. This is not Reddit. Unfortunately I feel visiting HN is now lot of waste of time. I wish somebody from YC or HN does something about this soon.

<comment><AUTHOR> No agenda. Why would I have one?

<comment><USER11> Well, other than sharing interesting stories, right?

<comment><AUTHOR> Wait! That's the one. I had forgotten.</comment></comment></comment></comment>