UC Berkeley chooses Google apps over office 365 based on this analysis - <AUTHOR> http://technology.berkeley.edu/productivity-suite/google/matrix.html <comment><USER2> The report says this:

Limitation of Liability:

"Google does not limit their liability in regards to its confidentiality obligations and includes Customer Data in definition of Confidential Information. Google is responsible for any actions of its employees and agents."

"Microsoft caps its liability for "free" services at $5K including any damages related to Institution Data short of Gross Negligence or Willful Misconduct."

Google is far superior.

But <http://www.google.com/apps/intl/en/terms/user_terms.html>

says this: "YOU EXPRESSLY UNDERSTAND AND AGREE THAT GOOGLE AND PARTNERS SHALL NOT BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR EXEMPLARY DAMAGES, INCLUDING BUT NOT LIMITED TO ..."

Am I looking at the wrong TOS?

<comment><USER3> Yes, I think you're looking at the wrong terms of service; you're looking at their ordinary end user terms of service, not their educational terms of service. The educational terms of service, [http://www.google.com/apps/intl/en/terms/education_terms.htm...](http://www.google.com/apps/intl/en/terms/education_terms.html) say:

> 12.2 Limitation on Amount of Liability. NEITHER PARTY MAY BE HELD LIABLE > UNDER THIS AGREEMENT FOR MORE THAN THE AMOUNT PAID BY CUSTOMER TO GOOGLE > DURING THE TWELVE MONTHS PRIOR TO THE EVENT GIVING RISE TO LIABILITY. > 12.3 > Exceptions to Limitations. These limitations of liability apply to the > fullest extent permitted by applicable law, but do not apply to breaches of > confidentiality obligations, violations of a party's Intellectual Property > Rights by the other party, or indemnification obligations.

So, they put a cap on liability of how much they've been paid in the past year (which in most cases for their educational program is nothing), but do not limit liability for breaches of confidentiality obligations.

<comment><USER2> I wonder if UC Berkeley is confusing "confidential obligations" with "confidential data".

It doesn't seem like google is saying, "if we lose an email, and it costs you a million dollars, we're liable for it".

From this section: "Confidential Information.

5.1 Obligations. Each party will: (a) protect the other party's Confidential Information with the same standard of care it uses to protect its own Confidential Information; and (b) not disclose the Confidential Information, except to Affiliates, employees and agents who need to know it and who have agreed in writing to keep it confidential. Each party (and any Affiliates, employees and agents to whom it has disclosed Confidential Information) may use Confidential Information only to exercise rights and fulfill its obligations under this Agreement, while using reasonable care to protect it. Each party is responsible for any actions of its Affiliates, employees and agents in violation of this Section.

5.2 Exceptions. Confidential Information does not include information that: (a) the recipient of the Confidential Information already knew; (b) becomes public through no fault of the recipient; (c) was independently developed by the recipient; or (d) was rightfully given to the recipient by another party. "

It sounds like what Google is really saying is, "If we make some kind of secret deal with your organization, you better protect that secrecy as well as you protect all of your confidential information, or there is no limit to how much we can sue you for the breach."

<comment><USER4> But see the definitions: "Confidential Information" means information disclosed by a party to the other party under this Agreement that is marked as confidential or would normally be considered confidential under the circumstances. Customer Data is Customer's Confidential Information.

<comment><USER5> Does anyone know what this switch implies for Patriot Act information requests? I had heard that the UC puts up some resistance to the FBI when asked for user data, and that Google does less so.

</comment></comment></comment></comment></comment><comment><USER6> It strikes me as accurate. Google apps is better for mail, and Microsoft is the cats ass when it comes to calendaring. If you are the secretary who must schedule 20 execs for a meeting the first Wednesday of every month, then there is nothing better than Outlook/Exchange.

Thank god I no longer live in a world where I am tied to my calendar. As google gets bigger, google calendar will get better, and the worse it will be a place to work. Understanding the need for a great calendaring app requires a company that has lost its soul.

<comment><USER7> Does Google really make its employees depend on Gmail and Gcal?

<comment><USER8> Yes, they do. But that shouldn't be surprising.

<comment><USER7> Well, I mean I know they are Google and all- but I can't imagine Gmail and Gcal supporting a corporate user very effectively. I use it for my personal stuff, but I am extremely appreciative of Exchange at work.

<comment><USER9> Calendar seems to have a similar featureset to that of Outlook. You can put appointments on other people's calendars and you can schedule rooms, anyway. And, at Google, the conference rooms have motion sensors that allow the room reservation system to "garbage collect" unused rooms. (You also get an LED indication of whether or not you'll be running into someone else's time if your meeting runs late. Gotta love the hardware support!)

<comment><USER10> Who makes these devices?

<comment><USER11> Googlers?

Edit: maybe that? <http://code.google.com/p/radishdisplay/>

<comment><USER12> Actually that's the code for an older 20% project:

<http://code.google.com/apis/gdata/articles/radish.html>

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> The "analysis" seems to be strongly favoring Google products in the Email/Calendar Cloud/Local Solution section.

\- Integration with Collaboration Tools: SharePoint Online is all about collaboration. As they want to continue using Microsoft Office, there is no better way (at the moment) than SharePoint. Google's cloud connect Add-In for Office is the first try at this, but it is still rather ugly (who wants a large bar on all documents, what about using the ribbon?) They also misspelled Lync, which is the Google Talk of Microsoft, goes much further than GTalk and integrates in ALL Office applications. You can chat with the people currently editing the same document you are editing on your local computer.

\- Ease of Tools Development: "Google has a more robust and documented API" - NO, simply not true. "aligned with the skillsets on campus" is a valid argument. But saying it is easier to develop for Google products than for the MS stack is a stretch.

\- User Familiarity: They say, that they use MS Office. Familiarity therefore is high with Office 365.

</comment><comment><USER14> Glad to see Cal quitting MSFT. Perhaps they're looking for a donation from Schmidt (alum). Gates hasn't given to Berkeley afaik while he has to Stanford. Yes I acknowledge this is a cynical perspective but perhaps makes sense considering public universities are so strapped for cash while privates keep on getting donations.

<comment><USER15> On the other hand, they just signed a contract providing Microsoft licenses to all the students, which is unfortunate. Earlier they did the same with Adobe. This is their solution to lowering costs--lets provide Microsoft and Adobe products to everybody! I, naturally, disapprove, but that has never mattered...

<comment><USER16> Their students are smart. The'll figure out there are better solutions.

And having seen how not to do it is also an invaluable lesson. ;-)

(Burn, karma, burn)

<comment><USER15> I don't think most of the people taking, say, a class on Nabokov will complain much about needing Microsoft Word. In fact, I suspect the only inconvenienced person will be the lone EECS major using Linux (me :)).

As long as professors want hard copies, it won't be an issue. (Using MLA format with LaTeX is really, really easy.) However, some do expect digital copies for whatever reason. I've seen them ask for docs uploaded to the internal class website system, which is itself pretty horrible. I just hope I don't have to deal with it--even opening LibreOffice is more annoying than one could hope.

<comment><USER16> It could be worse. The class website could be a Sharepoint portal.

</comment></comment></comment></comment></comment><comment><USER17> I've always found these large organizational analyses to look completely sound and well-reasoned on paper, yet never really align with reality when deployments start going up.

IIRC, the City of Los Angeles did a similar evaluation, yet they shuttled a migration to Google due to security concerns.

In less-public implementations, I've seen planned migrations from MS to Google Apps dropped after pilot tests, mostly due to user complaints about usability.

I don't fault MS or Google in any of these areas, but rather those conducting the analysis.

<comment><USER18> > IIRC, the City of Los Angeles did a similar evaluation, yet they shuttled a > migration to Google due to security concerns.

Actually most of the city is using Google Apps and the contract has been renewed. The LAPD decided it wanted additional security and hasn't migrated yet.

<comment><USER19> To be clear it's more than just the LAPD. From [http://informationweek.com/news/government/cloud- saas/232300...](http://informationweek.com/news/government/cloud- saas/232300646)

 _The amendment will remove the LAPD, portions of the city attorney's office, the Los Angeles Fire Department, Department of General Services, and Department of Transportation from the contract_

Only a little more than half of the expected number of users have been migrated.

Still a nice win for Google, but also a cautionary tale.

<comment><USER20> So what are the excluded departments using? Hosted Exchange? I'm going to go out on a limb and assume that whatever solution they're using isn't any more secure. Even if you host your own mail servers inside Fort Knox, and hire and vet everyone who goes near those servers, you still have all the same trustworthiness problems.

On the other hand, I'm probably missing something there, since Google somehow failed to convince them to stay on board.

</comment></comment></comment></comment><comment><USER21> Seems that they found Google to be a better end-user experience for email and for casual calendar users, but Microsoft won out on power calendar users and in terms of legal/compliance issues. Doesn't seem clear to me why they chose Google here -- costs?

<comment><USER22> Notice the priority column at the far left. Legal and compliance issues were ranked far below integration.

</comment></comment><comment><USER23> Glad they're changing: the homegrown Calmail system had ~5 consecutive days of downtime this November alone.

</comment><comment><USER24> I am surprised they haven't evaluated the office apps that are associated with the suits. As a long time Google Docs user, I was stunned by how superior the online version of Excel is to Google's offering.

<comment><USER25> They're not migrating the other apps. From the FAQ [1] - _Q: Is Microsoft Office being replaced by Google Docs? A: No, Microsoft Office will continue to be a desktop standard at the University. ..._

Nice Freudian slip, btw.

[1] [https://kb.berkeley.edu/jivekb/entry.jspa?externalID=2980...](https://kb.berkeley.edu/jivekb/entry.jspa?externalID=2980&categoryID=107)

<comment><USER15> It's really unfortunate that they insist on using Microsoft office. Happily the rational departments--the ones I usually deal with :)--don't expect office docs, but I wouldn't put it past my future literature (or whatever) professors to expect them. And that would suck.

</comment></comment></comment><comment><USER26> I am Google Apps addict but they still don't have a web-based Excel-killer. I think that when bankers and corporates start using Google apps to build their financial models then Microsoft should worry.

<comment><USER27> This is true.

I've seen entire hedge funds run on excel sheets. Our traders do alot of trading right out of excel sheets with custom C# addin's.

<comment><USER9> This is not the intended use case for Excel. Rather, it's a product of 10 years of bad decisions leading to more bad decisions.

(Someone I work with created a "market data bus" using Excel. It had the side effect of preventing Outlook from receiving email while it was running, so you had to have two machines if you wanted to read email AND get market data via this method. Now that's shoddy software engineering.)

<comment><USER27> > This is not the intended use case for Excel. Rather, it's a product of 10 > years of bad decisions leading to more bad decisions.

This is true :)

</comment></comment></comment></comment><comment><USER28> I think they got it wrong, Google offerings SUCK (REALLY BAD) when it comes to collaboration. In a large organization like UCB, I wonder why they don't use any collab tools?

"Both Google/Microsoft offerings were NOT reviewed"

"These applications were not extensively reviewed during this process, but are already used in other departments for teaching and learning purposes."

"Office 365 comes with its Office online apps and the option of deploying SharePoint Online and Lync* (which were not reviewed)."

I want white-boarding in GMAIL!!!

<comment><USER29> Isn't this a red herring? They stated they were evaluating Email/Calendering solutions.

<comment><USER28> Sure. I am just frustrated with Talk/Docs. Talk, a bare minimum IM client. Docs, are we in 1999?

</comment></comment></comment><comment><USER30> Huh, there is some points to check : Points 2.Location of Data 3.Data Management and Transfer

have exact same description (copy-pasted?) but have different advantage results.

For 2.Location of Data advantage goes to "Microsoft (slight). Google has more qualifications on U.S. data storage obligation in minor respects."

For 3.Data Management and Transfer it suddenly become "Draw"

</comment><comment><USER31> At least they aren't doing what Indiana University did - use both and let the students decide which system they want to use. Now they can choose between Imail (Microsoft) or Umail (Gmail). But on the administration side, IU is still pretty much a Microsoft shop though with the faculty having an Exchange setup.

</comment><comment><USER32> This should teach you: "do an analysis before choosing" ;)</comment>


Synopsis – Open-Source Computational Cinematography - <AUTHOR> https://synopsis.video/ <comment><USER2> Noticed this is now supported in VJ software VDMX, [https://vdmx.vidvox.net/blog/synopsis- beta-1](https://vdmx.vidvox.net/blog/synopsis-beta-1)

</comment><comment><USER3> Seems powerful, but I don't really understand exactly what this is. Probably industry specific. Could someone explain to a non film creator?

<comment><USER4> From my minimal understanding, computational photography/cinematography is an application of tools like GANs to various problems in video editing/production. Can anyone confirm/elaborate?

</comment></comment><comment><USER5> Certain merit!</comment>


Confessions of an accidental job destroyer - <AUTHOR> https://www.technologyreview.com/s/611477/confessions-of-an-accidental-job-destroyer/ <comment><USER2> " In society's narrative of the war between robots and humans, I'm probably the bad guy."

Question: Would Erin have done what she did if the fellow was her father? I hope the same sort of circumstances happens to the writer just to make it a fair, symmetrical, ending.</comment>


Walkway.js - <AUTHOR> http://www.connoratherton.com/walkway

<comment><USER2> Hey! Guy who did the Polygon PS4 and Xbox review SVG animations here.

This is really great stuff; has a super friendly API to boot.

FWIW, we also released a generalized library for some of the things we were doing WRT frame-based and duration-based animation:

[https://github.com/voxmedia/metronome](https://github.com/voxmedia/metronome)

(Reason being that for some animations, we wanted it to finish in a precise amount of time, but for others, maintaining frame rate was paramount, even if total elapsed time was longer)

</comment><comment><USER3> The current link does not work for me (Firefox 36.0a1 on Debian). The correct content seems to be here: [http://connor- personal.herokuapp.com/walkway](http://connor-personal.herokuapp.com/walkway)

Why doesn't it work? I imagine it's this funky DNS config:

      sergio@sergio-laptop:~ > dig www.connoratherton.com

      ;; QUESTION SECTION:
      ;www.connoratherton.com.		IN	A

      ;; ANSWER SECTION:
      www.connoratherton.com.	3529	IN	CNAME	http://connor-personal.herokuapp.com.
      http://connor-personal.herokuapp.com. 229 IN CNAME us-east-1-a.route.herokuapp.com.
      us-east-1-a.route.herokuapp.com. 55 IN	A	126.96.36.199

This is not valid CNAME content. Does this work on some browsers???

<comment><AUTHOR> Switched to Google domains a few days ago, still configuring it properly. Thanks for letting me know about the link, looking into it.

<comment><USER4> Do you happen to have any extra invites for Google domains?

<comment><USER5> I do if still needed, drop me a line (email in profile)

</comment></comment></comment></comment><comment><USER6> Firefox 33.0.2 on OSX 10.9 here.

It renders well if the tab is in focus (fantastic work!) but there's a bug: if I open it in a tab in the background, and visit it later (after the animation would have completed?) I see only a nearly-blank page. Confused me for a bit, as I open mass-tabs here on HN...

<comment><USER7> I guess OP doesn't use requestAnimationFrame, otherwise, it would wait for the tab being focused.

Edit: Checked the code. He uses requestAnimationFrame.

<comment><USER8> You meant the opposite though, right?

</comment></comment></comment><comment><AUTHOR> Sorry guys, I really didn't expect this much traffic. I still have to do some compatibility testing, I'm aiming to do it this weekend.

<comment><USER9> Works well on Windows Phone 8.1.

I'm telling you because I'm pretty sure I'm the only Windows Phone user worldwide.

<comment><USER10> No you're not :)

</comment></comment></comment><comment><USER11> I made this [http://grayarea.org/theater/](http://grayarea.org/theater/) after being inspired by the Polygon animations.

I'm using Two.js [https://github.com/jonobr1/two.js](https://github.com/jonobr1/two.js) to do the SVG interpreting and processing.

</comment><comment><USER12> This article by Jake Archibald does a good job of explaining the underlying mechanism used by libraries such as walkway.js: [http://jakearchibald.com/2013/animated-line-drawing- svg/](http://jakearchibald.com/2013/animated-line-drawing-svg/).

</comment><comment><USER13> Hey Connor, awesome work. You would have saved us a lot of work had you created this a year ago ;)

</comment><comment><USER14> Really silly request, but could someone provide an implementation where I can just drop in an svg and it would have this effect? Please, and thank you.

<comment><USER15> Have a look at lazy line painter

[http://lazylinepainter.info/](http://lazylinepainter.info/)

<comment><USER14> Gotcha. The next step is making an animatable SVG. I'm trying to follow [https://inkscape.org/doc/tracing/tutorial- tracing.html](https://inkscape.org/doc/tracing/tutorial-tracing.html)

</comment></comment></comment><comment><USER16> A bit of trouble on OS X Yosemite, running Chrome. if i open it an new tab, nothing happens. cmd + r and it works.

</comment><comment><USER17> Nice work, looks great.

<comment><USER18> agreed!

</comment></comment><comment><USER19> If I open this in a tab and wait a while to go there, it never plays the animation.

[https://www.dropbox.com/s/z57g1fmmn10qmej/Screenshot%202014-...](https://www.dropbox.com/s/z57g1fmmn10qmej/Screenshot%202014-10-31%2010.14.42.png?dl=0)

</comment><comment><USER20> This seems like an awful lot of code for animating the path's strokeDashArray and strokeDashOffset attributes. Especially if you're already using another library with built in transitions, such as D3 or even jQuery.

</comment><comment><USER21> Very nice. It's funny, I keep clicking on these 'new JS library' links thinking, oh geez, not another one... but then I leave thinking, wow, that is very impressive and super useful.

</comment><comment><USER22> make sure to leave and come back/hit refresh etc to try to get the animation to fire. looks cool, although there was nothing on the screen the first time i went to the tab on Chrome/MacOSX

</comment><comment><USER23> it seems that it doesnt run if I open this in a new tab and stay on the current page but if I reload the page it works (all done with chrome) Seems like a page focus event gone wrong

</comment><comment><USER24> Cool! We've been using a similar technique in our in-app sales page, also inspired by the Verge articles. I'll upload a video when I get a chance.

</comment><comment><USER25> Works great for me on Safari 8 (Yosemite)!

</comment><comment><USER26> On my machine the animation never plays in Opera 12, and in Chrome and Firefox the black color is never applied.

<comment><USER27> In Chrome on my machine, the black colour is not applied also.

Furthermore if I middleclick to open a new tab, the animation does not play.

</comment></comment><comment><USER28> "server not found".

I don't know if HN crashed the server or if there is some irony involved.

</comment><comment><USER29> How would somebody do this w/ D3? Are there any similar bl.ocks that anybody knows about?

<comment><USER30> Since you can write more or less, arbitrary "tween" functions in D3 (I learnt more than I wanted to about these when I wrote the dynamic "semaphore pie" we used here [1] ) it's relatively viable to do it for paths created from within D3. For paths already present and wanting to be selected... I guess it would work pretty much the same, since d3 selectors are more or less good to go with embedded SVGs, IIRC.

It's actually a very nice trick (and I love moving things around in graphs), and I'm intrigued by how I'd do it with D3. If I have any free time this weekend I'll look into it ;)

[1] VanillaSEO: [http://vanillaseo.com/report/20141030fdec7fda61eafac3b8dca58...](http://vanillaseo.com/report/20141030fdec7fda61eafac3b8dca58899defb9e)

Ninja edit (or almost): Here's the relevant part of d3 docs about tweens: [https://github.com/mbostock/d3/wiki/Transitions](https://github.com/mbostock/d3/wiki/Transitions)

</comment></comment><comment><USER31> Site's not available?

</comment><comment><USER32> was gonna ask if inspired by Verge/Polygon console reviews from awhile back... nice.

</comment><comment><USER33> Should be called Hawt.JS

</comment><comment><USER34> not working in Safari with Yosemite.

<comment><USER35> Works fine for me with the same setup (Safari 8 on Yosemite).

</comment></comment><comment><USER36> Link broken?</comment>


Your data fits in RAM - <AUTHOR> http://yourdatafitsinram.com/ <comment><USER2> It's probably worth extending "Your data fits in RAM" to "Your data doesn't fit in RAM, but it does fit on an SSD". So many problems will still work with quite reasonable performance when using an SSD instead. By using a single machine with an array of SSDs, you also avoid the complexity and overhead of distributed systems.

My favourite realization of this: Frank McSherry shows how simplicity and a few optimisations can win out on graph analysis in his COST work. In his first post[1], he shows how to beat a cluster of machines with a laptop. In his second post[2], he applies even more optimizations, both space and speed, to process the largest publicly available graph dataset - terabyte sized with over a hundred billion edges - all on his laptop's SSD.

[1]: [http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html)

[2]: [http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html)

<comment><USER3> This is classic case of "Algorithm/Problem Selection" if your algorithm/problem is tailored to a task such as PageRank, surely a single threaded highly optimized code will beat a cluster designed for ETL tasks. In real organizations where there are multiple workflows/algorithms, distributed systems always win out. Systems like Hadoop take care of Administration, Redundancy, Monitoring and Scheduling in a manner that a single machine cannot. Sure you can "Grep" faster on a laptop than AWS EMR with 4 Medium instances, but in reality where you have 12 types of jobs which are run by team of 6 people, you are much better off with a distributed system.

<comment><USER4> Ditto for computationally intensive work: if it is CPU dominated, more CPU's calculating in parallel will be of advantage, even if the data could fit some RAM.

There's no a single simple answer, but sure, whenever less computers are enough, less should be used.

The recent problem is, some people love "clouds" so much today that they push there the work that could really be done locally.

<comment><USER5> Part of the problem is that a lot of problems that are CPU dominated on a single system becomes IO dominated once you start distributing it at very low node counts without _very_ careful attention to detail.

<comment><USER6> The entire reason for the popularity of distributed systems is because application developers in general are very bad at managing I/O load. Most developers only think of CPU/memory constraints, but usually not about disk I/O. There's nothing wrong with that; because if your services are stateless then the only I/O you should have is logging.

In a stateless microservice architecture, disk I/O is only an issue on your database servers. Which is why database servers are often still run on bare metal as it gives you better control over your disk I/O - which you can usually saturate anyway on a database server.

In most advanced organizations, those database servers are often managed by a specialized team. Application servers are CPU/memory bound and can be located pretty much anywhere and managed with a DevOps model. DBAs have to worry about many more things, and there is a deeper reliance on hardware as well. And it doesn't matter which database you use; NoSQL is equally as finicky as a few of my developers recently learned when they tried to deploy a large Couchbase cluster on SAN-backed virtual machines.

<comment><USER7> I own a ASUS gaming laptop...

It has two flaws: One, it has nVidia Optimus (that just suck, whoever implemented it should be shot).

Two, the I/O is not that good, even with a 7200 RPM disk, and Windows 8.1 make it much worse (windows for some reason keep running his anti-virus, superfetch, and other disk intensive stuff ALL THE TIME).

This is noticeable when playing emulated games: games that use emulator, even cartridge ones, need to read from the disc/cartridge on the real hardware, on the computer they need to read from disc quite frequently, the frequent slowndowns, specially during cutscenes or map changes is very noticeable.

The funny thing is: I had old computers, with poorer CPU (this one has a i7) that could run those games much better. It seems even hardware manufacturers forget I/O

<comment><USER4> I can bet in your case it's not the disk I/O actually, unless you have some very strange emulator the file access should still go through the OS disk cache. VMware for example surely benefits from it. How many GB do you have on the machine? How much is still left free when you run the emulator and the other software you need?

<comment><USER7> I noticed it as disk i/o because I would leave task manager running on second screen and every time the game lagged memory and CPU use were below 30% and disk was 100%, and if I left sorting by disk usage, the first places are the windows stuff, and after them, the emulator.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> So, let's say my system is currently backed by MySQL or PostgreSQL, and that is not fungible. How would one move that data into RAM, including writes? And, how would one maintain some level of safety in the event of a crash? i.e. I don't really care if I lose X amount of time worth of data (say, five minutes), but I do care that when I reboot the system, the database comes back from disk into RAM in a consistent state.

Is there some off-the-shelf solution to this problem? And, if so, why isn't it talked about more? Every CMS ever, for example, would be very well-served by something like this. My entire website's database, all ~100k comments and pages and issues and all 60k users, is only 1.4GB, and performance is always a problem. I don't care if I lose a couple minutes worth of comments in the event of a system reboot or crash. So, why can't I just turn that feature (in- memory with eventual on-disk consistency, or whatever you'd want to call it) on and forget about it?

<comment><USER9> Does your RDBMS's built-in caching not handle this pretty well? Just up the cache size, e.g. in PostgreSQL changing effective_cache_size [https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serv...](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server)

<comment><USER8> For reads, yes. It's fine on reading data. The problems are writes which always wait for disk consistency to return.

</comment></comment></comment><comment><USER10> And if your data doesn't fit in a single server's RAM, just buy some more and run Apache Spark [1] on them. It's an in-memory computation engine that's really nice to program for: you don't have to worry about low-level clustering details (like MapReduce). And it's way (10-100x) faster than Hadoop.

[1] [https://spark.apache.org](https://spark.apache.org)

<comment><USER11> Spark is fast becoming the default tool for big data.

The recent addition of SparkR in 1.4 means that now data scientists can leverage in memory data in the cluster that has been put there by output from either Scala or DW developers.

Combine it with Tachyon ([http://tachyon-project.org](http://tachyon- project.org)) and it's not hard to imagine petabytes of data all processed in memory.

<comment><USER12> Can you explain what Tachyon does that's different from what Spark already provides?

I haven't used either Spark or Tachyon. I thought the Spark solution was to just put my dataset in memory. But the Tachyon page seems to say the same thing

<comment><USER13> There's a slide deck[1] that explains it rather well.

Basically, Tachyon acts as a distributed, reliable, in memory _file_ system.

To generalise enormously, programs have problems sharing data in RAM. Tachyon lets you share data between (say) your Spark jobs and your Hadoop Map/Reduce jobs at RAM speed, even across machines (it understands data-locality, so will attempt to keep data close to where it is being used).

[1] [http://www.cs.berkeley.edu/~haoyuan/talks/Tachyon_2014-10-16...](http://www.cs.berkeley.edu/~haoyuan/talks/Tachyon_2014-10-16-Strata.pdf)

<comment><USER12> Neat, thanks for the link, the code examples towards the end make it clear that this is pretty simple to use.

<comment><USER13> Yeah, most things coming from the Spark team are excellent in that respect.

I've never used Tachyon, but based on the wonderful "getting started" experience Spark gives I'd be confident it would be similarly well thought out.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> Inspired by [https://twitter.com/garybernhardt/status/600783770925420546](https://twitter.com/garybernhardt/status/600783770925420546)

<comment><USER14> Can someone explain in a bit more detail what this is about? Is the 'joke' that running data computation in RAM is faster than what? From disk?

<comment><USER15> The subtext is that running a fancy distributed system is more exciting and beneficial for ones resume than simply buying a massive bloody server and putting postgres on it, and that people are making tech decisions on this basis.

<comment><USER16> This of course ignores that it's much easier to get your hands on a cluster of average machines than one massive bloody server, and all the non-performance- oriented benefits of running a cluster (availability etc.).

Much easier to request a client provisions 20 of their standard machines, or get them from AWS. People don't like custom hardware, and for good reason.

<comment><USER17> Amazon offers some bloody huge servers... 32 core, 256GB RAM, and 48TB HDD space. d2.8x large

<comment><USER18> That is 4k a MONTH for 256gb of ram.

If you could do the same job on a fleet of 8-16GB servers.. you can get a lot more CPU for a lot less dollars. Depends if you really need everything on 1 machine or not (as of course nothing will beat same machine in memory locality)

<comment><USER19> Not true, 8x16GB costs as much as 1x256 on Amazon. The issue here is that Amazon is hilariously expensive in general. Hetzner will rent you a 256GB server for €460 per month. Or you can buy one from Dell for $5000. These are not high numbers, in 1990 you paid more than that for a "cheap" home computer. For the price of a floppy drive back then you can now get a 32GB server.

</comment></comment></comment></comment></comment></comment></comment><comment><USER20> Yes! As someone who frequently runs memory-intensive algorithms on large(ish) datasets, I have a hard time explaining to many technical people that moving from a single server to a cluster increases complexity and cost by an incredible amount. It affects key decisions like algorithm and language, and generally requires a lot of tweaking.

When a problem becomes big enough, moving to a cluster is absolutely the right decision. Meanwhile, RAM is cheap and follows Moore's Law.

<comment><USER21> Complexity, sure. But cost? I thought a single 1 TB RAM server is more expensive than 10x 100 GB RAM servers.

And many people don't want to deal with physical hardware. Dealing with physical hardware increases operational complexity too. They want to rent a virtual/cloud server. Which provider allows you to rent a virtual server with 1 TB RAM?

<comment><USER22> I took a look around for "high-ram" servers, and it seems one I can buy today, is HP ProLiant DL580 Gen9. With just 256 GB of ram, it clocks in at 540.995,- NOK (71.5k USD). It has 96 ram slots, and I can't seem to find anything bigger than 32 GB DDR4 RAM, and rounding the price _up_ 96x32GB comes to roughly 672.000,- NOK (~90k USD). Adding that up (throwing away the puny ram installed), gets us to a little over double the original price, or 1.212.995,- (~161k USD). This has 4 18 core E7s (72 cores) clocked at 2.5Ghz -- and 3TB of ram (half of max, because of 32 GB dimms).

It _is_ true that while the jump from 256GB to 3TB is "just" ~2x -- I could get a server for 1/10 of the price of the original configuration -- but only with 4GB of RAM, and nowhere near even 18 hardware threads.

If you are CPU limited (even at 72 hw threads) you might need more, smaller servers.

But such a monster should scale "pretty far" I'd say. Does cost about half as much as a small apartment, or one developer/year.

<comment><USER23> Dell sells servers with 96x64GB RAM. There is a huge (7x) premium for the 64GB DIMMs instead of 32GB, so it runs around 500k, with almost the entire price going to RAM.

</comment></comment></comment></comment><comment> chao- I love it. I was just doing some Fermi estimates for a friend on the data for a project he has in the pipeline. I was curious whether or not it would be cost efficient for his project's budget to go with NVMe SSDs or have to stick with traditional SATA ones, and turns out it doesn't even matter (for now) because at least the first three months of data will fit in 256GB of RAM, even allowing for a 2.5x factor stemming from some (estimated) inefficient storage or data structure use in a scripting language like Ruby or Python.

Edit: And after those first three months he'll know more about the use and performance demands of the project and will be able to make far more accurate decisions about storage categories.

<comment><USER24> Where's 2.5x from? I'd be curious to see any actual data on comparing memory footprint for a problem in C/Go/Rust to Python/Ruby. I'm sure it varies widely, but 2.5x might not be far off.

</comment></comment><comment><USER25> Today I'm working on dataset of 1GB, which fits in memory. But it is not enough. If a variable is category/factor you need to introduce dummy values and your dataset starts picking the weight. Next - do you want apply ML algorithm in parallel? Upst, you need more memory. Done that? Now please use test dataset for prediction. My point that "data in memory" is just the beginning...

</comment><comment><USER26> The problem with giant boxes (full of RAM / SSD / Disk) is giant failures and huge recovery times. This is worsened in case of RAM because now every power blip is a full on recovery situation. Have a big enough data set focussed on a single box (or two for backup purposes), your customers are going to blow a gasket the moment one of them go down because workloads usually grow to accommodate available capacity.

FB has a nice paper that talks about this problem. [https://research.facebook.com/publications/300734513398948/x...](https://research.facebook.com/publications/300734513398948/xoring- elephants-novel-erasure-codes-for-big-data/)

<comment><USER27> Well, you wouldn't run such a server without a hefty uninterruptable power supply system. On your bigger server you can expect a smaller frequency of failures due to fewer points of failure, and can make your system more resilient (rendundant RAM, filesystems, power etc).

</comment></comment><comment><USER28> More accurate title would be fit in RAM of single machine.

Maybe some bonus category:

0\. Spreadsheet is all you need.

1\. Python script is good enough.

2\. Java/Scala is way to go.

3\. Need to manage memory (gc doesn't cut), some custom organization.

4\. Actually needs a cluster.

<comment><USER29> > 0\. Spreadsheet is all you need.

I HATE when people use Spreadsheets to do anything besides simple math.

[http://lemire.me/blog/archives/2014/05/23/you-shouldnt- use-a...](http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a- spreadsheet-for-important-work-i-mean-it/)

TL:DR your work is not reproducible and we can't see what you did to get to your numbers. A million examples of why this is bad.

Also

> 1\. Python script is good enough

You mean Python with pandas and numpy?

I use R which is also a great choice

> 2\. Java/Scala is way to go.

For you but the vast majority of Data Scientist don't use either and their choice for people is not universal. Julia looks like a great new comer. I again mainly use R.

> 3 & 4 are good points.

<comment><USER30> Sadly, I recall those arguments against spreadsheet computation being made in the early 1990s. People simply won't learn.

Ray Panko, University of Hawaii.

[http://panko.shidler.hawaii.edu/SSR/](http://panko.shidler.hawaii.edu/SSR/)

[http://panko.shidler.hawaii.edu/SSR/](http://panko.shidler.hawaii.edu/SSR/)

Goes back to 1993:

[http://panko.shidler.hawaii.edu/SSR/auditexp.htm](http://panko.shidler.hawaii.edu/SSR/auditexp.htm)

<comment><USER31> There is even a European Spreadsheet Risks Interest Group: [http://www.eusprig.org/](http://www.eusprig.org/)

</comment></comment></comment></comment><comment> a-saleh I am affraid the in our research lab we didn't have 10 000$ up front/200$ a month to get a pc with 1TB ram ... we did have a large computer hall and BOINC though :)

</comment><comment><USER32> Looks like 1.5TB RAM with 15 cores costs $50K. But it shouldn't be just about RAM. The problems I'm working on requires 250 cores on similar amount of data. If there was an option to get say 150 cores with 2TB RAM, things would fly for sure.

<comment><USER33> Another 4 to 6 years and that should be a reality.

<comment><USER34> 4-6 months and you'll have a Knight's Landing Xeon Phi with at least 72 cores and 288 hardware threads, with vector instructions, and you'll be able to stick 3 of them in single blade.

</comment></comment></comment><comment><USER17> Seems a bit naive, saying 2.1PB probably doesn't fit in ram, "but it could"...

I get who this is aimed at, and why, but just saying that it fits in RAM isn't as useful as it could be. This is an opportunity to teach, not just snark.

<comment><USER35> A bit of clever reformatting of your data and 2.1 Pb could probably easily be reduced in size to something that would fit in RAM. Are you actually needing every byte?

</comment></comment><comment><USER36> Outside of scale-out, scale-up there is also solution: scale-in. Optimize your memory usage, so your data occupies less space.

I work on something like that.

</comment><comment><USER37> So this seems to use 6.144TiB as the limit that will fit in RAM. That's 1.536TiB x 4 when using the latest Xeon I could find[1]. According to the specs though you should be able to use 8, so the total limit should actually be 1.536 x 8 = 12.288 TiB. 12TiB of RAM, that's quite amazing.

[1] [http://ark.intel.com/products/84688/Intel-Xeon- Processor-E7-...](http://ark.intel.com/products/84688/Intel-Xeon- Processor-E7-8893-v3-45M-Cache-3_20-GHz)

<comment><USER38> It seemed to use 6.000000000000000444089209850...??? TiB when I tried values.

<comment><USER37> It seems to use different values of cutoff depending on if you are using MiB/GiB/TiB/etc. I tested with GiB and 6144 is OK, 6145 is not.

</comment></comment></comment><comment><USER39> I think its wrong. It says 64 TB does not fit in RAM, but you can get 64TB machines from SGI as well 32 TB ones from Oracle.

The SGI one with up to 2048 cores are larger in their single system images than most people have in their clusters.

The benefit of these systems is not really the ease of programming but the speed of interconnect.

List price of the Oracle one was 3 million a few years ago. But most of that is actually in the high density dimms. These days I think the price must be lower, but I won't waste my Oracle sales contact time in figuring out what it is today. Of course it will still be expensive, it is an Oracle product after all.

However, an equivalent dell list price cluster of simple 1U boxes (512 6C/64GB ones!) will go for 1.5 million. The fact that to house 512 boxes i.e. 25 racks or so plus networking. Of course you do get 1/3rd more cores than the SGI one.

For many of us that are between the just use a single normal server and yet too small for the google solutions. These big memory solutions from Oracle and SGI can make sense even if they are not the first thing that comes to mind!

</comment><comment><USER40> Everything fits in RAM if you have the budget for it.

<comment><USER33> No, the point is that usually fitting things in RAM _lowers_ the budget. So it's well worth doing proper analysis on whether or not you can (a) fit all your data in RAM and (b) if a cluster of machines does not become it's own reason for existence.

Replacing a large number of nodes with a single machine with a lot of RAM is usually a cost savings measure rather than a larger expense (and it saves power too!), and due to a lack of communications overhead and exploitation of the fact that you now have access to _all_ the data in one go you may very well find that your algorithms run _much_ faster.

A distributed solution should be a means of last resort.

<comment><USER41> What does 6TB of RAM go for these days?

<comment><USER33> Less than 6 machines with 1T each ;) (Assuming the 6 will operate only on local data and will never need to communicate in which case you may well end up with more than 6).

But seriously: the price of RAM for servers is now ~10$ / G.

<comment><USER42> That would translate to ~$60k for 6TB of RAM. Plus the cost of the server itself ($10k?)

<comment><USER33> No, that's a _very_ expensive server, and Dell will charge you a hefty premium for the memory. There are quite a few options that will cost you less than that (of course the maximum capacity will vary).

It would be nice to see an article comparing all the high RAM machines side by side with specs and prices.

The largest machine I have right now will hold 512G and was a run-of-the-mill machine, it was about $5K, I'd expect the more exotic ones to be substantially more expensive but probably not as expensive as the machines linked here.

<comment><USER43> Can you point me to cheap 64GB LRDIMM Octal rank memory? Dell's prices for this seem to be the market rate, but maybe I don't know where to shop.

<comment><USER33> I can get Octal rank in bulk for ~$1K so that's $16 or thereabouts / G, still not bad for RAM that is obviously going to be sold in smaller quantities. Note that HP or Dell will probably not be happy if you use 3rd party RAM in their machines (if they didn't pull tricks to make sure only their own stuff works!).

(For contrast, that $1K if you'd spend it on HP branded RAM would not even get you four 16G dual rank units...).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER44> Taken from the github repository:

var MAX_SENSIBLE = 6 * TB; function doesMyDataFitInRam(dataSize) { return dataSize <= MAX_SENSIBLE; }

</comment><comment><USER45> And if your data doesn't fit into the RAM of a single machine you can buy a few more and use vSMP ([http://www.scalemp.com/](http://www.scalemp.com/)) to create a shared memory single system image.

</comment><comment><USER3> In my opinion the correct answer is 255Gb. (i.e. AWS r3X8 High Memory instances ).

While one can purchase servers with larger memory most likely you will run into limitation on number of cores. Also note that there is at least some overhead in processing data, so you would need at least 2X the size of raw data.

Finally while its a good thing to tweet, joke about and make fun of buzzword while trying to appear smart. The reality is that purchasing such servers (> 255 Gb RAM) is costly process. Further you would ideally need two of them to remove single point of failure. it is likely that the job is batch and while it might take a terabyte of RAM you only need to run it once a week, in all these cases you are much better off relying on a distributed system where each node has very large memory, and the task can easily split. Just because you have cluster does not mean that each node has to be a small instance (4 processors ~16 Gb RAM).

<comment><USER33> > Further you would ideally need two of them to remove single point of > failure.

That's assuming that everything needs to be 'high availability' and buying _two_ of everything is a must. This is definitely not always the case. In plenty of situations buying a single item and simply repairing it when it breaks is a perfectly good strategy.

<comment><USER3> Its not about having two of everything at all times, but rather about having a capacity whenever you need it. At 244Gb you hit a sweet point where you can have access to large capability at a flexible price (Spot Market / On Demand / On Premise). This is what separates engineers with business acumen from run of the mill "consultants" with a search engine.

<comment><USER33> You mentioned 'single point of failure'.

</comment></comment></comment></comment><comment><USER46> "Your data fits in RAM", vs "Your data fits in RAM on around X machines", would be better. Any dataset fits in RAM.... but if its going to take more machines then I am willing to buy it really doesn't.

</comment><comment><USER47> Before core, there was tape. Tape used to be backup medium, then disk became the new tape. Bubble memory begat SSD, so memory has in some sense become the new disk.

RAM is the new disk: now for some, later for others.

</comment><comment><USER48> "Yes, your data fits in RAM... if you feel like buying a server at the same price as 3 Tesla Model S automobiles, a mansion in the Southern U.S., or a bachelor pad in San Francisco."

</comment><comment><USER49> HP hints its new memristor memory computer will have the cost of flash and the speed of registers. An will mostly eliminate the multi-level memory hierarchies we have today.

<comment><USER27> Unlikely; the limiting factor is already distance - poor scaling from interconnects (wires) already means that we can't have all that much global state. This might increase the amount of state we can have, but unless you can fit gigabytes into a single chip you won't be eliminating the multi level memory hierarchy.

Like right now the L1 cache will have latencies of 1 or 2 cycles, and the L2 cache 15; this is due to the overheads of cache coherency protocols, moving the data around the chip; it's not that the memory's slower, it's all SRAM.

They are probably referring to enterprise workloads. Here you have large working sets (so caches are less useful) and you want maximum throughput. Clever multithreading (finegrained) can reduce effective latency by scheduling many (32?) processes at the same time, executing an instruction from each in round-robin fashion (see Sun Niagara). In that case, you can sometimes dump the L1 cache, and you would be able to get rid of the memory hierarchy.

There's also probably a benefit wrt hard drives/secondary storage; you can obviously make system storage very fast, which might improve random access times considerably. BUT this is probably not going to be transformative; it'll improve certain types of accesses, but current algorithms are already very highly tuned to spatial and temporal locality of reference. Furthermore, you'll still see these structures win out, because they can take advantage of hardware prefetching more easily.

<comment><USER50> The property of memristors having real values instead of 0 or 1, and the fact that their value can be path dependent, leads me to think that at least information density can be increased over conventional memory today.

</comment></comment></comment><comment><USER51> Cute but "Big Data" is really just data that's not in the building and isn't feasible to just move around from one machine to another in your department.

</comment><comment><USER52> Even if your data doesn't fit in RAM... and even if it does... when you're developing, you should be using a sample of your data that fits into RAM.

</comment><comment><USER53> This is good marketing, but you know what would be even better marketing? Give me access to that server for a week. Let me setup a demo of my biggest customer, and then run my tasks. We've started (and are in progress) of investing thousands of dollars in moving to Azure. A server this large is not something I can buy, and experiment on easily. Hard numbers would convince my superiors that its a better solution, but they're not going to give me $10k to do the experiment.

<comment><USER34> 10k? Those sticks of RAM alone will cost you something like 75k USD. Then you'll need the processors, arguably 4 of the top of the line 18-core XEONs at 5000 USD each. Then you'll need to put it all together with software and a (properly cooled) rack, not to mention the terminal(s) to access it, plus the personnel to put this baby together for you. This box could easily cost you 150 grand.

<comment><USER54> Its not cost effective to use non E5-class Xeons, or go above 32GB DIMMs right now.... So you want a Dual-Processor setup, 16 DIMM slots, so 16x 32GB = 512GB w/ Dual Proc -- which you can do for about $10,000.

<comment><USER34> That's a very nice piece of kits for 10k I have to say. Thanks for the "sweetspot" price/perf advice. Seems like excellent value. I've had my heart on a badass mac pro but these specs put it to shame.

</comment></comment></comment></comment><comment><USER55> If I select 1KB, why does the link point to an HP server with up to 6TB of RAM? Linking to an 80's PC seems more appropriate :)

</comment><comment><USER56> Wow, I wish I had the spare change for one of these beasts. I think I have enough NP-hard problems to fill any RAM to the brim :)

</comment><comment><USER57> [http://www.downloadmoreram.com/](http://www.downloadmoreram.com/)

</comment><comment><USER58> Although we can theoretically handle up to 2^64 bytes of RAM (16 exabytes), the practical limit is much lower. I think someone on Wikipedia said it's somewhere around 8TB, but I imagine the performance of random access into 8TB RAM is much worse than a motherboard designed for up to 32GB RAM.

It's not as easy as just buying more RAM. You'll have to pay more attention to how you make use of the various caches in between your CPU and RAM.

<comment><USER59> I imagine that on a motherboard with 96x RAM slots, the access time between the first one in row and the last one will be actually quite different, due to the physical distance between them.

</comment></comment><comment><USER60> Sorry for the simple question but if you store it in ram what is the strategy for when the server is turned off?

<comment><AUTHOR> The idea is more that when you process data, if you can fit it all in memory (and you don't need lots of CPU power, etc, etc, etc) then just use one machine and don't worry about "clusterising" it.

If you're expecting growth in the size of your dataset (beyond growth in RAM size availability), then, well, maybe don't just use a single machine. Same goes for a whole bunch of similar "it's too large for a single machine" considerations.

Storing data should probably still be persisted to disk, and backed up.

</comment></comment><comment><USER61> Damn. my data is 6597069766657 bytes. Apparently if it was 6597069766656 bytes it would have fitted in RAM.

<comment><AUTHOR> Well, hate to break it to you, but you probably have some overhead associated with your data, like your operating system or structures related to processing your data.

</comment></comment><comment><USER62> Our data fits in ram but it proved to have no speed benefit. So the ram just sits there, being empty.

</comment><comment><USER63> Redis as a primary data store!

</comment><comment><USER64> 600 PiB "No, it probably doesn't fit in RAM (but it might)."

Well, well, well.

</comment><comment><USER65> What is the point of this site other budget shaming?

</comment><comment><USER66> Great googly moogly! terabytes of ram!

</comment><comment><USER67> But does it fit in the L1 cache?

</comment><comment><USER68> Brilliant!

</comment><comment><USER69> 6.000000000000000444 TiB

</comment><comment><USER70> Any point on the stupidly big ass font? It does not fit in my screen.

<comment><USER71> BUT IT FITS IN RAM!

</comment></comment><comment><USER72> That's like saying "you can fly first class".

If you don't have money, you can't. Very few people can afford it.

<comment><USER73> It's more akin to saying "if you're looking at buying several economy tickets to go from A to B, a first class ticket on a direct flight might be cheaper and faster than stitching together several economy tickets"

</comment></comment><comment><USER74> If you are programming in R, you sure better hope it does!

<comment><USER29> After reading the title I was sure there was something about R in the comments.

You can program R in Spark you can now program in R [http://blog.revolutionanalytics.com/2015/01/a-first-look- at-...](http://blog.revolutionanalytics.com/2015/01/a-first-look-at- spark.html)

Now you can work directly with SQL Server as announced this week by MS. [http://www.computerworld.com/article/2923214/big-data/sql- se...](http://www.computerworld.com/article/2923214/big-data/sql- server-2016-to-include-r.html)

I have had a ton of arguments about R's "biggest weakness" being that it uses RAM. I haven't once in the almost 3 years of working in R that I ran into this road block, but I am sure others have. Which there are several good distributed choices that will keep getting better and better.

Using RAM instead of Distributed is better in R as well as really any other language in terms of complexity and flexibility.

<comment><USER75> For my workloads, R has always choked on its single thread long before it choked on memory. And the parallelism options are terrible hacks.

</comment></comment></comment><comment><USER76> or you learn about data structures and algorithms and try to need less :-); Randomized algorithms for example are intriguing.

</comment><comment><USER77> We build object stores... so, no it most definitely does not.

</comment><comment><USER78> I sincerely hope nobody is using a tool like this to decide which enterprise servers to buy...

<comment><AUTHOR> Me too. The links are mostly to back up my claim rather than as a suggestion of servers to buy (or I'd have found some affiliate links!)</comment></comment>


Fire near LANL Whole town evacuated - <AUTHOR> http://nmfireinfo.wordpress.com There is a fire very close to Los Alamos National Laboratory. They have begun to evacuate the whole town. Here is some info from someone on the ground: Our main concern is that the Las Conchas fire is about 3 1/2 miles from Area G, the dumpsite that has been in operation since the late 1950s/early 1960s. There are 20,000 to 30,000 55-gallons drums of plutonium contaminated waste (containing solvents, chemicals and toxic materials) sitting in fabric tents above ground. These drums are destined for WIPP. We understand that LANL has been working since late last night to build a fire line in Water Canyon, between the fire and Area G. Over the last 26 hours the fire has grown from 0 acres to about 45,000 acres – about the size of the Cerro Grande fire in 2000. <comment><AUTHOR> There is a fire very close to Los Alamos National Laboratory. They have begun to evacuate the whole town. Here is some info from someone on the ground: Our main concern is that the Las Conchas fire is about 3 1/2 miles from Area G, the dumpsite that has been in operation since the late 1950s/early 1960s. There are 20,000 to 30,000 55-gallons drums of plutonium contaminated waste (containing solvents, chemicals and toxic materials) sitting in fabric tents above ground. These drums are destined for WIPP. We understand that LANL has been working since late last night to build a fire line in Water Canyon, between the fire and Area G. Over the last 26 hours the fire has grown from 0 acres to about 45,000 acres – about the size of the Cerro Grande fire in 2000.

</comment><comment><AUTHOR> <http://twitpic.com/5hvqed>

</comment><comment><AUTHOR> Holy crap it's serious! I may have to bug out!</comment>


Internet Curated: A collection of 100+ hand-picked curated resources and tools - <AUTHOR> http://internetcurated.com <comment><USER2> Looks like a some useful, but I've seldom seen the word "curated" over-used more. In a cursory glance of a couple categories, it looks like "curated" is used in close to 50% of the collection-item descriptions.

When you over-use a word that much, it tends to lose any meaning.</comment>


ASK HN: Does Apple store the facerecognition images on their platform? - <AUTHOR> If they do, they have your personal info and an image of your face ( several ). I cant even start to explain the dangers and power they gain with this. <comment><USER2> The answer is no, this concern is addressed in the recent interview to Craig Federighi: [https://techcrunch.com/2017/09/15/interview-apples-craig- fed...](https://techcrunch.com/2017/09/15/interview-apples-craig-federighi- answers-some-burning-questions-about-face-id/amp/)

</comment><comment><USER3> _I cant even start to explain the dangers and power they gain with this._

I'm not sure they'd gain anything. If a user has Facebook or Instagram (or any social network really) on their phone, which _most_ do, then Apple has access to a corpus of personal information and face images that's big enough to build a massively detailed picture of someone's life with an accurate biometric model. Apple users, rightly or wrongly, trust Apple not to abuse that information. Likewise Google's users, Facebook, etc.

It's too late to be talking about the dangers of letting corporations gain access to personal data and facial biometrics. The question now is how do we protect ourselves and our loved ones in a world where that data, for most people, is out there _by default_.

</comment><comment><USER4> For unlocking, the data will be stored on the Phone's secure enclave which is locked with your passcode. They say they won't upload it to their servers (which you have to trust them on, because it's hard to see if they do it).

However, there's an API for this tech as well and apps could abuse it. I bet they'll introduce a permission request for this though.

</comment><comment><USER5> Define 'platform'. Do you mean on their servers? If you do, then no, they don't. If you mean on the device itself, then yes, they do. They do it in such a way that it is inaccessible (using the secure enclave).

<comment><AUTHOR> I meant on their servers.

<comment><AUTHOR> But you are absolutely sure those images are never send to their servers?</comment></comment></comment>


Ask HN: I bought my first-ever smartphone. Now What? - the-rahul Hi,
After spending the last 9 years using a Nokia 1100, I just bought very first Smartphone yesterday - an HTC Desire 816. It can safely be said that I've missed most of the "mobile" revolution and basically used my old mobile phone as just that - a phone, I made and received calls and sent and received text messages. Now , after purchasing this device (which runs Android 4.4.2, by the way), I am at a loss about what to do, what to install etc.
I am a sysAdmin who writes writes small pieces of software by night. I have installed JuiceSSH and Evernote thus far. As for social apps, it came pre-installed with the FB app and Whatsapp, of which I've tried the latter for the first time last night.
I don't know what else to install/ use this for. So I'm asking the community for help to understand this device/OS and what it can be used for. Any help suggestions will be appreciated. <comment><AUTHOR> I dunno, dude. I was like you until recently, but smartphones are really personalized so my experience may not help you. What would you put on a laptop that fits in your pocket? I'm not much for social media apps and am actively hostile to Facebook's app because it has a history of spidering through your contact list and sucking up all your data. My phone has a couple of games (Ziggurat, King of Dragon Pass, and Lords of Midnight), an ebook reader, some useful map stuff, a couple apps that track the various public transport options in London so I know how long I have to wait for the bus. It's entirely possible that none of those are useful to you, if you don't play videogames or don't, you know, live in London. I also have google chrome. And skype. I find my phone (Samsung Galaxy s4) useless for writing longform, although email is more convenient when you can fire one off on the train.

</comment><comment><USER2> Try not to think like a system administrator.

Instead, think about your unique life. What do you do in your daily life that you might like to enhance? There are apps for just about everything, many suck, some are good. You might also write a better one.

Track your health? Know where your kids are? Calendar and synchronizing with other humans and organizations?

Where might you eat lunch where you are right now? What stars are visible in the night sky, in the direction that you're pointing your phone? What else can you know based on where you are or where you're pointing?

Oh, and take a shitload of pictures. I set my camera to the maximum resolution, so that I can crop and still have a nice result. I also take videos at work when something weird or notable is happening with the [thing] that I test.

Sometimes a picture is the quickest way to note something. I know some people snap every meal they eat, for example. Or take a picture of your kids if you're out somewhere, in case they get lost.

</comment><comment><USER3> Welcome onboard. If you are a news junkie - 'Feedly' & 'Instagram' are for you and if you often get reading time without internet, 'Pocket' is for you. Photo-editing Aviary, Photo-Sharing Instagram, Flickr. Zomato for finding hotels. Tell your interest so we can suggest based on that.

</comment><comment><USER4> Try Android programming[1] atleast once. Who knows, maybe you'll enjoy it. That's what happened to me.

[1]: [http://www.smashingmagazine.com/2010/10/25/get-started- devel...](http://www.smashingmagazine.com/2010/10/25/get-started-developing- for-android-with-eclipse/)</comment>


GM, LG to Spend $2.3B on Venture to Make Electric-Car Batteries - <AUTHOR> https://www.morningstar.com/news/dow-jones/2019120510724/gm-lg-to-spend-23-billion-on-venture-to-make-electric-car-batteries-5th-update <comment><USER2> Disclaimer: I work for GM, any opinions are my own; I have no special knowledge of this deal.

Vehicle launches are generally prepared starting 5 years out with defined components. This creates a special challenge when interfacing with quickly changing technology.

I don't have a publicly available source for this, so take it with a grain of salt, but I've heard that for every new battery pack released, there have been serious, funded plans for 2 or 3 other pack, battery form factor, or chemistry updates. The technology for each simply moves too fast to integrate at the pack level, much less the vehicle level.

As an example: the Model Year 2017 Bolt EV released with 238 miles of range, and the 2020 bumped that to 259, without changing the module or pack form factor. It's an incremental advance to be sure, but ~10% without adding modules is pretty dang good.

<comment><USER3> Things to consider.

BEVs are far simpler than regular cars. Particularly ones that aren't retrofitted on older chassis. There are simply far fewer parts to deal with.

Chemistry for batteries doesn't really matter all that much when thinking about compatibility. Do you care about the chemistry of any given AA? Probably not. Batteries and battery packs are right there in the same wheelhouse. Whether a chemistry gives you 1wH or 20wH really doesn't matter if the form factor and voltages are all the same.

Tesla packs are a good example of this. The chemistry has changed pretty drastically from 2012->today. Yet the pack form factor for Model S and X is the same. It only recently changed with the Model 3 (and they are still producing S and Xes).

<comment><USER2> > BEVs are far simpler than regular cars. Particularly ones that aren't > retrofitted on older chassis. There are simply far fewer parts to deal with.

Sure, but you've still got to figure out which parts to use and how to put them together. Automakers have over a century of calibrating ICE propulsion, so the incremental change in calibration for an ICE is very small. Calibrating EVs may be much easier, but the incremental change is huge for legacy automakers. As an aside, even Tesla cheerleaders should be happy that legacy automakers are entering the EV space. Competition is great for the consumer.

> Chemistry for batteries doesn't really matter all that much when thinking > about compatibility.

I mean, sure, all batteries will deliver "electricity", but engineering is all about balancing conflicting requirements. You want low volume, low weight, low risk of "thermal events", high energy capacity, high discharge rate, high charge rate, high charge cycle count, etc etc etc.

> Tesla packs are a good example of this. ... Yet the pack form factor for > Model S and X is the same.

I'd say this is a good example of the cost of engineering. On the one hand, if it ain't broke don't fix it. On the other hand, customers continue to demand better vehicles, especially as competitors enter the market.

<comment><USER3> I think you missed my point just a little.

It's true that if you designed a EV battery pack to be one solid anode and cathode, you might have to worry considerably about the chemistry that makes up the pack.

That, however, isn't how anyone is building packs (that I know of). EV manufacturers are all pushing for packs made of modules made of cells. The cells that they are making are quite literally a typical form factor (Typically 18650s)

Yes, you have to figure out how to put together your pack of 18650s and how to cool/heat them. However, what you don't have to worry about is what chemistry goes into them. That's an upstream supply concern. A 10% bump in storage can be gained with better chemistry, it doesn't require that you completely redesign your pack with each new chemistry. As a bonus, that also means you can source your cells from multiple manufactures, because they are all pushing out the same form factor.

<comment><USER4> > cells that they are making are quite literally a typical form factor > (Typically 18650s)

Typically cylindrical cells like 18650 or 21700 for Tesla, everyone else is using pouches or prismatic cells. Of course, based on sales the typical EV is a Tesla.

</comment></comment></comment></comment></comment><comment><USER5> Man, to think everyone in the industry said the Tesla gigafactory was a mistake or a waste.

<comment><USER6> The industry was right back in 2015-2016. The gigactory was supposed to have capacity 150Gwh/year by now. This is what they called a mistake and a waste. The current gigafactory operates at less than 40Gwh/year (as of August 2019), which is the reasonable quote. And a darker interpretation of this is that Tesla knowingly deceived the state about the size in order to be granted more incentives with no intention of ever achieving this size.

<comment><USER7> Source? From October 2014:

> The automaker says that by 2020, the factory will be shipping out 35GWh of > cells and 50GWh of packs per year.

[https://www.zdnet.com/article/panasonic-to-pour-billions- of-...](https://www.zdnet.com/article/panasonic-to-pour-billions-of-yen-in- teslas-gigafactory-as-initial-investment/)

<comment><USER8> There seems to be contradictions on the Wiki article about Gigafactory 1...

"Its projected capacity for 2018 is 50 (GWh)/yr of battery packs, and its final capacity upon completion was, as of May 2016, planned to be 150 GWh/yr of battery packs.[39] This would enable Tesla to produce 1,500,000 cars per year."

But after this quote, they failed to verify the initial numbers, so maybe there is some whitewashing of those facts?

</comment></comment></comment></comment><comment><USER9> Good. They make some interesting batteries with unusual specs.

Here's a 38 pound battery pack that can allegedly source 800 amps (briefly), with a nominal voltage of 60.8. That's crazy. You could run a car off of about three of these. It wouldn't go very far, but still. It's kind of the exact opposite of the Tesla approach, which is to use cells with low-ish power density, but to make up for it by using huge battery packs.

[https://www.evwest.com/catalog/product_info.php?cPath=4&prod...](https://www.evwest.com/catalog/product_info.php?cPath=4&products_id=481)

<comment><USER2> FYI, LG Chem makes the current batteries for the Chevy Bolt EV

<comment><USER9> Also, the Chrysler Pacifica hybrid. I'm not sure if those are the same cells used in the Bolt, but Pacifica battery packs are where a lot those LG chem batteries on the used market come from.

</comment></comment></comment><comment><USER10> There are battery factories in China coming online in the next couple of years that make the Gigafactory look small. I expect that batteries will follow a boom/bust cycle like DRAM does.

<comment><USER11> Depends on the timing with respect to market demand / production volume of EVs. As battery prices come down due to increased production / economies of scale, EV demand will go up so it _could_ be a smooth transition. It could also turn into a glut/bust scenario like you say, but I think we'll see the factories scale up production to meet demand rather than running at 100% from day 1.

<comment><USER12> Batteries have uses other than EVs, with a big one being power storage. There's a big push to reduce fossil fuel usage and get more power from renewables (solar/wind), but these aren't constant sources of power, so massive banks of batteries would be very useful here for utility-scale use of renewables without needing nuclear or fossil fuel plants for a baseline.

<comment><USER13> With V2G, EVs and power storage are one and the same.

<comment><USER14> the last time I checked the CTO of Tesla said that vehicle2grid had negative return on investment but that might have been politically motivated.

</comment></comment></comment></comment></comment><comment><USER15> [http://archive.is/85M0J](http://archive.is/85M0J)

</comment><comment><USER8> Does anyone know more details about what kind of batteries these are going to be?

<comment><USER5> Probably some version of the pouch cell type.

<comment><USER16> Yeah, that's what most cars using LG's batteries use, right?

Though they must have the cylindrical too, since LG is currently making the cells for the Model 3 in China before Tesla starts their own production there.

</comment></comment></comment><comment><USER17> What are the current avenues for us regular people to invest in companies doing battery R&D? I found the lgclf stock, but that doesn't seem right from my low knowledge level.

<comment><USER13> Invest in Tesla? They have made several acquisitions related to battery R&D and are likely moving to producing their own batteries.

<comment><USER17> That's one way, for sure. But that stock is already well known.

<comment><USER18> I'm not sure if it is still true but up until the past month or two Tesla was the most shorted stock in the market and you couldn't watch CNBC for 30 minutes w/o a talking head positing that it would go bankrupt before the year was out. Fast forward to today and every automaker is falling all over themselves to copy the Tesla model and Tesla has positive cash flow and $5.3 billion cash on hand.

<comment><USER19> Not the most shorted, but up there. It was #50 on a list back in August ([https://seekingalpha.com/article/4285222-heavily-shorted- sto...](https://seekingalpha.com/article/4285222-heavily-shorted-stocks- august-2019)). Today it has dropped off that list: its short ratio is still 20% but the bottom of that list is now at 24%. That 20% figure has been relatively constant.

Top of the list: GameStop, followed by Bed Bath & Beyond.

<comment><USER20> I love how almost every company on that list has a market cap around $1 billion, and then there is Tesla with a market cap over $40 billion.

</comment></comment></comment></comment></comment></comment><comment><USER21> Sorry, I don't believe it. I just sold my GM stock at 10% loss. I bought it thinking they were actually going to invest in EVs and self driving. Instead they have been pumping out trucks and SUV, while killing the Volt. Screw GM; the federal government should have let them sink.

<comment><USER8> One could argue they needed to continue making some trucks and SUVs until they have a positive margin on their EVs...They can't just shutter all their profit-makers immediately, how would they fund their EV future? What I DO agree about, though, is that they should have kept the Volt and they should have made the Bolt look less ugly.

<comment><USER22> > Bolt look less ugly

People I talk to say that it looks "tiny" and toy-like. Not so much ugly. Also, fun to drive.

Maybe they can beef up the look a bit -- this Hyundai Kona EV is the same size, but looks better (the SUV look may appeal to GM buyers): [https://www.consumerreports.org/hybrids-evs/2019-hyundai- kon...](https://www.consumerreports.org/hybrids-evs/2019-hyundai-kona-ev- first-drive-review/)

<comment><USER23> I test drove the Kona EV and it's quite a fun car. Hard to find one, though.</comment></comment></comment></comment>


Tomorrow's World – The Information Superhighway (1994) - <AUTHOR> https://www.youtube.com/watch?v=V8cnP-RtRHU <comment><AUTHOR> I remember well this hype back in the day, if not this actual piece (though a regular watcher of Tomorrow's World). Their analysis and predictions really hit the nail on the head.

The UK can claim to have been in on the Internet at the ground floor thanks to Tim Berners-Lee and their involvement in the pan-European CERN project.

Interesting reference to Net Neutrality at 2:50</comment>


Boxee Welcomes FCC Rulemaking That Opens Door for Innovation - <AUTHOR> http://blog.boxee.tv/2012/10/14/boxee-welcomes-fcc-rulemaking-that-opens-door-for-innovation/ <comment><USER2> So how about some details around the solutions, time frames, etc?

This was disappointingly lacking any actual info. Just some 'we are happy the FCC did something we like' empty words.</comment>


WindowsLies/BlockWindows - <AUTHOR> https://github.com/WindowsLies/BlockWindows <comment><USER2> It seems incredibly risky to block Microsoft hosts and stop updating Windows. I'd rather Microsoft abused my machine than an attacker.

I'm also curious: if someone has the privileges to run something like this, they're probably not under the thumb of an IT department, so why not switch to OS X or Linux?

<comment><USER3> Maybe this is suprising to the HN crowd, but there are many people who would never want to move to OSX from Windows (or Linux). I had to use OSX for two years, the only compelling part of it is the proper *nix shell, other than that, it was very unconvenient (and slow) for me (on a top-end MBP).

<comment><USER4> Agreed. I've used Windows since 3.1 (MS-DOS before that) and I bought a Macbook a few years ago to learn iOS development. I dabbled with Linux as a desktop OS for a few years too (but that's another story for another time...).

As a generally keyboard-centric user I've found OS X to be awful. Inconsistent behaviour when using Cmd+Tab (which switches to most recently used app) versus Cmd+` (which cycles through app windows instead) is frustrating to say the least. Finder is vastly inferior to Windows Explorer and I find myself dropping into the shell way too much just to do simple file management tasks (cut and paste anyone?). Being unable to tile, snap or maximise windows is also rather frustrating; yes, you can fullscreen an app and yes, you can drag and drop another window onto a fullscreen app [in El Capitan] to tile them side by side but that's so clunky it's almost a joke when on Windows I can hold down the Win key and use the arrow keys to snap windows to the screen edges.

I could go on, but I'll stop there...

</comment></comment></comment><comment><USER5> Does it install Linux?

<comment><USER6> That would be next version of this.

</comment></comment><comment><USER7> Hmm lets see... Autochk\Proxy (acproxy.dll) - Proxy Auto Config, also provides uPNP services for NAT traversal, could cause issues if you have a system proxy configured, some applications (mainly games) might also experience NAT traversal issues, IIRC Steamworks games like CS:GO/CS:S use it for NAT traversal.

Disables quite a few Media Center services including services that are needed if you are using your PC for HTPC or consuming some services Netflix's Silverlight player used to use ehDRMInit not sure if this is still the case.

It uninstalls several performance and compatibility updates, as well as updates to Windows Update itself which might actually prevent you from receiving updates properly.

Host block, well blocks MSN, bing, Skype and some other services which probably have nothing to do with Windows Telemetry, and worse has a script that actually modifies the routes on your machine which are going to be horrible to debug, not to mention outright stupid.

echo block www.microsoft.com route -p add 22.214.171.124/32 127.0.0.1

Seriously?

</comment><comment><USER8> Don't block Windows updates. Don't be retarded.

</comment><comment><USER9> Just stop using Windows. If you don't play PC games there is little reason to still be using it.

<comment><USER10> There is lots of Windows only software that people rely on other than games.

<comment><USER9> I'm more referring to the people that might look at this and say, "Oh, this looks great. Let me run this." If you have that much of a problem with Windows (as I do) just don't run it--not just a general comment on Windows.

</comment></comment></comment><comment><USER4> "M$"? Is the author 12?

<comment><USER6> Impossible. "M$" spelling is an artefact of at least 10-15 years ago. And author should have been at least 12 back then to pick this up.

Regardless, author does have a valid point here, despite somewhat puerile literary style.</comment></comment>


The United States of Venture Capital: The Most Active VC in Each State - <AUTHOR> https://www.cbinsights.com/blog/most-active-tech-venture-capital-firms-us-map/ <comment><USER2> It's pretty interesting to see what the biggest VCs are in states that aren't massive business centers. I didn't even know what to expect for my home state of Virginia.

Now I know what CIT does in that giant oddly shaped building near Dulles.</comment>


A history of the Amiga – The demo scene (2013) - <AUTHOR> http://arstechnica.com/gadgets/2013/04/a-history-of-the-amiga-part-9-the-demo-scene/

<comment><USER2> I remember this scene pretty well: I was a Mac guy in that era (i hated the rectangular pixels of the Amiga) but I had many Amiga friends (amigos) who would always show me these demos which blew me out of the water. Something to keep in mind was that these demos came out before Doom and many featured 3D graphics. The other cool thing about the Amiga was the ability to display 4000+ colors with the HAM chip (this was when you'd be overjoyed to get 256 colors on Mac II):

[http://en.wikipedia.org/wiki/Hold-And- Modify](http://en.wikipedia.org/wiki/Hold-And-Modify)</comment>