Using AWK and R to parse 25TB - <AUTHOR> https://livefreeordichotomize.com/2019/06/04/using_awk_and_r_to_parse_25tb/ <comment><USER3> There was a similar article (2014) that is also interesting. I think too many of us see new and shiny and immediately glom onto it, forgetting that the UNIX/regex fathers knew a thing or two about crunching data.

[https://adamdrake.com/command-line-tools-can- be-235x-faster-...](https://adamdrake.com/command-line-tools-can- be-235x-faster-than-your-hadoop-cluster.html)

<comment><USER4> > often people use Hadoop and other so-called Big Data ™ tools for real-world > processing and analysis jobs that can be done faster with simpler tools and > different techniques.

Right tool for the right job, as always. For a 2-3GB dataset size you don't need to bother with Hadoop just as for a 2-3PB dataset size you probably don't need to bother with awk.

<comment><USER5> If like to think that it is feasible most 2-3PB datasets can be easily partitioned to GB datasets. I rather guess it is more common to expand GB datasets into PB ones, though. :(

</comment></comment></comment><comment><USER6> I counted to 15 awk calls in our latest pipeline processing drug compounds to build predictive machine learning models of them: [https://github.com/pharmbio/ptp- project/blob/master/exp/2018...](https://github.com/pharmbio/ptp- project/blob/master/exp/20180426-wo-drugbank/wo_drugbank_wf.go)

One of the most eye-opening aspects of awk (goes for other pipeable commandline tools too), was how they support iterative development of pipelines regardless of data size.

We tried SQLite at some point for some of the stages because of pretty complicated selections sometimes, but it often won't give back a single result in minutes. Switching to AWK, I could immediately get _some_ output, so I could quickly validate and iterate on the awk code until I got what I expected. The actual execution was likeways always very fast.

</comment><comment><USER7> Awk is kinda my new favorite scripting language. I've been using it for the master's thesis I'm writing and am amazed at how quickly I can script in it. And how easy it was to learn. From the beginning out results-visualisation pipeline of awk+gnuplot was just supposed to be a quick hack to get something we could look at but it has kept up the whole thesis through and just been extended and made better instead of switching. We still use python when we need some lib help to get some data right but damn it goes quick to handle well structured data with awk. Sad I didn't learn it earlier.

</comment><comment><USER8> My biggest question is why on earth did they give you 25 Tb of TSV genetic data? :-)

I'm not sure what your sample was but seems like it would have been better to use one of the special binary file formats for genetic data. You wrote SNP chips, But in order to get to 25 Tb I assume there must be imputed calls, so it seems like a BGEN might have been a lot easier.

This is speculation of course, I'm not sure exactly what your situation was.

<comment><USER9> If you unpack all of [https://files.pushshift.io/reddit/comments/](https://files.pushshift.io/reddit/comments/) you have many Tb of JSONs that are just dumps of API responses that slowly change schema over the years. It's also an incredibly useful dataset.

In the end CPUs are fast enough and compression algorithms good enough that I would argue it doesn't really matter what format you use for storage, as long as it's reasonably easy to read back.

<comment><USER8> In the case of genomics, there have been at this point decades of work developing high performance file formats and there are large ecosystems of tools around them. Lots of bioinformatics is really manipulating these files. So using a supported file format makes a big difference.

</comment></comment></comment><comment><USER10> Hi! Author of the post here. I can attempt to answer any questions if need be although it looks like others have done a great job doing that already!

<comment><AUTHOR> Hi nstrayer thanks for the excellent article!

I work as a data analyst and I never got to worry about big data as the DWH takes care of the aggregation for us, plus I only work in Windows.

I see now that it would be very useful to learn *nix tools in general, as it seems that the skills to process (not to predict/analyze) terabytes+ data are very valuable and expensive to acquire and could be one's butter and bread.

<comment><USER11> Windows OS is making it easier to access nix terminal commands, but in my experience making the OS switch (mac machine but unix terminal, or cloud linux) has been game changing for me.

</comment></comment></comment><comment><USER10> As a followup to this. We have now successfully run complex statistical models across all 2.5 million snps on a single AWS instance in less than 3 hours just by writing R code using the package I describe at the end of the article.

<comment><USER12> Have you considered using AWS EC2 Spot Instances? The price can be 50 to 70% cheaper.

You would have to add some additional reliability to your pipeline so it could continue processing when instances are unexpectedly terminated. But this might be well worth it as it sounds like your research group is cost-constrained.

AWS made some changes this year so the spot prices are more stable and instances don't get shut down as frequently.

<comment><USER10> I did use spot instances for most of the clusters and a few of the processing jobs! I got out of the habit of using them earlier due to loosing them but now that they have the 'pay up to the on-demand price' option they're great!

</comment></comment></comment><comment><USER13> Have used Mawk [1] in similar cases for a runtime savings, provided that the script works without any GNU Awk extensions.

[1] [https://invisible-island.net/mawk/](https://invisible-island.net/mawk/)

<comment><USER14> Thank you! For those that don't know, 15 years ago GNU awk was sometimes oddly REALLY slow. Mike's awk was not. Plus when things were overall slower back then, it mattered more.

But there were bugs in mawk and it seemed basically unmaintained. So you'd run into something and have to use gawk or perl instead.

That's no longer the case, the xterm guy adopted it, ten years ago, and now I know!

<comment><USER13> Last release was late 2017, but it has been very stable for me. Plus the author responds to bug reports.

Mawk is my go-to version because of speed. GNU Awk when its extensions are needed, or the task is over "small data" and the system default version is sufficient.

</comment></comment></comment><comment><USER15> I remember frantically needing to parse a few hundred megs of log files and join them up to the db rows that fired the individual errors.

Initially I was trying to use SQLite for it but I kept running out of memory and crashing the system. Turned out using grep, join, sort, and paste got the job done in seconds.

</comment><comment><USER16> Very well written.

I think using 'make' with the -j parameter (# of parallel jobs) is more useful than using gnu 'parallel'. The reason is that if one of the job fails for some reason, you just re-run 'make' and only the required jobs are started instead of restarting the entire computation.

<comment><USER17> With --results or --joblog and --resume-failed GNU Parallel can do this, too.

</comment></comment><comment><USER18> Great post and thanks for sharing your learnings.

A couple of quick questions:

Was the 25TB raw data gathered from a single human genome?

What would be the size in bytes of a unique genomic fingerprint once raw data is all fully processed into high confidence base values? (including non-coding regions)

If we just look at coding regions and further compress by only looking at SNPs, how many bytes is that?

Considering that each base has ~2B of information... it would be super interesting to know how much space it takes to describe our uniqueness!

<comment><AUTHOR> Sorry I saw this article and thought it was pretty interesting. This is NOT my article but I'd like to know what others would do under this situation.

BTW not sure, but is it OK to post other's article here? Maybe I should add a short commentary in the title.

<comment><USER19> The default assumption is that it's not your article, unless you prepend "Show HN" (or there's something obvious like your username matching the domain name).

</comment></comment></comment><comment><USER20> Lol , this is basically how I roll most the time . However what Linux is really missing right now is command line tools that saturate a GPU.

Just counterparts to all the favourites that utilise the GPU ... imagine GPU awk.

<comment><USER21> > imagine GPU awk

My intuition tells me that awk and other text processing tools won't scale well to a GPGPU. I might be wrong though. Is there any example of something like grep etc working well on a GPU?

<comment><USER20> Pffff naysayers ...

[https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/c...](https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/competition/bkase.github.com/CUDA- grep/finalreport.html)

<comment><USER21> That's very surprising, assuming they didn't doctor the results by choosing the workload all too carefully.

I would have expected a GPU regex too perform much worse, given that regex matching is probably very branchy code. Especially since computation is generally way faster than IO.

<comment><USER22> Modern GPU have no issues with branching.

<comment><USER23> What specifically are you referring to? Branching on GPUs has not substantially changed for a decade. If all threads on warp skip a branch, it's free. If one takes it, the rest also pay the penalty and mask out the vector units.

What's at play here is that the needle in a haystack search of regex is going to spend almost all its time 0 or 1 deep in the state machine, so the threads skip the branches and the penalty is not large.

</comment></comment></comment></comment></comment></comment><comment><USER24> Here is an alternative solution, although yours is a good one.

If you only need a single SNP or a group of SNPs within a region, you can use tabix[1] to index gzipped TSVs and query by genomic position. The position of SNPs can be obtained from a lookup table (2.5M is not very big even for R) or from an API if you were to say - query by rsid.

tabix also works over http (and s3) and can utilize RANGE queries to select a subset of a file...so you only wind up downloading a or reading a small portion once it is indexed and can do something like this:

tabix [https://www.file.url.tsv](https://www.file.url.tsv) chr1:1-1000

The command above would return variants on chromosome 1 between 1 and 1000.

The following variant browser works in this way: [https://elegansvariation.org/data/browser/](https://elegansvariation.org/data/browser/) \- Theres no formal database (e.g. MySQL) running here, just tabix (actually bcftools which uses tabix) to select variants in a particular region, wrap them in JSON, and return to the client.

Setting this up on S3 requires configuring CORS... the igv browser also uses tabix indexes and provides guidance on how to set this up [2]

[1] [https://www.htslib.org/doc/tabix.html](https://www.htslib.org/doc/tabix.html) [2] [https://github.com/igvteam/igv.js/wiki/Data-Server- Requireme...](https://github.com/igvteam/igv.js/wiki/Data-Server-Requirements)

I created a similar solution to what you have done using this alternative approach, writing a wrapper in R that invoked bcftools under the hood. The dataset I was working with was a lot smaller (1.6M Snps x 252 individuals), but should work with larger genotype sets as well.

</comment><comment><USER25> I was onto a very similar idea recently! I used split to split a large CSV file and then used disk.frame ([https://github.com/xiaodaigh/disk.frame](https://github.com/xiaodaigh/disk.frame)), which is my package, to read in a large file. Not 25TB though!

</comment><comment><AUTHOR> I found this article particularly interesting as the author discusses a lot of (failed) methods.

Since I never dealt with big data before, I'm wondering what would you do in this situation?

<comment><USER26> It seems he had a lot of issues due to Spark executors failing which seems a setting issue. My guess is that the executors were being killed by the system OOM killer. Spark's memory management is counter-intuitive. Spark spills intelligently to disk so executors don't need a lot of memory to process data if you're not doing interactive queries. However spark will use all the memory it's given and sometimes it will use more than that (might be the OS actually, not sure). So the trick is to give Spark's executors LESS memory (as a percentage of the node's memory) so there's a buffer in case Spark uses more memory than allocated.

<comment><USER10> Pretty much. I am sure if I truly understood the inner workings of spark I would have been able to get it to work. I didn't go into it too much in the article but I did tweak the executor memory a lot. Going as far as transcribing the aws article on tuning into an R script that generated a config exactly as they stated. Also when I tried GLUE with its supposedly no- configure setup I still got the same problems.

<comment><USER26> Interesting. Another reason I can think for it failing is lack of disc space on the nodes. Spark will spill data to disc if it doesn't fit into memory and your nodes may not have had enough disc space for 25TB of data.

</comment></comment></comment></comment><comment><USER27> Unix pipelines, AWK, gnu parallel, R, all great stuff.

If you have such an specific task, why not just write an "actual program" (as opposed to pipeline of scripts)? From the looks of it, it sounds like this problem could have been solved with, say, 50 lines of Java, C, Go, etc, etc. Maybe a bit more verbose but it would give you full control, you wouldn't need to lookup how to use command line parameters on S/O, and would probably give you a bit more performance.

<comment><USER17> I have done that more than once. I often end up with a solution that works on the test set but which breaks after 10 TB just because <" "@example.com> is a valid email address according RFC-822 (Who the f __* thought it was a good idea to allow spaces in email addresses?). Or some other exception that was not part of the test set, and that was not identified before starting.

Dealing with exceptions is extremely error prone if these exceptions are not mapped beforehand. Thus it can be very costly.

Similarly doing stuff in parallel is extremely error prone due to race conditions: What does not happen when running on your 1 GB test set, may very well happen when running on your 25 TB production data.

<comment><USER27> I get your point but the same error handling problems can appear in scripts and pipelines, no?

In a program I'd try/catch defensively "just in case", if missing one line out of 25TB is not a bit deal.

For parallel processing I'd reach for the nearest standard library at hand on the language of choice.

<comment><USER17> > For parallel processing I'd reach for the nearest standard library at hand > on the language of choice.

That is a good example of what I mean: The nearest standard library is likely to either buffer output in memory or not buffer at all (in which case you can have the start of one line ending with another line). This means you cannot deal with output bigger than physical RAM. And your test set will often be so small that this problem will not show up.

GNU Parallel buffers on disk. It checks whether the disk runs full during a run and exits with a failure if that happens. It also removes the temporary files immediately, so if GNU Parallel is killed, you do not have to clean up any mess left behind.

You _could_ do all that yourself, but then we are not talking 50 lines of code. Parallelizing is _hard_ to get right for all the corner cases - <USER2> with a standard library.

And while you would not have to look up how to use command line parameters on S/O you _would_ be doing exactly the same for the standard libraries.

Assuming you can get better performance is also not given: GNU Sort has built- in parallel sorting. So you clearly would not want to use a standard non- parallelized sort.

Basically I see you have 2 choices: Built it yourself from libraries, or build it as a shell script from commands.

You would have to spend time understanding how to use the libraries and the commands in both cases, and you are limited by whatever the library or the command can do in both cases.

I agree that if you need tighter control than a shell script will give you, then you need to switch to another language.

<comment><USER27> I agree with everything you said, as always, everything is a trade off. Good point about trickiness of memory management w/parallel processing! Would have to be extra careful to avoid hoarding RAM.

</comment></comment></comment></comment></comment><comment><USER28> One note, this type of data isn't traditionally considered "sequencing" data, as it's a much simpler form of point measurements. Yes, you get the base call at a place of common variation, but "sequencing" is generally reserved for those cases where one gets extended sequences of bases, rather that just data from here and there.

</comment><comment><USER29> Welcome to Bioinformatics! Excellent write up!

</comment><comment><USER30> While I get the whole "you can do it on one node without all the complexity" thing, I do still wonder if map-reduce-synchronizer + coreutils is better than the behemoths that are the distributed ETL platforms right now. All the system would need to do is make a data file available on a node and capture stdout of the unix pipeline. I know gnu parallel does some of this.

</comment><comment><USER31> Just wondering if this problem could have been solved by a properly indexed table? The article says: "Eight minutes and 4+ terabytes of data queried later I had my results". 4+ TB seems way too much for 60k patients and sounds like an inefficient table scan was performed.

<comment><USER31> Also, wouldn't partitioning only make sense if there is a sensible way to separate data that is more likely to be accessed vs data less likely to be accessed? Like is common with date data, since recent entries are often more relevant compared to old entries. For example, you could categorize SNPs by priority and eg. partition SNPs of high importance (frequently accessed) vs medium importance (sometimes accessed) vs low importance (rarely accessed).

</comment></comment><comment><USER26> One correction to the article, snappy is not splittable however parquet files using snappy ARE splittable. Parquet compresses blocks of data within a file rather than compressing the file as a whole. Each block can then be read and decompressed independently.

</comment><comment><USER32> What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.

<comment><USER33> I am not kidding: A graphical terminal emulator.

Mastering the usual command line interface (terminal emulator, interactive shell, maybe a terminal multiplexer) is non-optional if you want to use CLI tools at or close to peak effectiveness.

<comment><USER32> I mean task resumption after interruption etc. Like airflow type of tools. Not quite unix task suspend options, this is about data pipelines. For Hadoop- style MapReduce, you can split the task into jobs which can be resumed and discarded etc. Shell scripting is not an elegant way to deal with this, a proper orchestrator tool is better.

<comment><USER34> You could try the tool my group builds, Arvados ([https://arvados.org](https://arvados.org)). We use Common Workflow Language (CWL) as the workflow language. Arvados works great for very large computations and data management at petabyte scale. It really shines in a production environment where data provenance is key. Intelligent handling of failures (which are inevitable at scale) is a key part of the Arvados design.

</comment></comment></comment></comment><comment><USER35> Can someone add "join" to the language/standard please. Its awkward to have split but not join.

<comment><USER36> There's a "join" tool in the GNU coreutils. "man join"

<comment><USER37> Somehow, I hadn't known of the existence of the `join` utility until this moment. I really should devote some time to play around with it and paste, sort, awk, etc.

</comment></comment></comment><comment><USER38> (my) Lesson Learned: don't use spark.

</comment><comment><USER39> You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

<comment><USER40> Furthermore, as computers get faster and cheaper in every dimension what makes economic sense to use "Big Data" tooling and efforts gets substantially larger with it. The limits of single nodes 15 years ago were pretty serious but most problems businesses have even in the so-called enterprise can currently easily fit on a workstation costing maybe $5k and be crunched through in a couple hours or maybe minutes - a lot easier to deal with than multiple Spark or Hana nodes. Operationalizing the analysis to more than a single group of users or problem is where things get more interesting but I've seen very, very few companies that have the business needs to necessitate all this stuff at scale - most business leaders still seem to treat analytics results in discrete blocks via monthly / weekly reports and seem quite content with reports and findings that take hours to run. Usually when some crunching takes days to run it's not because the processing itself takes a lot of CPU but because some ancient systems never intended to be used at that scale are the bottleneck or manual processes are still required so the critical path isn't being touched at all by investing more in modern tools.

I can support "misguided" Big Data projects from a political perspective if they help fund fixing the fundamental problems (similar to Agile consultants) that plague an organization, but most consultants are not going to do very well by suggesting going back and fixing something unrelated to their core value proposition itself. For example, if you hire a bunch of machine learning engineers and they all say "we need to spend months or even years cleaning up and tagging your completely unstructured data slop because nothing we have can work without clean data" that'll probably frustrate the people paying them $1MM+ / year each to get some results ASAP. The basics are missing by default and it's why the non-tech companies are falling further and further behind despite massive investments in technology - technology is not a silver bullet to crippling organizational and business problems (this is pretty much the TL;DR of 15+ years of "devops" for me at least).

<comment><USER41> > The basics are missing by default

Absolutely. I really want to see advanced AI/ML tools developed to address THIS problem. Don't make me solve the data before I use ML, give me ML to fix my data!

That's hard though, because data chaos is unbounded and computers are still dumb. I think there's still tons of room for improvement though.

<comment><USER40> I watched a talk by someone in the intelligence community space nearly 8 years ago talking about the data dirt that most companies and spy agencies are combing through and the kind of abstract research that will be necessary to turn that into something consumable by all the stuff that private sector seems to be selling and hyping. So I think the old guard big data folks collecting yottabytes of crap across the world and trying to make sense of it are well aware and may actually get to it sometime soon. My unsubstantiated fear is that we can't attack the data quality problem with any form of scale because we need a massive revolution that won't be funded by any VC or that nobody will try to tackle because it's too hard / not sexy - government funding is super bad and brain drain is a serious problem. In academia, who the heck gets a doctorate for advancements in cleaning up arbitrary data to feed into ML models when pumping out some more model and hyperparameter incremental improvements will get you a better chance of getting your papers through or employment? I'm sure plenty of companies would love to pay decent money to clean up data with lower cost labor than to have their highly paid ML scientists clean it up, so I'm completely mystified what's going on that we're not seeing massive investments here across disciplines and sectors. Is it like the climate change political problem of computing?

<comment><USER42> > In academia, who the heck gets a doctorate for advancements in cleaning up > arbitrary data to feed into ML models

Well - Alex Ratner [stanford], for one: [https://ajratner.github.io/](https://ajratner.github.io/)

And several of Chris Re's other students have as well: [https://cs.stanford.edu/~chrismre/](https://cs.stanford.edu/~chrismre/)

Trifacta is Joseph Hellerstein's [berkeley] startup for data wrangling: [https://www.trifacta.com/](https://www.trifacta.com/)

Sanjay Krishnan [berkeley]: [http://sanjayk.io/](http://sanjayk.io/)

<comment><USER40> I was asking somewhat rhetorically but am glad to see that there's some serious efforts going into weak supervision. At the risk of goalpost moving, I am curious who besides those in the Bay Area at the cutting edge are working on this pervasive problem? My more substantive point is that given the massive data quality problem among the ML community I would expect these researchers to be superhero class but why aren't they?

<comment><USER42> ... they are?

There are a lot of people tackling bits and pieces of the problem. Tom Mitchell's NELL project was an early one, using the web in all its messy glory...[http://rtw.ml.cmu.edu/rtw/](http://rtw.ml.cmu.edu/rtw/)

Lots of other folks here (CMU). Particularly if you add an active learning. Hard messy problem that crosses databases and ML.</comment></comment></comment></comment></comment></comment></comment>


Elon Musk calls for Amazon to be broken up: 'Monopolies are wrong ' - jor-el https://thehill.com/policy/technology/501273-elon-musk-calls-for-amazon-to-be-broken-up-monopolies-are-wrong <comment><AUTHOR> Well dang, when Musk said he'd be off Twitter for "a while", he only meant three days.

I fail to see which market Amazon monopolizes. Certainly not book retail, which is what triggered this comment.

<comment><USER2> This might have something to do with the impetus: [https://www.forbes.com/sites/alanohnsman/2020/05/26/amazon-s...](https://www.forbes.com/sites/alanohnsman/2020/05/26/amazon- said-to-be-in-talks-to-buy-self-driving-startup-zoox/#6335cc9370c8) If amazon enters the autonomous driving space is bad for Tesla</comment></comment>


Apple reportedly considering buying Beats Electronics for $3.2 billion - <AUTHOR> http://www.theverge.com/2014/5/8/5696652/apple-reportedly-considering-buying-beats-electronics-for-3-2-billion <comment><USER2> Discussion: [https://news.ycombinator.com/item?id=7717961](https://news.ycombinator.com/item?id=7717961)

Other submissions, some have up-votes, some have comments:

[https://news.ycombinator.com/item?id=7718106](https://news.ycombinator.com/item?id=7718106)

[https://news.ycombinator.com/item?id=7718020](https://news.ycombinator.com/item?id=7718020)

[https://news.ycombinator.com/item?id=7717995](https://news.ycombinator.com/item?id=7717995)

</comment><comment><USER3> This would be Apple's largest acquisition in company history. What would Beats in-house at Apple provide that's compelling?

<comment><USER4> That's the interesting question. Because the answer is: nothing.</comment></comment>


Dominate Dragons with Git - <AUTHOR> http://tech.hulu.com/blog/2012/11/26/dominate-dragons-with-git/#

<comment><USER2> This is a remarkably accessible way of explaining source code control to people.</comment>


The human imperative of stabilizing global climate change at 1.5°C - <AUTHOR> https://science.sciencemag.org/content/365/6459/eaaw6974 <comment><USER2> The issue has become heavily politicised and even worse, is now a fault line in identity politics. This is the death sentence for meaningful action. If only publishing journal articles was enough.

I'm not sure that we are hoping for the right category of solution. All countries and people in the world uniting and making sacrifices... has this ever happened for anything ever? I am hoping there are examples, I just can't think of any. Maybe I'm overly pessimistic, but I just can't see the 'Unified Earth' solution happening, at least not because everyone just agrees to be that way.

Maybe the right analogy is something like the internet. If you were concerned about tyranny and oppression of free speech in the 1930s, it is doubtful you would ever conceive that the internet could be one 'solution' to that problem. It isn't a perfect or complete solution, and it actually makes the problem worse in some ways, but nevertheless, on that particular issue, the internet has changed the landscape completely. Maybe this category of solution is the best we can hope for. The trouble is that it is very hard to conceive of how it will look or what we should do.

<comment><USER3> People often talk about sacrifices, but we don't even make much progress on the stuff that's actively harmful and therefore a net benefit, like phasing out coal for cleaner energy, or building well insulated homes so I don't think that's the whole reason. In fact it seems like a talking point from one side.

</comment></comment><comment><USER4> We are halfway to the first doubling of co2 right now. 280 is preindustrial, 560 is the first doubling, 410 is where we are. Now emissions are rising exponentially. If things turned around on a dime and emissions fell as fast as they went up, then the total co2 would be about 1 doubling. If the climate sensitivity is 3 degrees, then that's 3 degrees of warming right there. Holding it to 1.5 or 2 would require emissions to go down substantially faster than that.

But my money is on nothing getting done, and emissions continuing to rise exponentially for the next 20 years at least. That means probably 6 degrees or more of warming.

I will fight to try to stop that from happening.. but that's where we're at.

The real problem is that the worlds fossil fuel deposits and infrastructure represents at least 6 trillion dollars worth of someone's private property, and they will fight to the end to make sure that that investment pays out. That means using _all_ the worlds fossil fuel deposits, no matter the cost to society.

</comment><comment><USER5> Not gonna happen, unfortunately.

I agree that achieving those goals, even at the envisioned costs, would be desirable. But the short term pain of much of the necessary measures would be too great to make it politically infeasible. And any "defector" of significant size (China? Russia? US?) would make it even more difficult, politically.

The developing nations will feel a lot of the pressure to cut down on their growing emissions. Which is unfair and would also result in more Human suffering, short and midterm.

I'm hoping for somebody to prove me wrong!

<comment><USER6> Or more realistically business will go on as usual, with a little green- washing here and there, and we will cope as well as we can with the resulting mess. Which means after a period of relative calm (less wars nowadays than before) we'll probably have climate/water exodus and wars in the future. But this will happen gradually.

<comment><USER7> My hope is all in climate engineering. Not because it's the best solution but because it's the only politically viable one. Not until we've felt the pain will we invest in solving the problem. Governments also have an easier time investing in massive infrastructure projects than curbing pollution.

Will will be able to do anything at that point is not certain.

<comment><USER5> At the moment the traditional ways of fighting carbon emissions seem to be more practicable...

And who is going to pay for those infrastructure? Are the rich countries paying for them to be build in the poorer countries?</comment></comment></comment></comment>


Windows disk space requirements (log-log scale) - <AUTHOR> https://wh0.github.io/2018/03/15/windows-size.html <comment><USER2> Ugh, I felt an itch because of the x-axis ordering. Obviously it jokingly skips the XP, Me and NT builds because x-axis is only numeric.

<comment><USER3> They do have numeric versions too. This only has XP onwards but proves the point. [https://docs.microsoft.com/en- us/windows/win32/sysinfo/opera...](https://docs.microsoft.com/en- us/windows/win32/sysinfo/operating-system-version)

<comment><USER4> Windows 10 was 6.4 for a time briefly too.

</comment></comment></comment><comment><USER5> Does anyone know exactly WTF happened after XP to make the disk space requirements balloon so much, so fast? I know they started including more drivers, but, I mean, so does Linux and you can go pretty nuts with driver modules there and still not move the needle much on disk space requirements. Certainly not to the tune of gigabytes. New features _definitely_ don't seem to justify the increase. Even supporting both 32 and 64 bit, that still doesn't come close to justifying it. Combine Win2k and Win98, double that for no real reason, double it again for 32 and 64 bit, and you still don't touch Win7.

<comment><USER6> For compatibility purposes, WinSxS basically contains every version of every DLL ever shipped with Windows from 95 to present day.

<comment><USER7> If only they also included the old UIs...

<comment><USER8> You can make Win7 look like Win95 by turning off all eye candy. Still usable today if that's your thing.

<comment><USER4> Windows 7 is about to be EOL'd. You can enable the classic theme on 10 with some security descriptor shenanigans [http://winclassic.boards.net/thread/413/reversibly-enable- di...](http://winclassic.boards.net/thread/413/reversibly-enable-disable- classic-powershell)

<comment><USER7> Do/have you use(d) this? Any screencaps? :) No classic theme was one of the deal breakers from me going from 7 to 10.

Worth mentioning here that XP support only really ended last April (POSReady).

7 is supported at least 3 more years:

> Windows 7 ESU include security updates for critical and important issues as > defined by Microsoft Security Response Center (MSRC) for a maximum of three > years after January 14, 2020.

[https://support.microsoft.com/en-us/help/4527878/faq- about-e...](https://support.microsoft.com/en-us/help/4527878/faq-about- extended-security-updates-for-windows-7)

But MS really doesn't want you to use Windows 7, unless you are an "IT Professional" :)

</comment></comment></comment></comment></comment></comment><comment><USER9> I know this is a joke but Windows has build versions which might work better than the number marketing name.

<comment><USER10> They'd work better, but wouldn't be perfect either since they had some arbitrary jumps. For a long time, release build version numbers were divisible by 400 - they had to be divisible by 16 for servicing reasons[1], and divisible for 100 for marketing reasons (to look nice) [2] so build numbers were divisible by 400 for a long time.

This meant that the final versions had some correlation to how much time had passed between versions, but it wasn't reliable. For instance, Win7 RTMed on 2009-10-22 as build 7600. Win8 RTMed on 2012-08-01. At one build a day, you'd expect a build number of 8614....and internally, we were pretty close to that, but the actual release number was 9200 (so we instantly gained almost 2 years of build numbers). For the story of why 9200, see [2].

They eventually got rid of both requirements, which is why the latest Windows release is 18363.

[1]: [https://devblogs.microsoft.com/oldnewthing/20160330-00/?p=93...](https://devblogs.microsoft.com/oldnewthing/20160330-00/?p=93223) [2]: [https://docs.microsoft.com/en-us/previous- versions/technet-m...](https://docs.microsoft.com/en-us/previous- versions/technet-magazine/jj663498\(v=msdn.10\)?redirectedfrom=MSDN)

</comment></comment><comment><USER11> I have enterprise windows on a domain joined machine and literally gigabytes of data are devoted to shadow versions of games. Does anyone have a solution for this? Why is this happening on Enterprise?

[https://imgur.com/PsFySH9](https://imgur.com/PsFySH9)

Windows has been a compliance nightmare with a small team.

Edit: I can't delete these to be clear, even with a local admin and domain admin account. I saw some solutions online but I need something I can have an admin create a script and/or gpo for.

<comment><USER12> I've been dealing with that garbage since migrating my office to Windows 10. Each update, things would reinstall. Or scripts I used to automated uninstallation stopped working because they just change the game roster. Or various GPOs I used no longer worked if you merely had Windows 10 Pro, so I had to shell out for new Enterprise licenses.

I've really liked some things MS is doing recently, like VS Code. But the Windows team--I just don't understand what they're doing. They're treating an OS--an OS that's frequently used for serious business purposes and ought to be predictable and stable--like some startuppy web app that changes everyday according to the latest A/B test. It's infuriating.

<comment><USER13> > But the Windows team--I just don't understand what they're doing. They're > treating an OS--an OS that's frequently used for serious business purposes > and ought to be predictable and stable--like some startuppy web app that > changes everyday according to the latest A/B test. It's infuriating.

And there's your answer. Microsoft has become infected with these kind of people and they're fucking up Windows 10. The situation hasn't been fixed because attention is on more profitable portions of the business like Azure.

</comment></comment></comment><comment><USER14> Did he really treat the "x" in "x64" as a multiplication operator?

<comment><USER15> [heart-eyes emoji]

</comment></comment><comment><USER16> On a more serious note, I wonder what the size of Windows looks like when compared to the amount of storage you could buy for $200 at each point

<comment><USER17> The price of storage has pretty much plummeted compared to the size of Windows. A quick glance at a magazine shows that in 1987, a few years before Windows 3.0, a 20MB hard drive was about $300 (in 1987 dollars).

<comment><USER18> It depends on what your comparing.

Win 3.0 came out March 1990, which puts HDD at 4$/MB or 50MB for 200$. [https://jcmit.net/diskprice.htm](https://jcmit.net/diskprice.htm) So 6MB win 3.0 should be ~12% of a hypothetical 200$ disk.

Now HDD prices dropped a lot, but high end PC's used SSD's in 2009. A 120GB SSD cost ~250$ when windows 7 came out, which is ~13% of the drive.

Inflation pushes these further apart, but $200 in 1990 is only $324.15 in 2009.

<comment><USER17> Fair enough. Though that's probably on the early side for most people to have put an SSD in their computer. (Inflation also means the 1990 number is also about 50% higher relative to 2009.)

TBH, I probably spend as much on disk drives as I ever did but that's because of the redundancy I can afford, all the media, and because I just leave a lot of stuff laying around because it's easier than pruning.

</comment></comment></comment></comment><comment><USER19> > My last post was about either The SSH protocol or The reason you need to > install a karaoke captioning library if you want to change your desktop > wallpaper.

I was really hoping for the latter.

<comment><USER20> I totally believed that was real until I clicked through to some of the other last-post links.

> The reason you need to install a karaoke captioning library if you want to > change your desktop wallpaper

> Why using a can opener backwards doesn't close the can back up

> Best haircuts for Wi-Fi reception

> Losing weight in 14 minutes a day with PostgreSQL

</comment></comment><comment><USER21> Not sure what to make of this since my gaming machine that I sometimes use for programming has the C:\Windows directory at ~40GB. This has been the case ever since I installed the Win10 Beta somewhere in 2015 (okay, it started at about 25GB but grew pretty quickly, over the course of just a few months). I run both variants of the disk cleaner (user and system) after every update.

Sure the PC is like 9 years old at this point but has seen only a minor amount of external plug-and-play hardware. I can appreciate all this bloat including all drivers for all well-known hardware but _damn_ , that's 40GB still.

</comment><comment><USER22> Let's make a project to trim Win10 to 1G of disk usage.

<comment><USER23> Back in Windows 98 days there was 98lite project, that allowed to customize Windows 98 components, even replace explorer.exe with one from Windows 95. Smallest version was just under 50MB. I wonder how much Windows 10 can be shrunk? And probably hardest thing of all - keep updates working after that.

<comment><USER24> You've got a few responses spelling out a few ways it can be done.

Almost all of them are based off something called "WinPE" or the Windows Preinstallation Environment. [0]

This is a similar OS, used when installing your OS. It usually shares the same kernel, but with everything stripped away, including useful things like the desktop, etc. You can add some, but not all, of these things back.

As WinPE uses the Fat32 file system by default (you can change it with some difficulty), most of these installations will be less than 4GB.

Building a WinPE-based environment is a fun weekend project. Totally against the license agreement to make it into a useable OS, but that shouldn't matter if you never share it and are just using it to get to grips with how WinPE works.

[0] [https://docs.microsoft.com/en-us/windows- hardware/manufactur...](https://docs.microsoft.com/en-us/windows- hardware/manufacture/desktop/winpe-intro)

<comment><USER25> Microsoft also today has containers and images that they themselves streamline as much as possible, for certain use cases. The two most common I've seen referenced:

1\. Windows IoT Core [0] designed for IoT devices and embedded scenarios. The RPi build of Windows 10 IoT Core is around 800 MBs. (The Raspberry Pi build is an interesting one to note because you can explore yourself easily in a weekend project.)

2\. Windows Nano Server [1] is a fascinating Windows 10 build intended for among other things Docker containerization [2] and virtualization projects. I've heard some images of Nano server are as small as about 400 MBs. Given its intent for containerization/virtualization, it's pretty amenable to experimenting with in weekend projects, too.

Microsoft has seemed quite interested in the last few years with how small and modular they can make Windows under the hood. It's not always apparent at the consumer level because of backward compatibility and all of the required "modules" (like Win32) that folks think of "as Windows".

[0] [https://www.microsoft.com/en-us/software- download/windows10i...](https://www.microsoft.com/en-us/software- download/windows10iotcore#)!

[1] [https://docs.microsoft.com/en-us/windows-server/get- started/...](https://docs.microsoft.com/en-us/windows-server/get- started/getting-started-with-nano-server)

[2] [https://hub.docker.com/_/microsoft-windows- nanoserver](https://hub.docker.com/_/microsoft-windows-nanoserver)

</comment></comment></comment></comment><comment><USER26> Funny, but it still would be more interesting to see the same chart with an x axis making more sense. Windows 2000 is just a brand name, it's actual version number is 5.something.

</comment><comment><USER27> Why only 7 has separate x64 point?

<comment><USER28> Right? And Why has Windows 7 got a higher disk space requirement than Windows 8.x (the only time it _actually_ went down over time)

<comment><USER29> Part of the Windows 8 (8.1 actually) objectives was to reduce the memory and disk footprint so that cheap tablets could be viable. With WIMBOOT enabled a full 8.1 install could fit in slightly over 3 GB on certain devices (the evolution of WIMBOOT is automativally enabled in 10 when the installer sees fit)

<comment><USER30> Is there anyway to force it to enable?

<comment><USER29> Looks like it is, but it's not straightforward as a checkbox during installation: [https://docs.microsoft.com/it-it/windows- hardware/manufactur...](https://docs.microsoft.com/it-it/windows- hardware/manufacture/desktop/compact-os?redirectedfrom=MSDN)

</comment></comment></comment></comment></comment><comment><USER31> Anyone else notice that this is starting to form a letter 'M'? Perhaps, with the correct versioning scheme[0] and appropriate install size fluctuations, they could write out the company name.

[0] Almost definitely _not_ semver.

</comment><comment><USER32> Somewhat related to a recently shared post about software bloat: [https://news.ycombinator.com/item?id=21929709](https://news.ycombinator.com/item?id=21929709)

</comment><comment><USER33> This would actually be interesting if the author wasn't trying to be funny.

<comment><USER34> I'm still trying to understand if the author is making fun of Windows or making fun of himself.

<comment><USER35> Maybe they're making fun of you for overthinking it.

</comment></comment></comment><comment><USER36> Looks like the log log size's an upside down W for Windows.

</comment><comment><USER37> That X-axis is obnoxiously useless.

</comment><comment><USER38> (2018)

</comment><comment><USER39> I distinctly remember fresh install of Win 95 taking 73 MiB of disk space (definitely between 70 and 79 MiB; with a 7 in the front). So I'm not sure where the 50 MiB number comes from.

Anyone with me on this? (not that it's a huge difference, but still).

ps: Also horrible chart due to x-axis. No offense.

<comment><USER40> 50 is the minimum. It could vary quite a bit depending on which features you chose to install.

<comment><USER41> And you could still get win95(a) below 30 if you cleaned up manually. I remember that my setup was 27 and some (small disk, needed space for linux...)</comment></comment></comment>


Gog.com giving away Dungeon Keeper for free - <AUTHOR> http://www.gog.com/

<comment><USER2> Whatever you do, do not let your enjoyment of Dungeon Keeper tempt you into even installing Dungeon Keeper Mobile: [http://www.youtube.com/watch?v=GpdoBwezFVA](http://www.youtube.com/watch?v=GpdoBwezFVA)

It's the new poster child for the suckage that is Free to Play games with In App Purchases.

<comment><USER3> I'm curious where most HNers would draw the line between Doom's "Get 3 levels for free, pay us $50 for the rest" and something like Candy Crush's "Play the whole game for free, pay us $x for any x to get more out of it."

I'm playing a bit of devil's advocate here, because I am very much against in- app purchases. But, when you think about it, you can play Dungeon Keeper Mobile for _free_ , or you can pay something like $20 (or $10, or $5, maybe even $1) to get more out of it (boosts, etc).

So where does the issue lie? I here a lot of people saying "Well they reel you in with a free app and then make you pay for it!" but isn't that exactly how shareware works?

<comment><USER2> There's a fundamental difference between shareware / free trial apps and apps that require constant in-app purchases to... well... play. Doom and others use the free trial model, which is what we used to call shareware (before the internet, they wanted people to share the free trial with each other to inspire them to then go buy the full version). As long as these are properly listed as such, there's nothing sleazy about them at all. And it's fine to use an in-app purchase to go from trial to full version. Again, as long as its clear to the user that this is the way it works.

Games that have in-app purchases run the spectrum. Some use in-app purchases to do customizations and other fun things that don't affect gameplay. Then there are those like Dungeon Keeper which are basically unplayable without in- app purchases. Watch the video I linked above for details on why. You have to pay over and over and over just to keep the game playable.

This is very different from a shareware/free trial model. You pay ONE TIME and you own the whole game. It's a classic commercial game scenario except they give you a taste of it... a trial... for free. It's the same thing with 'lite' versions of commercial software that are also free.

The in-app purchasing model is more akin to getting a free version of an FTP client that only allowed you to upload software at 10KB/s. You can then pay $0.99 to upload that one file at full speed. But, your very next upload would again be 10KB/s unless you also paid $0.99 to upload that one at full speed. But wait, for an even better value, you can buy a 20-pack of full-speed uploads for only $9.99 or a 100-pack of full-speed uploads for only $29.99. Once those run out, of course, you're back to 10KB/s. Unless you spend even more money.

Now, it seems silly with the FTP client example above. Who would ever use that software? But this is exactly how many of the free to play games work. And this is the way a lot of people coming in to gaming for the first time think games work.

<comment><USER3> Great explanations, thanks. I think the biggest difference is that some games (like Dungeon Keeper Mobile IIRC) have IAP into perpetuity - meaning you can essentially _never_ unlock the full game. I like your FTP client example, I'll be using that.

</comment></comment></comment></comment><comment><USER4> I've always loved GOG's value proposition: DRM-free, old-school exe installers on _your_ disk, no activation needed, no limit on the number of installs, can redownload at any time with all associated assets.

No confusion over who actually owns the games. You buy a copy of a game, not some ill-defined permission to play a game.

I'm just concerned these constant promotions will condition their customers to always wait for the next massive sale, never paying full price for anything. Isn't that unsustainable?

<comment><USER5> I think it's sustainable, it's not exclusively about getting the games cheaply it's also about removing the consequences for buying games you don't like. If I buy a game on GOG or HumbleBundle and never play it I don't feel any remorse, and I do it again and again and again.

</comment></comment><comment><USER6> I did not like Dungeon Keeper first time round. The optimal strategy is turtle, making missions 3 hour affairs. Furthermore, you get to keep your best monster in the next level, so you need to mine every map to death if min/maxing. Multiplayer was very wonky strategically too. The outcome was decided in the first 10 mins based on whether you could out recruit monsters, but the death blow could not be dealt until about an hour later :s

The actual concept was pretty amazing though. It certainly stuck in my mind for one of the coolest concepts for a game. Playing the baddie is always fun. Just the actual dynamics of the game were pretty flawed. I bought Dungeon Keeper 2 though anyway, haha.

<comment><USER7> In the days when I had too much time on my hands, I used the editor to pit every combination of monster against each other in the arenas. The balancing was way-off, and monster's fighting abilities are inconsistent with their description in the manual.

For instance, dragons, supposedly tough hombres, are singularly useless. No matter how many spells they learn, they only ever use their puny fire breath, presumably due to a bug.

Your best troops are orcs by a large margin, followed by mistresses. At high levels, orcs can kill everything except the horned reaper in single combat, and are incredibly cheap to train (1/10 the expense of reapers, and 1/2 of mistresses). Build an army of these guys as soon as you can and you're sorted.

It's a great concept, but for this sort of game, I thought Bullfrog's Populous: the Beginning, was infinitely superior. Get hold of that in GOG! It plays beautifully under WINE (even better than on Windows, and not arbitrarily limited to a maximum 800x600 resolution).

<comment><USER6> That's really interesting.

Powermonger I liked better than populous. Populous was too many levels and maps were too big IMHO. I never understood the ecenomic strategy (build big houses or small ?) but I was quite young (small then big I spose).

<comment><USER7> Ah, you want Populous: The Beginning, not the original Populous.

The original Populous is a simulation. Populous: The Beginning is a fully 3D real-time strategy game, where you control individual units.

<comment><USER6> AHhhhh. I'll check it out!

</comment></comment></comment></comment></comment><comment><USER8> They've also got Temple of Elemental Evil at 80% off at the moment. That's one of the most under-appreciated D&D games I've ever played. The turn-based combat is rich and interesting without dragging to a crawl. Unfortunately it's marred by bugs, though the unofficial patches alleviate most (but not quite all) of that problem. It's the closest to a true pencil and paper experience I've ever had on the computer.

<comment><USER9> For those interested note that ToEE, while good, is about combat and combat only. The story and npc are not interesting (compared to the other games in that sale).

</comment></comment><comment><USER10> And Dungeon Keeper 2 [1] discounted to $1.49 which is a bargain :)

[1] [http://www.gog.com/game/dungeon_keeper_2](http://www.gog.com/game/dungeon_keeper_2)

<comment><USER11> Shame it's Windows only :(

<comment><USER12> Dungeon Keeper 2 works fine on Linux using Wine: [http://appdb.winehq.org/objectManager.php?sClass=version&iId...](http://appdb.winehq.org/objectManager.php?sClass=version&iId=26254)

Dungeon Keeper 1 works fine on Linux using Wine: [http://appdb.winehq.org/objectManager.php?sClass=version&iId...](http://appdb.winehq.org/objectManager.php?sClass=version&iId=28495)

<comment><USER13> What about OS X?

<comment><USER14> Crossover.

</comment></comment></comment></comment></comment><comment><USER15> Hm, I got a "You'll get an e-mail when your copy is ready", something about supply depleted or so. Can't even visit the game link[1] anymore. Guess it's in high demand? =)

[1] [http://www.gog.com/game/dungeon_keeper](http://www.gog.com/game/dungeon_keeper)

<comment><USER16> Might have been a burp; I got that e-mail right away, and downloaded a couple of them. :)

<comment><USER17> I got the eMail straight away, with some great free games I wasn't even expecting.

Ultima 4, DragonSphere, Treasure Hunter...

But as yet no Dungeons for me :-)

<comment><USER18> Yeah I got that first miscellaneous-free-games email immediately; got the Dungeon Keeper email about 90 minutes later.

</comment></comment></comment></comment><comment><USER19> And this is true free to play Dungeon Keeper, not the EA abomination on mobile.

<comment><USER20> What a brilliant stunt this is. I only heard about Dungeon Keeper in the backlash to what EA is doing, now I am curiuos and GOG comes in to scoop up some good will. Nice.

<comment><USER21> Looks like EA is directly involved in this[1]. A less charitable person would say it's their attempt to save face after their gaming media drubbing over their money-sink mobile version.

[1] [http://www.reddit.com/r/Games/comments/1xw0kw/dungeon_keeper...](http://www.reddit.com/r/Games/comments/1xw0kw/dungeon_keeper_is_free_on_gog_for_next_48_hours/cff50bh)

<comment><USER22> I seriously doubt it. I am having difficulty imagining EA giving away anything for free and without DRM.

<comment><USER21> Well the alternative is that GOG is lying, since one of their verified employees said this in the linked thread.

<comment><USER22> Given my biased and prejudicial opinion of EA, I haven't ruled that out as a possibility.

On the other hand, EA is very large, so I also cannot rule out the possibility that its left hand is facepalming even while its right hand is giving all gamers everywhere the longest, fattest middle finger yet seen in the world of microtransaction-driven gaming.

</comment></comment></comment></comment></comment></comment><comment><USER23> I've spent quite some money on GOG but haven't really played many of the games. DRM-free is great though and I'm just snatching up stuff on sale here and there.

I enjoy retroish games and have a pipeline of entertainment for years to come lined up so that's good :)

Edit: Just checked, my collection contains 137 games

<comment><USER24> I've done similar with Steam seasonal sales and HumbleBundle's. I've got pile of games just waiting for the arrival of the mythical "spare time".

</comment></comment><comment><USER12> It would be great if GOG would provide also the Win32 version (which works with minor glitches on modern Windows).

GOG provides the DOS version that runs using Dosbox and feels a bit slower.

Btw. to increase the resolution make it fullscreen with Alt+Enter and then press Strg+G to double the output.

Hint: to rotate the map, press Del / PageDwn. To zoom press Pos1 / End.

<comment><USER7> I'm pretty sure the D3D versions are included, but by default it fires up DosBox. Good luck getting them to work on the latest windows though. I remember having issues getting them to work even on XP.

<comment><USER12> It's only the DOS version, as I said.

The original CD back in 1997 contained both the DOS and Win32 version. The DOS version has a lower resultion and runs slower in Dosbox even on a high-end PC.

Nevertheless, I love the game and had played it the last few days, one level per night ;)

</comment></comment></comment><comment><USER25> Dungeon Keeper is the only game I've ever enjoyed in a localized, swedish voice version. It's really very well produced.

<comment><USER12> Yes, true. The german voice-over is in my opinion even better than the good original english one.

GOG provides only the English edition, good that I have still the orginal german CD edition.

</comment></comment><comment><USER26> I had some great fun playing DK and it's sequel years ago. What amazes me is that there hasn't been any A-title clones made of it in all these years. Closest I've seen is the indie title War of the Overlord, still in development though.

</comment><comment><USER27> GOG has been pulling off some fantastic marketing lately. Good for them.

</comment><comment><USER28> Between this and De La Soul, looks like my entire weekend's entertainment is going to be free..!

</comment><comment><USER29> I've been watching DK on gog for a long time, thank you so much for saving me some bucks!

</comment><comment><USER30> Good to know! brb, telling this to my friends. All of them. Which is 3.

</comment><comment><USER31> Some classics in that list.</comment>


'AI is very, very stupid,' says Google's AI leader, at least compared to humans - <AUTHOR> https://www.cnet.com/news/ai-is-very-stupid-says-google-ai-leader-compared-to-humans/ <comment><USER2> Yes, the AI is stupid, but a computer machine is much faster than figuring it out manually, and sometimes that is what is help.</comment>


Google Details Electricity Usage of Its Data Centers - <AUTHOR> https://www.nytimes.com/2011/09/09/technology/google-details-electricity-output-of-its-data-centers.html?hp <comment><AUTHOR> >> Google also released an estimate that an average search uses 0.3 watt-hours of electricity, a figure that may be difficult for many people to understand intuitively. But when multiplied by Google's estimate of more than a billion searches a day, the figure yields a somewhat surprising result: approximately 12.5 million watts of Google's 260 million watt total can be accounted for by searches, the company's bread-and-butter service</comment>


Ask HN: Is there a better period of the year to apply for a H1B position? - <AUTHOR> The US Government opens the program for applications on April and then it reaches the quota usually in a few days. Does that mean that if I apply for a job in the US in May (and I need a H1B visa) I'm out of luck? Do I need to hurry and apply for a job let's say before March? <comment><USER2> Yes, if you need a new H1B visa (not a transfer), best to have an offer before April with enough time for the lawyers to apply for your visa. After that companies will have to wait until next year October for you to start with them in US.

Some companies like Google and Facebook might send you to a non US location until you get an H1B.

<comment><USER3> Does this apply for E3 too?

<comment><USER4> No, H1B visa has a lottery in April, which E3 doesn't have as far as I know.</comment></comment></comment>


TSA: Some on US-bound flights must turn on phones - <AUTHOR> http://bigstory.ap.org/article/tsa-some-us-bound-airports-must-turn-phones <comment><USER2> _" TSA will not disclose which airports will be conducting the additional screening, although it will be at some airports with direct flights to the U.S. Industry data show that more than 250 foreign airports offer nonstop service to the U.S."_

Prediction: after some time, they will just roll out this policy everywhere - including US domestic airports.

<comment><USER3> Prediction: they will not say anything further and use it in completely arbitrary ways whenever and wherever they want.

</comment></comment><comment><USER4> Would someone please start a religion that denies the authority of the TSA so we can opt out of this security theater on religious grounds?

<comment><USER5> If you are an Orthodox Jew, I don't think you can turn on a cellphone on the Sabbath.

<comment><USER6> This is actually a very good point...I wonder if there will be a religious exemption. Some Orthodox Jews can be exceptionally strict (I once was on a delayed flight where we had to offload some of the passengers, since due to the delays their connecting flight would have taken place on the Sabbath).

<comment><USER7> While an Orthodox Jew cannot directly ask you to take his cellphone and turn it on, he or she will be indifferent to you taking and power cycling it voluntarily. It's the intent and activity that's considered sinful, not state of the device per se.

Also it's perfectly normal to leave a phone powered on for a Sabbath, if you don't actively use it. Same applies to e.g. elevators, that just cruise in an endless loop and stop on every floor. You can ride in it, as long as you aren't the operator.

Source: living in Israel for many years.

</comment></comment></comment></comment><comment><USER8> They do this at Ben Gurion Airport too - make you turn on all your electronics, the Israeli airports are praised for their security procedures - right?

eg: [http://www.huffingtonpost.com/2013/12/23/airport- security_n_...](http://www.huffingtonpost.com/2013/12/23/airport- security_n_4494308.html)

<comment><USER9> Whenever that model is considered nice I try to paint a different picture. Example: [https://news.ycombinator.com/item?id=3283768#up_3285789](https://news.ycombinator.com/item?id=3283768#up_3285789)

I've lived there for a year and travelled to Ben Gurion before and after that time. It's never pleasant and often downright insane.

<comment><USER10> It's interesting the different experiences people have.

I've found LLBG to be downright pleasant. I get there at most 90 minutes prior to an international flight, breeze through security in about 15-30 minutes depending on queues and often have time to spare to get a coffee and browse through the duty free shops at my leisure.

Full disclosure: I am an Israeli citizen, but I've had this experience even prior to becoming a citizen, when I was only a US citizen.

<comment><USER9> Regarding your last sentence: May I ask whether you

a) speak Hebrew

b) are jewish?

Both seems to help a lot (got some friends from DE and the US that had less trouble than I did and .. well, that might make a difference).

I'd certainly even agree that I'd prefer the treatment in IL to any body scanner/groping experience "elsewhere", but I don't like the experience one bit and get singled out more or less every time.

</comment></comment></comment></comment><comment><USER11> I assume this is so they can log cell phone/tablet IMEI numbers and/or MAC addresses at the airport while the foreigners queue to be finger printed and photographed?

If they log the IMEI they can track most foreigners as they move around the country. Even if they are not bringing sim cards, the phones will still try to connect to a local network to allow emergency calls.

Or maybe so they can confiscate and download the contents of electronics devices quickly with as little delay as possible (it can take 5-10+ minutes before a completely dead smartphone has enough charge to complete a boot up).

<comment><USER12> Then include your device, powered on in airplane mode, in an easy to reach pocket.

Easy to prove it's on, just don't take it out of airplane mode until you arrive at your destination. In flight you could power it off & back on before touchdown to preserve battery.

</comment></comment><comment><USER13> Wow, that's a blast from the past. I recall having to do the same with my Game Boy and cell phone when traveling through ATL in the 90's, well before the TSA was created.

</comment><comment><USER14> Except last time I checked, possession of a mobile device is not required by law? So, what the fuck.

      Not to mention: [

          "the battery's dead and the proprietary charger
          is stuck in checked baggage", "whoops i lost the
          battery", "whoops i dropped the phone and now it
          is broken", "whoops cannot power on without
          password for encryption", "aw shucks my phone is
          keister stashed so come get it big boy", "hello
          look at this childrens toy which very closely
          resembles a cellular phone ha ha", "gee why does
          this powered on mobile device emit far more radio
          spectrum energy than expected even though i am
          usually a law abiding citizen?"

      ]

<comment><USER15>
        var terroristWarningFlags = // sarcasm...?

Either way, with the TSA you don't win with reason or truth.

<comment><USER14> Well, the non-cutesy response is to cull local IMEI's (with false base station attacks) and spoof random IMEI's that spam local base stations with control channel traffic, and swamp normal signal data with noise. (not quite "jamming", per se, but maybe something close to it)

Someone could likely accomplish this, and enable others to participate in the protest of security theater, by developing and distributing an open-source application that interfaces with a number of readily available software defined radio antennas (preferably those costing less than one hundred dollars).

Although, to do so might be terr'ism.

</comment></comment></comment><comment><USER16> This has to be one of the silliest moves yet.

What a monstrous waste the TSA is. A waste in resources better utilized elsewhere.

<comment><USER17> TSA is not a waste. It's a business model (for some).

<comment><USER18> Is just a start-up that went big; the security business was just begging for disruption.

</comment></comment></comment><comment><USER19> Can't the TSA just do what the Canadians do and swab electronics and use the ion scanner?

<comment><USER20> If explosives were what they were really searching for you would be absolutely correct.

</comment></comment><comment><USER21> [http://xkcd.com/651/](http://xkcd.com/651/)

and by turning it on you just proved you have the materials for a bomb. touche.

</comment><comment><USER18> You also have to press all the possible combination of numbers on your device, just to make sure there isn't one to activate a bomb. You also have to open all the apps in your smartphone, just in case one of them is "open to blow the bomb".

<comment><USER22> So you've got 17 apps on your phone. And all you need to do to see Allah is to open 5 of them in sequence. 17x16x15x14x13 = 742560.

Or maybe all you have to do is get your unlock password wrong a certain way.

Fact is, the fact your phone boots makes it more of a threat, not less. LOL!

Fact is, this is not about security. It is an exercise in security _theater_.

</comment></comment><comment><USER23> Because it's impossible to build a cell phone bomb that also functions?

<comment><USER24> To be extra cautious, they should profile people with older blockier phones. Oh no, now I said it, the TSA will likely do it. xD

</comment></comment><comment><USER25> I wonder what will be done in the case of laptops with dead/no batteries.

<comment><USER26> > It says devices that won't power up won't be allowed on planes, and those > travelers may have to undergo additional screening.

Better bring your charger.

<comment><USER25> But what's the probability they'll let you use it?

</comment></comment></comment><comment><USER27> Another on the long list of reasons I don't every fly to, from or through the USA even though it sometimes makes my routes longer and more expensive.

<comment><USER28> Right, because only USA airports (or flying to US) has stupid rules? I've gone through the same inane things in other countries. In fact the only time I've ever had to turn on my computer was outside of US flying to another country outside the US.

</comment></comment><comment><USER29> Duplicate: [https://news.ycombinator.com/item?id=7996380](https://news.ycombinator.com/item?id=7996380)

</comment><comment><USER30> So, is this effectively saying that the scanners and such have been seriously flawed this whole time?

</comment><comment><USER31> This is step one. Then in a few months they will introduce new requirements to hand over the login password so the TSA agent can confirm the laptop indeed works.</comment>


[Closed beta] Saasmetrics.co - <AUTHOR> http://www.saasmetrics.co

<comment><AUTHOR> Hi folks, for those of you who did build or is building a subscription business, you know that dealing with metrics is a big deal.

When learning about SaaS metrics some time ago I ended up on the famous David Skok's article "SaaS Metrics 2.0" and found that most part of my friends co- founders were using the very same spreadsheet from this article.

The result of this studying is [http://saasmetrics.co](http://saasmetrics.co), an SaaS product to manage your subscription business metrics.

The first version is an MVP that allows you to manually input your data or use or well document API to build your own integrations. But we know that the hard thing about metrics is to keep them up to date, so our near future road map includes connection with different platforms to get data 100% automatically.

We know a few good tools out there focusing this problem - such as baremetrics.io - but we really want to cover the whole thing about saas metrics, and allow you not just to connect your payment platform (such as stripe), but also your marketing tools such as Google Analytics, Hubspot, your financial apps and ERPs such as quickbooks, and more to come. And of course you can always build your own integrations to input data using our API.

We're in closed beta with a few customers selected by hand, but you can see a full demo working here: [http://demo.saasmetrics.co](http://demo.saasmetrics.co)

We're also working on a comprehensive wiki for SaaS metrics. Check out this example: [http://saasmetrics.co/monthly-recurring- revenue](http://saasmetrics.co/monthly-recurring-revenue)

I'd like to hear your thoughts and comments.

Thanks, Leo.</comment>


Experimenting with Infinite Scrolling & HTML5 History API - <AUTHOR> http://warpspire.com/experiments/history-api/

<comment><USER2> It is not fluid user experience at all. Click and drag the scroll bar to the bottom, and keep trying to scroll down. It violates expectations. The scroll bar should be sized to the amount of data that exists, whether or not it is fully loaded/rendered.

</comment><comment><USER3> The only thing I don't like is that the URLs seem "dirty." IE I don't like seeing kajillion digit IDs with no human meaning.

Anyone have any solutions to that?

<comment><USER4> Since those IDs are just the IDs of the tweets underneath, I don't think it has much to do with the elegance of this solution, per se.

Perhaps it would be cleaner use the Flickr approach to set URL naming:

      /<item-id>/in/<set-id>/</comment></comment>


Wingman: Git + GitHub + OS X - <AUTHOR> http://www.wingmanapp.com/

<comment><USER3> I'm shocked that an indie Mac app that isn't in the app store is trying to get away with selling a $29 license that has no free trial. To Github developers, of all people. Shenanigans.

It'd be one thing if you had an awesome video whereby I could see in this in action. Or tons of awesome reviews from people I respect. Hell, I would pay $5 to try it out.

But you want me to pay $30 bucks in hopes that this doesn't suck? There is too much competition in this market to even attempt this. This developer is shooting themselves in the foot.

<comment><USER4> Why is it such a big deal that it isn't in the app store? As far as I can see automatic updates is the only advantage that the app store brings... and even that has been handled for years without problems by third party updating systems like Sparkle.

<comment><USER5> sadly, I'd say because it's a centralized place for reviews that makes it less likely a company might publish a half-written something

<comment><USER6> Yeah, the app store model keeps the reviews out of the hands of the developers. Almost like a consumer protection.

</comment></comment></comment></comment><comment><USER7> This seems like a cool idea but i'm bewildered by one thing. Why is this a menu item? And does it have hotkeys?

Because, for better or worse, my menu bar is essentially full, and I save and manipulate my code and repositories via my keyboard, not my mouse (well that's mostly true. I still browse my git repos using GitX).

The workflow ideas seem worth trying, but the interface isn't something I can get with (although i will cop to not having any good ideas on how to improve this. I wonder what a quicksilver-like workflow would be like).

<comment><USER8> I recently discovered Bartender to help me with my menu bar and cannot recommend it enough.

<comment><USER9> Link: <http://www.macbartender.com>

I'm a big fan of the app, and the developer finally fixed some annoying bugs from the 1.0 release.

</comment></comment></comment><comment><USER10> Really surprised to see people balk at $29 for a software productivity tool. I mean, I get it if it's not worth it for you, but I happily paid a similar amount for SourceTree (before it was free ;) ), and pay similar amounts for all sorts of development tools. It doesn't take much productivity gain for these sorts of tools to pay for themselves quickly. This is the sort of thing that a dev lead might buy for his team to teach/enforce the git-flow workflow quickly and easily.

We happen to use Mercurial around here, so that was my biggest turnoff, but the rest of the value proposition made plenty of sense to me: Solve one problem. Solve it well. Charge real money.

I wish these guys success. :)

<comment><USER11> So is there an equivalent for Mercurial, i.e. automates best-practices recipes for common dev workflow steps?

</comment></comment><comment><USER12> I suggest moving your imagery over to a CDN. Chrome network tab clocked your site at taking 30 seconds to load all the images. (I am on a 100mbit connection)

<comment><USER13> The images and the download itself are both crawling for me.

<comment><USER14> Same issue here, I'm getting an estimated download time of four hours for the app itself.

</comment></comment></comment><comment><USER15> Seems awesome - like a GUI for git-flow.

What is the pricing for this? Seems there's a free download at the top, but the bottom says "Buy for $29.99". Might want to make the pricing structure clearer. :)

<comment><USER16> From what I can gather it's some kind of dodgy Shareware thing. Although I can only guess since they really have made little effort to make it any more clear.

Neither the "purchase" site [1] nor anything inside the app itself [2] clearly explains the terms of the free download, or why one should even consider purchasing a licence.

I understand this is a fresh product and everything but it's expected one makes an effort to make these terms far clearer before asking for people's money.

Is this a simply a case of neglect or an example of a dark pattern [3] in the wild?

[1] <http://wingmanapp.com/purchase>

[2] <http://i.imgur.com/aD5g8.png>

[3] <http://wiki.darkpatterns.org>

<comment><USER17> What makes shareware dodgy now?

<comment><USER16> Nothing. You've misinterpreted. Or I've miscommunicated. Perhaps both.

Read what I wrote again without presuming I'm universally referring to all Shareware as being dodgy; just this particular instance.

</comment></comment></comment></comment><comment><USER18> I'd love to try this, but I'm uncomfortable entering my github username and password - shouldn't apps like this be using oauth instead of basic authentication?

Also, the sign in button appears disabled until you actually purchase it (I assume it's not just a bug), which makes the prominent Download link on their site with mostly hidden mention of pricing feel pretty shifty. Behavior like this doesn't instill confidence or trust.

<comment><USER19> I have a native iOS app that works with GitHub's API. I had the exact same concern as you. As a user, I would much rather use an oath of flow instead of entering my password.

Unfortunately, GitHub actually suggests you use basic auth for native applications. Their reasoning is that the application secret would need to be shipped with the binary, thus making it… not secret (even with various obfuscation).

Source: [http://developer.github.com/v3/oauth/#non-web-application- fl...](http://developer.github.com/v3/oauth/#non-web-application-flow)

<comment><USER20> I don't know what their reasoning is, but there's another obvious reason - <USER2> you're trying to protect against a malicious app, unless you get it from the App Store so you can be assured it's sandboxed _and_ it uses an external web browser to show the OAuth dialog, you don't have any real guarantee that it can't steal your password anyway. Although it might be easier to write an app that steals your password if you enter it directly, that's not much real security - to that extent, the sense of protection that the OAuth dialog gives you is false. (This is especially silly for iPhone Twitter apps, where the standard is to pop up a web view which the app can easily inject JavaScript into.)

<comment><USER21> One nice thing about Windows 8 app development is that they provide a method through WebAuthenticationBroker to have the browser instance hosted in a secure separate process from your app to prevent any shenanigans.

</comment></comment></comment></comment><comment><USER22> SourceTree has something similar: a GUI for git-flow. Mac has the best tools for git and in linux i can't find something with the quality of SourceTree.

link: [http://blog.sourcetreeapp.com/2012/08/01/smart-branching- wit...](http://blog.sourcetreeapp.com/2012/08/01/smart-branching-with- sourcetree-and-git-flow/)

</comment><comment><USER23> I'm truly intrigued and curious... my startup Gitpilot (<http://gitpilot.com/>) is trying to solve a similar problem. Was Wingman conceived from any inspiration of Gitpilot? Truly curious, especially because of the airplane naming scheme.

<comment><USER24> Wingman is a friend you take to the bar, not an airplane. But props on your app/startup - pretty damn slick UI.

<comment><USER25> A wingman is the other pilot flying a sortie with you, supporting you in a dangerous mission. The bar-cruising friend only wishes he could be that awesome.

<comment><USER24> I stand corrected (<http://en.wikipedia.org/wiki/Wingman>).

</comment></comment></comment></comment><comment><USER14> Command + Return probably isn't the best key combination for this. Most source code editors on OS X use that key combination to insert a new line below the current one regardless of cursor position.

</comment><comment><USER26> Downloaded it. Ran it. Entered my github credentials... and the login button is disabled.

hm.

<comment><USER27> Login is enabled once you buy it. It's not clear, but there's no trial you have to pay to login.

</comment></comment><comment><USER28> No way I'm dropping $30 without trying it first, especially if it is not on the app store. This isn't exactly a 99 cent flashlight app.

</comment><comment><USER29> The page what this brings me on top of Git+Github. Is it just a GUI around Git to avoid having to learn the ~5 everyday commands?

</comment><comment><USER30> So I was ready to buy Wingman, but then paused and thought. Why isn't this on the App Store? Am I really going to use it? I am all for productivity tools, and honestly willing to pay for anything that makes my life easier, but I think I just need a trial of Wingman, just to make sure it is what I am envisioning.

</comment><comment><USER31> Alienating developers from git? What's next, hiding them the source code? I do not really see the point of this app.

</comment><comment><USER32> So this is graphical representation of gitflow... and tied to github... it sounds cool, I am a little suspicious how many people will really need this. From angry comments I see this is not tied to appstore, kudos for that, hope more devs would do it like this.

</comment><comment><USER33> Does this work with git-flow already? I wouldn't switch away from an existing git based workflow for this but I'll consider it for future projects.

</comment><comment><USER34> I'm not into these stuff, but this might be an alternative: <http://codejournalapp.com/>

</comment><comment><USER35> For all the comments about no trial version, they cover this at the bottom of the page by saying they'll gladly refund if you're unsatisfied.

</comment><comment><USER36> 1) I use CMD + Enter for Alfred 2) How is this faster than just going into command line? 3) $30? I'll just stick with command line.

<comment><USER37> With spotlight at command-space, why not option-space for Alfred?

<comment><USER36> Because Alfred is basically a replacement for Spotlight, so I just use Alfred instead.

</comment></comment></comment><comment><USER38> This looks cool. Is there any similar software for Windows?

</comment><comment><USER39> Are there settings for a --no-ff merge?

</comment><comment><USER40> 29$ ruined it for me

<comment><USER41> $29 is a small price to pay to increase your productivity.

<comment><USER42> How do I know it will? I was turned off by the lack of a trial.

</comment></comment></comment><comment><USER43> Ugh, github. No thanks.

</comment><comment><USER44> Tip.

A lot of people will see the name and have the response that I did. Which is that this is likely to be an app that aims to help sleazy pickup artists out.

Like it or not that is the immediate association that people have with "wingman".

<comment><USER45> I'm calling shenanigans.

First of all, please don't pass off personal perspectives as fact. An "immediate association that people have"? Really? What actual data do you have to back such an assertion? I had no such image in my head whatsoever when I first saw the name. Am I in the majority or the minority? I have no idea. Neither do you.

Secondly, if we avoided each and every possible warped euphemism in the English language, we wouldn't be able to name anything. Teabag, facial, taint, snorkel... the list of potential innuendo is endless. These are perfectly good words that have been ruined by some jerk who thought he was being clever.

Don't buy into this. Let the 13 year-olds snigger in the back of the room if they want to. The only way to combat this prepubescent inanity is to use these words in their proper context -- as often as possible.

<comment><USER44> It is unarguable that there will be people who have that immediate association. I stand as an existence proof for that. It is also unarguable that not everyone has that association - you are proof of that.

How many have the association? A few? A lot? From the fact that I did, my inclination is to suspect that it is not uncommon. Perhaps I'm wrong.

Will the name be a barrier to success? Possibly, possibly not. But it is important to know that some people will have that association. Perhaps you think that using the name anyways is a worthwhile public service. Perhaps you think that anything that causes people to get curious is good. Perhaps you think it is not the image you want.

The authors of the tool should take it however they want to. It is their product, not mine. But they should at least KNOW that, for better or for worse, there are people out there who will have the reaction that I did.</comment></comment></comment>


Designing a Browser that isn't a Browser - <AUTHOR> https://medium.com/project-tofino/designing-a-browser-that-isn-t-a-browser-685b63c4b6f1#.amawrkcjp <comment><USER2> We are going to do something. We don't know what. Not very interesting.</comment>


It's official: HP bids $25 a share for EDS, will take on IBM in tech services - <AUTHOR> http://www.alleyinsider.com/2008/5/live_hp_hpq_buys_eds_eds_for_25_a_share_raises_guidance

<comment><USER2> Ross Perot is a rich man!

<comment><USER3> Didn't he sell EDS to GM in the 90s?

<comment><USER2> Doh! That he sure did! Then GM spun off EDS again!</comment></comment></comment>


AT&T Early Termination Lawsuit Settlement (PDF - click comments for details) - <AUTHOR> http://www.attmetfsettlement.com/pdfs/ClaimForm.pdf

<comment><AUTHOR> Hey guys, just got emailed this from AT&T:

<http://www.mobilitywelcome.com/files/7584_ATTinsrt_R.pdf>

Posting for two reasons. First, it shows the general nonsense and BS of class action lawsuits. Second, you can get a free phone card if you're an AT&T user or lower the cost to cancel your plan from $175/flat to $175 - $5/month you were a member.

\--

o By checking here, I choose to receive an AT&T Prepaid Long Distance Card with up to 200 minutes; OR

o By checking here, I choose to be subject to a prorated ETF provision instead of the flat-rate ETF provision that is currently in my contract. I understand that I will be provided with other benefits subject to the terms of the Plan of Allocation if, at the time that benefit becomes effective, I am no longer an AT&T Mobility subscriber with a flat-rate ETF provision.

\--

From the link in this comment:

\--

If the Settlement is approved, a settlement fund of $16,000,000 in cash and $2,000,000 in non-cash benefits will be created.

\--

Most of that will go to lawyers' fees - you can get a bit if you canceled your contract, or get a phone card, but mostly it's a lawyer play. Anyways, I always found I learned when I went over legal documents, so hopefully someone finds this valuable. If you're going to cancel an AT&T plan in the near future, this could save you a decent chunk of change for the 15 minutes of your life too.</comment>


Quake devastates mountain towns in central Italy, at least 20 believed killed - <AUTHOR> http://www.reuters.com/article/us-italy-quake-idUSKCN10Z04H?il=0 <comment><USER2> Myanmar, too: [http://www.reuters.com/article/us-myanmar-quake- idUSKCN10Z17...](http://www.reuters.com/article/us-myanmar-quake- idUSKCN10Z176)</comment>


DRAM errors vastly more frequent than previously thought - <AUTHOR> http://blogs.zdnet.com/storage/?p=638

<comment><USER2> I've worked for a major DRAM company and it's a wonder that the parts are reliable at all. The storage cells are being pushed to their limit as they are often the bottleneck in scaling the overall size of the bit structure. Not to mention the engineers are pressed to the max to get them out the door, often before they are fully qualified. Throw in the millions of process steps that are required to make the part, and the fact that your computer boots at all is remarkable.

Then throw in a high speed bus, processor and disk array....

<comment><USER3> And try and convince a programmer that it's possible that their program's memory can be wrong.

They understand in theory but refuse to code for the possibility. Especialy when you get into HPC and there are clusters of 50-60 machines with 4Gb each, the chance of not having corrupt memory is almost 0.

<comment><USER4> I'm honestly curious... what kind of defensive programming techniques could you use to try and deal with this?

<comment><USER5> Do everything twice and make sure the results match.

<comment><USER6> What happens when the code doing the comparison becomes corrupted? Do the comparison twice? What happens when the code controlling the evaluation of both comparisons becomes corrupted?

Your data and your instruction set are in the same memory. Even if they are separated into different areas of memory to prevent buffer overflow exploits, it's all still in memory. Once the memory starts going, you're kind of screwed. It's the same as how -- in respect to computer security -- once someone has physical access to the machine, you're screwed.

With respect to memory errors in distributed environments, usually such environments are distributed to increased the processing power for number crunching. If you run all calculations twice and have code comparing them for acceptance, you're more than doubling your processing requirements.

But at the end of the day, it's all a matter of what level of risk is acceptable (or tolerable). There is no magic bullet to fix these issues.

<comment><USER7> You're ignoring that the voting code would be a very small fraction of your RAM and thus less likely to be corrupted. But it's academic since no one runs twice to avoid the cost of ECC.

<comment><USER6> I realize that. My point is that there is no 100% solution.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> I remember djb exhorting people to buy ECC memory and supported motherboards back in 2001 (<http://cr.yp.to/hardware/ecc.html>) for their standard workstations and me thinking he must be somewhat crazy since no one else seemed to be making a fuss about it.

Perhaps he was right after all.

<comment><USER9> He is right.

Have you ever had fsck detect errors on a filesystem that you haven't abused? Guess what, that's memory corruption -- saved to disk forever.

I remember having a machine with especially flaky memory (memtest86 failed in about 30 seconds)... I detected it because dpkg's database was corrupted enough for it to cause errors in the application. I never even _tried_ to save that filesystem...

</comment></comment><comment><USER10> As a user of hardware, not a hardware engineer, I wonder if I've seen these errors. Crashes -- I don't see any of those except the ones related to web browsers. I do a lot of long compiles and don't see any crashes from gcc. Corrupted data -- well, I've had several large downloads this year that didn't match the advertised md5 sums. I redownloaded and got matching checksums. Does any of this have to do with DRAM errors? There are lots of other potential sources of error in my computers. I could name half a dozen off the top of my head, but I'm sure I would only prove that I'm ignorant of another half dozen that are an order of magnitude more important than the ones I named. I await an answer to one question: who should care about these DRAM errors? Does that group include me?

<comment><USER11> Did you check whether the downloads had been truncated? Downloads finishing prematurely is a frequent cause for a checksum mismatch.

</comment></comment><comment><USER12> What's the difference between a "hard" and "soft" error that the article mentions?

<comment><USER13> I've heard the term used in two different ways, and I'm not sure which way is the 'correct' one:

In one usage, "soft" errors are ones that are 'caught' and transparently fixed by ECC, and thus have no effect (on a system that has ECC memory). "Hard" errors, by contrast, are ones that affect multiple bits and aren't corrected by ECC.

In the other usage, which I think is the more technically correct one, a "soft" error is a transient condition (bit flipped by cosmic ray, etc.) and the memory cell continues to operate normally on the next cycle. A "hard" error is where the cell is basically stuck in one state or another, and indicates that it's probably time to replace the module. I think you detect a "hard" error by looking for a series of "soft" errors, although maybe some architectures/chipsets detect the difference and report them in different ways...?

If anyone can substantiate either set of definitions, I'd be interested as well.</comment></comment>


How to Drink at the Twitter Firehose and not Drown - <AUTHOR> http://community.devexpress.com/blogs/garyshort/archive/2010/03/16/how-to-drink-at-the-twitter-firehose-and-not-drown.aspx

<comment><USER2> That pre-launch checklist is a big caveat to this article...</comment>


Nostalgia doesn't need real memories – an imagined past works too - <AUTHOR> https://aeon.co/essays/nostalgia-doesnt-need-real-memories-an-imagined-past-works-as-well <comment><USER2> > "Improving the accuracy of our memory for the past could indeed be the best > strategy to curb the uncharitable deceptions of the politics of nostalgia."

I agree, and will add that there's been no time like now for ease of improving the accuracy of our impressions of the present, due to the sheer availability of primary sources. Click. Click. Click. It's real easy, man.

As to secondary sources: [https://news.ycombinator.com/item?id=23858477](https://news.ycombinator.com/item?id=23858477)

> "If you agree with it, it's truth. If you don't agree, it's propaganda. > Pretend that it is all propaganda. See what happens on your analysis > reports."

Minitrues may still be as active as ever ( [https://news.ycombinator.com/item?id=23895444](https://news.ycombinator.com/item?id=23895444) ) but reasonable propaganda analysis (PROPANAL) skills help triangulate. See also para 15 for examples of domestic propaganda in the age before social media. (Other sources such as Bernays go further into the details of how one exploited the social graph without automation.)

Compare the analytical outline on para 45, pp 18-20: [https://collections.nlm.nih.gov/ext/dw/01130770R/PDF/0113077...](https://collections.nlm.nih.gov/ext/dw/01130770R/PDF/01130770R.pdf)

</comment><comment><USER3> An essay about nostalgia that is very interesting to read. I became rigid to understand it there. Writing that spurred me to remember what I dreamed of as a child and still continues to exist until now. Thank you to Felipe and Nigel for publishing this great essay.</comment>