Amazon Redshift is 10x faster and cheaper than Hadoop and Hive - <AUTHOR> http://www.slideshare.net/Hapyrus/amazon-redshift-is-10x-faster-and-cheaper-than-hadoop-hive

<comment><USER2> Disclaimer: I'm a committer on the Apache Hive project.

A couple points in no particular order:

* EMR Hive is a closed source fork of the upstream Apache Hive code base. The EMR docs imply that the latest version of EMR Hive is based on Apache Hive 0.8.1 (which was released more than a year ago), which means EMR users aren't benefitting from the performance improvements that appeared in the 0.9 and 0.10 releases.

* It is implied (though not explicitly stated) that the Hive queries were run against gzip compressed TSV files stored in S3, while Redshift was allowed to spend _17 hours_ converting the same data to its own optimized internal format. Hive supports an optimized columnar format too (RCFile). Why wasn't that used in this performance comparison?

<comment><USER3> _Why wasn't that used in this performance comparison?_

Because then the stupid headline wouldn't be so sensationalist, would it?

// I have no dog in this fight, but hate twisted claims

</comment></comment><comment><USER4> Comparing a column-oriented RDBMS with parallel query execution versus hadoop is a joke in the first place. Hadoop is extremely slow. That's nothing new. This is not an apples-to-apples comparison whatsoever.

How does it compare against Greenplum or Aster or Vertica and is it more cost- effective? Those are important questions.

<comment><USER5> Comparing Redshift against Hadoop+Hive is reasonable. As you pointed out... the technologies are very different. However, there is a large overlap in use cases.

<comment><USER6> I strongly disagree. I am (or was) using both Hadoop and HBase and they are useful for very different purposes (huge amounts of nonstructured data, possibly with difficult-to-predict use cases versus structured data). Also note that Hive is just a layer over Hadoop with DB-like syntax, it doesn't make Hadoop a DB. It is still running MR queries beneath it.

</comment></comment></comment><comment><USER7> So redshift took 155 seconds + 17 hours (17 * 3600) = 61355 secs total

vs 1491 Hadoop

Looks like to me Hadoop is about 40 times faster...

<comment><USER8> That's like saying a bicycle is faster than a car because I can buy a bike in 10 minutes while it may take me a couple of hours to get through the car's paperwork.

If you do enough queries, redshift will come out faster (assuming the numbers are correct).

<comment><USER9> If you do enough queries, you should spend the time to use RCFile for Hive, in which case redshift wont come out _that_ much faster. The point is the 17 hours is not negligible.

<comment><USER10> That is a good case since customers who typically need a datawarehouse aren't just going to upload data once... they probably are going to upload frequently.

<comment><USER9> You're missing my point and resorting to sarcasm - very nice </sarcasm>. My point is not that Hive is the better choice because everyone is going to reload their data frequently. My point is that if you want a fair benchmark, don't use an obviously slow data format for Hive. They spent time importing data optimized for RedShift, but they took a very naive approach for Hive. I'm sure RedShift will still be faster, but not 10 times faster.

</comment></comment></comment></comment></comment><comment><USER11> There's a motive here. hapyrus.com is pushing themselves as Redshift consultants. Oh you're using Hadoop? Redshift is better, cheaper and you can pay us to help you use it.

</comment><comment><USER12> I haven't tried redshift before, but coming from a MR/Hadoop/Hive background, this seems to me like quite a sensational claim. I'd be very keen to hear other's thoughts on how widely these kinds of gains would apply for BigData processing.

As Carl Sagan said..

"Extraordinary claims require extraordinary evidence"

<http://en.wikipedia.org/wiki/Carl_Sagan>

<comment><USER13> Hive is not particularly fast in and of itself; it just has horizontal scaling and a SQL-ish front-end. Looking at AWS RedShift's homepage[1] (emphasis added):

> Amazon Redshift delivers fast query and I/O performance for virtually any > size dataset by using _columnar storage_ technology and parallelizing and > distributing queries across multiple nodes.

Column stores databases[2] can be _screamingly_ fast for analytics operations compared to RDBMS or other DB types (ala assorted NoSQL). See Kdb[3] or MonetDB[4] for examples of specific implementations. I'd fully expect a competent column store designed for horizontal scaling to obliterate Hive for a wide range of problems.

The usual big-data caveat: you need to pay attention to the fit of your tools against your problem and your data. I don't expect RedShift to be any different. Still, it's pretty exciting to see a new analysis DB tech cropping up like this. And doubly interesting to see this coming from Amazon.

[1] <https://aws.amazon.com/redshift/>

[2] <https://en.wikipedia.org/wiki/Column-oriented_DBMS>

[3a] <http://kx.com/kdb-plus.php>

[3b] [https://en.wikipedia.org/wiki/K_%28programming_language%29#K...](https://en.wikipedia.org/wiki/K_%28programming_language%29#K_financial_products)

[4] <http://www.monetdb.org/Home>

<comment><USER14> SAP HANA has a column store, and a row store, and does OLAP (Analytics) and OLTP.

There is a lot of new DB tech, Redshift doesn't seem particularly competitive at the moment unless you only need to use it a portion of the time, where Amazon excels.

</comment></comment></comment><comment><USER15> 1.2 TB really is not very much data in the context of "Big Data". The supposed advantage of Hadoop is that it can scale horizontally with linear performance.

<comment><USER16> They note on slide 9 that this is only biggish data -- but sometimes, that's what you need to work with.

</comment></comment><comment><USER17> I wish the post had gone into depth on _why_ Redshift was significantly faster, but I'm betting it uses in-memory joins whereas (hence the size limitations it mentions) whereas Hive joins are just MapReduce jobs that keep only minimal subsets of data in memory at a given point. The upshot is the Hive/MapReduce strategy isn't limited by physical memory.

Of course, if your data set can fit in memory, then Redshift or similar technologies probably is a better choice than Hive. But it's important to remember that the performance gains here come as the result of a tradeoff.

<comment><USER18> It was significantly faster because as was mentioned above the graph ignores the the 17 HOURS it took for RedShift to import the data.

The comparison is complete and utter joke.

</comment></comment><comment><USER19> I wonder how it compares to BigQuery: <https://developers.google.com/bigquery/docs/pricing>

</comment><comment><USER20> Worth noting this presentation was made by Hapyrus, a Hadoop specialized startup from 500startups. They know quite a bit about running Hadoop. Following the results of their tests they are now adding Redshift support to their services.

<comment><USER21> ... and want to sell their Redshift services starting with a bang.

</comment></comment><comment><USER22> They should compare redshift with hadoop + Imapala, OR hbase with Phoexnix from Salesforce. Comparing with hadoop + hive is not a correct comparison

</comment><comment><USER23> I am still new to large data, but isn't a solution like Redshift similar to Google's Big Query in that it only works with data that has a schema? How might one use Redshift with a db thats originally in Mongo?

<comment><USER24> You won't fit that much data into Mongo anyways, so does it matter?

<comment><USER18> People have been apparently storing 3TB of data in MongoDB.

So I guess it does matter.

</comment></comment></comment><comment><USER25> Indeed, this comparison seems fishy.

Nevertheless, I'll take a moment to predict that articles like this will be only becoming more and more frequent in time. Hadoop has entered its "enterprisey" stage, with massively complex, cumbersome code, arcane performance tuning, bullshit consulting business built around it (complete with books and "certificates")...

The more agile competitors will be snapping at its flanks (and ankles), sometimes without merit, and sometimes with.

</comment><comment><USER14> I'd be interested in seeing a comparison between Redshift and SAP HANA, but a more fair comparison than this one by someone who isn't partisan.

</comment><comment><USER26> Slide 2&6, one query every 30 minutes.

It turns out usage based billing can be cheaper if you don't use a resource.

<comment><USER27> I'm willing to bet that's a not-uncommon scenario for a lot of organizations, however. If you're doing continuous querying of large amounts of data, then it's probably worth building your own hadoop cluster (physically or via Amazon), but a lot of people are just going to accumulate data and then make queries against it. Lots of 'active users per day', 'traffic by hour', 'purchases by popularity', etc. only get run to create data for the CEO every morning, or by the marketing manager every afternoon, that sort of thing.

</comment></comment><comment><AUTHOR> We wrote the blog post about this benchmark. [http://www.hapyrus.com/blog/posts/behind-amazon-redshift- is-...](http://www.hapyrus.com/blog/posts/behind-amazon-redshift- is-10x-faster-and-cheaper-than-hadoop-hive-slides)

</comment><comment><USER28> Seems like stupid marketing shit

</comment><comment><USER29> Are they benchmarking hash join on hadoop and redshift?

</comment><comment><USER30> Anyone interested in SQL queries on Hadoop should be checking out Cloudera Impala. It's open source.</comment>


The parable of the fisherman - <AUTHOR> http://blog.figuringshitout.com/the-parable-of-the-fisherman/ <comment><USER2> Tragedy of the commons => common anti-capitalist fodder, whether fish, pollution, fur, logging, etc.

Also what is not shown here are the millions of people that were able to eat fish due to its drastic drop in price. The omega balance raising IQs and resulting in more eco-scientists, sociologists, and general income for the government or charities for retooling of bronze-age fisherpeople.

<comment><USER3> Yeah, factoring in where the money came from takes the moral in a completely different direction.

In the end, a lot of people are getting cheaper food, and more fish is being produced with fewer workers. The suffering is caused by unexpected _change_ , not because the corporate fishing fleet is inherently _bad_.

I would argue that the poor working conditions are caused by the temporary oversupply of fisherman, not by anything inherent to the job structure. Once that oversupply is corrected, you end up with fisherman working for a good wage to provide cheaper food to thousands of people. The community as a whole can work fewer hours and enjoy more of their life.

So capitalism ruthlessly exploited the people it pushed out of a niche, a one- time cost to a small group of people, in exchange for permanent good effects for everyone.

Obviously it would be better if the company was giving good wages from the start, and a social safety net kept that small group from suffering at all. But even with that terrible side effect, the search for efficiency was a good thing.

<comment><USER4> > So capitalism ruthlessly exploited the people it pushed out of a niche, a > one-time cost to a small group of people, in exchange for permanent good > effects for everyone.

That's pretty much the definition of totalitarianism (the end justify the means, like in The Prince).

> Obviously it would be better if the company was giving good wages from the > start, and a social safety net kept that small group from suffering at all. > But even with that terrible side effect, the search for efficiency was a > good thing.

Our moral compass should tell us that if achieving efficiency without exploitation is possible then it must always be chosen over achieving efficiency with exploitation.

There's another subtle thing to consider here: the capitalists seem to justify and enforce the means when the terrible side effects only apply to others. The line between socialist despotism and liberal despotism is a bit blurry.

<comment><USER5> That's a utilitarianism, not totalitarianism. Utilitarianism is a philosophical outlook, totalitarianism describes a category of government. I don't think this counts as totalitarian because it was not the government who did the pushing here.

<comment><USER4> Ah, yes! Good point, upvoted.

</comment></comment></comment></comment></comment><comment><USER6> This isn't a story about all capitalism. It's a cautionary tale, a story of what happens when capital is not responsive to local conditions and social factors. This is why the concept of Triple Bottom Line accounting was created [http://www.economist.com/node/14301663](http://www.economist.com/node/14301663) . However, power in late American capitalism still stems from short-term decisions in finance, ignoring the issues of natural finite resources, ably conceptualised by the Rocky Mountain Institute: [http://en.wikipedia.org/wiki/Natural_Capitalism:_Creating_th...](http://en.wikipedia.org/wiki/Natural_Capitalism:_Creating_the_Next_Industrial_Revolution) .

I also recommend the writings of American economist and sociologist Thorstein Veblen from 100 years ago on how it is remote capital that increases the likelihood of businesses to have terribly deleterious effects on the physical, biological and social environment. [http://www.gutenberg.org/ebooks/author/438](http://www.gutenberg.org/ebooks/author/438) [http://en.wikipedia.org/wiki/Thorstein_Veblen](http://en.wikipedia.org/wiki/Thorstein_Veblen)

Change happens, and often it brings benefits: but unless we are mindful, as say Elon Musk attempts to be, in our desires to improve the world with our technolgies, then dystopias will follow.

I can also recommend the various works of David Brin on the near to medium future, whose mind is quite as sharp as Musk's and other meta-systemic thinkers we know and appreciate. [http://www.davidbrin.com/](http://www.davidbrin.com/)

</comment><comment><USER7> This got very dark very quickly.

It does make me wonder though, to what extent are we responsible for the repercussions of our actions? Where along that chain did it stop becoming the fault of the businessman, if ever?

<comment><USER8> _Where along that chain did it stop becoming the fault of the businessman, if ever?_

Current ethics would probably say, it was never the fault of the businessman. The profits were ripe for the taking; under a capitalist worldview, _someone_ would have taken them, whether or not it was this businessman.

As far as we can tell, the businessman did nothing expressly immoral- the consequences were externalities, and our moral systems are rarely developed to encompass anything beyond externalities of the first degree, because let's face it- externalities are _damn hard_ to predict, understand, or even monitor.

<comment><USER9> The profits were already being enjoyed by the fisherman. The problem is that the profits were rapidly extracted at a rate that the environment could not support. Sucked dry.

Enter stage left: Sustainability.

As someone posted above, the Triple Bottom Line presents a different concept of capitalism that takes into account the limited supply of resources that make profitable life possible. At our current rate of growth, if we do not refocus efforts on sustainable consumption, we will all end up like the fisherman.

<comment><USER10> Last sentence particularly rings true for me! Where ever there are profits, there is also a deficit elsewhere, this is one of the core problems with our monetary system. We are brought up not to be greedy and to be generous, then as soon as we enter the real world as adults that need to earn a living, many are forced to make decisions that are not ethical in order to survive/support family etc. I have been spending a lot of time looking into the Resource Based Economy recently and this model seems far superior to anything we have at the moment, unfortunately, it is so radically different from what we have that it is going to take a lot of educating and a lot of work to start abandoning our current failing system in favour of another. Still, I have a lot of hope for the future, we have everything we need right now to change the world, we just need everyone to be willing and ready to accept this change!

</comment></comment></comment></comment><comment><USER11> Brilliant re-write. If you skipped this link, thinking it was going to just be the well-known tale, look again.

<comment><USER12> Indeed, I thought it was a repeat, until verifying today's posting date. For reference, here's a variant of the 'original':

[http://renewablewealth.com/the-parable-of-the-mexican- fisher...](http://renewablewealth.com/the-parable-of-the-mexican-fisherman/)

</comment></comment><comment><USER13> Nice re-write. The most interesting part: the businessman never knew what impact his actions had on the fisherman.

<comment><USER14> I doubt it would have changed his course of action. In his mind, if he didn't take advantage of the niche opportunity, someone else would.

<comment><USER13> True, it wouldn't have changed his actions, but it's interesting to think of how all that we do has an impact somewhere else. People reading this now are creating jobs for some and destroying jobs for others. Such is business life.

</comment></comment></comment><comment><USER15> This is a famous German short story by Heinrich Böll, from 1963. [https://en.wikipedia.org/wiki/Anekdote_zur_Senkung_der_Arbei...](https://en.wikipedia.org/wiki/Anekdote_zur_Senkung_der_Arbeitsmoral)

<comment><USER16> nope read it again, this one has an alternate ending

</comment></comment><comment><USER17> "democracy cannot survive overpopulation. Human dignity cannot survive it. Convenience and decency cannot survive it. As you put more and more people onto the world, the value of life not only declines, it disappears. It doesn't matter if someone dies." by Isaac Asimov

[http://en.wikiquote.org/wiki/Overpopulation](http://en.wikiquote.org/wiki/Overpopulation)

</comment><comment><AUTHOR> Server is struggling, here's the text:

A vacationing American businessman standing on the pier of a quaint coastal fishing village in southern Mexico watched as a small boat with just one young Mexican fisherman pulled into the dock. Inside the small boat were several large yellowfin tuna. Enjoying the warmth of the early afternoon sun, the American complimented the Mexican on the quality of his fish.

"How long did it take you to catch them?" the American casually asked.

"Oh, a few hours," the Mexican fisherman replied.

"Why don't you stay out longer and catch more fish?" the American businessman then asked.

The Mexican warmly replied, "With this I have more than enough to support my family's needs."

The businessman then became serious, "But what do you do with the rest of your time?"

Responding with a smile, the Mexican fisherman answered, "I sleep late, play with my children, watch ballgames, and take siesta with my wife. Sometimes in the evenings I take a stroll into the village to see my friends, play the guitar, sing a few songs…"

The American businessman impatiently interrupted, "Look, I have an MBA from Harvard, and I can help you to be more profitable. You can start by fishing several hours longer every day. You can then sell the extra fish you catch. With the extra money, you can buy a bigger boat. With the additional income that larger boat will bring, before long you can buy a second boat, then a third one, and so on, until you have an entire fleet of fishing boats."

Proud of his own sharp thinking, he excitedly elaborated a grand scheme which could bring even bigger profits, "Then, instead of selling your catch to a middleman you'll be able to sell your fish directly to the processor, or even open your own cannery. Eventually, you could control the product, processing and distribution. You could leave this tiny coastal village and move to Mexico City, or possibly even Los Angeles or New York City, where you could even further expand your enterprise."

Having never thought of such things, the Mexican fisherman asked, "But how long will all this take?"

After a rapid mental calculation, the Harvard MBA pronounced, "Probably about 15-20 years, maybe less if you work really hard."

"And then what, señor?" asked the fisherman.

"Why, that's the best part!" answered the businessman with a laugh. "When the time is right, you would sell your company stock to the public and become very rich. You would make millions."

"Millions? Really? What would I do with it all?" asked the young fisherman in disbelief.

The businessman boasted, "Then you could happily retire with all the money you've made. You could move to a quaint coastal fishing village where you could sleep late, play with your grandchildren, watch ballgames, and take siesta with your wife. You could stroll to the village in the evenings where you could play the guitar and sing with your friends all you want."

Sensing skepticism from the fisherman, the businessman moves onto the next boat and finds a more receptive fisherman. The two, sensing an obvious business opportunity, decide to go into business together. They raise a venture capital round and a year later, return to the pier outfitted with a dozen high tech fishing boats.

Immediately, the price of tuna at the pier drops threefold with increased supply, forcing the young Mexican fisherman to increase his hours at sea just to maintain his existing standard of living.

Shortly thereafter, all of the shallow water tuna have been caught and the young Mexican fisherman discovers his tiny boat is incapable of deep water fishing. Because of his limited savings, he does not have enough capital to invest in a deep water fishing boat and he is forced to sell his tiny fishing boat for pennies on the dollar as scrap because advances in technology have made it obsolete.

After discovering that there is limited demand for an employee whose only skills are watching ballgames, playing the guitar and taking siestas, the young Mexican fisherman finds his only option is to take a job working minimum wage on one of the businessman's fishing vessels.

Several years later, the fisherman's joints are shot through from the hard manual labor of operating on a commercial fishing vessel and an ill timed lift of a 150lb pallet of tuna finally causes his back to give way, causing permanent crippling. The fisherman discovers intensive lobbying from the businessman has weakened workplace protection rules and the fisherman is summarily let go with only a paltry settlement.

After years of expensive medical treatments and crippling bills, the fisherman is finally forced to sell his land, passed along to him from generation to generation, to a development conglomerate run by the businessman who is buying large tracts of the entire village.

Unbeknownst to the fisherman, the businessman has lobbied for the village to turn into a protected nature reserve, allowing for the rehabilitation of the environment and the restocking of fish in it's pristine waters. The businessman painstakingly recreates the quaint, costal charm of the village he once visited, making it a paradise where the wealthy flock to when they want to retire into a life of easy indolence.

Finally, 15 – 20 years after the original conversation, the fisherman and his wife are found dead in a homeless shelter. Meanwhile, the businessman retires to the village having made two successive fortunes first in fisheries and then in real estate development. He spends his days sleeping late, playing with his grandchildren, watching high def ESPN ballgames on a 70″ TV, and taking siesta with his wife. He occasionally strolls down to the village in the evenings where he regales his fellow millionaires with the story of how he found an unexploited niche in the marketplace and then took full advantage of it to make the fortune that got him to the comfortable retirement he enjoys today.

<comment><USER18> With no other opportunity and with a deep resentment towards the millionaire, the fisherman's sons become involved in the drug trade. Violence escalates in the village to the point that kidnappings for ransom become a popular way to finance the importation of drugs for resale. The sons kidnap the millionaire's wife for ransom and kill her when the businessman fails to follow instructions. Later the millionaire and his new trophy wife are gunned down in the street. See Acapulco:

[http://www.borderlandbeat.com/2013/07/violence-in- acapulco-i...](http://www.borderlandbeat.com/2013/07/violence-in-acapulco- intensifies.html)

</comment></comment><comment><USER19> Much more realistic than the usual version that ends early.

</comment><comment><USER20> I think the better story would have been of the 2nd fisherman that was approached. Same end result only it would have given a better contrasting perspective.

<comment><USER21> I'm not sure how it could have been more interesting.

The success story ? The happy ending ?

</comment></comment><comment><USER22> A tangental question (and not meant to be a critique): Is this a parable? It too closely mirrors reality and there is no openended quality (ie Talmudic writings or Jesus parables). Neither is it an allegory.

What do you refer to a story like this? An Epitome? ANecdote? Vignette?

</comment><comment><USER23> This is the post Hacker News needed.

Thank you.

Thank you so much.

\-- A fellow Hacker

<comment><USER24> Why?

<comment><USER25> Because HN can get so hung up on success at nearly any cost? Programmers forget their code has real world impact.

</comment></comment></comment><comment><USER24> Nothing is permanent, innovate, disrupt, adapt or die... So this parable is about Silicon Valley (businessman) crushing the complacent entrenched corporations (fisherman) with new technology? Am I missing something?

<comment><USER19> No, the fisherman represents fishermen, and the businessman represents businessmen.

It's about just what it says it's about: the businessman crushing everyone else to satisfy his own greed, and destroying the quality of life for anyone that's not rich.

You know, like real life, as oppposed to the self-serving silicon valley fantasy where getting rich is somehow socially beneficial.

<comment><USER26> The businessman didn't "crush" anyone, he just made the fishing industry a thousand times more efficient through technology. You see the fisherman losing his job, not the thousands of villagers who now have cheaper food.

The problem is all the wealth created from this went to the businessman alone and the fisherman was left with nothing. Because we stupidly live in a society where everyone depends on the value of their labor to get by. When technology makes your labor obsolete, as it _should_ , then you have nothing.

<comment><USER27> Call me naive, but I actually believe it is possible nowadays to make labour obsolete, without making the _labourer_ obsolete.

<comment><USER28> Really? Because the only people able to get paid after their work has been automated (textiles, manufacturing) are those who are able to collect from social safety nets, and they _barely_ get by.

I'm all for a more equitable distribution of wealth though, through a combination of a much more solid web of social programs and higher taxes levied on top recipients of income (I dare not confuse those with "earners").

<comment><USER25> That depends on the state that makes the social net. Scandinavian countries make way better social nets.

</comment></comment></comment></comment></comment></comment><comment><USER29> I was under the impression that the story was actually about taking chances when they're offered to you, and about the consequences of letting great opportunities pass you by. Not so much about the whole ethical responsibilities of big business.

</comment><comment><USER30> dynamite. sounds like real life. modern greek tragedy.

unless we humans decide to harness greed, and quickly, we will shortly follow all of the other large mammals to the mass extinction which we engineered ourselves.

-bowerbird

</comment><comment><USER31> in the original, the fisherman returns to a life he had before the MBA came into his life. The original story was meant to teach us the evils of money. But!! The thing that is missed is that the fisherman made a fortune and now has Health insurance good medical care can pay for his kids to be well educated can fund his nations defense his tax dollars help fund stuff like figuring out the cure for cancer Has helped to produce wealth money is not evil folks.

</comment><comment><USER32> it isn't just about capitalism, it is generic principle of living systems evolution - it isn't possible to stay in static balance with environment forever as environment will change [will be "disrupted" or "paradigm shifted" in local parlance] sooner or later.

</comment><comment><USER33> nice story, Hang.

</comment><comment><USER34> Communist propaganda. Shame on you successful one percenters.

</comment><comment><USER35> Well that was a surprise turn around. Great for the entrepreneur, not so great for the kind natured protagonist.

Also [http://assets0.ordienetworks.com/misc/patfitty.gif](http://assets0.ordienetworks.com/misc/patfitty.gif)</comment>


Ask HN: Would you still recommend a new learner to start from an old language? - <AUTHOR>

I have a friend that understands programming. He wrote a little in C over a decade ago. He now wants to get back in development. He asked me if he should start back from C/Java or jump in with the new languages Ruby/Go? And if that will make his job hunting tougher if he did go with the new languages.<p>What would you recommend? <comment><USER2> I don't think it makes sense to start this endeavour by choosing a programming language.

I think he should choose something he wants to develop, and then choose the programming language as a technology choice in the development of that project.

If he wants to do web, Python/Ruby would be good choices. If he wants to do a native iPhone or OSX app, he should probably start by brushing up on his C then learn Objective-C.

When you're programming, the biggest motivator for me isn't to learn a new programming language, it's to work on a new project, and being motivated is important. I always choose language as a tech choice of the project. YMMV

</comment><comment><USER3> If the only criteria is only to get "a" job, then only the ratio job applicants / job offers categorized by programming language should be taken into account.

If you add other criteria, such as salary, happiness, difficulty, locality, etc, then the choice will become more difficult.

That said, the programming language should just not be a consideration for a programmer or to get a programming job.

Do you ask your surgeon whether he uses knifes or clams from such or such provider?

Do you think that hospitals will reject a surgeon because he never worked with knifes of the same made as the ones they have and use at the hospital?

A real programmer will be able to use any programming language. If you ask an experienced real programmer, he will tell you what programming language to use to write what application, and it may very well not be the same language for all your applications.

(Actually, this is because languages have different features that are required or more adapted for different applications, that languages such as lisp or scheme that have a good macro system, ie. compiler hook, allowing the programmer to _change_ the language and implement domain specific language features (or use libraries implementing domain specific languages), that language as lisp are so good and should be used more, if you asked experimented programmers, instead of newbies or managers; this is also a reason why newer languages like ruby or python that try to be more like lisp have such current development, but people would do better with lisp).

So as a future professional programmer, your friend would be well advised to become acquainted with several programming languages. For example, he could write a small smartphone applications with a backend server, where he would have to write android java, iOS Objective-C, some C or C++ library used by either, a Common Lisp server, using some PostgreSQL database (programmed in SQL), with a Ruby on Rails front end, and a few bash glue scripts. To be efficient in his work, he would also learn emacs lisp to customize and automatize his emacs workflow. In addition to the smartphone apps, he would write some administrative web server, including some Javascript stuff generating HTML and CSS.

[http://www.jasonbock.net/jb/News/Item/7c334037d1a9437d9fa650...](http://www.jasonbock.net/jb/News/Item/7c334037d1a9437d9fa6506e2f35eaac)</comment>


Bitcoin P2P Cryptocurrency - <AUTHOR> http://bitcoin.org/ <comment><USER2> This is an interesting idea, but it seems too complicated to gain the kind of ubiquity it needs to work as a viable currency. As a case study or a prototype for future attempts it looks great, though.</comment>


Two epigenetic regulators interfere with healthy aging - <AUTHOR> https://padiracinnovation.org/News/2020/02/two-epigenetic-regulators-interfere-with-healthy-aging <comment> 1e-9 The researchers discovered that two particular proteins appear to degrade health in worms and mice by harming mitochondria. Worms, mice, and humans all produce these proteins in increasing amounts as they age. The researchers improved the health of aging worms and mice by suppressing the proteins. They found reason to believe this might also be possible for humans.

<comment><USER3> Thank you for this fantastic summary.

<comment><USER4> not sure why you're getting downvoted, but I'd agree, it's really useful to have this summary!

<comment><USER3> I think posts "without content" such as simple +1s or thank yous are not liked here, because they don't add much to the conversation. Still, I found the summary exceptionally useful and wanted to thank the guy/gal.

</comment></comment></comment></comment><comment><USER5> That healthspan might be extended along with lifespan is tremendous news. I think people are afraid of living longer than 75 or so because they assume that the rest of their life will be a spent in senility. But if we can figure out how to keep minds and bodies useful as long as a person lives by influencing their epigenetics, we could make people productive and happy for decades longer.

</comment><comment><USER6> Perhaps a more layman-focused explanation: [https://www.alzforum.org/news/research-news/aging- epigenetic...](https://www.alzforum.org/news/research-news/aging-epigenetic- wet-blanket-douses-mitochondria)

<comment><USER7> Thanks, this was very helpful.

</comment></comment><comment><USER8> If anyone want to know the heads or tails of this, read the book 'Lifespan' by David Sinclair. It has good explanations and awesome research findings on ageing like this study.

</comment><comment><USER5> Does anyone know what the implication for humans is? Do we know what might be activating these epigenetic regulators in humans and how we can reverse them?

<comment><USER9> According to David Sinclair, there are some things that are likely to work:

\- Lots of exercise

\- Diet changes

\- Intermittent fasting

Wikipedia about him: [https://en.wikipedia.org/wiki/David_Andrew_Sinclair](https://en.wikipedia.org/wiki/David_Andrew_Sinclair)

His book: [https://en.wikipedia.org/wiki/Lifespan:_Why_We_Age_- _and_Why...](https://en.wikipedia.org/wiki/Lifespan:_Why_We_Age_- _and_Why_We_Don%27t_Have_To)

Podcast about this (you can skip the first 8 or so minutes of ads): [https://pca.st/rpnecpmb](https://pca.st/rpnecpmb)

<comment><USER10> He also eats NMN, resveratrol and metformin (not for diabetes, but for anti- aging). He talks about these measures that he takes and the research he uses to motivate them in his podcast with Joe Rogan (can be found by "Joe Rogan David Sinclair"). The research is not as conclusive yet as to be fully agreed on upon everyone after rigorous testing up to WHO standards, but it is solid enough for this researcher (in the field of aging no less) to consider these measures valuable.

</comment></comment></comment><comment><USER11> I've one smart body composition scale. It shows my mom's age 1/3 of her original age and my age has 5 year plus.

It means I am older than my mom according to Mi smart Scale.

Anyone knows what they use to determine that?

<comment><USER12> They use wishful thinking. Body composition scales are crap, don't trust their numbers. Most just run a little current through your legs and then guess some numbers. The error bars on that are so huge that it's basically meaningless to compare between people. The have limited utility for comparing your body composition to your past self.

<comment><USER13> I agree smart scales have a massive margin of error for BMI, muscle weight, water %, etc..

That said, they are useful when used on a daily basis over many months. Clear trends show up accurately (I started using a Withings scale 8 months ago and have lost 30lbs in that period, no single measurement is useful and they often swing wildly up and down, but the trend lines over time are what I found valuable, and as accurate as needed, at least for my purposes)

<comment><USER14> Margin of error for BMI is small as weight and height are relatively easy to measure accurately.

As for BF % guesswork - you'll probably do better with simple calipers - <USER2> least those measurements are based on something tangible

</comment></comment></comment></comment><comment><USER15> Is the site down?

<comment><USER16> Yep. Looks like we missed the immortality bus by just a few hours. Darn :(

</comment></comment><comment><USER17> Can someone put it in Layman's terms?</comment>


Facilitation Tools for Meetings and Workshops (2013) [pdf] - <AUTHOR> https://seedsforchange.org.uk/tools.pdf <comment><USER2> Some of these feel pretty dated. For something a bit more recently updated, Hyper Island have quite a good toolbox of workshop exercises: [https://toolbox.hyperisland.com/](https://toolbox.hyperisland.com/)</comment>


Testing release of a platform for hosting pure functional web applications - <AUTHOR> http://lambda-the-ultimate.org/node/3836

<comment><USER2> Ur/web looks very cool - I just skimmed through some of the documentation. I have often thought about creating my own web framework and writing a DSL using Gambit-C (probably by building on Blackhole, Sprock, etc.) Something with the ease of use simplicity of Sinatra.

<comment><USER3> Yeah, great documentation. It's rare to see such extensive documentation for a newly released project, and I must say I am enjoying reading through it.

Funny, just heard about LunaScript[1] the other day, and it seems they both make HTML/XML first class. Personally, I want to go the opposite way. I don't want to know about HTML or CSS being generated at all. Templates are a good first step, to separate logic and presentation, but I would prefer just to describe GUI widgets and callbacks. Django's forms are a nice step towards this type of thing, but I feel it can be taken farther.

[1] <http://www.asana.com/luna></comment></comment>


Google Outage  - <AUTHOR> http://www.google.com/appsstatus#hl=en&v=status&ts=1366204363099

<comment><USER2> For a second, thanks to that headline, I thought this was a new service from Google (and frankly, I was excited for it).

<comment><USER3> 'Helps you spend more time away from the computer, doing the things you really care about!'

Sign me up :)

<comment><USER4> You probably just spoiled Google's April fools for next year.

<comment><USER4> I assume the downvote means "fuck you now you really spoiled it". Thanks google employee ;)

If not, please comment what's wrong, then I might be able to post better comments in the future.

</comment></comment></comment></comment><comment><USER5> It's oddly amusing that a similar outage occurred exactly one year ago, 17 April 2012: [http://www.theverge.com/2012/4/17/2954949/gmail-outage- april...](http://www.theverge.com/2012/4/17/2954949/gmail-outage-april-2012)

</comment><comment><USER6> > Google Documents service has already been restored for some users, and we > expect a resolution for all users in the near future. Please note this time > frame is an estimate and may change.

So many words for "we're still working on it."

</comment><comment><USER7> My hosting provider had some serious connection issues moments ago, and I'm seeing some major latency and packet loss at L3: <http://internethealthreport.com/>

Anyone have any insight?

</comment><comment><USER8> Well that's an unfortunate coincidence. They only just sent out this email a couple of days back: <http://i.imgur.com/yZOdKTb.png>

<comment><USER9> To be fair, they only quoted 3 9's and I doubt they'll lose that because of this

<comment><USER10> uptime percentages are a useless measure

<comment><USER11> It depends. You can look back at a year and say "We had 100% uptime". You can also make an assurance that you have %90 uptime, %99, 99.9, or even %99.99 and beyond which all actually carry a meaning about load balancing, fall over, and redundancy.

</comment></comment></comment></comment><comment><USER12> You gotta feel bad for that intern that tripped over a server cable.

<comment><USER13> Google is better designed than that. No single cable failure could cause a user-facing outage of more than a few minutes.

<comment><USER14> I hope not.. If it takes more than a couple of minutes for you to figure out how to plug in a cable, I would wonder why google let you in the building

<comment><USER13> If it's a main multimegawatt power cable, it could take a while to plug back in (first the charred corpse needs to be removed). But it doesn't take long to redirect all traffic to another datacenter.

</comment></comment></comment></comment><comment><USER15> Both apps- and non-apps working fine here in the UK.

From the status page: "This issue is affecting less than 0.007% of the Google Mail user base"

<comment><USER16> Maybe that's true in the usual way we'd understand that percentage (the percentage of total users subject to this particular issue). But based on the fact that all seven of my Google accounts were inaccessible via IMAP, I'm wondering if they only had 0.007% of users fail in their attempt to access the service during that window, while many other potentially-affected users were simply blissfully ignorant of the issue.

</comment></comment><comment><USER17> The UK University I work at is experiencing problems with IMAP connections and intermittent problems with Google Drive. I believe other UK Universities are also experiencing similar problems.

Web access to the same services seems OK though.

</comment><comment><AUTHOR> Drive and Docs are down now!

</comment><comment><USER18> Of my 10 google apps for domain accounts, 9 were down. They seem to have all trickled back online over the past 30 minutes.

Looks like things are coming back online gradually.

</comment><comment><USER4> I was expecting a blogpost about it with explanation like Cloudflare always has. But no, Google's not like that I guess.

</comment><comment><AUTHOR> Experiencing problems with IMAP connections!

</comment><comment><USER19> My university google accounts were down earlier. They're back now though.

</comment><comment><USER20> Google Groups also failing.

<comment><USER3> Something something rimshot.

Ironically, Google Reader has been fine.

</comment></comment><comment><USER21> heh, as I was first scanning the HN homepage, I read this as "Google Outrage" and immediately thought it was talking about something like the scroogled campaign.</comment>


Contoso Fooled by Google, Returns to Microsoft - <AUTHOR> https://blogs.technet.microsoft.com/whymicrosoft/2012/03/30/contoso-fooled-by-google-returns-to-microsoft/ <comment><USER2> For non-Microsoft develepers: [https://en.m.wikipedia.org/wiki/Contoso](https://en.m.wikipedia.org/wiki/Contoso)

 _" Contoso Ltd. (also known as Contoso and Contoso University) is a fictional company used by Microsoft as an example company and domain."_

</comment><comment><USER3>
       We later learned that because Google doesn't support
       Information Rights Management, email message recipients
       can easily edit, forward, or print sensitive
       information. One of our employees had leaked out our
       secret without us knowing it!

That's... not going to stop anyone from leaking an email. They'll just take a photo of the screen with their cell phone. The analog hole can't be closed.

[https://en.wikipedia.org/wiki/Analog_hole](https://en.wikipedia.org/wiki/Analog_hole)

</comment><comment><USER4> 2012 piece but was amused by "Frustrated and confused, many of our employees turned to knitting to drown their sorrows, decreasing their productivity even further."</comment>


Web Cruft: NY Times Homepage takes 3.5 seconds to load, 322 requests - <AUTHOR> http://tools.pingdom.com/fpt/#!/cQEwCh/http://www.nytimes.com/ <comment><USER2> I got 1.2s, 327 requests , 2.7MB, Dallas origin</comment>


Login to Starbucks WiFi is broken on macOS Catalina - <AUTHOR> http://medium.com/p/45610cc016b5 <comment><USER2> Not just stating the facts, he also added a temporary fix!</comment>


Censorship by Google - <AUTHOR> https://wikipedia.org/wiki/Censorship_by_Google <comment><USER2> This is far too generic and popular a topic for a Wikipedia page to make a good HN post about it. See recent explanations at:

[https://news.ycombinator.com/item?id=22990237](https://news.ycombinator.com/item?id=22990237)

[https://news.ycombinator.com/item?id=23117614](https://news.ycombinator.com/item?id=23117614)

[https://news.ycombinator.com/item?id=23089041](https://news.ycombinator.com/item?id=23089041)

When I say generic and popular, here's what I mean: the #1 and #4 stories on HN yesterday were about this theme: [https://news.ycombinator.com/front?day=2020-05-18](https://news.ycombinator.com/front?day=2020-05-18). Even apart from the Wikipedia aspect, good HN submissions on a theme like this need to meet two criteria:

(1) It needs to not be generic - [https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20generic%20discussion&sort=byDate&type=comment)

(2) It needs Significant New Information (SNI) - [https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20%22significant%20new%20information%22&sort=byDate&type=comment)

When the theme is one of the hottest controversies of the moment, multiply all of the above 10x.

If you're worried that this means we're censoring Google-censorship stories, we're not, and HN search is your friend:

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E100%20google&sort=byDate&type=story&storyText=none)

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=comments%3E100%20youtube&sort=byDate&type=story)

<comment><USER3> Censorship by one company isn't specific?

<comment><USER2> Not when it has been discussed many times from many angles. And probably not when the company is a planet.

HN discussion thrives when there are details for the mind to sink its teeth into. This actually brings up another point: lists. Lists don't make good HN submissions. See previous explanations: [https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20denominator%20list&sort=byDate&type=comment)

<comment><USER3> Ah, ok. So an article about a specific incident or a specific, small subsidiary would work?

<comment><USER2> A specific incident, yes. Whether it's a small subsidiary or not probably is a red herring, unless there's something unexpected about that small subsidiary being involved in such a thing.

Really, though, this is just about avoiding repetition. The first principle of HN is curiosity. Everything else follows from that ([https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20curiosity%20optimiz&sort=byDate&type=comment)). Curiosity withers under repetition ([https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adang%20curiosity%20repetition&sort=byDate&type=comment)). Generic discussions are bad because they're repetitive. There's a close relationship between genericness and repetition: generic is what you get when you take a bunch of things that have been seen and heard before, and blend them in a blender.

When a topic has been repeated enough to become generic, SNI is our only friend (see link upthread). That's basically the same as your specific incident. By the way, these specific incidents quickly become generic in their own right. An ongoing example:

[https://news.ycombinator.com/item?id=23223219](https://news.ycombinator.com/item?id=23223219)

[https://news.ycombinator.com/item?id=23221264](https://news.ycombinator.com/item?id=23221264)

[https://news.ycombinator.com/item?id=23213213](https://news.ycombinator.com/item?id=23213213)

[https://news.ycombinator.com/item?id=23172564](https://news.ycombinator.com/item?id=23172564)

[https://news.ycombinator.com/item?id=23232115](https://news.ycombinator.com/item?id=23232115)

(those are just the ones with significant comments that I happen to know about)

Edit - by the way, it has taken years to hammer out these principles to the point that they've become building blocks of moderation that can be linked to and expressed in clear, concrete ways. I think it's probably time to collect this material into a more orderly form.

<comment><AUTHOR> For what it's worth, me posting like to the Wikipedia page was the result of a random Google search trying to find examples of new Google SERP features; Google search being...

[new message in Google search -"support.google.com"]

...with results limited to the past 24-hours. I had not even heard of the YouTube issue and was honestly surprised how much information was on wiki's about Google censorship.

All that said, lol, didn't have an issue with your reasoning, and when I saw how fast the post was climbing wasn't shocked to see it fall just as fast as it happens all the time and you'd pinned your comment too.

</comment></comment></comment></comment></comment></comment><comment><USER3> One big example was Google's censorship during the 2016 American election. The autocomplete suggestions were wildly different than every other major search engine (DDG, Bing, etc.). Typing "Crooked" into every other search engine would suggest first the completion "Crooked Hillary". Searching on google about the Wikileaks Clinton email server leaks wouldn't yield results that would take you to the links. Most notably, the Chrome browser would try to divert you from the website entirely claiming that wikileaks was unsafe to visit.

I'm sure there are plenty more examples and I'm not attempting to portray any particular motive or larger scheme, but there is boatloads of evidence that Google censors political information and news during American elections. For all the talk of Russian interference, you'd think we'd talk more about Google's interference.

<comment><USER4> Is it that Google "censored" it or that other search engines were engineered into posting astroturfed content?

I mean, maybe the proper link would be to crooked.com?

</comment></comment><comment><USER5> Some of this censorship seems to be at the request of various governments. For example the "right to be forgotten" in some EU countries. What do people expect Google to do in such cases?

<comment><AUTHOR> Stating the obvious, Google's response depends on the context, that being: what Google makes of the request, the country, changes requested, demand for Google in that country, blowback from other countries, etc.

Unless Internet is available via a connection not controlled by a given country, which would be hard given even satellites report to countries — and the end user had a safe way to access the connection without upsetting the country they're in, including if they're in the ocean, since (most) boats & airplanes are still bound to a given country and/or country space the happen to be in.

</comment></comment><comment><USER6> Even they removed search suggestions for "黃色" (Yellow).

It's related to the color of the Hong Kong protests: [https://en.wikipedia.org/wiki/Yellow_Economic_Circle](https://en.wikipedia.org/wiki/Yellow_Economic_Circle)

</comment><comment><USER7> Will the wiki link be censored if it mentions the censored stuff?

<comment><AUTHOR> Appears Google even suggests it, or at least that's how I ran across it:

[https://news.ycombinator.com/item?id=23242571](https://news.ycombinator.com/item?id=23242571)

</comment></comment><comment><USER8> Its a good article</comment>


Giant bags of mostly water: Securing your IT infrastructure - <AUTHOR> http://slides.com/mricon/giant-bags-of-mostly-water#/ <comment><USER2> Sorry if I'm being thick, but how is this twodimensional slideset supposed to work? I just want to go to the next slide, should I go all the way down and then right? Or do I need to go up again before going right? Or can I ignore down?

I've seen this being presented before, it seems to be a new thing, but never navigated one myself.

Edit: Just saw a comment about pressing space bar, that seems to work linearly. Thanks! (Source: [https://news.ycombinator.com/item?id=11653281](https://news.ycombinator.com/item?id=11653281) )

<comment><USER3> Just use your spacebar and it'll navigate you linearly, taking you down ad across the pseudotree as you need. Edit, or click downarrow, if no spacebar.

<comment><USER4> See my comment here [0] with its genuine, but as yet unanswered question:

      If you hit space, how do you know
          you end up seeing every slide?

Every seems to be chipping in with these navigation hints, and that's great, but no one seems to answer my question. The site seems to be a triumph of presentation over usability, and I'd really like an insight into the minds of those people who think it's all obvious and usable.

For example, someone else said[1] that if you hit ESC you get an overview. How do you find that? What if you don't have an escape key, such as on my tablet?

I'm not trying to be grumpy or anti-anything, I'm trying to gain an insight into how people think this is a good thing, and to provide, in return, an insight into why I think it's unusable.

[0] [https://news.ycombinator.com/item?id=11653447](https://news.ycombinator.com/item?id=11653447)

[1] [https://news.ycombinator.com/item?id=11653323](https://news.ycombinator.com/item?id=11653323)

<comment><USER5> Slides.com is focused on the presentation. The audience is not typically interacting directly with the deck.

When you create a deck, there is a pretty good tutorial explaining how the interface works, including what spacebar and escape do.

The real advantage of the service, for me, is the support for embedding any digital media you want into a slide via iFrames, and the ability to use your phone/tablet to advanced slides and see speakers notes if your venue did not provide you with a clicker.

It also has another pretty neat feature where audience members can pull up the presentation on their laptops/tablets to follow along, and their slides will automatically advance to match my progress through the deck.

I'm just trying to explain why its very much a tool for the presenter, not the presentee.

Edit: Slides.com is a locked down instance of reveal.js

Here is a link to their deck, explaining all of its features [http://lab.hakim.se/reveal-js/#/](http://lab.hakim.se/reveal-js/#/)

</comment></comment></comment></comment><comment><USER6> I absolutely loathe this style of presentations. It always feels like I've missed a part. Call me old fashioned but the UI and UX of linear presentations is so much easier to use.

<comment><USER7> You can simply press "space" to go to the next slide, almost like a linear presentation.

<comment><USER4> How do you know? How can you be sure you reach every slide simply by pressing space?

What about on my tablet, which doesn't have a space bar?

Edit: These are genuine questions, and I am _deeply_ frustrated that it's being downvoted without any discussion. People are claiming that hitting the space bar takes you linearly through the entire presentation. _How do you know you reach every slide like that?_

Really, seriously, how do you know?

As far as I'm concerned the navigation is opaque - if I hit space, I don't know that I'll reach every slide.

 _How do you know ?!?_

I suspect that people are enamoured with the attractiveness of the presentation - it's cool, it's slick, it's gorgeous, it's wonderful - and when I question its usability or discoverability, people are downvoting because they have no answer, and just get pissed off at being questioned.

Do you actually have an answer? If so, tell me.

<comment><USER8> You have raised a very important issue here. Today's web has been festered with such sucking UI/UX. It just caters to the demands of novelty-seeking users with no concern for usability/discoverability.

<comment><USER9> > It just caters to the demands of novelty-seeking users

Not even sure if users or just developers and website owners. As in - _oh man, that looks so cool, let 's do that!_

<comment><USER8> Indeed, you made a very sharp statement. It might be driven, to a very large extent, by the crazy novelty-seeking ideas of the website owners and devs.

Then if some equally stupid "journalist" gives praise and publicity to their foolish UI/UX this trend increases and perpetuates. Unfortunately, it seems to be happening to a very large extent. The irony of this article is that it talks about security, and calls for anti-bloat things.

</comment></comment></comment></comment></comment></comment><comment><USER10> I'm sorry you all have to read just the slide deck. It's an hour-long presentation and a lot of content is simply not in the deck. :( Unfortunately, every time I've presented it, the talk was not recorded -- hopefully I'll eventually present it somewhere else that will capture it for me.

Additionally, here's a small op-ed piece that is supposed to go with it: [http://mricon.com/i/airbags-and-steel- frames.html](http://mricon.com/i/airbags-and-steel-frames.html)

<comment><USER11> It's a good presentation with many good points outside the horrid formatting. Just turn it into a PDF with slides for goodness sakes. Write key pages on a piece of paper for audience questions where you have to go back. Should work fine. :)

Btw, one thing worth correcting is false claim that QubesOS was or is only attempt at workstation security. I've evaluated almost a dozen over past 10 years with some still existing. List those here:

[https://news.ycombinator.com/item?id=11654680](https://news.ycombinator.com/item?id=11654680)

You really need to look up separation kernels as isolating most critical stuff in a dedicated partition protected with 4-12kloc kernel is one of strongest approaches. seL4 and Muen are examples with GenodeOS an example of FOSS attempt to do a Nizza-like architecture with strong foundation and best-of- breed components (esp Nitpicker GUI). High-assurance security is moving forward with hardware-software architectures with one maybe getting SOC release (plus source code) in 1-2 years. Yet, our prior work with separation kernels/VMM's plus safe code (esp SPARK Ada or C w/ Astree Analyzer) for trusted components is _still_ stronger than any crap mainstream FOSS, VMware, etc are making. They rarely learn from the past.

Note: Email me if you want more examples of past and current high-assurance work. I have collected them for most focus areas with papers, prototypes and/or products.

<comment><USER10> > Just turn it into a PDF with slides for goodness sakes.

Hey, I'm not the ones who linked to slides.com. :) The PDF version is linked off the main conference page: [http://kernsec.org/files/lss2015/giant-bags-of- mostly-water....](http://kernsec.org/files/lss2015/giant-bags-of-mostly- water.pdf)

> Btw, one thing worth correcting is false claim that QubesOS was or is only > attempt at workstation security.

You must look at my statements in the context of presenting this at the Linux Security Summit. You know a lot more about this than me, obviously, but from what I can tell, each of the other solutions you mention run custom non-Linux microkernels that provide virtualization to other consumer OSes. I'm ready to be educated here, but I believe I didn't misstate that QubesOS was one of the first pure-Linux mainstream attempts at workstation security through compartmentalization.

<comment><USER12> Oh, you're a legend, no more reveal.js thanks for the link!

EDIT: It was 28MB so I compressed it down to 1.7MB here (image quality wont be as good but meh): [https://www.dropbox.com/s/8bu3rkj6pjbneiv/giant-bags-of- most...](https://www.dropbox.com/s/8bu3rkj6pjbneiv/giant-bags-of-mostly- water.compressed.pdf?dl=0)

</comment></comment></comment></comment><comment><USER13> On the workstation part, it recommends QubesOS.

Am I the only one who is skeptical about it?

From what I saw superficially reading their source code, there are some frightening stuff going on:

* tons of C code with nearly zero unit tests, same with the python code

* lots glue in form of bash or python scripts

* some not so beautiful stuff like:

\- [https://github.com/QubesOS/qubes-core-agent- linux/blob/maste...](https://github.com/QubesOS/qubes-core-agent- linux/blob/master/vm-init.d/qubes-firewall#L29) (kill -9 on a daemon...)

\- [https://github.com/QubesOS/qubes-core-agent- linux/blob/maste...](https://github.com/QubesOS/qubes-core-agent- linux/blob/master/vm-init.d/qubes-firewall#L18) (a daemon is a little bit more than an exe launched with '&'

\- [https://github.com/QubesOS/qubes-core-agent- linux/blob/maste...](https://github.com/QubesOS/qubes-core-agent- linux/blob/master/vm-init.d/qubes-core-appvm#L34) (changing a config file in an init script, humm, weird...)

\- [https://github.com/QubesOS/qubes-core-agent- linux/blob/maste...](https://github.com/QubesOS/qubes-core-agent- linux/blob/master/vm-init.d/qubes-core-appvm#L32) (starting a service inside the init of another service...)

\- [https://github.com/QubesOS/qubes-core-agent- linux/blob/maste...](https://github.com/QubesOS/qubes-core-agent- linux/blob/master/vm-init.d/qubes-qrexec-agent#L13) ("logging" with stderr redirection in a file)

And it's just the init scripts... I'm too lazy to take a look further inside the C or python stuff.

IMHO, as a proof of concept, it's interesting, as a finished, reliable and secure OS, it's frightening...

<comment><USER10> It's not really a recommendation. It's presented as one of the free software projects attempting to tackle workstation security. Another one is SubgraphOS.

<comment><USER11> That's not how I read it:

"The only serious attempt at workstation security"

"The Volvo of blah blah"

Quite a slam to those of past and present that handed NSA or DOD pentesters their asses back to them. Maybe be more accurate if you said "a FOSS attempt at workstation security" minus Volvo part. Volvo probably goes to INTEGRITY-178 as SKPP cert requires more attack areas to be covered plus 2 years of pentesting for kernel. Genode Architecture is prime contender for FOSS far as foundations go. Next time a FOSS project claims to be designed securely just ask for a covert storage and timing channel analysis of any components that handle secrets. They'll either say "Huh? What's a covert channel analysis?" or "We don't really have anyone doing that as we're too understaffed or it doesn't really matter." ;)

</comment></comment></comment><comment><USER4> Perhaps I'm wrong, but I've always remembered this as

        *Ugly* bags of mostly water.

Am I wrong?

And along with several other commenters here, I strongly dislike this sort of multi-direction navigation with no overall map showing what's there, where I've been, and what I haven't yet visited. Beautifully designed and presented, with no concern for the user experience.

A bit like the cars being described.

        The car is designed  | The presentation is
            perfectly.       |     designed perfectly

        Any crashes are the  | Any inability to navigate
            driver's fault.  |     is the user's fault.

_EDIT: The Wikipedia page about this episode agrees with me, but other "quotation" pages have "ugly giant bags of mostly water." Does anyone have the episode to hand and the time to watch it? Am I paying too much attention to trivial details?_

 _EDIT II: Found a script[0]: First reference is in an initial translation and is:_

        Ugly... Ugly... Giants...
        Bags of Mostly Water...

_Subsequently as the translator gets better it 's:_

        Ugly Bag of Mostly Water

_So there we are._

 _EDIT III: Wow - downvotes! No complaint, obviously people feel either that this comment is wrong, or doesn 't belong. I'd appreciate knowing why people might think that, but I guess I'll never know. Which is a shame, I'd welcome the opportunity to learn._

[0] [http://www.st-minutiae.com/resources/scripts/117.txt](http://www.st- minutiae.com/resources/scripts/117.txt)

<comment><USER14> Press escape to get the slides overview.

<comment><USER15> There's no escape on my tablet. Also, the whole experience is horrible on tablet also wrt the other parts of navigation. Doesn't fit right on screen, navigates when trying to adjust, jumps around when trying to navigate. Hard to hit the arrows. I bailed out on this after three slides. Unusable.

</comment></comment></comment><comment><USER16> Let me know when your team of 10 security people that the corporation spends a million per year on is ready to tackle all these issues. Not one overworked IT guy that keeps getting shit from dumb people like the CEO who don't get it. Otherwise forget it, security is not happening, because there is no budget for it. At the end of the day the problem with security is not security, it's money, we have more than enough tools to make everything airtight. What we don't have are the budgets to make this happen, so instead of proactive security you get reactive security, and the CEOs and other executives don't care about that until it happens because it costs money.

<comment><USER10> Of course, if the company leadership doesn't care, then you will have a hard time convincing them why the upfront effort of "doing it right" is worth it. When dealing with this situation, I found it useful to compare IT security people to lawyers. Wait, hear me out before you shout me down. :)

To the non-initiated, lawyers and infosec people are seen with nearly-equal amount of both dislike and trepidation. They are seen as a force of lawful evil that descends on your team and starts telling you that all those cool things you're trying to do cannot actually be done, or must be done in a non- obvious roundabout way. When asked for reasons, both lawyers and infosec start talking about concepts that are entirely unfamiliar to most devs (code provenance, license agreements, trademarks, patent litigation, IP isolation, containers and namespaces, RBAC policies, multifactor authentication). All you care about is that this is a person who is telling you that your project, 99% complete after your team worked multiple 60-hour weeks, must be delayed until a bunch of things -- that you don't consider broken! -- are fixed.

However, this is where things usually go differently. If a lawyer comes to management and says "this project cannot launch because a bunch of code was copy-pasted from stackoverflow and links with an incompatibly-licensed library," the management is likely to listen even if they don't understand a word of what was said -- because they know the importance of lawyers and know that, in the long run, litigation is extremely expensive. However, if an infosec person comes to them and says "this project cannot launch because they have a PHP script running as root that listens on external port 80," management will not value this input nearly to the same degree, even though, in the long run, a bad security vulnerability can have just as much of a detrimental impact on a company as litigation -- and probably worse, because you won't be able to hush-hush and "settle out of court."

The reasons for this are multiple -- infosec is in infancy compared to the legal field, and, sadly, many IT security practitioners tend to look and act in a way that makes their recommendations carry so much less weight with upper management.

So, where I'm going with this is -- if you work for a company in an infosec field and you genuinely want to improve things to the point where management actually starts to listen (which translates into $$ for your team and your projects), then you need to both convince them that your expertise is equally as important as the lawyers', and probably present yourself with the same amount of gravitas as those working on the legal team.

</comment></comment><comment><USER17> Anyone here has good pointers to get started with and understand [0] selinux?

[0]: in that order preferably : )

<comment><USER18> I found that this talk (selinux for mere mortals) was good for me to start off - [https://www.youtube.com/watch?v=cNoVgDqqJmM](https://www.youtube.com/watch?v=cNoVgDqqJmM)

After that, Red Hat's documentation was probably the next most useful thing for getting started; [https://access.redhat.com/documentation/en- US/Red_Hat_Enterp...](https://access.redhat.com/documentation/en- US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/)

But mostly it was a lot of proding and poking whilst setting various things up that helped a lot, and remembering that when something doesn't work, to check selinux first.

The 'settroubleshoot-server' package for the Red Hat based distributions is also good whilst you're getting started in dev, as it takes the avc logs and 'guesses' what the likely cause of the problems you've had are, giving percentage likelihoods of the policies and booleans that might be causing problems.

<comment><USER19> When I started using Linux very heavily for a lot of things, nothing really clicked until I started paying real attention to logging.

If you configuring logging correctly -- or just be aware of it, really, you can always optimize later -- you are no longer SysAdmin-ing blind.

</comment></comment></comment><comment><USER20> Is there a link to the talk? The slides just list "Networks", "Servers", "Workstations" and so on... Should there be audio? I don't hear anything.

<comment><USER21> You might be missing the "down" part of each slide? In the bottom right corner there's the 4-arrow thing, and if down is 'not disabled', there are more slides to see.

I can't stand this kind of "2-d" presentation...

<comment><USER20> Oh thank you! Yeah, I was just pressing the right arrow key so was missing all the information.

</comment></comment></comment><comment><USER22> This aricle falls foul of what I might call 'security shopping'\-- passing mentions of lots of brightly coloured complex sounding security things with very little regard to what exact problem they're solving.

They mention a VPN or insecure access panel having bad permissions, but recommend a mixed bag of differently coloured jellybeans as the solution without once reccomending shutting down the PHP script, allowing access from the VPN only through certificate, password and hardware two-factor authentication, and ensuring good access controls and employee on- and off boarding systems.

Far more importantly, I question the efficacy of any security recommendation that doesn't mention threat modelling at all. What is it you want to protect? What's it going to cost to protect these things? What's it going to cost to lose them? What's the simplest and most effective best way of protecting these? Is it really moving your entire system to a different platform and upgrading all your cypto -- ask yourself -- are we really installing air bags, or are we building our car out of armour plates? Some kid is going to spend 2 hours on XSS in your app if you spend all your resources investing in in- datacentre encryption and service-to-service authentication.

<comment><USER23> Im sure threat modelling is something everyone does implicitly.

As someone who practices security, I found the keywords you can pull from the slide reasonable in their suggestions to follow up on. There were a couple of places he went into the weeds, and I think he probably could have talked up iOS security a bit more instead of smart cards which are a bit overkill relative to his other suggestions.

But, this is just a slide deck. Try not to rush to judgement considering we didn't hear the talk that came with it.

<comment><USER22> > I'm sure threat modelling is something everybody does implicitly.

You may work somewhere that this is the case, but I can't count the number of times I have tested an application where someone has equated security to having an A+ HTTPS rating.

> This is a slide deck

Understood, and something I didn't consider before. That said, I think my comments will still be useful to those here who have also not seen the original talk.

</comment></comment></comment><comment><USER24> First off: just hit space. You'll see the whole presentation.

\----

As a developer, I definitely liked the framing of the presentation. Though I don't think it goes far enough in emphasizing defense in depth. Put simply, user workstations should ideally _never_ be trusted. Getting into the network shouldn't help an attacker much if everything requires authentication even once you're in.

<comment><USER25> How do I hit space on my tablet?

</comment></comment><comment><USER26> The analogy between IT security and car safety is really insightful. However, it's also interesting that cars are two to three orders of magnitude less safe than rail and air travel, respectively, in terms of injuries and fatalities per passenger mile. Maybe we should be thinking about how to make using computers as safe as air travel, rather than as safe as car travel.

<comment><USER27> Why are different modes of travel's safety measured in passenger mile instead of passenger hours? To measure utility of distance traveled, it makes sense, but passenger mile doesn't reveal how safe it is per hour that I sit in a particular vehicle (train, plane, car, etc.).

<comment><USER24> > Why are different modes of travel's safety measured in passenger mile > instead of passenger hours?

Because it's really the only sensible way to compare different forms of transportation.

Imagine a teleportation device with an accidental death rate of 1 per 10 trillion miles. But it only takes 2ms to operate (regardless of distance), so the death rate measured in hours would be horrible.

Compare that to covered wagons. Per mile, they're quite dangerous—going on a long trip in one might mean a 1 in 10 chance of death. But they're also extremely slow. So measuring death in terms of hours would make them look safe.

From a safety perspective, which would you rather use?

</comment></comment></comment><comment><USER28> What I love is the smartcard... Well ... Main AAA (Authentication/Auhtorizaation/Accounting) problem is the clould.

How can we federate identity and manage them safely?

Plus, once you decide to be connected how to make AAA system talk to each others?

And then, sometimes you need to make money ... and you know, safely pass token forth and back... And what standard solution do we have that is not a framework?

Well, 3GPP proposed IMS based on IETF Diameter. Still not there.

Some proposed Role Based Accounting and proxy authz based on LDAP ... well not really deployed.

So we are also waiting for a new standard of inter communication of centralized Enterprise Directory that has tokens, tickets, multi policy for authentication according to origin, roaming, sane schema...

LDAP is honestly a good tool, is it actually relying on a secured, strongly typed NOSQL. But, I hardly saw any devs understand the anonymous bind then authed bind mechanism.

I do feel our biggest problem is not in the tools/technology, but rather in too much education of people that are useless in production.

<comment> jon-wood > Main AAA (Authentication/Auhtorizaation/Accounting) problem is the clould. > > How can we federate identity and manage them safely?

AWS, and I assume Google's stuff, will let you integrate their authentication system with your internal one using SAML to authenticate.

<comment><USER28> SAML is based on SOAP no? SOAP that google dropped for security and overcomplexity concern while security studies tends to say the bigger risk is using system you cannot fully understand, no?

It seems to me like modern security by trust in "expert" rather than understanding the basics. But I guess I am wrong in my appreciation?

<comment><USER29> SOAP is just a message format though, so does it matter? SAML works, so long as the implementation is good (assumption, yes)

<comment><USER28> and Oauth is said to be a security framework that can easily go wrong... in its implementation of a solution. there is no concern here neither?

<comment><USER29> Dont know, not familiar enough, my point was writing something because it talks in SOAP seems...petty?

<comment><USER28> Hell often lies in the details

</comment></comment></comment></comment></comment></comment></comment><comment><USER30> Giant bags of mostly water? I thought we were mostly empty space with a dash of perturbations.

</comment><comment><USER31> The slides mention IP reputation and a great free resource for that is [http://GetIPIntel.net](http://GetIPIntel.net) to make sure unexpected IPs don't connect to your services.

</comment><comment><USER32> IMO, the key challenge to IT security is business justification. The costs are very explicit: it's a shit ton of work to enable SELinux, and 2factor auth isn't free. Meanwhile the benefits are fairly nebulous: 2 factor auth doesn't prevent shellcode injection into imagemagick.

The end result is you have a low probability of a huge impacts occurring, without enough data to suggest the distribution of low probabilities, or the nature or distribution of huge impacts. You're sort of selling tiger repellent rocks, and when the big event does happen, you now have to convince the executives the counterfactual that if you had the budget to implement X, Y and Z, the event would have been prevented. Meanwhile, the costs on team velocity are silently ignored.

The best case scenarios here probably revolve around insurance. Compliance auditors can impose non-compliance fees, and giving security teams a direct financial consequence avoided to point to would help their budget justifications.

<comment><USER33> I was watching S01E05 of Mr. Robot, which involves the crew breaking in to a huge data warehouse facility reminiscent of Fort Knox. My friend asks me if places like that really exist and I laughed, "No company cares enough to pay that much to protect their data".

</comment></comment><comment><USER34> This just in: people on HN doesn't know how to navigate a slidedeck.

What gives? More comments on the format rather than content

<comment><USER35> Exactly, I've been annoyed more than once by pure whining on the format/color/navigation of a site and little to no comments on content. [Now going meta whining about the whining, but I'll try not to do it again, I promise]

<comment><USER36> I don't mind the meta comments when they're confined to one thread, but it gets annoying when there are more meta threads than on topic ones.

People, read the comments before you comment. Chances are someone already complained about the slide format or the font color or whatever.

<comment><USER34> Correction: Chances are someone already complained about something or whatever.

</comment></comment></comment></comment><comment><USER37> Very good read on security and how we can improve systems

<comment><USER23> Agreed, some good ideas here. Reading the comments here though I think you probably have to care about security to get anything from it though.

</comment></comment><comment><USER38> "This is what DevOps is about: running Ops like you're Developing an app, not letting your devs run your ops."

This is a very common attitude of sysadmins who think that configuration management is the only thing they need to do to "become DevOps". Sadly, years after DevOps movement has started, the majority of people who are "doing DevOps" are those sysadmins who just added Puppet or Chef to their toolbox.

Security is a very difficult subject when it comes to DevOps practices, but the approach given in this presentation is definitely something I would not want to be part of. Unless what they are securing is a nuclear reactor control center.

<comment><USER39> On the other hand, I have yet to work with, or even meet a developer who thinks that DevOps _isn 't_ dumping .tar files or Docker images directly into production. The developers love it because it gives them _carte blache_ to just hack production any way they see fit. For instance, I had some Java developer attempt to argue with me that .tar files are the same as .rpm's, and that was just one of many incidents. The worst part is, the developers actively spread that propaganda, along with OS packaging requiring root privileges to deploy payload as being unsuitable for continuous delivery.

<comment><USER38> I think this is solvable by educating devs, not just by enforcing policies. And path to production should be a well defined process, that has to evolve from collaboration between devs and ops, so nobody should be able to just "hack production" on their own, and ops could also be doing dev code reviews to see what exactly they are doing.

<comment><USER40> Education is critical, along with ownership. A lot of the bad practices on either side happen when someone can just fob a mess off onto the other group rather than trying to fix it. Once they have that responsibility it's easier to get someone to care about e.g. how a tarball doesn't address dependency management.

</comment></comment></comment></comment><comment><USER12> Any chance you could export the slides to a PDF or similar?

It looks like the reveal.js pdf export function of appending /reveal-js?print- pdf to the URL seems to be broken.

EDIT: Thanks to someone in the comments below who linked to the PDF, it was 28MB so I compressed it down to 1.7MB here (image quality wont be as good but meh): [https://www.dropbox.com/s/8bu3rkj6pjbneiv/giant-bags-of- most...](https://www.dropbox.com/s/8bu3rkj6pjbneiv/giant-bags-of-mostly- water.compressed.pdf?dl=0)

</comment><comment><USER41> Lots of "you're doing it wrong" with no solutions. This is why nobody bothers.

<comment><USER17> Did you read the second last(?) page? It contains this link: [https://github.com/lfit/itpol](https://github.com/lfit/itpol)

</comment></comment><comment><USER42> Anyone got a link to a PDF?

<comment><USER43> [http://kernsec.org/files/lss2015/giant-bags-of-mostly- water....](http://kernsec.org/files/lss2015/giant-bags-of-mostly-water.pdf)

</comment></comment><comment><USER44> The slides said something about an explanation. And then they link to a github repo with one of the most minimal readmes you can create. Well... Good attempt I guess.

</comment><comment><USER45> So what is "IP Score" in the car analogy? A World-wide DMV? With all user information exposed to either a closed-source entity, or an open source effort?

</comment><comment><USER46> Where's the presentation?

</comment><comment><USER47> _Ugly_ giant bags of mostly water.

<comment><USER10> I'm sure we are all beautiful people, which is why I omitted that part. :)

</comment></comment><comment><USER45> The car-analogy is a good sign that the speaker is not able to express the thought directly. The networks are not that simple, and the analogy breaks down at depth for every comparison. Like the modern drivers who consciously chose not wear a seat belt. Or that half of the 1.25 millions of death a year are the "vulnerable users", called so by the who: [http://www.who.int/mediacentre/factsheets/fs358/en/](http://www.who.int/mediacentre/factsheets/fs358/en/) , and fatalities of laws of momentum, where a 150 lbs pedestrian absorbs more energy than a 3000 lbs Scion...

Stop talking cars and and analogies! Talk straight!

</comment><comment><USER39> The author of the presentation lost my respect when I read he is advocating for using SELinux and Linux containers instead of zones and SmartOS, which tells me he is not up to date on security. The whole thing was even made worse by him advocating for the use of continous integration: no, .tar files and Docker images are not the same thing as operating system packages, nor do they come anywhere near the functionality, therefore "DevOps" is insane, especially when the whole thing can be pulled off correctly and securely with SmartOS zones, OS packaging for both delivery and configuration management, and change management process modeling. Those are your airbags, crumple zones, and safety systems, not SELinux, "DevOps", nor Linux "containers". The only thing of technical value in the presentation is how to put SSH keys on SmartCards, and even that is just explained as a series of manual steps (hacking in system engineering context), rather than as a turnkey, OS packaging process (full, high quality, repeatable integration). Damn it how I loathe half-baked stuff from people who should know better but don't: if you are going to be presenting on security, you better know what you are talking about, or else it is _blind leading the blind_.

<comment><USER48> You may be right, but from my perspective you're falling into the same trap that most IT security people do. You're assuming that the mainstream devs should put security first.

They just never will. Follow the money. The money is paying for revenue generating activity.

Using your example that "zones and SmartOS" are preferable to "SELinux and Linux Containers"....that's not going to happen. The people with the money have already chosen the container direction.

A much better strategy would be to give up on what you want, and focus your energy on making the direction that's been chosen more secure.

I suspect if enough talented, security minded engineers descended as contributors for docker, rkt, etc...the situation would improve much faster than the current direction of just complaining they aren't secure.

Perfect is the enemy of good.

<comment><USER39> > Using your example that "zones and SmartOS" are preferable to "SELinux and > Linux Containers"....that's not going to happen.

That has actually already taken place, and is taking place. There is unlikely to be one winner-take-all.

Following the money does not mean that the application cannot be programmed from the ground up to support SmartCards and roles, or that it has to be full of security holes.

> I suspect if enough talented, security minded engineers descended as > contributors for docker, rkt, etc.

Depends on what is under etcetera. Docker and rkt are not the Silver bullets everyone who has not gotten busted by them think they are, they are just a trend. With all of those you instantenously lose lifecycle management, because they are just images of massive file dumps, not images of software and configuration installed with packages. When you use Solaris zones in SmartOS, Docker and rkt become completely superfluous, because you suddenly get a fully working yet completely isolated UNIX server, running at the speed of bare metal no less. Add some OS packages on top of that, make them into an image for imgadm(1M), and in few seconds you're done. What does one need Docker for in that scenario?

And I should certainly hope that perfect is the enemy of good, because life has taught me, the hard way, that good isn't good enough. I absolutely _hate_ being woken up during the night because of an incident, and will go out of my way to get as close to perfect as possible in order to be able to sleep through the night.

<comment><USER48> >That has actually already taken place, and is taking place.

"Not going to happen" meaning it's not going to overtake docker/containers in terms of overall mindshare, activity, etc.

If you're focused on driving the best solution in some sub niche, yes...you can be successful with that.

But, the larger market is going to continue prioritize things that directly generate revenue over everything else.

<comment><USER39> > If you're focused on driving the best solution in some sub niche, yes...you > can be successful with that.

I would be hard pressed to call securing containers, virtualization and cloud a niche, since that is exactly what SmartOS has been designed for, from the ground up.

A large part of generating revenue is not having downtime caused by data corruption incidents or security breaches (or both), which means picking and using a substrate which provides guards against that.

<comment><USER48> The sub-niche would be the smaller number of people that recognize your list of concerns and choose SmartOS over Docker.

Just as Betamax was a technically better solution than VHS.</comment></comment></comment></comment></comment></comment>