The C10M problem - <AUTHOR> http://c10m.robertgraham.com/p/manifesto.html <comment><USER2> What's significant to me is that you can do this stuff today on stock Linux. No need to run weird single-purpose kernels, strange hypervisors, etc.

You can SSH into your box. You can debug with gdb. Valgrind. Everything is normal...except the performance, which is just insane.

Given how easy it is, there isn't really a good excuse anymore to not write data plane applications the "right" way, instead of jamming everything through the kernel like we've been doing. Especially with Intel's latest E5 processors, the performance is just phenomenal.

If you want a fun, accessible project to play around with these concepts, Snabb Switch[0] makes it easy to write these kinds of apps with LuaJIT, which also has a super easy way to bind to C libraries. It's fast too: 40 million packets a second using a scripting language(!).

I wrote a little bit about a recent project I completed that used these principles here: [https://news.ycombinator.com/item?id=7231407](https://news.ycombinator.com/item?id=7231407)

[0] [https://github.com/SnabbCo/snabbswitch](https://github.com/SnabbCo/snabbswitch)

<comment><USER3> Snabb Switch looks awesome.

Is there a list of "things" (not sure of the terminology?) people have built with it?

I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).

<comment><USER4> I'm the Snabb Switch originator.

The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack.

Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing Intel VMDq hardware acceleration and providing zero-copy Virtio-net to the VMs. So the virtual machine will see normal Virtio-net but we will make that operate really fast.

Inside the VMs we can either access a hardware NIC directly (via IOMMU "PCI passthrough") or we can write a device driver for the Virtio-net device.

So, early days, first major product being built, and lots of potential both inside and outside VMs, lots of fantastic products to build with nobody yet building them :-)

[1] [http://blog.ipspace.net/2013/11/deutsche-telekom- terastream-...](http://blog.ipspace.net/2013/11/deutsche-telekom-terastream- designed.html)

[2] [https://ripe67.ripe.net/archives/video/3/](https://ripe67.ripe.net/archives/video/3/)

[3] [https://github.com/SnabbCo/snabbswitch/blob/snabbnfv- readme/...](https://github.com/SnabbCo/snabbswitch/blob/snabbnfv- readme/src/designs/nfv/README.md)

<comment><USER5> How about this use case: I have a ChromeCast on my home network, but I want sandbox/log its traffic. I would want to write some logic to ignore video data, because that's big. But I want to see the metadata and which servers it's talking to. I want to see when it's auto-updating itself with new binaries and record them.

Is that a good use case for Snabb Switch, or is there is an easier way to accomplish what I want?

<comment><USER4> That sounds pretty reasonable to me.

If you can express how you want to filter with a fancy pcap-filter expression the tcpdump is the easy answer. Otherwise you might want to code it up in Lua with snabbswitch.

Here is our basic trace store/replay library today: [https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/p...](https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/pcap/pcap.lua)

<comment><USER5> OK and I forgot to say I might want to deny some traffic... like disable auto updates but still allow it to contact other servers to play video. AFAIK tcpdump doesn't let you do that.

Thanks for the very cool project! I will have to learn more about it.

</comment></comment></comment></comment></comment></comment><comment><USER6> On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me.

On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto.

I sometimes wonder if my urge toward efficiency something like my grandmother's Depression-era tendency to save string? Is this kind of efficiency effectively obsolete for general-purpose programming? I hope not, but I'm definitely not confident.

<comment><USER7> I believe it's what used to be called 'craftsmanship'. Taking pride in creating things that are efficient and not wasteful for no other reason than the desire to make the best product possible.

<comment><USER8> It is wasteful to spend time doing something that benefits nobody, or where the benefits outweigh the cost. Your time is worth something.

<comment><USER9> I think the "just good enough for business" attitude contributed to the demise of the American car and the rise of the Japanese ones. The American tradition was to use engineering tolerances that would maximize throughput under the constraint that it produced a pretty functional car. The Japanese tradition on the other hand was to use tighter tolerances, well because there was room for tightening the tolerance. At the surface level, or the MBA level it seems that the Japanese way is just dumb. It does not make monetary sense when measured in sales per year. But it turns out that the benefits express themselves at a different timescale: better brand and a culture of devotion to improvement, rather than to being "just good enough to work so that I can move to the next thing".

Part of this applies to the craft of software too. One can be sloppy and churn out functional websites by the dozens. At the superficial level, the goal to extract the most from a 500MHX Pentium-III might seem brain dead, with little or no pay off. But it pays back by instilling an attitude of deeply learning your craft, that learning does pay back although that specific well tuned web server might not. You dont have to build all your servers that way, but if it doesnt hurt you a little when you know exactly what you could have done to extract some more juice out of it, you will not reach that level of understanding. If you are impatient or sloppy, you will build impatience into your product, and it will show.

Besides it is always easy to over value ones time, it sometimes correlates with conceit.

@sitkack what makes you think I was talking about the 50s. Going by the downvote I seem to have touched a nerve. You are right Japanese products around that time were synonymous with bad quality, not just in US. Much water has flowed under the bridge since then.

<comment><USER10> Your recollection of Japanese manufacturing is clouded by mythology. Japanese quality sucked in the 1950s, they had a horrible world wide reputation for shoddy goods.

<comment><USER7> Maybe German manufacturing would be a better example?

<comment> guard-of-terra German quality sucked some half century earlier, so much that England had to require them write "Made in" on their goods to warn consumers.

Quality comes and goes.

<comment><USER10> Exactly. When people complain about the quality of Chinese goods as a proxy for China and Chinese people as a whole, I ask them where their MacBook Pro is made and they quickly shut up.

All of these lazy, low tolerance sweeping generalizations need to go!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> It seems we've already passed this problem: "We also show that with Mio, McNettle (an SDN controller written in Haskell) can scale effectively to 40+ cores, reach a throughput of over 20 million new requests per second on a single machine, and hence become the fastest of all existing SDN controllers."[1] (reddit discussion at [2])

This new IO manager was added to GHC 7.8 which is due for final release very soon (currently in RC stage). That said, I'm not sure if it can be said if all (or even most) of the criteria have been met. But hey, at least they're already doing 20M connections per second.

[1] [http://haskell.cs.yale.edu/wp- content/uploads/2013/08/hask03...](http://haskell.cs.yale.edu/wp- content/uploads/2013/08/hask035-voellmy.pdf) [2] [http://www.reddit.com/r/haskell/comments/1k6fsl/mio_a_highpe...](http://www.reddit.com/r/haskell/comments/1k6fsl/mio_a_highperformance_multicore_io_manager_for/)

</comment><comment><USER12> Previous Discussion: [https://news.ycombinator.com/item?id=5699552](https://news.ycombinator.com/item?id=5699552) (9 months ago)

High Scalability Post: [http://highscalability.com/blog/2013/5/13/the-secret- to-10-m...](http://highscalability.com/blog/2013/5/13/the-secret- to-10-million-concurrent-connections-the-kernel-i.html)

Original Shmoocon Presentation: [http://www.youtube.com/watch?v=73XNtI0w7jA](http://www.youtube.com/watch?v=73XNtI0w7jA)

</comment><comment><USER13> WhatsApp is achieving ~3M concurrent connections on a single node. [1][2]

The architecture is FreeBSD and Erlang.

It does make me wonder, and I've asked this question before [3], why can WhatsApp handle so much load per node when Twitter struggled for so many years (e.g. Fail Whale)?

[1] [http://blog.whatsapp.com/index.php/2012/01/1-million-is- so-2...](http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/)

[2, slide 16] [http://www.erlang- factory.com/upload/presentations/558/efsf2...](http://www.erlang- factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf)

[3] [https://news.ycombinator.com/item?id=7171613](https://news.ycombinator.com/item?id=7171613)

<comment><USER14> The problem of 1:1 messaging is slightly different to Twitter, which is more m:n. 1:1 messaging can be handled reasonably easily with a mailbox per user, and there is no shared state. Messaging with m:n has different optimal patterns depending on the relative ratios of m and n. Twitter has many users with millions of followers; if Twitter used a 1:1 mailbox approach like a chat app, these users would be whole countries worth of load on their own.

That's not to say that Twitter's scaling issues where wholly forgivable. They weren't fatal to the service, but I don't think they were necessary with good design from the start. High popularity is a good problem to have though.

</comment></comment><comment><USER15> If you are going to write a big article on a 'problem', then it would be a good idea to spend some time explaining the problem, perhaps with some scenarios (real world or otherwise) to solve. Instead, this article just leaps ahead with a blind-faith 'we must do this!' attitude.

That's great if you are just toying with this sort of thing for fun, but perhaps worthless if you are advocating a style of server design for others.

Also, the decade-ago 10k problem could draw some interesting parallels. First of all, are machines today 1000 times faster? If they are, then even if you hit the 10M magic number, you will still only be able to do the same amount of work per-connection that you could have done 10 years ago. I am guessing that many internet services are much more complicated than a decade ago...

And if you can achieve 10M connections per server, you really should be asking yourself whether you actually want to. Why not split it down to 1M each over 10 servers? No need for insane high-end machines, and the failover when a single machine dies is much less painful. You'll likely get a much improved latency per-connection as well.

</comment><comment><USER16> Here is how C2M<x<C3M connections problem was solved in 2011 using Erlang and FreeBSD:

[http://www.erlang- factory.com/upload/presentations/558/efsf2...](http://www.erlang- factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf)

It shows good practical tricks and pitfalls. It was 3 years ago so I can only assume it got better, but who knows.

Here is the thing though, do you need to solve C*M problem on a single machine? Sometimes you do but sometimes you don't. But if you don't and you distribute your system you have to fight against sequential points in your system. So you put a load balancer and spread your requests across 100 servers each 100K connections. Feels like a win, except if all those connections have to live at the same time and then access a common ACID DB back-end. So now you have to think about your storage backend, can that scale? If your existing db can't handle, now you have to think about your data model. And then if you redesign your data model, now you might have to redesign your application's behavior and so on.

<comment><USER17> If you could do 10M connections on one machine, then why not 1B on 100? Does it even make sense to have a billion simultaneous connections?

<comment><USER2> If by "connection", you mean TCP, probably not. But that's not the only way to maintain connections, and it's certainly not the only reliable network protocol.

My latest project keeps every "connection" open at all times, but it's a custom UDP based reliable messaging protocol, not TCP. At Facebook's scale, we'd have the equivalent of one billion connections "open". It's easy to keep them open, despite changing IP addresses, because every packet is public-key authenticated and encrypted, so you don't have to rely on IP addresses to know who you're talking to...

It also means you only pay for a connection setup time once. For mobile devices, the improvement in latency is palpable.

<comment> mh- Can I ask some questions about that? Extremely interested..

I've considered doing something similar for our messaging/signalling protocol (currently standard TCP sockets established to several million mobile devices.)

I had concerns about what the deliverability of UDP would be on mobile networks; many carriers are going towards NAT'ing everything, (forced) transparency proxies, etc.

Are you only using UDP from device->infrastructure? If not, do you rely on the devices providing an ip:port over a heartbeat of sorts (to keep up with IP changes?)

Do you have any issues with deliverability, in either direction? (not due to UDP's inherent properties, but because of carrier network behavior)

thanks very much for anything you're able to answer.

<comment><USER2> Good questions.

I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything.

It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client<->server.

All I've seen so far is the usual UDP stuff: dropped packets, re-ordered packets, and duplicate packets. Nothing out of the ordinary. Our network protocol handles those things without any difficulties.

We did it specifically because all of our clients are mobile devices, and we didn't want to have to do the lengthy TCP connection setup (or worse, SSL setup) each time the network changed—which is often.

The biggest downside of UDP at the moment is that Apple only allows TCP connections in the background. That seems like a silly decision, but whatever, it's what they've done. I may, someday, set up a bunch of TCP forwarders for iOS devices running in the background. Our messages can be decoded just fine over byte-oriented streams, so it wouldn't change much.

It's a tough call. On the one hand, our UDP clients do not need to reconnect, since the connection is always set up. So when they wake, they send a packet to the server (Hello), and the server immediately sends back any immediate updates, such as new chat messages.

Our read path is perhaps over-optimized, so it's exactly the network latency of one round trip to get the updates since you last opened the app. It takes longer to get the UI up in some cases, so that's why we haven't done the TCP background thing. For others, that might be a much more important consideration.

<comment><USER18> A lot of these features sound similar to MinimaLT: [http://cr.yp.to/tcpip/minimalt-20130522.pdf](http://cr.yp.to/tcpip/minimalt-20130522.pdf)

Do you have explicit DoS protections?

<comment><USER2> That paper (as well as CurveCP) was definitely a huge inspiration for what I'm doing.

A big difference is I'm not running a packet scheduler (e.g. Chicago). Frankly, our data rate on a per device basis is just minuscule, and our internal protocol has back pressure built into it anyway, so I just skipped that part. If it becomes an issue (unlikely), I'll of course actually add an explicit scheduler so our UDP traffic plays nicely with others.

I'm not doing MLT's puzzle step (although I really like the concept). At the moment, all I can do is deny connections from unknown devices if we're under attack (I can drop packets from an unknown device with a single hash lookup). We're also in the process of moving to OVH, which is able to block DDoS stuff at the network edge, should it happen.

There's more stuff I've got planned, but that's it for now.

<comment><USER18> It's interesting how much you can streamline a protocol for a single use case. Did you retain the crypto mostly intact-- including the PFS?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> Projects such as the Erlang VM running right on top of xen seem like promising initiatives to get the kind of performance mentioned ([http://erlangonxen.org/](http://erlangonxen.org/)).

<comment><USER16> I wish they open sourced it and let others look at the code and experiment with it. For a lot of developers if it isn't open = it doesn't exist. Now it is their code and they do whatever they want but that is my view of the project.

</comment></comment><comment><USER20> An implementation of the idea: [http://www.openmirage.org/](http://www.openmirage.org/)

A good talk about it by one of the developers/researchers: [http://vimeo.com/16189862](http://vimeo.com/16189862)

</comment><comment><USER21> > There is no way for the primary service (such as a web server) to get > priority on the system, leaving everything else (like the SSH console) as a > secondary priority.

Just for the record -- the SSH console _is_ the primary priority. If the web server always beats the SSH console and the web server is currently chewing 100% CPU due to a coding bug..

</comment><comment><USER22> Those two articles, [http://blog.erratasec.com/2013/02/multi-core-scaling-its- not...](http://blog.erratasec.com/2013/02/multi-core-scaling-its-not- multi.html) (from Robert Graham) and [http://paultyma.blogspot.com.br/2008/03/writing-java- multith...](http://paultyma.blogspot.com.br/2008/03/writing-java- multithreaded-servers.html), seem to say opposing things about how threads should be used.

Having no experience with writing Java servers, I wonder if any you guys have an opinion on this.

</comment><comment><USER23> I think cheetah OS, the MIT exo kernel project proved this and halvm by Galois does pretty well for network speed that xen provides, but I forget by how much.

The netmap freebsd/linux interface is awesome! I'm looking forward to seeing more examples of its use.

<comment><USER24> i would just love to see that exo kernel from MIT in practice some day in some OS.. i think the research is from the nineties, isnt?

Also, netmap from freebds was the first thing that come to my head, as a relief from the IO bottleneck from moderns systems..

As in the original C10k, freebsd to the rescue here.. since it was the first OS with the kqueue interface.. and now is netmap.. the numbers from the speedup in the original paper are astounding

</comment></comment><comment><USER25> Just what are these resources that we are using more efficiently? CPU? RAM?

Are they that important? Should we not be trying to use electricity more efficiently since that is a real world consumable resource. How many connections can you handle per kilowatt hour?

<comment><USER7> Generally electricity use is roughly proportional to CPU and RAM usage, as they're powered by electricity. If you have two otherwise identical programs, one of which uses 50% of the cpu and the other of which use 10%, chances are the latter will use less electricity.

</comment></comment><comment><USER26> At the risk of sounding dumb, aren't we still limited to 65,534 ports on an interface?

<comment><USER27> Port numbers must only be unique for ip:port pairs. A TCP connection is identified by the "quadruple" source_ip:source_port, dest_ip:dest_port. You can have as many connections as you want on the same source_ip on port 80 as long as there aren't 65,535 to the same dest_ip (ie as long as the quadruple is unique).

<comment><USER28> Also with IPv6 you can easily route a whole /64 net (2 __64 IPs!) onto a single machine.

<comment><USER29> 2^64 IPs. :)

</comment></comment></comment></comment><comment><USER30> >Content Blocked (content_filter_denied)

>Content Category: "Piracy/Copyright Concerns"

I'm starting to use these blocks at my workplace as a measure of site quality (this will be a high quality article). Can someone dump the text for me?

<comment><USER31> That page is essentially a glorified intro to his series of blog entries, and they are on another domain, so perhaps they are not blocked:

[http://blog.erratasec.com/search/label/C10M](http://blog.erratasec.com/search/label/C10M)

<comment><USER30> Ah, erratasec. I'm surprised that isn't blocked here, too.

Thanks for the link.

</comment></comment></comment><comment><USER32> What's the current state of internet switches? Back when I used to run the Napster backend, one of our biggest problems was that switches, regardless of whether or not they claimed "line-speed" networking, would blow up once you pumped too many pps at them. We went through every single piece of equipment Cisco sold (all the way to having two fully loaded 12K BFRs) and still had issues.

Mind you, this was partially because of the specifics of our system - a couple million logged in users with tens of thousands of users logging in every second pushing large file lists, a widely used chat system which meant lots of tiny packets, a very large number of searches (small packets coming in, small to large going out) and a huge number of users that were on dialup fragmenting packets to heck (tiny MTUs!).

I imagine a lot of the kind of systems you'd want 10M simultaneous connections for would hit similar situations (games and chat for instance) though I'm not sure I'd want to (I can't imagine power knocking out the machine or an upgrade and having all 10 million users auto-reconnect at once).

<comment><USER33> 10 Gbps switches are pretty good and are generally line rate (as long as you avoid ten-year-old chassis).

</comment></comment><comment><USER34> "There is no way for the primary service (such as a web server) to get priority on the system, leaving everything else (like the SSH console) as a secondary priority" \- Can't we use the nice command (nice +n command) when these process are started to change its priority? I am sorry if it is so naive question

<comment><USER35> He probably meant that from a TCP point of view, as in there is no way to give a higher priority to incoming TCP connections going into the server than to those going into SSH, even if you could use nice to assign more cpu time to your server.

Or perhaps he meant that the infrastructure used to do multitasking still have to interrupt both his server and SSH, but then described how the kernel can be set to leave some cores free of work then to set the server to use only those and then run absolutely uninterrupted.

Not the only bizarre and confusing statement he wrote, anyways.

</comment></comment><comment><USER36> This article on High Scalability also covers part of the problem: [http://highscalability.com/blog/2014/2/5/littles-law- scalabi...](http://highscalability.com/blog/2014/2/5/littles-law-scalability- and-fault-tolerance-the-os-is-your-b.html)

</comment><comment><USER37> I think OSv or something similar would be part of that solution. Single User / Purpose OS designed to do one / few things and those only.

I could only hope OSv development would move faster.

</comment><comment><USER38> So, he is trying to suggest that pthread-mutex based approach won't scale (what a news!) and, consequently JVM is crap after all?) The next step would be to admit that the very idea to "parallelize" sequential code which imperatively processes sequential data by merely wrapping it into threads is, a nonsense too?) Where this world is heading to?

</comment><comment><USER39> So an exokernel?

<comment><USER33> People don't use actual exokernels; they just use Linux like an exokernel. Aka "1975 programming".

</comment></comment><comment><USER40> One more problem is cloud. We host on cloud. cloud service providers might be using old hardware. Newest hardware or specific OS might be winner but no options on cloud. How do you tackle that ?

<comment> jon-wood If this sort of thing matters that much to you then you bite the bullet and rent a rack to fill with physical servers somewhere.

</comment></comment><comment><USER41> What about academic operating system research that was done years ago? Exokernel, SPIN, all aim to solve the "os is the problem" issue. Why don't we see more in that direction?

</comment><comment><USER35> The two bottom-most articles (protocol parsing and commodity x86) are seriously pure dump, but fortunately the ones about multi-core scaling are pretty damn interesting.

</comment><comment><USER42> Isn't this more-or-less how mainframes work?</comment>


Do Weird Stuff in case it works - <AUTHOR> http://robertheaton.com/?p=111

<comment><AUTHOR> It's so easy to forget to mix things up and yet it makes _SO_ much difference when it works. Also, a change really can be as good as a rest</comment>


Free Thought by Outlandish Thinking - <AUTHOR> https://outlandishthinking.com/free-thought.html <comment><AUTHOR> There's free speech and the question of free will, but lately I've been wondering how free our thoughts are. And I'm pretty sure they're not free to roam wherever they may like, but constrained by unseen guardrails and self- censoring mechanisms that arrest the development of interesting ideas before they can be fully articulated.</comment>


Ask HN: Have you ever had a founder quit or had to fire one? - <AUTHOR>

What advice would you have for someone who's in this position?
News delivered Tuesday. Haven't heard from him since then.
He wrote a critical piece of code that is difficult to understand. Losing him is a major setback.
There were 5 of us; now 4. <comment><USER2> Unless the code is difficult to understand because of what it actually -does-, rewrite it and move on. You'll be better for it in the long run with people who write understandable code.

At least you didn't have one founder quit, then wind up having to fire the other (woo, Apple).

<comment><AUTHOR> It's a mixture of both. The code is complicated, and written in an incomprehensible way (not documented).

He's great at producing code fast, but he's poor at architecture/design and that slows us all down in the long run. However, since we're looking to launch soon, technical debt is not a high-order concern compared to the success of the imminent launch.

</comment></comment><comment><USER3> Ask the others if you can contact him to document the code.</comment>


Ask HN: What books have most influenced the way you see the world? - <AUTHOR> <comment><USER2> The Road Less Traveled by M. Scott Peck - great insight into the human condition

One Minute Manager by Kenneth Blanchard and Spencer Johnson - concise and pithy while containing lots of useful pointers

The Divine Center by Steven Covey - the spiritual grounding that informs his later more secular books and much more interesting if you don't mind religious thought mixed in with your motivation

The Foundation trilogy by Issac Asimov - Caused a huge detour in my life. Immersed myself in speculative fiction for decades due to the brain-quake caused by this material.

Stranger in a Strange Land by Robert Heinlein - A detour of a different sort. Great introduction to alternative modes of thought to my uncultured religious teen mind.

LDS/Christian Scriptures - regardless of your belief level, the ideas/thoughts/stories/literature encompassed in scripture is enriching to the mind

</comment><comment><USER3> Meta comment: It would be more helpful if people included a reason why they are listing these specific books. Seeing a title of a book they never heard of probably isn't enough to get anyone to read it, but a few quick words about why you found it important might intrigue people.

<comment><USER4> can't agree more.. It seems like many are simply listing their favorite books, not necessarily the ones that "most influenced the way" they see the world

</comment></comment><comment><USER5> The two books I've read that have shaped my worldview the most:

 _Stranger in a Strange Land_ by Robert Heinlein - A classic SciFi novel that digs deep on relationships, politics, and religion. Like a lot of old SciFi, it's filled with blatant sexism. If you can look past that, it has a lot of great lessons.

 _Island_ by Aldous Huxley - a beautiful look at alternative societal structures, psychedelics, and the cruelty of the western world overcoming sacred places.

<comment><USER6> Men also smoked more often that women at that time.

</comment></comment><comment><USER7> Zen and the Art of Motorcycle Maintenance, by Robert Pirsig

<comment><USER8> Came here to say this.

The squel "Lila" is also quite profound.

</comment></comment><comment><USER9> Antifragile - Nicholas Nassim Taleb

Mother Night - Kurt Vonnegut

Pavarotti: My World - Luciano Pavarotti, William Wright

Genghis Khan and the Making of the Modern World - Jack Weatherford

<comment><USER2> The Hardcore History podcast's coverage of Genghis Khan/the Mongols is additionally a wonderful resource.

</comment></comment><comment><USER10> Atlas Shrugged (Ayn Rand) - classic book of objectivism , self proud and wealth

Why Nations Fail: The Origins of Power, Prosperity, and Poverty (Daron Acemoglu and James Robinson) - awesome analytics book about wealth source

The Wealth of Nations (Adam Smith) - classic book about source of money and economic processes

</comment><comment><USER11> In no particular order:

Ecclesiastes

The Meditations of Marcus Aurelius

The Prince

How To Win Friends and Influence People

<comment><USER2> I wish I'd included Meditations in my contribution- speaks directly, wisely and usefully even to the modern man's condition.

</comment></comment><comment><USER12> It's quite trendy at the moment, but Homo Deus by Yuval Noah Harari was quite the life changing book. As well as An Astronaut's Guide to Life on Earth by Commander Chris Hadfield was a rather inspirational tale.

<comment><USER13> HomoDeus is indeed wonderful

</comment></comment><comment><USER14> _The Life-Changing Magic of Tidying Up_ by Marie Kondo has life changing philosophical implications when applied outside of simple home life.

 _Tao Te Ching_ by Lao Tzu is a classic text on harmony and humility.

</comment><comment><USER15> The Epistles of the Bible: Romans, 1-2 Corinthians, Galatians, Ephesians, Philippians, Colossians, 1-2 Thessalonians, 1-2 Timothy, Titus, Philemon, Hebrews, James, 1-2 Peter, 1-3 John, Jude

</comment><comment><USER16> \- A day in the life of Ivan Denisovich - aleksandr solzhenitsyn \- Breakfast of champions - Kurt Vonnegut \- Plato's the republic \- The unfettered mind - Takuan Soho

</comment><comment><USER17> Bunch of Eastern stuff...Hagakure, Book of 5 rings, art of war, tao te ching

Subtle art of not giving a fuck, The art of happiness

But the one book I always keep...The boy scout handbook

</comment><comment><USER18> Crime and punishment - made me open minded and try to put myself in other person's shoes. This helped me alot to be less judging.

</comment><comment><USER19> The Ancestor's Tale by Richard Dawkins. None of his militant atheism. Just a brilliant and engaging work on evolutionary biology.

</comment><comment><USER20> The Demon-Haunted World: Science as a Candle in the Dark by Carl Sagan. It helped me be have more cohesion in my thinking in regards to my love for science and my very religious upbringing.

</comment><comment><USER21> The Better Angels of our Nature - Steven Pinker (The world isn't so bad, and it's getting better all the time)

Vagabonding - Rolf Potts (It's easy and cheap to move around the world these days)

</comment><comment><USER22> The Asian Saga by James Clavell, reading as a kid on how to rule your life, business and create a dynasty...

[https://en.wikipedia.org/wiki/The_Asian_Saga](https://en.wikipedia.org/wiki/The_Asian_Saga)

As grown up, on now to rule the world:

More Money Than God: Hedge Funds and the Making of a New Elite by Sebastian Mallaby

[https://en.wikipedia.org/wiki/More_Money_Than_God](https://en.wikipedia.org/wiki/More_Money_Than_God)

</comment><comment><USER23> The Dune series, especially "Dune Messias", made me a vocal critic of organized religion.

Books by Vernor Vinge made me a liberal (european meaning, pro free markets and individual responsibility).

Ken McLeod made me aware of the rich history and valid points of communism and socialism, even when I'm still not a fan.

Iain M. Banks showed me that an AI-enabled future doesn't have to be a distopia. He also made me aware of how pale and boring the often cited "Star Trek Utopia" really is.

<comment><USER18> I really liked dune 1 . But God I hated dune 2 and dune 3. I tried so hard to like 2 and 3. What's your opinion. I would be curious to know

</comment></comment><comment><USER24> I don't read very much, but: 1984 2001

</comment><comment><USER25> The Bible for sure is #1

<comment><USER26> As an atheist, I am curious what in the Bible has influenced you.

I read it and I found that without the faith part (which makes it a book important to you, no matter if you like it or not) it reads as a story like others. I am not trying to be critical in any way, it is that I failed to find really deep thoughts inside, which would make me think after reading it.

It is full of stories which have a moral, not different from many others and not thought provoquing (of course this is my personal opinion) - as a casual reader I be glad to be pointed to such parts there.

<comment><USER25> In fact, there's quite a lot of practical and interesting insights into relationship, finances, discipline and more, for example the book of Proverbs is full of literal advice on dealing with family, children, marriage, business ethics. The letters of Paul to the Romans and Corinthians contain very interesting thought provoking statements on love, relationship, respect. Even if you remove faith completely from the picture you can definitely find very interesting content.

<comment><USER27> When I was younger, I read the Bible from cover to cover, twice.

Now I am much older and as an atheist, I wish I hadn't wasted my time on such simplistic themes written by simplistic men. There are much deeper and more profound books also written by man, and far more complex.

</comment></comment></comment></comment><comment><USER4> The Elusive Quest for Growth by W. Easterly -- read it when I was in college hoping to study development (poverty, growth) economics; changed the way I look at poverty, economics and politics in general. Likewise I helped me ditch modern liberalism for classical one.

</comment><comment><USER28> 100 Years of Solitude by Gabriel Garcia Marquez

The Black Swan by Nicholas Nassim Taleb

The Wind-Up Bird Chronicle by Haruki Murakami

</comment><comment><USER29> Godal Escher Bach - I think influenced myself and a lot of people who think about AI.

</comment><comment><USER30> The Surrender Experiment - about following the flow of life and seeing events unfolding in front of you and how amazing the perfection of life is if you take a step back observe how everything plays out

</comment><comment><USER31> "What makes it go, what makes it work, what makes it fly, what makes it float" taught me that gadgets and appliances have internal mechanics.

"The Lucifer principle" introduced me to evolution and evo-psych thinking.

</comment><comment><USER32> The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life - Mark Manson

Surely You're Joking, Mr. Feynman! (Adventures of a Curious Character) by Richard Feynman

Born Standing Up by Steve Martin

</comment><comment><USER33> Surely you're joking Mr. Feynman - got a second degree in Physics

So good they can't ignore you - the one career self-management book you should read

The essays of Waren Buffet - how to approach investment

</comment><comment><USER34> _The Social Brain - Michael S. Gazzaniga_

(Relays a bunch of super interesting experiments done on 'split brain' patients with important consequences for normally functioning brains. Not at all gimmicky, doesn't read like a popularization—I think my dad said he read it in grad school for cognitive psych. But I read it as a teenager so it's generally accessible, and it'll likely change the way you think about people and their brains.)

 _The Structure of Scientific Revolutions - Thomas S. Kuhn_

(Looks at large scale trends in scientific progress [as well as looking more closely at the notion of 'progress' in this context]. Probably the most important features he extracts have to do with distinct phases that scientific work can typically be classified under: 'normal' science and 'revolutionary' science. These concepts are easiest to understand in relation to the notion of a 'framework'. In revolutions, we are developing new frameworks and potentially abandoning lots of old work; during normal science, we are elaborating within an established framework. Phrases like 'the dominant paradigm' in this context are due to Kuhn [IIRC he uses 'paradigm' rather than 'framework']. The book isn't so long and his argument for and presentation of these ideas is well worth reading.)

 _Hackers: Heroes of the Computer Revolution - Steven Levy_

(Traces the major trends, personalities, and events leading to personal computing as we know it today, starting with IBM mainframes mediated by an IBM employ. Also describes the origin and meaning of the 'hacker ethic'. The hacker ethic, especially as exemplified by early MIT programmers in the book, was hugely influential on how I came to think about programming and creative work generally.)

 _The Varieties of Religious Experience - William James_

(It's largely an attempt at a scientific account of the potential cognitive/emotional impacts of adopting various beliefs, though especially religious/philosophical beliefs. James was an early experimental psychologist and philosopher. The book is mostly a transcription of his Gifford Lecture series of the same name.)

 _The Philosophy of Physical Science - Sir Arthur Eddington_ (Got me thinking about deep issues where we still trip ourselves up by assuming that parts of our own mental makeup are parts of the objective universe instead. Also clarified my understanding of the notions of 'structure' and 'analysis'. Also gives an interesting example of using Group Theory for physics work. Not as scary (or long) as it sounds, though it takes some work.)

Also: Three Scientist's and their Gods, Metamagical Themas, The Society of Mind, Consilience: The Unity of Knowledge, Chaos: Making a New Science, and Darwin's Dangerous Idea.

</comment><comment><USER35> The Discworld Novels by Terry Pratchett and the Fencer Trilogy by K.J. Parker.

I read a lot of everything I can find but always get back to these.

</comment><comment><USER36> Beyond Good and Evil by Friedrich Nietzsche. No single book has changed my view of reality, specifically ethical reality, more.

<comment><USER37> What about Beyond Good and Evil made such a big impact on you? I read it and enjoyed it a lot as well but struggle to put it into words how it has impacted me.

</comment></comment><comment><USER38> many years ago and long before 'matrix' his books changed my view on the world: William Gibson, the guy who invented 'cyberspace'

[https://en.wikipedia.org/wiki/List_of_works_by_William_Gibso...](https://en.wikipedia.org/wiki/List_of_works_by_William_Gibson)

</comment><comment><USER39> Narcissus and Goldmund - Hermann Hesse

</comment><comment><USER40> beautiful boy - david sheff

crime and punishment - fyodor dostoevsky

the trial - franz kafka

catch-22 - joseph heller

how to win friends and influence people - dale carnegie

mindset: the new psychology of success - carol dweck

I really enjoy novels a lot more than self-help and business books, but lately everything I read is about some business concept or backstory.

</comment><comment><USER41> Paulo Coelho - The Alchemist - because it wonderfully describes the journey of an entrepreneur

</comment><comment><USER42> The Four Agreements - Don Miguel Ruiz

The War of Art - Steven Pressfield

Thinking, Fast and Slow - Daniel Kahneman

</comment><comment><USER43> Not Fade Away: A Short Life Well Lived by Laurence Shames & Peter Barton

</comment><comment><USER44> Start With Why - Simon Sinek

</comment><comment><USER45> East of Eden - timshel "thou mayest"

</comment><comment><USER46> The Beginning of Infinity - David Deutsch

</comment><comment><USER47> The Selfish Gene - Richard Dawkins

</comment><comment><USER48> The Selfish Gene, Richard Dawkins

</comment><comment><USER49> Lying - Sam Harris.

Behave - Robert Sapolsky.

Gene: An Intimate History - Siddhartha Mukherjee

</comment><comment><USER50> The Prince, Niccolo Machiavelli.

</comment><comment><USER51> The Art of War by Sun Tzu

</comment><comment><USER52> The Prince, Machiavelli

</comment><comment><USER53> in no particular order:

Lord of the Rings

Cryptonomicon

Anathem

Moby Dick

The Master and Margarita

Walden

Arabian Sands

Gawain and the Green Knight

Beowulf

<comment><USER35> I love The Master and Margarita. A lithuanian friend introduced me to it and she told me that it was censored back when Lithuania was still part of the Soviet Union and they used to share those parts in secret (copied on a typewriter).

<comment><USER53> true samizdat!

I have the english translation by Burgin and O'Connor, which is excellent. [https://www.amazon.com/Master-Margarita-Mikhail- Bulgakov/dp/...](https://www.amazon.com/Master-Margarita-Mikhail- Bulgakov/dp/0679760806/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1516992816&sr=8-1)

<comment><USER35> Ha, that was the word I was looking for, thanks :)

Ok, I'll have to read this version too then (only know a german one so far). And probably learn russian.

</comment></comment></comment></comment><comment><USER54> Surprised nobody has yet mentioned "Waking Up" by Sam Harris.</comment>


Ask HN: Anyone else do the Admissions Test for the W2009 Founder Institute? - <AUTHOR>

First, congrats.  Second, a question....
I was working on the Admissions Test for the Founder Institute (http://www.founderinstitute.com/) on Monday night and experienced some considerable lag loading some pages after submitting a set of answers in the timed section (2 or 3 minutes in some cases).  I'm wondering if anyone else did it on Monday or last night and noticed the same thing.
FYI, I emailed Adeo, who was reasonable and responsive but didn't seem like he had any control over it (or possibly any way to prove my claim).  He did forward my email to the testing provider (Jordan Peterson), who hasn't responded yet.
I'd be interested to hear if anyone else experienced this and if so what you did about it.  Thanks.
(edited for formatting) <comment><AUTHOR> Huh. Serves me right for being overly anxious.

Thanks and good luck to you too bhousel.

</comment><comment><USER2> I took it Tuesday early AM, but didn't experience any lag. Good luck!</comment>


Interaction of time crystals observed for first time, confirms Josephson effect - <AUTHOR> https://www.inverse.com/innovation/time-crystals-quantum-computing-study <comment><AUTHOR> >AC Josephson effect between two superfluid time crystals

[https://www.nature.com/articles/s41563-020-0780-y](https://www.nature.com/articles/s41563-020-0780-y)</comment>


Scientists find Amazonian tribe that has no words to express numbers - <AUTHOR> http://www.brainmysteries.com/research/Team_finds_language_without_numbers.asp <comment><USER2> Find a much more detailed article here: [http://www.newyorker.com/reporting/2007/04/16/070416fa_fact_...](http://www.newyorker.com/reporting/2007/04/16/070416fa_fact_colapinto)

</comment><comment><USER3> and here: RECURSION AND HUMAN THOUGHT: WHY THE PIRAHÃ DON'T HAVE NUMBERS <http://edge.org/3rd_culture/everett07/everett07_index.html></comment>


Sabre (computer system) - <AUTHOR> https://en.wikipedia.org/wiki/Sabre_(computer_system) <comment><USER2> Having worked for a major booking website several years ago, Sabre was a nightmare to work with. It felt more like I was in a beta program to integrate with them rather than a production system.

<comment><USER3> can confirm. Though the other gds aren't _that_ much better

<comment><USER4> Do you think this might be a function of the same thing the electronic medical record (EMR) field faces?

For reference, I used to work in the EMR space for a "value-added reseller" of a certain EMR package. It wasn't great. But when you looked at the other options, they weren't great either.

Eventually I learned that medical software is a ghetto because those who care about the medical part of it don't _really_ care about the software part of it. At least not enough to treat it as the craft it is. And those who care about the software development tend to leave because you can do better work elsewhere.

So do you think GDS products suffer because no one cares about actually developing software, but rather the end goal?

<comment><USER5> GDS suck because of compability. Compability to decade old systems and workers, and compability to evertything and anything. The first GDS where 40-50 years ago created, under the stress of very limited resources, aimed for very specific handling. In that time, workflows and concepts were established and brought into productions, which are still used today.

If you think unix-commandline is cryptic in it's usage, then you should try gds-commands, which are often optimized to the single characters. Today you can use XML and JSON, but they are only build on top of those old systems, leading to many old crafty things.

</comment></comment></comment></comment><comment><USER6> Sayber or Sawbre [https://www.youtube.com/watch?v=BoT-6Z597Tc](https://www.youtube.com/watch?v=BoT-6Z597Tc)

<comment><USER7> I have no clue why the Sabre Pyramid never took off. It's an ergonomic and engineering marvel, touted as 'Your portable connection to everything in space and time.'

[http://theoffice.wikia.com/wiki/Pyramid](http://theoffice.wikia.com/wiki/Pyramid)

</comment></comment><comment><USER8> Ah. memories. My first business back in the early 90's was a small local IT company called Sabre Systems. On a weekly basis, we would get calls from travel agents all over the world asking for technical support for their online reservation system which I didn't even know existed until we started getting those calls!

</comment><comment><USER5> [https://en.wikipedia.org/wiki/Amadeus_IT_Group](https://en.wikipedia.org/wiki/Amadeus_IT_Group)

The other big reservation provider.

<comment><USER9> Sabre's market share is relatively strong in the United States and North America in general, but it loses out to Amadeus's CRS [0] everywhere else.

[0] [https://en.wikipedia.org/wiki/Amadeus_CRS](https://en.wikipedia.org/wiki/Amadeus_CRS)

</comment></comment><comment><USER10> And here I thought maybe you were linking to this ...

[http://www.softwarepreservation.org/projects/interactive_c/s...](http://www.softwarepreservation.org/projects/interactive_c/saberc/saberc)

... which was amazing in the day.

</comment><comment><USER11> Yeah and these GDS systems are also strikingly insecure [https://media.ccc.de/v/33c3-7964-where_in_the_world_is_carme...](https://media.ccc.de/v/33c3-7964-where_in_the_world_is_carmen_sandiego)

</comment><comment><USER12> My sister in law is a travel agent and has to use this. It looks god awful, but she is quite efficient in it. Similar to finding companies still using AS400 systems...not modern in the slightest, but efficient for skilled users.

<comment><USER13> In some ways AS/400 is still more modern than most systems. Flat, persistent address space and capability security? I'm still hoping that the ideas in AS/400 will become mainstream.

<comment><USER14> Our main customer has a global network of AS/400 systems. The employees make fun of the "old" looking text interface, but that system is pretty much bullet proof. The only times I ever see it go down are when their Windows based DNS servers crash.

</comment></comment></comment><comment><USER15> One of my friends told me it is a Mainframe-based application. That is pretty cool.

<comment><USER16> It's a direct descendant, really almost a twin, of the SAGE air defense system from the 1950s. This was one of, if not the, first large-scale networked computer system.

[https://www-03.ibm.com/ibm/history/ibm100/us/en/icons/sage/b...](https://www-03.ibm.com/ibm/history/ibm100/us/en/icons/sage/breakthroughs/)

<comment><USER17> Although SABRE uses ideas (like networking) from SAGE, the two systems are entirely different. SAGE used the AN/FSQ-7, a monstrous vacuum tube system. SABRE used transistorized IBM 7090 mainframes, a descendant of the 709 mainframe.

You can see parts of the SAGE system at the Computer History Museum in Mountain View. An interesting feature of the displays is a "light gun", used by the weapons director to select targets. Also of interest is the cigarette lighter and ashtray built right into the console.

<comment><USER18> > light gun

The same technology in light pens and the NES Super Scope, involving a sensor in the gun and a sync-pulse on the display? Or was this an actual example of a CRT being used as a dual-mode transducer, both producing light and observing "unexpected" light above-and-beyond that produced?

<comment><USER17> The light gun had a sensor. When it was pointed at something on the display, it would trigger at the time the item was redrawn, indicating to the system which item was detected. Details: [http://ed-thelen.org/comp- hist/CasteelSageRecollections.html...](http://ed-thelen.org/comp- hist/CasteelSageRecollections.html#LightGun)

</comment></comment></comment></comment></comment><comment><USER19> The book "Hard Landing: The Epic Contest for Power and Profits That Plunged the Airlines into Chaos" provides a good overview of how these airline reservation systems came to be.

</comment><comment><USER20> > Congress investigated these practices and in 1983 Bob Crandall, president of > American, was the most vocal supporter of the systems. "The preferential > display of our flights, and the corresponding increase in our market share, > is the competitive _raison d 'être_ for having created the system in the > first place," he told them

[http://infolab.stanford.edu/~backrub/google.html](http://infolab.stanford.edu/~backrub/google.html)

> Google is designed to crawl and index the Web efficiently and produce much > more satisfying search results than existing systems. The prototype with a > full text and hyperlink database of at least 24 million pages is available > at [http://google.stanford.edu/](http://google.stanford.edu/)

[..]

> Currently, the predominant business model for commercial search engines is > advertising. The goals of the advertising business model do not always > correspond to providing quality search to users. For example, in our > prototype search engine one of the top results for cellular phone is "The > Effect of Cellular Phone Use Upon Driver Attention", a study which explains > in great detail the distractions and risk associated with conversing on a > cell phone while driving. This search result came up first because of its > high importance as judged by the PageRank algorithm, an approximation of > citation importance on the web [Page, 98]. It is clear that a search engine > which was taking money for showing cellular phone ads would have difficulty > justifying the page that our system returned to its paying advertisers. For > this type of reason and historical experience with other media [Bagdikian > 83], we expect that advertising funded search engines will be inherently > biased towards the advertisers and away from the needs of the consumers.

> Since it is very difficult even for experts to evaluate search engines, > search engine bias is particularly insidious. A good example was OpenText, > which was reported to be selling companies the right to be listed at the top > of the search results for particular queries [Marchiori 97]. This type of > bias is much more insidious than advertising, because it is not clear who > "deserves" to be there, and who is willing to pay money to be listed. This > business model resulted in an uproar, and OpenText has ceased to be a viable > search engine. But less blatant bias are likely to be tolerated by the > market. For example, a search engine could add a small factor to search > results from "friendly" companies, and subtract a factor from results from > competitors. This type of bias is very difficult to detect but could still > have a significant effect on the market. Furthermore, advertising income > often provides an incentive to provide poor quality search results. For > example, we noticed a major search engine would not return a large airline's > homepage when the airline's name was given as a query. It so happened that > the airline had placed an expensive ad, linked to the query that was its > name. A better search engine would not have required this ad, and possibly > resulted in the loss of the revenue from the airline to the search engine. > In general, it could be argued from the consumer point of view that the > better the search engine is, the fewer advertisements will be needed for the > consumer to find what they want. This of course erodes the advertising > supported business model of the existing search engines. However, there will > always be money from advertisers who want a customer to switch products, or > have something that is genuinely new. But we believe the issue of > advertising causes enough mixed incentives that it is crucial to have a > competitive search engine that is transparent and in the academic realm.

\-- Sergey Brin and Lawrence Page, "The Anatomy of a Large-Scale Hypertextual Web Search Engine"

From that (lip service) to the ElsaGate autoplay fiasco in under 2 decades, and of course using public resources to get the foot in the door. Par of the course.</comment>


Development story of puzzl - <AUTHOR> http://pravj.github.io/blog/development-story-of-puzzl/

<comment><USER2> I just wrote a blog post about using A* to solve puzzles in my game: Mondrian. [http://jefarmstrong.com/posts/building- mondrian2-astar/](http://jefarmstrong.com/posts/building-mondrian2-astar/) I think you might find it interesting.

<comment><AUTHOR> @jefarmstrong Mondrian sounds an awesome thing dude. I liked your both posts on it.

Can I ask, how can you miss the exact shortest path for puzzles in its version-1? I mean did you just used hard-coded levels or what?</comment></comment>


Is Extended Random a Malicious NSA Plot? - <AUTHOR> http://sockpuppet.org/blog/2015/08/04/is-extended-random-malicious/ <comment><USER2> For your web inspector console:

      $("body").html($("body").html().replace(/Clyde Frog/g, "the NSA"))

Update: more proper

      $("body").html($("body").html().replace(/Clyde[\s\r\n]Frog/g, "the NSA").replace(/\. t/g, ". T"))

<comment><USER3> That's not perfectly accurate; the reason I think an abstract name is helpful is that GCHQ is just as bad, if not worse: it's handy to have a name that captures all of them.

<comment><USER2> In my defense, when I first skimmed the article I missed the note about the secondary meaning and was quite confused. "New hacker group? Old hacker group I don't know about? Kids these days..."

<comment><USER4> I did too. I assumed it was referring to an individual working for USG (although "Frog" is a pretty uncommon surname, I'd think).

</comment></comment></comment></comment><comment><USER5> Tangential forward-looking paranoia:

I've got to wonder if the DUAL_EC debacle only appears so ham-fisted because the public understanding of public key crypto is much further ahead than our understanding of symmetric ciphers. Universities employ armies of mathematicians studying mathematical structures for their own right, whereas shuffling bits isn't sexy.

Conversely, "Clyde Frog" has been studying symmetric ciphers much longer and harder (symmetric is sufficient for nation-state security) and could have a deep symbolic understanding of common symmetric constructions akin to how we see the public-key math. They would then know how to choose constants that admit similar backdoors, and the entropy of "nothing up my sleeve numbers" isn't exactly well quantified.

Rather than a proactive attempt, DUAL_EC could have been a _reaction_ to worries about movement to RNGs based on asymmetric math.

<comment> why-el > symmetric is sufficient for nation-state security

Is that a verifiable assertion? Would love to read more about it.

<comment><USER5> Governments can afford to employ people to transport N^2 keying material. Additionally, they have predefined communications patterns.

Even if part of the government moves to asymmetric algorithms for key distribution (only possible after the discovery of Diffie-Hellman), the top secret portions can continue using couriers to avoid relying on an additional algorithm.

Combined with its standard use for bulk ciphering, symmetric is obviously the more valuable target to secure/break.

<comment><USER6> Public Key crypto was discovered by GCHQ (and then given to the NSA) several years before it was publicly discovered by Diffie-Hellman and RSA. I think this was to avoid having to have the symmetric keys under armed guard. The public discovery is also what kicked off the 'crypto wars'. I'd be surprised if modern nation state intelligence communities found symmetric encryption sufficient.

A lot of interesting information about the history I learned from Steven Levy's crypto: [http://www.amazon.com/Crypto-Rebels-Government-Privacy- Digit...](http://www.amazon.com/Crypto-Rebels-Government-Privacy- Digital/dp/0140244328)

<comment><USER5> Oh yeah, I'd forgotten about Cocks. Blame my selective memory for withholding credit from people who don't share.

Spooks would of course welcome any discovery, and asymmetric crypto does solve problems for them (getting government crypto distributed as wide as possible). I am saying purely symmetric is "sufficient" for their core functionality - the communications that really _need_ to be secret. Coupled with the head start before asymmetric was even discovered, that is where their focus is going to be.

Put another way: if you were in charge of securing communications and had to prioritize resources, would you rather research a trustworthy asymmetric algorithm or a trusty symmetric algorithm? Likewise if you wanted to snoop on others' communications, would you prioritize breaking symmetric or asymmetric techniques?

</comment></comment></comment></comment></comment><comment><USER7> Just out of curiosity, are those jabber chat rooms public? tptacek mentions some jabber logs of the TLS working group.

<comment><USER8> [http://datatracker.ietf.org/wg/tls/charter/](http://datatracker.ietf.org/wg/tls/charter/)

</comment></comment><comment><USER9> May I just say, I am extremely happy that the NSA has to jump through such incredibly laborious hoops to gain a glimpse into anything, a capability which they would then fail to acknowledge at any price.

This is the OPPOSITE of a dictatorship, where there would simply be a heavy- handed order to put in an explicit, acknowledged back door or be jailed without trial, or executed.

This is what freedom looks like. Enjoy it!

I personally also enjoy the fact that nobody with a few million dollars in spare change can surf the dark web as Dr. evil. But that's just me.

 _EDIT: this comment is at -1, perhaps people thought I was making a ham- fisted sarcastic statement. I 'm speaking literally. You all can keep either your dictatorship, or the society in which someone can commit an act of terrorism for the going black market rate without any repercussions; if it's a false dichotomy, you'll have to explain why._

 _EDIT 2: this comment is fluctuating wildly (-2, +2, 0, etc) especially since my edit. Thoughtful replies would probably be more helpful than voting here._

</comment><comment><USER10> _Except for Hoffman's last proposal, the extensions are cordoned off to the US Government. The sponsors of the standards and their authors make very little effort to provide a use case for normal Internet users._

If this were an X-Files episode, then the group who really runs the world would be forcing the USG to subvert it's own crypto.

</comment><comment><USER11> Why would Certicom bother filing a patent(s) on this. The only likely buyer/licensee would be a nation state - which can easily appropriate whatever IP it desires. Further, NSA paying/licensing with a foreign company (Canadian Certicom) only adds to the number of people in the know. Likely Certicom realized this and contributes to the reason why some of the patent applications were never pursued beyond provisional patent applications.

</comment><comment><USER12> Certainly seems like a very well crafted but poorly executed plot to me. The tricky thing is how the hell do you really expose it? There are so many levels of obfuscation both by the people who are putting forth the proposal and the technical details as well.

</comment><comment><USER13> Can anyone figure out whether USG is Unix Systems Group or United States Government. (I think we're safe in assuming they aren't United States Gypsum (though, from my trips through Empire to Gerlach, that was the first thing that came to mind)). [Edit - if you read through the entire (epic and wonderful resource) article, United States Government is used where USG might be - so I think we are safe in assuming it is United States Government. tptacek, might be worth introducing the acronym at the beginning.]

<comment><USER14> I thought it was the US Government.

</comment></comment><comment><USER11> PKRNG - if the attacker obtains the private key, why do they need the 28+bytes?

<comment><USER15> The attacker doesn't have the TLS private key, they have the RNG key. But they don't have the RNG seed. Recovering the seed is necessary to predict other RNG outputs and break TLS, but requires observing more RNG output than one typically sees.

<comment><USER16> FWIW, the operative theory here is that "Extended Random" was designed to work in concert with the DUAL_EC DBRG/RNG, which almost certainly allows "Clyde Frog" to predict all future output on the basis of _very_ few samples.

<comment><USER11> seems they aren't limited to just future output:

"Using that private key, they can observe CSPRNG output on the wire, "decrypt it", and use that to rewind and fast-forward other people's CSPRNGs, discovering their keys."

</comment></comment></comment></comment><comment><USER17> I hate to ask a dumb question, but the article discusses the actions of Clyde Frog a lot. Is Clyde Frog a person, a company, a government project, or what? A web search found a TV show and a stuffed animal, so I'm honestly puzzled.

Edit: thanks cmg. I was reading the article on my phone and the side notes were off screen so I totally missed the explanation.

<comment><USER18> In small (11px), light-ish (#777777) next to the first paragraph:

> If I call NSA "Clyde Frog" long enough, eventually other people will too. > Someone has to start the meme! I think Dual_EC is a backdoor.

<comment><USER19> First time I've ever seen sidenotes done like that.

BTW, screen reader users (i.e. blind people) can't possibly miss the sidenotes; in fact, each sidenote will interrupt the text at the point where the note is most relevant. So a screen reader will render the first sentence like this:

Did Clyde Frog If I call NSA "Clyde Frog" long enough, eventually other people will too. Someone has to start the meme! subvert crypto standards with a backdoored random number generator called Dual_EC?

A little jarring when first encountered. (In my case, because I have some usable vision, I could tell what was going on.) I'd suggest sticking with more conventional footnotes, but I can see why this form of sidenote was appealing.

<comment><USER20> Proper support for footnotes / sidenotes in HTML itself would be dandy. There's not, so people write hacks.I've done a few myself.

Presently footnoting is either manual or requires a preprocessor -- LaTeX, Markdown, CMS, etc.

</comment></comment></comment></comment><comment><USER21> That is because Jerry Solinas works for the NSA. Jerry Solinas @ NSA @ firstname.lastname@example.com.

Notice that the company "Clyde Frog" doesn't have a company website. Notice that Jerry Solinas don't have a Linked-In profile.

<comment><USER3> woosh

</comment></comment><comment><USER22> It may be my (somewhat archaic) sense of crypto humor, but any time I read the term "Dual EC", my mind says "CE lauD", making it sound like someone saying the word "cloud" with an accent expressing a lot of disdain[1].

Anyway, the Dual EC backdoor, if real, along with the extra randomness, may yet prove to be part of "the gubment's" very own cloudbusting operation, to make cloud services rain users' secrets at the push of a button...

[1]: _cf._ "my butt"

</comment><comment><USER23> Doesn't this essay absolutely bury one of the most important parts of this scandal, that RSA used DUAL_EC as the default random number generator in their FIPS certified encryption product for almost a decade!?! I note that this is glossed over with a description so marginal I would tempted to call it dishonest if I were not trying to apply the principle of charity to its author. "RSA BSAFE had support for DUAL_EC." Support!? Uh no, it used it as the default generator.

"I lean towards "not"; the structure of these proposals makes Clyde Frog's job needlessly harder, if only by practically ensuring that OpenSSL and Schannel would never default to enabling them. But people smarter than me are convicted of the idea that this was a backdoor attempt." Well yeah it would make their job harder unless one of the largest security companies in the world used that random generator in their flagship encryption product!!!

I feel like maybe their are better arguments for why this was not a subversion attempt, but honestly the points for seem so, _so_ strong and the points against seem like a mountain of wishy-washy humming and hawwing and extending the principle of charity even in the face of the above mentioned giant blaring klaxon of wrong-doing. I will still not say that reasonable people can't disagree over the question at hand but the arguments presented in this article don't strike me as being anywhere near strong enough to make this the sort of grey area the author would like.

<comment><USER24> > I will still not say that reasonable people can't disagree over the question > at hand but the arguments presented in this article don't strike me as being > anywhere near strong enough to make this the sort of grey area the author > would like.

You have three negations in this sentence, which means it's nearly impossible to parse or understand. It's been my observation statements like these follow rationalizations about a point in which there exists dissonance. Given you seem to be disagreeing with something Thomas said or the way he said it, but not actually disagreeing with a point he made, I'd say that is the case here as well.

Violations of our privacy via rationalizations of security makes me sad and bored. I think we can all agree that things could be better with the situation, and I for one appreciate Thomas' efforts in bringing the truth to light.

<comment><USER25> > I will still not say that reasonable people can't disagree over the question > at hand but the arguments presented in this article don't strike me as being > anywhere near strong enough to make this the sort of grey area the author > would like.

I didn't think it was that hard to read. It made sense on my first read, but here's my translation:

"I can see how arguments exist on both sides, but I don't think the author supported his argument with enough evidence to make it very relevant."</comment></comment></comment>