Distributed Logging Architecture in the Container Era - <AUTHOR> https://blog.treasuredata.com/blog/2016/08/03/distributed-logging-architecture-in-the-container-era/ <comment><USER2> I'm surprised the article didn't mention anything about the nuances of actual logging in a distributed architecture. In particular, when a request starts to span multiple containers or microservices, you need a way to correlate all log data to a given request. You typically do this by propagating a trace ID through your logging system, and a vector clock-type thing for causal ordering. If you can't tell which log statements belong to which request and the order in which they occur, debugging is quite hard :-).

<comment><USER3> I've just started down a path for us and been pretty happy with Loggly so far.

Just by adding a UUID to the requests as they're initially received I'm able to follow the chain of events across 5 different services, a message queue, and reconcile with any code exception reports from Bugsnag.

We don't have a lot of unique clients, but we do have a lot of requests.

Loggly can be set to parse logs (or have the data sent directly to it) and then tag messages with keys/values that can let you watch your request through every server.

<comment><USER4> How are you passing that UUID along the request chain? For instance, when one microservice calls another, does it have to be manually passed in?

<comment><USER5> My web stack checks for a specific header at each potential level of entry, and in the absence of a value, generates one. Then each layer's logger will log it back out. It's pretty straightforward for web requests - just use the plumbing that's already there, headers.

</comment></comment></comment></comment><comment><USER6> The first paragraph closes with a series a incorrect assertions contrasting containers with VMs, and "distributed logging architecture in the container era" is apparently nearly identical to the distributed logging architecture of the VM era in reality, but according to the article is a completely different thing and mindset altogether... again while looking basically identical.

<comment><USER7> VMs last much longer than containers, no? I can't remember the last time I spun up a VM that lived for less than a few seconds...

<comment><USER6> Mine live for milliseconds, but I'm a weird case where that's intentional as a security construct.

But that doesn't really have anything to do with the logging architecture. How long the thing lives doesn't matter when you're using aggregation, bypassing the local filesystem, and inspecting through a central portal.

People have been doing this in VM environments for a lot longer than they've been using containers. Whole companies and services were born to facilitate this, and some have even died already, in the time before containers hit full stride in the hype cycle.

<comment><USER8> Yes, I was left scratching my head as to why containers play any significant role in the issue of distributed logging. If your choice of container runtime/vm/servers are creating an issue with logging then there's most likely an issue with the system architecture rather than actual logging.

We've been building distributed systems (with containers and other approaches) for years and this is the first time I've heard of logging being a problem.

</comment></comment></comment></comment><comment><USER9> This is going to sound dumb, but I started dumping all my logs into mysql so that I wouldn't also need to run an elastic search cluster. I then query by the error level, latency, status code to find items of interest. And everything is stored by some sort of request ID so that I can correlate multi- entry logs.

It's not pretty -- there's no real-time alerts, but I'm not paying dollars per gb to AWS and I have no clue what google will end up charging... and I'm already running mysql. So there is that...

<comment><USER10> I'm guessing your log write volume is relatively low.

</comment></comment><comment><USER11> Fluentd is decent (though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go), but I'm still looking for a good solution to aggregate, inspect and search logs.

In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. There was one (Loggly?) which provided a CLI tool for tailing, but it wasn't very impressive. (All the SaaS apps I have looked at do quite poorly when it comes to rendering live logs in a browser, too.)

Any recommendations?

<comment><USER12> >In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators.

If you are talking about tailing log files live, Fluentd has supported it from Day 1: [http://docs.fluentd.org/articles/in_tail](http://docs.fluentd.org/articles/in_tail)

Also, as other sibling comments mention, there are tools, both SaaS and open source, that you can use as a destination of the logs Fluentd tails/listens/collects.

* Elasticsearch: [https://www.digitalocean.com/community/tutorials/elasticsear...](https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization)

* Graylog: [http://www.fluentd.org/guides/recipes/graylog2](http://www.fluentd.org/guides/recipes/graylog2)

* Scalyr: [https://github.com/scalyr/scalyr-fluentd](https://github.com/scalyr/scalyr-fluentd)

* Loggly: [https://www.loggly.com/blog/stream-filtering-loggly-fluentd/](https://www.loggly.com/blog/stream-filtering-loggly-fluentd/)

* SumoLogic: [https://gist.github.com/d-smith/8d3e7d53db772c6a7845](https://gist.github.com/d-smith/8d3e7d53db772c6a7845)

* Papertrail: [https://github.com/docebo/fluent-plugin-remote-syslog](https://github.com/docebo/fluent-plugin-remote-syslog)

(and literally hundreds of others)

>though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go

Heka was a great project, and a drop-in binary (as opposed to requiring a VM like Ruby) approach was interesting if not compelling in certain situations. That said, I never saw any benchmark that showed Heka was materially faster than Fluentd (or Logstash, for that matter). A lot of speed in this type of complex software comes from data structures/algorithms, an appropriate use of low-level language bindings, etc.

While language plays a role in the speed of software, it's hardly the only factor. As you said, it's only _in theory_ , not in practice =)

<comment><USER11> I meant tailing the aggregated logs themselves. I have looked at the CLI tools that exist for some of those SaaS services, and I have not found anything half decent.

For example, you will definitely want to filter on labels (including regexps) while tailing, and such filtrering should support adjustable context (both # of lines and time interval) and should support an optional time range to scroll back into history ("tail from 2pm"). And of course, grepping of historical entries.

Another thing I am not impressed with is pricing. Loggly seems the most reasonable in terms of price per volume, except it limits the number of "team members" to a ridiculous degree (5 users or something like that).

I set up Graylog once and wasn't impressed with it. Its reliance on Elasticsearch means it is quite static when it comes to the schema/input format. You can't change the settings; there is no reindexing support (or at least this was the case when I tried it, a year ago or so). Also, don't think it has any CLI tools?

</comment></comment></comment><comment><USER13> I recently just plugged python logger from [https://github.com/GoogleCloudPlatform/gcloud- python#google-...](https://github.com/GoogleCloudPlatform/gcloud- python#google-stackdriver-logging) into my python codebase. Configuration took 15 minutes, I have nice dashboard showing recent logs, I can log either strings or jsons. I can analyze data later using BigQuery. It works from docker containers.

I was contemplating trying out logstash or fluentd, but I don't see any major advantage over current simple solution. Can anyone more knowledgable than me help with explaining what I would get from logstash/fluentd?

<comment><USER14> Does any of your Python processes open a subprocess that then spawns another Python process as a child (and both gets logged)? If so, have you found any good way to associate the the parent and child logging messages?

If you just write to stdout everything makes sense in the file top to bottom, but if you're logging json objects into a database I had problems; if your parent process echos stdout from the child you lose a lot of context about the child, you can also double-log child process messages (because the parent is also echoing them), and you can't easily associate the parent/child objects. I came up with a hacky solution, but I wasn't logging these events to a centralized server at the time.

(I haven't used gcloud's dashboard, so I might be making wrong assumptions)

To respond your question: it sounds like you have something that works fine for logging messages. Personally, I split it into categories 1) logs (serial events that needed context for any usefulness) and 2) events (metrics or exceptions). I wrote 1) to traditional log files and wrote 2) to an ElasticSearch or statsd database to log exceptions to. Ideally, I wanted to use the same mechanism for both and peel off relevant data into separate databases.

A metrics database like Elasticsearch will let you query things like, "What modules give the most errors?" "Has this function been called more often this week than last?" "Is this process taking longer when using the newly released version compared to the old one?" etc.

<comment><USER15> Use the process group ID? ([http://unix.stackexchange.com/a/139230](http://unix.stackexchange.com/a/139230))

</comment></comment></comment><comment><USER16> Inbuilt docker + fluentd driver + plain ES falls flat on its face under scale. Not to mention docker daemon keeps dying :(.

Whats working for us now is local docker json logs -> heka -> kafka -> graylog -> ES. We even dockerized graylog to scale the processing up and down on demand.

Does 250k messages/sec at peak easily, More details in this presentation [https://www.youtube.com/watch?v=PB8dBnpaP8s](https://www.youtube.com/watch?v=PB8dBnpaP8s)

<comment><USER17> > Does 250k messages/sec at peak easily, More details in this presentation > [https://www.youtube.com/watch?v=PB8dBnpaP8s](https://www.youtube.com/watch?v=PB8dBnpaP8s)

250k messages/sec is pretty low.

It'd be useful (not to mention entirely feasible) to be able to handle 5M/sec bursts on a single system and aggregate sustained 50M/s+ (10Gbps network).

The system should be able to establish total order (serialization) locally and causal order over the network.

I did some tests and found out hardware limits are somewhere between 20-50M messages per second (serialized) on current consumer grade X86. Practical implementation would of course be slower.

Of course you need to go binary logging at that point, adhere to cache line boundaries, etc. mechanical sympathy. Maybe even do usermode networking at the aggregator server.

Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude.

In my quick tests I found the string processing hit to be surprisingly high, 10-50x. C "sprintf" and C++ stringstream are atrociously slow. (Surprisingly, considering sprintf has a pretty complicated "bytecode" format specifier parsing loop, it was still significantly faster than stringstream implementation.)

Timestamps are another huge performance issue. Typical system calls for high precision (a few microseconds or better) timestamp take several _microseconds_ each - using one of those will alone drop performance to 100-500k range. It's of course much faster to use RDTSC, but then you have the issue that different CPU sockets have different offsets (sometimes large) and possibly also some frequency difference between them. Also older CPUs don't have invariant TSC, so their speed changes when CPU frequency changes.

(Some word of warning about Windows QueryPerformanceCounter: When you test it on your development laptop, it appears fast, because it's using RDTSC behind the scenes. But when it runs on a NUMA server, it often changes behavior and becomes 20-100x slower, because Windows starts to use HPET instead. RDTSC takes maybe ~10 ns to execute and it's not a shared resource. HPET takes 1-2 microseconds to read and is a _shared system resource_ , concurrent access from multiple cores will make it slower.)

<comment><USER16> That is the peak we see right now at ~20-30% utilization overall network/cpu wise, it can do much more easily.

> Binary logging is a must, because even something like string formatting is > simply way too slow, by an order of magnitude.

true, string processing takes most of the resources.

<comment><USER17> Timestamps are even worse. They're slow to get and have a multitude of other problems, like timezones, system clock adjustments (manual or NTP), rate drift, offset errors, etc.

Some solve it by caching a low res timestamp, but that means losing ability to get side channel information about precisely how long time passed between two events. High resolution (micro/nanosecond range) timestamps give _so_ much more information than just time.

<comment><USER16> We haven't dealt with those yet. But in general we are always pushing everyone to log judicially, use metrics wherever possible (numbers are still easier to manage than strings), use sentry for exceptions logging and introducing zipkin for tracing.

</comment></comment></comment></comment></comment><comment><USER18> The Docker logging model of line-by-line stdout/stderr doesn't seem very useful beyond debugging in the local console. I prefer configuring the application logger to ship logs directly to a proper log system when possible.

E.g. Logback is perfectly happy shipping structured logs directly to Kafka or Elasticsearch, with no need to re-parse the formatted log output.

<comment><USER19> > _The Docker logging model of line-by-line stdout /stderr doesn't seem very > useful beyond debugging in the local console._

It's a well-understood pattern that Heroku made more visible under "The 12 Factor App".

stdout/stderr require no configuration in any language that I'm aware of. Why should the app care about how to wire up logging? That's a platform concern.

On Cloud Foundry you get Loggregator, which frankly needs improvement, but for the most part you don't care about how to wire up logging. You print to stdout or stderr and the platform wicks that away to a firehose service for you. You can hook up kafka to spout, or elasticsearch, or anything else you like.

Disclosure: I work at Pivotal, we donate the majority of engineering to Cloud Foundry.

</comment></comment><comment><USER20> I use graylog, and docker can ship logs using the GELF format.

Furthermore, I've got my graylog setup running from a customized docker- compose.yml file, and this thing sings.

</comment><comment><USER21> They say that without aggregation scale-out is impossible. That's simply not true. Using things like anycast and ECMP it's super easy as log destinations are usually stateless.

Also, never had network contention because I logged too much unless something was really broken - and then only at that node.

I'll continue sending events to my anycast address without any aggregation thank you very much.

<comment><USER22>
        > They say that without aggregation scale-out is
        > impossible. That's simply not true. Using things like
        > anycast and ECMP it's super easy . . .

Then you haven't worked at significant scale. That's fine! Just keep everything in context :)

<comment><USER21> A "probably" there would be nice. I don't want to have to defend myself, but I'm pretty confident about the scale I've worked on.

I guess it boils down to what you call aggregation.

Is having multiple stateless receivers behind anycast/ECMP aggregation writing to a distributed database aggregation? I'd argue not, but maybe this is where the difference in opinion lies.

</comment></comment></comment><comment><USER23> How does fluentd compare with logstash, the other logging swiss army knife?

<comment><USER12> There's been some comparisons in the past.

[http://logz.io/blog/fluentd-logstash/](http://logz.io/blog/fluentd-logstash/)

[https://www.pandastrike.com/posts/20150807-fluentd-vs- logsta...](https://www.pandastrike.com/posts/20150807-fluentd-vs-logstash)

[http://jasonwilder.com/blog/2013/11/19/fluentd-vs- logstash/](http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/)

</comment></comment><comment><USER24> > But unlike syslogd, Fluentd was built from the ground up to unify log > sources from microservices, so they can be used efficiently for production > and analytics.

syslogd(8) -r This option will enable the facility to receive message from the network using an internet domain socket with the syslog service (see services(5)).

Why won't this work with dynamic DNS ?</comment>


What's a "medium.com" and why is it all over HN? - <AUTHOR>

<comment><USER3> <https://news.ycombinator.com/item?id=5591238>

</comment><comment><USER4> To me it looks kind of similar to <https://svbtle.com/>, which is a network of chosen writers with multiple blogs aggregated to a single site.

</comment><comment><USER5> Medium.com is a publishing platform for the sharing of ideas and experiences.

It's founded by Evan Williams[1] of Blogger, Twitter etc.

[1] <https://twitter.com/ev>

<comment><USER6> and Biz Stone, another twitter cofounder

<comment><USER5> Biz Stone is a founder of Twitter, Obvious and now Jelly. I don't consider Biz Stone to be a founder of Medium as it operates independently of Obvious.

Obvious have invested/incubated Medium, Branch, Jelly, Lift, Beyond Meat, Good Fit and Neighborland. (There may be more but I'm pretty sure that's all of them)

Medium, Branch and Jelly are what the Obvious founders are now focusing on (although they sit on some of those other startups boards):

Ev - Medium, Branch - Jason and Jelly - <USER2>

<comment><USER6> You may very well be right, I had assumed more involvement on his part based on this article: <https://medium.com/obvious/e74637f2fe22></comment></comment></comment></comment>


Improving Docker with Unikernels: Introducing HyperKit, VPNKit and DataKit - <AUTHOR> https://blog.docker.com/2016/05/docker-mac-windows-open-source/ <comment><USER2> docker's go-9p now makes for the 3rd implementation of 9p in go:

docker/go-9p [https://github.com/docker/go-p9p](https://github.com/docker/go-p9p)

rminnich/ninep: [https://github.com/rminnich/ninep](https://github.com/rminnich/ninep)

rminnich/go9p: [https://github.com/rminnich/go9p](https://github.com/rminnich/go9p)

There's also the Andrey Mirtchovski and Latchesar Ionkov implementation of go9p, but all I can find is a dead Google Code link from here: [http://9p.cat-v.org/implementations](http://9p.cat-v.org/implementations)

<comment><USER3> We did try the other implementations first, but had a lot of issues doing what we wanted with them.

</comment></comment><comment><USER4> With lots of OCaml love it seems, from a quick glance through the source repositories.

<comment><USER5> And Mirage. This release is really awesome:

\- It puts Mirage in use in real production systems.

\- It strengthens the ecosystem around Hypervisor.framework. After xhyve and Veertu, this is really nice. So much for the meme that newer OS X versions only integrate iOS features.

<comment><USER6> They only took, like, the network stack from Mirage. Their virtualization library _is_ xhyve… well, turned into a library.

<comment><USER7> Open sourcing our libraries is obviously just the start. We're really keen to hear about any other uses that people might have and move HyperKit in the direction of easy integration with higher-level frameworks.

For instance, direct ports of unikernel projects to run against the HyperKit kexec APIs would be really fun. Any takers for MirageOS or HalVM? :-)

We've started keeping a list of "help wanted" issues for anyone interested in getting started with hacking in this area: [https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3A...](https://github.com/docker/hyperkit/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)

<comment><USER8> What was the motivation to create a new repo with a different name rather than maintaining a fork of xhyve?

<comment><USER7> We're taking it in a fairly different direction from upstream xhyve and bhyve, with the integration with other kits like VPNKit, DataKit (and soon, FSKit). We wanted to be able to diverge the build system in particular, and this makes it difficult to maintain a direct fork.

However, we're committed to upstreaming patches to their relevant projects where relevant, and so the HyperKit Git repository is as close as we can make it. If it diverges a lot in the future with file renames, we'll have to figure out the Git-fu to make cherry-picks easier...

</comment></comment></comment></comment></comment></comment><comment><USER9> I wonder if we'll see a move towards getting Docker working on FreeBSD using either Jails or bhyve finally, since it talks about using bhyve hypervisor... That would be really great.

<comment><USER6> There is a port of docker that works with jails+ZFS (and the linux.ko syscall translator to run linux images) [https://wiki.freebsd.org/Docker](https://wiki.freebsd.org/Docker)

There's also jetpack for the app container spec (rkt) [https://github.com/3ofcoins/jetpack](https://github.com/3ofcoins/jetpack)

I'm also experimenting with building a simpler system.

<comment><USER7> It should be possible to build a simpler port for FreeBSD with the recent Docker 1.11 release. It moved the container execution to containerd ([https://github.com/docker/containerd](https://github.com/docker/containerd)), so that's where a bunch of the Jails logic would go.

I'm quite keen to see Docker running on FreeBSD so I can use it for my CI pipelines. I'm less interested in Linux emulation to run Linux images -- I'd like Docker support on FreeBSD to run FreeBSD images first!

<comment><USER10> What advantages would Docker give you over jails themselves?

<comment><USER11> Can't speak for anyone else, but while I think jails are "better" for a variety of reasons it seems clear there is far more market momentum behind Docker. More particularly, its toolchain. Given the choice between building myself (or relying upon a small, but talented, pool of devs) and drafting off tens of millions in VC cash and a much larger community I think some would prefer the ability to have a *BSD CI chain that is just a config setting in a Docker setup.

</comment></comment></comment></comment></comment><comment><USER12> Seems like only a year ago Docker changed how it used Virtualbox to boot VMs using machine (and caused me endless amounts of suffering trying to figure out how to fix it). Now it would seem they are getting rid of Virtualbox entirely with their own VM...which needs contributions.

<comment><USER5> _getting rid of Virtualbox entirely with their own VM..._

Not really. This builds heavily upon Hypervisor.framework, which is included in new OS X versions. Hypervisor.framework is pretty cool, because it allows for building VM applications that are sandboxed and don't require kernel modules. As a result, Hypervisor.framework applications can even be distributed via the App Store (see e.g. Veertu).

<comment><USER13> The drawback is that now VM manager is at mercy of what Hypervisor.framework provides. For example, Veertu does not support redirection of USB devices into VM because the framework does not support it. That makes it rather useless for my usage and I stick with VirtualBox for now.

Of cause, this is not an issue for Docker as the framework covers all its needs.

</comment></comment></comment><comment><USER14> Very interesting work. I find go-9p quite fascinating and think it could really have broader applications. Docker if you see this, I actually think you're on to something for microservice development thats native to the docker world. I've been trying to come up with ways of replicating the unix philosophy around programs that do one thing well and the use of pipes but was always limited in my thinking in terms of http, json, etc, etc.

My advice, as a guy who's currently building something in the microservice space, explore this further. Spend some time building fit for purpose apps with this and see where it goes.

</comment><comment><USER15> I guess I just want to know the take away. eg. Will consume less memory on mac.

<comment><USER16> If you're asking about Docker for Mac/Windows, an earlier blog post has a few details.

[https://blog.docker.com/2016/03/docker-for-mac-windows- beta/](https://blog.docker.com/2016/03/docker-for-mac-windows-beta/)

In essence, it's a _much_ nicer experience using the app and since it's improving all the time (still in beta, fwiw).</comment></comment>


Equal-weight ETF allocation with automatic rebalancing - <AUTHOR> https://www.quantopian.com/sector-etf-rotation <comment><USER2> The strategy equally weights the stock sectors in the S&P 500 using ETFs. In a rising stock market this would incur capital gains taxes in a taxable account. If you are saving regularly, you could direct your new investments towards the sectors that have underperformed to get closer to equal weighting without selling the outperforming sectors.

</comment><comment><USER3> Very neat. After a brief foray into reading about HFT, I've started reading more about algorithmic (non-HFT) trading.

I'd love to hear if other HN readers are playing with algorithmic trading. I occasionally think quantitative trading would be fun because of the intersection of math/stats and CS. Of course, that's an "in the abstract" statement. Depending on it for my mortgage would be . . . different.

<comment><USER4> I do algorithmic trading, and write the underlying systems and algos, for a living, not high frequency, though I did once get to 200 messages a second:)

The stress is always there, though you become desensitized to it after a while, or you just quit in a stress induced rage:(

I'm Canadian so my knowledge is skewed to the Canadian markets.

My email is in my profile if you'd like to chat.

</comment></comment><comment><USER5> I'd feel VERY uncomfortable giving a 3rd party the password to my brokerage account. Too many bad things can happen.

Yes, I know you take property security. There's always some bug or flaw you don't know about. Because you're logging in to the clients brokerage account, that means your software has access to their UNENCRYPTED UNHASHED password.

Also, there are reasons to NOT automatically rebalance every 21 days. Long- term games are better than short-term gains (unless it's an IRA). In addition to commissions, there's the cost of the bid/ask spread, not included in your calculation.

<comment><USER6> Does it say somewhere that they store your IB username and password? Their documentation suggests otherwise.

 _You will need to authenticate to your Interactive Brokers_ account. (Note: Quantopian does not store your brokerage password.)*

[https://www.quantopian.com/help#overview- livetrading](https://www.quantopian.com/help#overview-livetrading)

</comment></comment><comment><USER7> I don't see the fundamental distinction between this and a S&P 500 index fund. Instead of carrying out one rebalancing (at whatever frequency) there's now two rebalacning steps -- one at the sector level and then another one to account for market cap changes within the sector. The net result of either strategy looks to be an imperfectly realized approximation of the performance a continuously adjusted market cap weighted portfolio of those 500 stocks.

Is the claim that this is somehow a better approximation? Are there some tax loss harvesting benefits? What am I missing?

<comment><USER8> The costs of this strategy are hidden in the trading. Every time you trade, you incur a cost. Lets say your brokerage commission is $0 - you still pay the spread when you trade and taxes when you sell. Fees for this look reasonable, but why not hold VTSAX which holds approximately 3700 stocks at 0.05% (5bp) expense a year.

<comment><USER9> They claim that their backtesting mechanism accounts for the spread. Your guess is as good as mine whether they do so perfectly.

</comment></comment></comment><comment><USER10> Out of curiousity does anyone do algo trading on options? Maybe through IBs java api?

</comment><comment><USER6> How does it execute the rebalance trades? Just MOC or something like that?

<comment><AUTHOR> No, it places orders market orders in any of the ETFs we need to rebalance mid-morning. Orders go out around 10:15am NY time on rebalance days.

Also, the chart in the page is from actual trading on a roughly $25k account, and updates each day.

<comment><USER6> It would be very interesting to see the chart of actual trading next to a similar chart generated by the Quantopian backtester.

Very impressed with the product btw. Good work.

<comment><USER11> That's a great question, I've been working up a more detailed analysis of implementation shortfall (and thinking about ways to make this automated) - but your question prompted me to post a quick comparison here that you can look at for some current results: [https://www.quantopian.com/posts/live- results-vs-backtest-re...](https://www.quantopian.com/posts/live-results-vs- backtest-results-at-a-glance)

Full disclosure, I work for Quantopian and I wrote this strategy.</comment></comment></comment></comment>


Theresa May to create new internet that would be controlled by government - <AUTHOR> http://www.independent.co.uk/life-style/gadgets-and-tech/news/theresa-may-internet-conservatives-government-a7744176.html <comment><USER2> There are previous major discussions:

Here: [https://news.ycombinator.com/item?id=14382184](https://news.ycombinator.com/item?id=14382184)

And here: [https://news.ycombinator.com/item?id=14374533](https://news.ycombinator.com/item?id=14374533)</comment>


Why Does Everyone Suddenly Have Fancy Fake Teeth? - <AUTHOR> https://www.theatlantic.com/health/archive/2019/02/dental-veneers-instagram-teeth-teeth-teeth/582010/ <comment><USER2> Veneers are awesome looking but when I investigated them for myself, I found two "gotchas." First, they have to grind down your real teeth to tiny cone- shaped stubs in order to apply them. You can never get your real teeth back.

Second, they don't last forever. This article says 10 years. That means replacing your fake teeth several times for someone with an average lifespan, or giving up and going to dentures which make you look really horrible when you take them out.</comment>


Nvidia's GauGAN Turns Doodles into Stunning, Photorealistic Landscapes - <AUTHOR> https://blogs.nvidia.com/blog/2019/03/18/gaugan-photorealistic-landscapes-nvidia-research/ <comment><AUTHOR> Direct link to the paper here: [https://arxiv.org/abs/1903.07291](https://arxiv.org/abs/1903.07291)</comment>


Ask HN: Is it worth upvoting old submissions? - <AUTHOR>

I've been very busy lately so I've added a lot of stories submitted here to my "read later" folders. Going over them today I find myself "resubmitting" them on HN so they get the upvote they deserve from me.<p>Is that of any use though? Does anyone look at old stories, if there is such a functionality even? <comment><USER2> Personally, I pretty often search HN with Google, and read old comments. (Basically, when researching some topics, I query Google twice, second time with _site:news.ycombinator.com_ added.) So upvoting probably makes sense.

That said, I also read a lot of posts 'later', but often forget to upvote. (Unless I return to read comments—Readability doesn't work for HN comment threads, unfortunately. Partly I blame page markup for that.)

</comment><comment><USER3> I don't think there's much value in them. Most of them are less relevant over time, you have to search an external site to find them, and the chances of anybody replying are pretty bad.

Most damning of all is Digg relaunched with a clean slate!</comment>


Programming Explained In Five Minutes - <AUTHOR> http://wanttt.com/posts/show/1513

<comment><USER2>
        Heroku | Backlog too deep

        The application currently has too many requests
        in its backlog.</comment>


Federal govt should be paid as part of the deal with TikTok - <AUTHOR> https://www.bloomberg.com/news/articles/2020-08-03/trump-says-tiktok-must-sell-u-s-operations-by-sept-15-or-close <comment><USER2> ''' Still, U.S. lawmakers and administration officials have favored shutting down the application altogether to send a message to China after Beijing restricted American companies like Facebook Inc., Twitter Inc. and Alphabet Inc.'s Google. White House adviser Peter Navarro on Monday said in a pair of interviews with CNN and Fox News that he wasn't sure Microsoft was the right company to buy TikTok's U.S. operations, saying it had helped China construct its internet firewall.

"Should we trust any company that operates in China?" Navarro told Fox News. '''

What else is there left to do but to prepare for war then? If your basic position is that maximalist. There's nothing left to negotiate over. It's you or it's me.

Now the implications are no longer just Chinese nationals or ethnic Chinese- Americans. It's potentially anyone who has business ties or otherwise to China.

</comment><comment><USER3> Headline updated:

Trump Says TikTok Must Sell U.S. Arm by Sept. 15, or Close</comment>


Meritocracy doesn't exist, and believing it does has bad effects - <AUTHOR> https://www.fastcompany.com/40510522/meritocracy-doesnt-exist-and-believing-it-does-is-bad-for-you <comment><USER2> I can't help feel like including any discussion of the ultra-rich such as Gates as an example when discussing meritocracy confuses the question. That level of wealth and success relative to others can only possibly be achieved by luck, as there is no possible way one person's skill or merit can exceed that of his peers by 6 to 7 orders of magnitude. Basically, that kind of wealth is more like winning a lottery where their skills bought them a ticket. I suppose the question then becomes, could you still say there is a meritocracy if there are outliers who win the lotto? Do you discard those outliers and look at the general population, is the existence of such of outliers counter-evidence to meritocracy, or does the fact that the kind of "lottery" Gates won still required a high level of skill to even play mean it may to some degree support the idea of a meritocracy?

<comment><USER3> I stopped reading that article when he implied that Gates' wealth should be considered or measured in relation to programming skill. Debates about meritocracy are important, but starting from flawed frame like that made me not want to invest more time reading that particular author. If he wanted to compare Gates to other technology entrepreneurs, that might have been worthwhile, but implying that we don't live in a meritocracy because there are better programmers than Gates shows the author is not interested in a rigorous look at the issue. I wasn't surprised to see Bill Gates clickbait at the top of the Apple News app, but I was a little bummed to also see it on the top of HN. Then again, I still clicked it and commented on it, so mission accomplished for Fast Company I suppose.

<comment><USER4> Maybe you shouldn't be commenting if you didn't read the article

<comment><USER5> I disagree. _Many_ people do not read the articles. You can easily tell by taking the number of unique users commenting on articles and comparing to standard marketing metrics for shared articles.

And they don't need to read anyway, because comments are the real reason people come to a site like this. If the comments praise the article highly or make strong references, then perhaps it warrants a read, otherwise the comments make for much better content. You can often tell what the article's main idea is from the headline anyway, otherwise it's just a bad headline and probably a bad article.

</comment></comment></comment></comment><comment><USER6> I think the author is using a much narrower definition of "meritocracy" than many people might. There's a difference between thinking that people who occupy some position of responsibility are "the best" and thinking that in general that they instead simply meet some basic bar of competence. The problem is that for many roles, you can't simply hire somebody off the street and expect them to perform acceptably. I can write software for you but I am in no way qualified to be your lawyer.

Figuring out who is the best is hard. But making sure people are capable of doing a job before we hire them (or promote them, or whatever) is at least a tractable problem. I have a big concern that people who are advocating that we "abandon meritocracy" are going to make it really hard to have any standards at all. This might be fine for hiring people to work behind the counter at McDonald's but it's not going to work if you're trying to assemble a team to find a cure for an infectious disease or, I don't know, design the first Global Positioning System, or you know, any number of things that allow us to live in a modern society.

</comment><comment><USER7> Just as bad for you is believing that hard work and intelligence will get you nowhere. The reality is somewhere in between.

<comment><USER8> I agree. Be a reasonably intelligent worker. Play the game well enough. Work an adequate amount. Above all else, be consistently reliable, even if that means consistently delivering B+ results.

<comment><USER9> > Play the game well enough.

An economic and social system with many games one must play does sound an awful lot like a gambling house.

(Edit: substituted "games one must play" for "many games for one to play" — it's about the necessity and lack of choice, not the freedom to play games if one desires to do so)

<comment><USER10> >An economic and social system with many games for one to play does sound an awful lot like a gambling house.

Humans play games. And business is just lots of humans playing lots of games. You can either accept it, go along, and build a career, or take your shot at doing your own thing. The latter is terrifying and incredibly unlikely to succeed, so most pick the former.

</comment></comment></comment></comment><comment><USER11> As already stated, Bill Gates and other billionaires are pretty irrelevant to this question. Nobody really cares what it takes to become a billionaire, almost nobody will become one anyway.

What matters is if talent and hard work pay off, and they most definitely do. Yes it helps if your parents are rich, but that's ususally because of _their_ hard work so that's not really a counterpoint, and most successful people come from average backgrounds.

So yes meritocracy most certainly exists.

<comment><USER12> If you forget about the question of why is it that someone's parents are rich, parental socioeconomic status has comparable power in explaining children success as the child's IQ has. Reference paper: "Intelligence and socioeconomic success: A meta-analytic review of longitudinal research", Tarmo Strenze.

</comment></comment><comment><USER13> I view it more or less that way:

log(success) = S * skill + P * privilege + L * luck

Were the coefficients S, P & L depend on the measure of success.

American dream-like meritocracy says it is almost all S.

For honest "equality of opportunities" (sadly, often it's a euphemism for the above), S, P and L (and want to change it to S and L).

People for "the equality of outcome" believe it is mostly P (and L).

You won't become a top scientist by will low skill. Yet, if raised in an environment with no value on education, and not spotted by anyone - they may not become a scientist at all. Or someone with high S and P, but focusing on a problem that turned out to spend their lives on problems that turned out to be dead ends?

Do you think that say Zuckerberg (given the same skill and drive) would have achieved the same success if he spent their forming years among less privileged environment, e.g. Poland (less mentoring, no-one wanting to invest millions, not that tempting to join a students' network of a not-Ivy league university?)

For executive positions (unlike technical ones), I guess it is much more P & L than most meritocrats would like, vide

"Regardless of intellect, positions at the level director and above seem to be assigned very unpredictably (luck/politics/privilege?)" from [https://news.ycombinator.com/item?id=19406432](https://news.ycombinator.com/item?id=19406432)

Also, as a side note and subjective): everything is luck. Your innate skills are a lottery, your environment (for nurturing and networking). This "luck" in the equation is more like "residual luck" (from "I worked on problem X and it turned out to be a game changer/time waste" or "They looked at that moment for a person of my profile").

<comment><USER14> I'm not really sure how else to see it. It's pretty straightforward to show that the simplest possible model is some kind of probability density function with inputs of skill and privilege and a wide distribution and a very long tail due to cumulative effects. Not having various skills will obviously preclude certain outcomes (probably can't buy 86-DOS and produce PC-DOS if you couldn't understand the technology), not having certain privileges will preclude others (can't sell PC-DOS if you aren't in a position to meet with anyone to try) and some outcomes beget further possibilities (unlikely to have grown Microsoft and produced Windows if you hadn't have sold PC-DOS). Luck being the realization of such a probabilistic model into finite outcomes.

Anyone attempting to reduce it further than that is selling (or got sold) some very dangerous and irresponsible ideas.

</comment></comment><comment><USER15> The authors demonstrate meritocracy doesn't exist by defining it out of existence: even if your success could be entirely due to intelligence and hard work, you don't deserve the genes and life experiences that made you intelligent and hard working.

<comment><USER16> so you're suggesting that the authors definition is to narrow or unrepresentative?

Consider this though, what is the say, American public's image of meritocracy in culture? It's the dishwasher who becomes a millionaire, or the underdog who makes it big, or the entrepreneur from humble beginnings, shunned by everyone else. In fact if there is an underlying theme to the story of meritocracy, it is that the greater the adversity to overcome, the better the story.

If Inherited wealth and genetic fortune should fall into the definition of meritocracy, why aren't American movies about extremely handsome and intelligent aristocrats born into wealth, carrying forth the family name?

I strongly disagree with the claim that the author redefines the term. He uses the exact definition that is the mind of societies that uphold meritocracy, and it is a blatant myth.

How many people think Gattaca is a meritocratic utopia?

</comment></comment><comment><USER17> How about believing it _can_ exist, with appropriate changes in fundamental market mechanics?

<comment><USER18> > _The management scholar Emilio Castilla at the Massachusetts Institute of > Technology and the sociologist Stephen Benard at Indiana University studied > attempts to implement meritocratic practices, such as performance-based > compensation in private companies. They found that, in companies that > explicitly held meritocracy as a core value, managers assigned greater > rewards to male employees over female employees with identical performance > evaluations. This preference disappeared where meritocracy was not > explicitly adopted as a value..._

 _They suggest that this "paradox of meritocracy" occurs because explicitly adopting meritocracy as a value convinces subjects of their own moral bona fides. Satisfied that they are just, they become less inclined to examine their own behavior for signs of prejudice._

Which "market mechanics" are going to change that? This is a factory issue monkey cognitive defect, as far as I can tell.

<comment><USER19> This study needs to be replicated before its conclusions are taken seriously, because much of sociology is known to be deciding on a conclusion in line with intersectionality ideology dogma before writing the paper and producing papers with extreme methodology flaws. See the sokal hoax for a recent very clear expose of this academic bankruptcy

<comment><USER20> The Sokal Hoax happened in a journal of "postmodern cultural studies", not sociology. Furthermore, it was an academic essay, not an empiric and quantitative study.

Can you point to anything akin to the Sokal Hoax that happened in the context of sociological empirical studies?

Also, if you don't trust sociologists in general, replicating the study will not convince you anyway. It is a good idea to replicate, but it is not a good idea to care about the opinion of those who dismiss entire scientific fields based on ideological prejudice, be the field sociology, immunology, climate science of evolutionary biology.

<comment><USER19> The point I am making is that the premise seems to be politically motivated and not scientifically. They even redefined meritocracy to fit an intersectional paradigm in the same way as the sokal hoax papers showed.

I fail to see how it's interesting that people with different capability as well as possibly willingness to work doesn't succeed at equal rate in intellectually demanding jobs.

Meritocracy is when such people of different class backgrounds succeed at about the same rate on average. Intersectionality muddles that water in an unhelpful way.

<comment><USER20> I understand your point. I work with sociologists and I also follow the ongoing "Internet cultural wars". People like Jordan Peterson (who I don't either dislike nor agree 100%) really brought these matters to the forefront. His criticisms of postmerdernism are somewhat relevant, but they are also exaggerated. Some corners of "postmodernist academia" are as bad as he paints them, but not all of them, by any means.

Even the Sokal scandal is somewhat simplified. It was not so easy for them to get the paper accepted, in fact they were rejected many times by other journals. This is rarely mentioned.

The cited work in TFA compares performance evaluations with raises. It found that _for the same performance evaluations_ , an explicit focus on meritocracy leads to worse monetary outcomes for women. That seems fairly objective and methodically sound to me, and not at all related to Sokal-style bullshit.

The income gap is a complex topic, and it is clearly not as simple as either extremes of the cultural wars paint it. It is not true that the pay gap is 100% due to discrimination against women, but it is also not 100% true that such discrimination plays no role. As usual in real life, there are nuances and complexities that cannot easily be reduced to simple slogans.

Silicon Valley has a religion-ideology that we have no name for yet. It is a very American kind of Protestantism with zero gods. I know the environment and I know what I'm talking about, even though I accept you might disagree. My view is that the self-proclaimed ultra-rationalists are not as rational as they would like to think.

It is true that Sociology has a left-wing bias. In my view, this is true of all academia. There was a time, not so long ago, when academia had a right- wing bias. This too is a subject of sociological research :)

<comment><USER19> Those are good points. However, I do think we will look back at the large amount of postmodern literature inspired by Foucault and deidre as obfuscatory to knowledge seeking.

They essentially wrap simple concepts within a format that require immense effort to parse, maybe to present an intellectual challenge or to obfuscate what is actually said. Great for staying fashionable as a tool for showing academic belonging in Parisian coffee shops due to the amount of studying needed to pierce the veil, but not so great for knowledge seeking.

To me it seems like what they are saying are:

\- culture is adapted to get the marginal to conform to the majority (Foucault marginal sexual identity that caused him to attempt suicide)

\- culture corrupts the pristine child's mind (Rousseau inspired)

\- either you apply cultural power or you are a victim to cultural power

These are interesting to some degree, but not when used to argue for a deconstruction of all experiences into identity groups like postmodernists have.

I think they've abandoned their academic duty to try to construct world views contiguous both within themselves and to the larger culture.

<comment><USER20> > Those are good points. However, I do think we will look back at the large > amount of postmodern literature inspired by Foucault and deidre as > obfuscatory to knowledge seeking.

Postmodernism, at its core, is precisely a criticism of conventional epistemology. I am not a postmoderinst, but I do find that some of their views are interesting, or at least worth considering. I would not be surprised if you are right, and that postmodernism goes down in history as a fool's errand. In any case, I think that it is good that people kick the foundations of knowledge to see what happens. If nobody does that, it becomes very easy to fall into religion/ideology without noticing.

To borrow a term from another thinker who also hates postmodernism (I think), knowledge should be anti-fragile.

> showing academic belonging in Parisian coffee shops

You are absolutely right about this part. Just don't think that Sillicon Valley's Matcha Tea Boutique Shops are so different. The environment is also full of terms and clichés that mostly serve to signal tribal alignment, that really say nothing new and only create a barrier of entry for outsiders: "let us create a curated list of use-cases for under-the-radar angel round startups that are attempting to leverage blockchain technologies for adtech verticals."

> culture is adapted to get the marginal to conform to the majority (Foucault > marginal sexual identity that caused him to attempt suicide)

Right, but he also has some interesting things to say about our relationship with the "mentally ill", how that relationship is different in other societies and why our approach may be pathological for all of us. For example.

> culture corrupts the pristine child's mind (Rousseau inspired)

Some might say this, but overall culture is just taken as a powerful force that shapes the reality we inhabit. The thing is that, if one doesn't cherry- pick, some postmodernist ideas actually became mainstream. "We inhabit narratives."

> either you apply cultural power or you are a victim to cultural power

I think this is mostly true. No problem if you disagree, but it doesn't seem like such a crazy hypothesis to me.

> but not when used to argue for a deconstruction of all experiences into > identity groups like postmodernists have

This is a confusion between postmodernism in general and "grievance studies".

> I think they've abandoned their academic duty to try to construct world > views contiguous both within themselves and to the larger culture.

For me, the academic duty is to think seriously and independently. Any other red line only guarantees that we get stuck in local maxima. I think that the unusual views should have a place, and knowledge seekers should make an effort to understand them if they are going to criticize them.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> Per Webster, Meritocracy: a system in which the talented are chosen and moved ahead on the basis of their achievement

Of course meritocracy exists. If it didn't, I'd be a professional athlete instead of a programmer and I'd make a huge salary.

Or maybe I'd be an astronaut, or a brain surgeon.

Thankfully, meritocracy exists and allows the truly gifted and hard-working people who excel at those jobs to hold those spots.

I think I make a pretty decent coder. But I'd just stink on an NBA court, in brain surgery, and probably in a space shuttle.

</comment><comment><USER22> _If it be true (as it certainly is) that a man can feel exquisite happiness in skinning a cat, then the religious philosopher can only draw one of two deductions. He must either deny the existence of God, as all atheists do; or he must deny the present union between God and man, as all Christians do.

The new theologians seem to think it a highly rationalistic solution to deny the cat._

Chesterton, 1908

<comment><USER23> The quote is great (added it into my collection), but what is the cat in the context of this discussion?

</comment></comment><comment><USER24> Multiple social priming studies are cited. This is a warning flag given how replication crisis was most pervasive in that entire field.

</comment><comment><USER17> What would happen if we'd apply PageRank to the problem? I.e., everybody gives N votes to other people. Then we'd put everything in a big matrix, compute the first eigenvector, and obtain a ranking. And then we'd distribute wealth according to that ranking.

Since N is limited, I'm guessing that "SEO" would not be a problem.

<comment><USER25> Even without SEO, I wouldn't say that the best websites are the top results - they tend to be those most broadly applicable. Which works okay perhaps for a search engine

</comment></comment><comment><USER26> I think aspiring to the meritorious (ethical, wise, knowledgeable, dedicated, etc.) being successful seems like it'd be good, rather the problem isn't so much "meritocracy" as it is believing that meritocracy is an existing norm, rather than being more of an aspirational Avalon.

</comment><comment><USER27> "On the Media" podcast this week addresses this as well. Worth a listen. [https://www.wnycstudios.org/story/on-the-media-myth-of- merit...](https://www.wnycstudios.org/story/on-the-media-myth-of-meritocracy)

</comment><comment><USER28> Commentator Alain de Botton explores the darker side of a western ideal: meritocracy.

[https://www.npr.org/templates/story/story.php?storyId=320002...](https://www.npr.org/templates/story/story.php?storyId=3200029)

</comment><comment><USER29> Pure communism doesn't exist, pure capitalism doesn't exist, pure fascism doesn't exist. Pure ideals of any kind don't exist. Meritocracy exists _to varying degrees_ all over the place, and it's something we should be striving for wherever possible. There are good arguments for temporary exceptions to meritocracy to, say, correct historical inequities, but it's extremely important that those be viewed as temporary roadblocks on the ultimate path towards a society that is as meritocratic as we can make it. Which, by the way, doesn't necessarily mean that those low on 'merit' need to be left to die, we can still provide a good life for even the least economically valuable individuals, but yes, those who are providing the most economic utility to society should get to consume more of that utility.

> According to Frank, this is especially true where the success in question is > great, and where the context in which it is achieved is competitive. There > are certainly programmers nearly as skilful as Gates who nonetheless failed > to become the richest person on Earth. In competitive contexts, many have > merit, but few succeed. What separates the two is luck.

This is also completely false. What separates Gates from most other programmers of his skill level is willingness to take on risk. Gates took on tremendous risk to do what he did. He could have failed spectacularly, but he chose to enter the startup lottery and he won. That's the deal you make, you go into that game knowing that it's a lottery, and of course luck is involved. But we have that lottery in place for a very good reason: New companies provide tremendous value to society, and so it's important that we incentivize people to create them. Nobody would take on the kinds of risks that he did if the rewards weren't enormous.

<comment><USER30> Gates didn't take much of a risk, he was never going to be homeless if his plans didn't work out. He is pretty much the perfect example if you want to prove we don't exist in a meritocracy.

<comment><USER29> Gates absolutely took a risk. He devoted many years of his life to something that may not have panned out. He could have spent those same years working his way up at say, IBM. Opportunity cost is a real risk. It's not the same as being homeless, but that doesn't make it not a risk.

</comment></comment></comment><comment><USER31> Laws of nature are in every person identical so it can't be both luck and hard work in different people in different amount.You cross street on green light and 160 mile per hour car kill you.Everyone would say bad luck determine that man life.If person get money on lotterry everyone would say good luck determine that man life.But if person get richer every day by medium amount people and end up being rich people say hard work determine that man life.If you accept that hard work determine man life than you must accept that lottery winner get his money through hard work.Otherwise you must accept that person that worked hard just got lucky most of days by medium ammount.Sorry for my english.

</comment><comment><USER32> Equality doesn't exist, and believing it does is bad for you.

That doesn't mean it's bad to have it as a goal / ideal.

<comment><USER22> As was pointedly observed in Harrison Bergeron, _sameness_ and _equal treatment under the law_ are not the same.

To attempt to have the former is folly; to attempt the latter, sanity.

</comment></comment><comment><USER33> > As with any ideology, part of its draw is that it justifies the status quo, > explaining why people belong where they happen to be in the social order.

I'd say that more than anything, what it offers is hope for those lower in the social order to improve their position, even if that's not very true to life. That's why the right's struggling middle class holds onto this worldview even as elites use it to take advantage of them.

I also think the author misses an opportunity by lumping all non-merit factors together as "luck". By far the biggest factor in success isn't sheer random luck, but your current standing (whether because of your family or because of wealth you've already accumulated). Capitalism as we know it is inherently exponential, which I think is its biggest problem. Success breeds success. You (or your family) only have to get lucky once and then the rest takes care of itself, forever. A person who contributes nothing to society can continue to rake in millions every year, give their children the finest education and connections, etc.

</comment><comment><USER34> > Talent and the capacity for determined effort, sometimes called "grit," > depend a great deal on one's genetic endowments and upbringing.

It is sad that some people are born dumb as a rock but we shouldn't make them math professors out of pity.

<comment><USER35> Agreed... but we also shouldn't make them suffer and starve.

The bad part of the 'meritocratic' world view is not that having people who are good at certain tasks do those tasks is a bad idea, it is that thinking those people are more 'deserving' of a good life is bad.

People that are good at math should be math professors, but we shouldn't think they are morally superior to the person born dumb as a rock, nor do they deserve a better life.

<comment><USER36> They deserve a better life as they can contribute more to the society, in effect building this better life.

<comment><USER35> That is one perspective on what make someone deserving or not. I disagree, but I understand it.

</comment></comment></comment></comment><comment> your-nanny what this author fails to do is propose an alternative that captures what we like about meritocracy, but avoidsvtgese unintended consequences.

</comment><comment><USER37> No one deserves or merits any success they have achieved.

Got it.

</comment><comment><USER38> What remains, hustling?

<comment><USER39> That's one of my three pieces of advice for young engineers.

Learn to fucking hustle. Learn to say no. Learn to know when to leave.

</comment></comment><comment><USER40> There are two issues I've identified:

1) For inclusion it is important that the people with power understand that people that currently have power are as capable as they are. So you see that with disenfranchised groups that have a history of systemic disenfranchisement based on irrelevant and sometimes non-existent biological differences needing to prove otherwise.

2) For the actual goal of companies and many organizations, meritocracy is counterproductive. So you have companies that require people with specialized skillsets, and those people think meritocracy is important. But the company is trying to cater to a broader and broader addressable market. It is more productive to have inclusion looks more like the market instead of an extreme selection of intellects. The flaw being to put intellect on a pedestal in isolation.

So its just as important that people are respected to have access to the pipeline and pedigree, as it is to understand that its also not that important and that other people with different attributes should be included anyway.

</comment><comment><USER41> Obligatory note that the word "meritocracy" was coined by a sociologist in mid-century England as a distopian ideal, distopian because it was used as a bad excuse for privilege and the withholding of social services. [https://en.m.wikipedia.org/wiki/Meritocracy](https://en.m.wikipedia.org/wiki/Meritocracy)</comment>


Intentional (Software) Qualities - <AUTHOR> https://medium.com/@elizarov/intentional-qualities-7e6a57bb87fc <comment><USER2> I agree that qualities like "fast"/"performant" are intentional, and are thus prominently displayed in the project's readme, but are qualities like "maintainable" often found as a stated goal in readmes?

Or if the readme doesn't mention such qualities, can one assume that if it isn't deliberately fast etc. then it is likely to be maintainable because that quality hasn't been sacrificed for a different one?

I'm tempted to say no because not all developers necessarily have the experience to ensure their code base is easily maintainable, but assuming the developer(s) do have experience in writing clean code, is it then safe to assume it [is maintainable]?

<comment><AUTHOR> Shouldn't clean code always be maintainable? Or maybe if setting everything up is too cumbersome or the build system is crappy it's not maintainable, hmm.</comment></comment>


UBM eliminating key employees, shutting down Blackhat security conference - <AUTHOR> https://twitter.com/treyford/status/408662334588985344 <comment><USER2> I haven't been able to find anything stating that the Black Hat conference is shutting down (which is very different than dismissing the GM). Can you provide your source for that information?</comment>


How Facebook can Make of Money while Delighting its Users and Not Turning Evil - <AUTHOR> http://blog.luckycal.com/?p=139

<comment><AUTHOR> This is an idea I've been kicking around since building our iPhone app.

</comment><comment><USER2> You make some very good points there, and I love the idea of being able to go to a new site, let it know who I am, and have it already configured for me.

The question I see though is how Facebook can turn around and regain the trust it lost through some of its misguided ventures like Beacon, etc.</comment>


UK Prime Minister Boris Johnson moved to intensive care - <AUTHOR> https://news.sky.com/story/coronavirus-prime-minister-boris-johnson-moved-to-intensive-care-after-condition-worsens-11969587 <comment><USER2> Discussion here: [https://news.ycombinator.com/item?id=22796906](https://news.ycombinator.com/item?id=22796906)</comment>


Centro hands out TI LaunchPad starter kits to students - <AUTHOR> http://www.somethingunimportant.com/2010/11/16/fun-with-tis-launchpad-starter-kit/

<comment><USER2> much better:

 _The eZ430-Chronos is a highly integrated, wearable wireless development system that comes in a sports watch. It may be used as a reference platform for watch systems, a personal display for personal area networks, as a wireless sensor node for remote data collection, or simply as a watch._

<http://processors.wiki.ti.com/index.php/EZ430-Chronos>

</comment><comment><USER3> My favorite sentence: "Since we don't know the students are that are getting these boards we decided we needed to do some assembly and, because electronics bought in bulk are so cheap we threw in some LEDs, resistors, wires and a breadboard."

Why are writing skills so abysmal? I didn't go through any special writing classes, yet I manage to form relatively brilliant sentences compared to my peers. And this, yes is more than jsut a typo too. The sentence is just structured poorly from the onset, has commas in the wrong places and is missing commas.

<comment><AUTHOR> My favorite sentence: "I manage to form relatively brilliant sentences compared to my peers."

I bet your peers absolutely love working with you.</comment></comment>


Android based camera from Nikon: Coolpix S800 - <AUTHOR> http://nikonrumors.com/2012/08/08/the-coolpix-s800-will-be-the-first-android-based-camera-from-nikon.aspx/ <comment><USER2> Finally, apps for cameras. Thom Hogan has been advocating this for some time now[1] and the success of the CHDK[2] somehow shows us that a lot of photographers may be interested in this. I hope Nikon starts a trend with this camera and go all the way to Android enabled DSLRs.

[1] <http://bythom.com/design2010.htm>

[2] <http://chdk.wikia.com/wiki/CHDK>

Some ideas: [http://news.stanford.edu/news/2009/august31/levoy- opensource...](http://news.stanford.edu/news/2009/august31/levoy-opensource- camera-090109.html)

</comment><comment><USER3> Interesting...the one nice thing I do like about the Google+ app running on Android is how it automatically uploads my pictures to a private folder online. Many times, I have come back from long hikes but the pictures remain in my slr for a few days because I was too lazy/tired to upload my pictures from the SD card to the computer. Having this functionality to automatically upload your pictures once you get home would be nice.

<comment><USER4> For the "antisocial", you can do the same with Dropbox.

<comment><AUTHOR> In fact, I prefer Dropbox's Camera Upload. With Dropbox, when I go home and sit in front of my computer, I find my photos as actual files existing on my computer. Google, on the other hand, is trying to make it very convenient to share them on Google+ and very inconvenient to download and store them locally. They don't even sync it to Google Drive as files. They are trying to eliminate the concept of "file" from the users' mind, just like Apple.

</comment></comment></comment><comment><USER5> This kind of thing is the reason I've started boning up on Android development. With the sheer volume of phones on the market and the versatility of the OS in completely unexpected applications I think there's going to be a lot of demand for people with Android skills in the future.

<comment><USER6> I'd love to learn it, but Java and XML scare me away.

<comment><USER5> It feels kind of verbose some times but with a good IDE like IntelliJ it's really not that bad. Hopefully Kotlin will take off on Android and eliminate a lot of the verbosity.

Obj-C on iOS isn't much better really though.

</comment></comment></comment><comment><USER7> Adding android to their p&s cameras is not what I expected from them. Nikon has a huge problem to tackle and adding android to their p&s is not the solution.

While Nikon makes great DSLR (but not necessarily better than the competition), their point and shoot cameras aren't any good. Not when compared to the Canon S95 in the cheap segment. Or the Fuji X10 in the expensive segment. Nikon can't even get the mirror-less interchangeable lens system right : what were they thinking with the Nikon 1 ? the sensor is way too small, much smaller than the already small sensors of the Four Third system from Olympus/Panasonic and the size of the sensor is one of the most important thing to consider when it comes to image quality. Canon obviously knows better than Nikon and their entry to that market will feature an APS-C sensor.

Whatever it is that Nikon does, they do not understand photography anymore, apart from the few who work in their DSLR division.

</comment><comment><USER8> Some random camera stats:

The iPhone has a 1/3.2" sensor (9.7x crop factor), a 4.28mm length f/2.4 max aperture.

That's like a 43mm f/23 full frame (basically, apature scales with crop factor just like length); or a 26mm f/14.5 APC-C.

Compared to a 1/2.3" sensor (which the S800 is likely to pack) it's like a 6mm f/3.3 lens (43mm full frame, as they don't like to mention how apature scales). The advantage of a S800 will be the range (25mm -> uselessly long without a tripod and good light), compared to the 43mm prime iPhone; apature and image quality will be similar. Also, the Nikon will have niceties like vibration reduction, a flash (maybe), smarter focusing / metering, better processing (maybe).

The real question is, who would want one? I'd say - people with dumb phones who carry cameras; and hipsters who want to be seen with a camera while they play Angry Birds on the bus. It might be bigger in non-US markets, where phones don't appear so cheap (due to hidden carrier costs).

<comment><USER9> Does aperture scale with the crop factor? I don't think that's the case. f/2.3 will always let the same light in. As you (hypothetically) scale up the focal length (the numerator), you also scale up the pupil size (the denominator), so the aperture remains constant.

The effect on DoF is different, sure, but not on light – and I think with those kinds of cameras you are more interested in their ability to let in light than their ability to produce shallow DoF (that's pretty much a given for any small sensor camera).

<comment><USER10> If you scale up everything, the aperture stays constant, correct.

But since the larger sensor can work with much more light, it therefore has a much higher S/N-ratio. If you switch that camera to a higher aperture, you get less light, leading to a lower ratio again.

So the lower S/N-ratio of the smaller sensor could be estimated as a higher aperture. Whether the numbers calculated by wisty are correct, I don't know, though.

<comment><USER9> Please don't do that. That's just confusing the issue and highly misleading.

The f-number is a property of the lens and nothing else. The S/N ratio of the sensor is independent of that.

It's correct: smaller pixels tend to have a lower S/N-ratio, which mostly means that you can't turn up the gain as much as with sensors that have larger pixels (you can't increase the ISO by as much) and you will see more noise at the same gain.

But that's no law of nature. It's possible to have a crappy large sensor and an excellent small sensor. It's the sensor that matters here – and there are many more variables involved there. Sensors with pixels a certain size don't have a certain S/N-ratio only depending on pixel size. And any exact mathematical formula (like f-number scaling with crop factor) doesn't make sense there, not at all.

<comment><USER8> Depth of field does scale perfectly with crop factor, and that's a big thing.

f-number is a ratio of apature to length. A 400mm f/5.6 will have a 71mm apature. An iPhone lens does not (despite having a better f-number).

The light is spread out more, so you'll a higher ISO to take fast photos, but let's say ISO is proportional to sensor size. A larger sensor captures more photons, so it can have a higher ISO. That is a law of nature.

As you say, ISO is neither deterministic nor linear, but I'm more confortable considering ISO to scale with crop size than pretending it's constant.

If we are ignoring ISO altogether that's fine, but then there's no point talking about the amount of light hitting the sensor.

That leaves diffraction, but I don't really understand diffraction.

</comment></comment></comment></comment></comment><comment><USER11> Every new device launched with an obsolete version of Android is part of the price that Google pays for making Android as free as it is. And frankly, I don't see the market for a slow, buggy Android experience tied to a Nikon camera.

<comment><USER12> This is not a slow, buggy Android that happens to come with a camera. This is first and foremost a camera.

And when its running Android, that can be win win for everyone involved, if Nikon makes the hardware sufficiently open. Anything is probably better than having to flash your custom firmware through a freaking IR led.

<comment><AUTHOR> But still, people are going to run apps on it and they are going to see the user interface everyday (which was vastly improved with Ice Cream Sandwich). They would never release this with Gingerbread if they really cared about user experience.

</comment></comment></comment><comment><USER13> This might only work if it remains a camera; tactile buttons, a non attention- stealing screen, its own app store(custom camera apps), and the apps function more like plugins/addons than apps. Otherwise it'll be just another android phone with a better camera instead of a simpler CDHK.

</comment><comment><USER14> Does this really make sense?

I mean, a camera is for taking pictures and the corresponing OS should only care about the job of taking pictures and show them later on.

Why do I need a fullfledge Android system running on my camera?

This seems more like a toy experiment.

<comment><USER15> There are so many things a camera could do that it can't right now: * Sharing photos on Facebook, Twitter * Post processing pictures (like Instagram) * Automatic backup to Dropbox * Live stream videos

<comment><USER14> That is what computers are for.

</comment></comment></comment><comment><USER16> Could anyone who knows about optics say how this compares to Nokia's 808 PureView camera/phone?

<http://en.wikipedia.org/wiki/Nokia_808_PureView>

</comment><comment><USER17> I was seriously let down that my new washing machine doesn't run on android.

</comment><comment><USER18> There's a rumor site focusing exclusively on Nikon? : o

<comment><USER19> Not only that, there are rumour sites for most of the camera systems:

<http://www.canonrumors.com/> <http://www.sonyalpharumors.com/> <http://www.fujirumors.com/> etc

<comment><USER10> Don't forget <http://leicarumors.com/></comment></comment></comment>


Show HN: Stargate – An open source API framework for data - <AUTHOR> https://stargate.io/2020/09/14/init-stargate.html <comment><USER2> Cool, congrats on this release.

Any advantages of having Cassandra as underlying db ? An example docker- compose would help to try it out quickly.

We[1] are in similar space but mostly for SQL databases. With automatic APIs we do provide GUI for schema management.

[https://github.com/xgenecloud/xgenecloud](https://github.com/xgenecloud/xgenecloud)

<comment><AUTHOR> Thanks!

The reason for Cassandra is the normal reasons someone would turn to Cassandra, fault tolerance, scalability, high write throughput. It's a powerful database we really like but can get a bad rap at times so with Stargate we wanted to make it easier for developers to get their data in and out with their preferred interface. Although we have left it open to extension so if Cassandra isn't your cup of tea we welcome PRs to add new persistence backends.

Example docker-compose and helm charts are definitely in the works! Stay tuned and we should have those out soon.

Oh that looks cool! I'm digging the UI. Thanks for sharing, I'll have to try that out later and maybe we can learn some things from each other.

<comment><USER2> Cool, feel free to reach out. Happy to 've a chat.

</comment></comment></comment><comment><USER3> This is very cool, and I really look forward to the future of the project. Have you thought about formalizing your API spec as an RFC? I generally think that the future of data is to allow complete access to all of your data on user programmatically, and this looks like an amazing tool that would help organizations accomplish exactly that. For example, a user story could be "I want to access then delete all of my comments on Facebook", or "I want to build an application that lets users view their data on Facebook as if it were an Excel spreadsheet".

</comment><comment><USER4> I love the name! It's pretty similar to one of my projects: Cloudgate a fast multi-threaded web application server and API gateway for Node.js

[https://github.com/elestio/cloudgate](https://github.com/elestio/cloudgate)

</comment><comment><USER5> So this is Hasura for Cassandra?

<comment><USER6> Similar, both are an API layer on top of the database, though Stargate is not strictly bound to GraphQL.

</comment></comment><comment><USER7> [https://xkcd.com/927/](https://xkcd.com/927/)

</comment><comment><USER8> It's a pet peeve of mine when people use real domains in example email addresses: [https://github.com/stargate/docs/blob/v0.1.0/modules/develop...](https://github.com/stargate/docs/blob/v0.1.0/modules/developers- guide/examples/stargate-rest-api.postman_collection.json#L194)

Please do consider using ".example" which was designed for that purpose, or "example.com" if the former is too jarring: [https://tools.ietf.org/html/rfc2606#page-2](https://tools.ietf.org/html/rfc2606#page-2)</comment>


Semiconductor Pioneer & Nobel Laureate Dr. Walter Kohn Has Died - <AUTHOR> https://en.wikipedia.org/wiki/Walter_Kohn <comment><AUTHOR> Dr. Kohn was a World War II era Austrian-Jewish Harvard trained immigrant to the United States largely responsible for cementing our understanding of the way electricity flows through matter. This theory, called the Density Functional Theory, has changed the way we understand solid matter's relationship to electrical current and has helped physicists to vastly improve computational power, as just one example among many. Dr. Walter Kohn was a brilliant and humble man, a professor, research scientist and gentleman of the first rate, and UCSB's very first Nobel Laureate in Chemistry (even though he was actually a physicist.)

[http://web.physics.ucsb.edu/~kohn/](http://web.physics.ucsb.edu/~kohn/)

[http://www.nobelprize.org/nobel_prizes/chemistry/laureates/1...](http://www.nobelprize.org/nobel_prizes/chemistry/laureates/1998/kohn- bio.html)

</comment><comment><AUTHOR> 'Physics isn't what I do; it is what I am.'

[http://dailynexus.com/2016-04-22/ucsb-professor-and-nobel- la...](http://dailynexus.com/2016-04-22/ucsb-professor-and-nobel-laureate- walter-kohn-passes-away-at-93/)</comment>


O'Reilly on "Web 2.0 and Cloud Computing" - <AUTHOR> http://radar.oreilly.com/2008/10/web-20-and-cloud-computing.html

<comment><USER2> There's an interesting note about amazon in there: "if Amazon intends to gain lock-in and true competitive advantage. . .".

The thing that's missing is any history of Amazon going for lock-in. In fact, Amazon looks like the anti-lock-in company. They allow their competitors to sell in their site, used products to be sold alongside their new products, etc. The closest thing to lock-in that Amazon has tried is Amazon Prime in which you pay for 2-day shipping in a lump sum rather than as you go.

Amazon's business has always been about efficiency and high-volume, low- margin. I don't see why their hosting business would end up different. In fact, Amazon seems perfectly comfortable with companies like RightScale selling on top of them.

<comment><USER3> _In fact, Amazon seems perfectly comfortable with companies like RightScale selling on top of them._

Just because Amazon has nothing to fear from their customers doesn't mean the reverse is true - Amazon just announced they are starting to add the kind of monitoring and management features to EC2 which RightScale provides, making EC2 more attractive and forcing RightScale to do more to differentiate themselves.</comment></comment>


Setting out plans to monitor all Internet use in the UK - <AUTHOR> http://www.bbc.co.uk/news/uk-politics-18434112 <comment><USER2> If you want to understand the state of state surveillance technology and thinking in the UK then I highly recommend Henry Porter's novel "The Dying Light".

[http://www.amazon.co.uk/The-Dying-Light-Henry- Porter/dp/0752...](http://www.amazon.co.uk/The-Dying-Light-Henry- Porter/dp/0752874845)

Although it's a novel, it's written by a well-known journalist with an interest in civil rights and liberty: <http://en.wikipedia.org/wiki/Henry_Porter_(journalist)>

</comment><comment><USER3> It's so trivial to work around this that I'm stunned they're even bothering to implement it. As we've heard over various stories recently terrorist groups were doing such varied things as steganography, if they've managed to work out how to do that I'm sure they've sussed VPNs, anonymous proxies and disposable network points.

Pointless and makes _my_ life as a British citizen more awkward as I'll having to put all my network traffic off through something rather than leaving it as a nice, simple internet connection.

<comment><USER4> If you are routing your traffic through VPN and anon proxies or use Tor then _you must be a terrorist_!

Totally retarded, but I bet they will look at your traffic harder if you did this, even though I have nothing to hide...

Interesting read from something that was posted on HN yesterday: <http://news.ycombinator.com/item?id=4105485>

<comment> mike-cardwell My understanding is that they need a warrant to look at the log of your traffic. Presumably they'd need to look at the log of your traffic in order to see if you're using a VPN or Tor.

So what you're suggesting means they'd need to look at your traffic without a warrant in order to determine if they need a warrant to look at your traffic.

[edit] Also, presumably they can already get a warrant to tap your traffic. All this new legislation means is that they can look at historic traffic too?

Don't get me wrong. I am British, and I do think it's fucking disgusting. But don't forget, it's just a proposal.

[edit2] I was wrong. It seems they wont need a warrant to see who you're talking to. Just a warrant to see what you're saying. So yes, I guess they'll be able to tell that you're using a VPN without getting a warrant first.

<comment><USER4> Don't know to be honest, unless EVERYONE is running encrypted traffic/VPN/Tor then the few that are using it stick out like a sore thumb. It is bound to raise _some_ suspicion. I am British too btw and hoping it is _just_ a proposal at the moment, but these things have a way of being watered down to make them "more acceptable" or revived under a different guise still have a whilst still have the core concepts in them.

Problem is I don't trust our government with our data, they've shown sheer incompetence far too many times when it comes to IT systems. And it's always a case of do as we say, not as we do as the expenses scandal and the current Leveson enquiry is showing.

</comment></comment></comment></comment><comment><USER5> > But senior Tory David Davis said it was "incredibly intrusive" and would > only "catch the innocent and incompetent".

I think this sums it up, this data will be of little help for stopping crime, but it would be of great help for corrupt officials to do evil not to mention the risks involved with storing this data.

<comment><USER6> "The only people who will avoid this are the actual criminals, because there are ways around this - you use an internet cafe, you hack into somebody's wi- fi, you use what's called proxy servers, and they are just the easy ways."

It's sad that it's shocking for a politician to be able to actually list ways that this sort of system can be worked around.

<comment><USER7> I'm also disappointed that the Lib Dem ministers are so easily placated by "warrants must be approved by the Home Office", and not a judge. This shows an unnerving readiness to erode the separation between legislature and the judiciary.

Given that another Conservative minister claimed, under oath, that he doesn't know what "quasi-judicial" means (he was appointed that exact role), I don't have much hope that the home secretary will apply legal advice before handing out warrants.

<comment><USER6> I'm not informed massively on the work-load that the home secretary is under but is it really a good idea to even expect that they have the time to do proper research and take advice into account when judging whether a warrant is worthy or not?

It seems an awful lot like catching someone in a hurry to get them to sign something because you don't want them to read it.

</comment></comment></comment></comment><comment><USER8> And as usual the criminals are going to find another way.

I await more "the terrorists are coming" and "think of the children" legislation.

However, we have one thing on our side: government IT incompetence. They have managed to screw every major IT project up in the last 20 years, so this will go the same way :)

<comment><USER9> Unfortunately this is a double edged sword. Their incompetence also caused the a lot of trouble like in this example <http://beusergroup.co.uk/index.php?id=695> where they proxied file sharing sites thus forbidding access.

</comment></comment><comment><USER10> There are two parts to the plans for Internet intercept support in the UK, the police and the security services. The police use is primarily for the purposes of stopping crime. The purposes for the security services are opaque and unlimited as long as they are for 'lawful purposes' - i.e. anything under the Intelligence Act 1994.

When opposing this it's important to understand the remits of the two organisations and that one is nominally accountable to the public, while one isn't.

<comment><USER11> The secret security services supposedly have a lot of oversight at all levels; including politicians, which in theory are accountable to the public.

(<http://www.gchq.gov.uk/AboutUs/Pages/index.aspx>)

I agree that's suboptimal.

I am much less worried about letting GCHQ trawl my traffic data than I am letting my local council noodle around in it. Mostly because GCHQ probably already do it - they certainly have the capability, but also because local councils have shown themselves to be corrupt and "data-leaky".

I'm not bothered about my traffic data being trawled. I oppose this bill because it's stupid - the people who have to keep data are not clearly defined; hiding some of that data is trivial for criminals; keeping that data is an unreasonable burden on some (but not all?) ISPs; etc etc.

It's frustrating that there are some really clueful politicians and advisors, yet governments keep pumping out really stupid laws about computers and networking.

</comment></comment><comment><USER12> For all you people who are not phased by it and think it's OK and harmless: you do realise it's your own tax money they are senselessly wasting here right? A lot of money. For nothing.

<comment><USER13> There's a lot of bigger wastes of public money TBH. Like the Leveson enquiry.

</comment></comment><comment><USER14> Brought to you by the same crowd that passed the "cookie law" that requires user opt in for any cookies - even if it is storing something as trivial as a session ID for the duration of one visit.

These legislators seem to be regulating what they do no understand.

</comment><comment><USER15> [http://bigbrotherwatch.org.uk/files/CCDP/CCDP_Briefing13June...](http://bigbrotherwatch.org.uk/files/CCDP/CCDP_Briefing13June.pdf) has some good information on the counter arguments to this bill. (Some of the media seems to be printing the Home Office briefing with only a little mention of critics).

If UK people are interested in helping stop this then the Open Rights Group is running training days on this topic: [http://www.openrightsgroup.org/events/2012/censorship-and- su...](http://www.openrightsgroup.org/events/2012/censorship-and-surveillance- campaign-training)

</comment><comment><USER16> Lets add my 2 centz

I dont consider myself any form of hacker, I dont think i do anything illegal on the interweb. However i am very for privacy on the internet and against goverment monitoring.

When SOPA and CISPA etc all came about my first port of call was to get off gmail and on to my own webserver on a VPS. 2 weeks ago i deicded it was time to create my own elite anon proxy using squid. Took a few days of tinkering (sidenote did you know that google can get your IP via user_agent header? took me ages to work out why all the sites but google were getting my VPS IP and yet google could see right passed it and get my orignal IP)

now I am posting to this topic using said proxy. I can bet that once all these systems go live I will be one of the first pulled up as a terroist. I have VPNs to 2 countries, and 2 machines route out over those, i have very little standard traffic going via my ISP, and i use external DNS (currently in the process of setting up my own bind server).

I am even in the process of setting up my own jabber server (what did google rename it to xxmp?) and using that as a replacement for MSN/Skype interaction thing with my friends.

All of the above will classify me as a terroist under the UKs ever watchful eyes, I think now I am going to route my proxy in to tor for extra funz

<comment><USER17> User agent headers don't include IP addresses. Are you sure Squid isn't setting the X_FORWARDED_FOR header and revealing your IP to every site you visit which is clever enough to look?

<http://www.squid-cache.org/Doc/config/forwarded_for/>

</comment></comment><comment><USER18>
      [...] the government would be able to request any service
      provider to keep data about internet usage, although
      initially it will involve about a dozen firms including
      BT, Virgin and Sky.

I guess that includes VPN services (like HMA, who are based in UK).

</comment><comment><USER19> Anyone else catch the irony that this is the same nation that passed an absurd "cookie law" in the name of protecting people's online privacy?

[http://www.theregister.co.uk/2012/05/26/eprivacy_cookie_comp...](http://www.theregister.co.uk/2012/05/26/eprivacy_cookie_compliance_begins/)

As I understand it they backed away from the stronger restrictions at the last minute... but still, the very idea of a surveillance nation like the UK "protecting" people by blocking third party cookies would be hilarious if it wasn't also tragic. Very generously they've said they won't prosecute offenders... probably... unless they don't like you... or have a political axe to grind... or you're the wrong ethnicity...

<comment> mike-cardwell There is no irony, because they are completely different things.

You think that if a government decides that its citizens should be granted extra privacy protections from corporations and individuals, then it's only right that they grant the same protections from law enforcement?

I guess it's also ironic that my government prevents McDonalds from locking me up, but will lock me up themselves if I commit a crime?

</comment></comment><comment><USER20> A lot of websites now offer SSL by default, or as an option - is this more of an obstacle or can ISPs use some sort of MITM technique to defeat the SSL for snooping purposes?

Forgive my ignorance, I'm very curious about this as a UK citizen.

<comment><USER21> Businesses (and governments) can legitimately buy root keys that allow MITMing any SSL connection or they could just be a CA themselves (any CA can MITM the whole internet).

Here's the best talk I know of this subject: BlackHat USA 2011: SSL And The Future Of Authenticity: <http://www.youtube.com/watch?v=Z7Wl2FW2TcA>

<comment><USER22> I find this hard to believe.

 _"Businesses (and governments) can legitimately buy root keys that allow MITMing any SSL connection"_

A business can buy their own SSL keys, but to MITM any SLL connection those keys would have to be trusted and installed at both ends of the connection and used to generate the SSL session key.

 _"any CA can MITM the whole internet"_

Again, the CA's keys would have to be trusted and installed by the communicating parties.

Or have I misunderstood your point?

<comment><USER21> You did not misunderstand me, but you misunderstand how SSL in the context of HTTP is used and what CAs offer for sale. Please see the video I linked in my original post.

<comment><USER22> I don't have the time or opportunity to watch a 50 minute video right now, but I found a summary of the talk on the presenter's blog [1]. Interesting stuff! He references a paper [2] that discusses a "compelled assistance" attach, where a government can use the law to compel a CA to hand-over a certificate that would certainly be usable for a MITM attack. Bruce Schneier also mentioned this [3].

But this is far from "Businesses (and governments) can legitimately buy root keys that allow MITMing any SSL connection". Businesses can't invoke CALEA (or the UK RIPA[4] law, since we're talking about UK government surveillance) - only governments can do that. And the keys are not really being _sold_ or _bought_. And I don't see how a government could compel a CA in a different legal jurisdiction anyway.

 _"any CA can MITM the whole internet"_

No. A CA has access to keys that can be used to MITM a connection that is secured by keys issued by them. Thats not the "whole internet".

[1] [http://blog.thoughtcrime.org/ssl-and-the-future-of- authentic...](http://blog.thoughtcrime.org/ssl-and-the-future-of-authenticity) [2] <http://files.cloudprivacy.net/ssl-mitm.pdf> [3] [http://www.schneier.com/blog/archives/2010/04/man-in-the- mid...](http://www.schneier.com/blog/archives/2010/04/man-in-the-midd_2.html) [4] [http://en.wikipedia.org/wiki/Key_disclosure_law#United_Kingd...](http://en.wikipedia.org/wiki/Key_disclosure_law#United_Kingdom)

<comment><USER23> > And I don't see how a government could compel a CA in a different legal > jurisdiction anyway.

Think Black Hawks and men in black suits.

</comment></comment></comment></comment></comment></comment><comment><USER24> My next startup's business plan, using £5000 of seed money to make £1B:

£2500 to bribe a young woman at a high-profile company to accuse their upper management of sexual harassment.

£2500 to bribe the cop for access to all of the company's data.

Sell to interested parties.

Repeat.

</comment><comment><USER25> Anyone here know enough about UK politics to know how likely it is that this will pass?

</comment><comment><USER26> "...The police and security services are concerned that criminals and terrorists are increasingly evading detection by using _social media_ and _online gaming sites_ to communicate with each other..."

of course they do - because the terrorists and the criminals have the brains and the survival instinct of a peanutbutter sandwich!</comment>