UnrealCV - <AUTHOR> https://unrealcv.github.io/ <comment><USER2> Aww, I don't get to create a walkthrough of my resume in Unreal? I'd love to let recruiters shoot my bosses.

(nevermind that the last Unreal I played was in 1998.)

<comment><USER3> I could actually make this, and it sounds like a cool idea. But I'm wary about putting the time into it.

My concern is how many recruiters would actually download and run a 1 or 2 GB application simply to see my resume, when they instead could visit my website and watch some quick videos.

My guess? None.

<comment><USER4> there's an experimental HTML5 feature so you could ostensibly embed it online.

</comment></comment></comment><comment><USER5> This is a great example of how we (collectively) are using virtual worlds as "dreams" for Deep learning.

In the Deep Vision world we as a group are trying to segment, classify and reinforce our NN training on labeled real world data. The challenge is, it's a very manual process to label data - specifically images. The more that we can do inside the computer, for example automatically labeling pixels inside an image, without having to acquire and label real world data (or making it easier to do with real world data) the easier and faster training becomes that can be applied to real world use cases.

The trick is making the virtual world match the real world as closely as possible so that the nets we make are accurate representations of real world scenarios.

<comment><USER6> It's not just for deep learning, it's for any computer vision research.

Synthetic datasets have been used in the past too, but this makes it much more accessible.

<comment><USER5> Indeed...

However CV is becoming more and more Deep Net based.

</comment></comment></comment><comment><USER7> I was hoping that this would be someone's really cool resume, but this is interesting, nonetheless.

<comment><USER8> Was actually hoping for an alternative to OpenCV that wasn't created in the 90s.

<comment><USER9> What problems do you see with opencv?

<comment><USER8> Nothing, actually, it's a quality library, just old.

I can think of a few algorithms that have changed or been created since the 1990s so I have to assume there's something that can be done to optimize it.

There's also a severe lack of "in-browser" CV solutions (which could be fun), instead opting for binding OpenCV in the server-side.

</comment></comment></comment></comment><comment><USER10> Can someone explain what this is for?

Is it so that computer vision researchers can create scenes in UE4 that can then be fed into their application?

<comment><USER11> You could use it to say, auto create large labeled data sets of images and their corresponding depth maps. Or generate large labeled data sets of objects and their segmentation masks. Or for training an autonomous agent (say self driving car) in a simulation environment

<comment><USER12> Exactly. That's the goal for this project.

<comment><USER11> Many thanks for your work, will be putting it to use! Please consider integrating it with [https://gym.openai.com/](https://gym.openai.com/) for easy reinforcement learning tasks.

Some similar efforts (though not with open source engines)

[http://download.visinf.tu- darmstadt.de/data/from_games/](http://download.visinf.tu- darmstadt.de/data/from_games/)

[https://arxiv.org/abs/1608.01745](https://arxiv.org/abs/1608.01745)

</comment></comment></comment></comment><comment><USER13> I thought about something like this for autonomous cars. The game engine could simulate all the sensors in a car. Then it would be possible to use this as training data. Would be fun to create a racing game out of it to let different algorithm compete against each other. Does this already exist? Google should have something like this, but probably not freely available.

<comment><USER14> AdasWorks is doing this, check the videos 'Data Generation Tool - Annotation' and 'Real-time Data Simulator' towards the bottom of this article (text is in Hungarian): [http://index.hu/tech/2016/09/13/obudan_epul_az_autoipar_noki...](http://index.hu/tech/2016/09/13/obudan_epul_az_autoipar_nokiaja/)

</comment></comment><comment><USER15> This is a brilliant idea, IMO, and a real game changer (in principle, not necessarily tied to the implementation). Without this, CV problems are largely only limited by how much data you can get. If you can simulate arbitrary scene configurations it's now possible to solve almost any non-human CV problem, even ones with sparse data.

For example, in security applications you have basically zero data on the sorts of anomalous events you want to trigger on. If you can simulate example cases of those things that means you can actually start applying supervised machine learning.

<comment><USER12> Many researchers have done interesting work using synthetic data (a lot of them are mentioned in this post). This tool just tries to make the creation of virtual world (which can be used by computer vision) easier.

</comment></comment><comment><USER16> Does anybody know if there's a plugin for unreal, like this, but supports creating geometry inside the game?

If not, is there a way of generating levels with code?

<comment><USER4> I don't think the plugin needs to directly support those features, as they are already in the engine natively.

The engine supports generating meshes dynamically at runtime: [https://wiki.unrealengine.com/Procedural_Mesh_Generation](https://wiki.unrealengine.com/Procedural_Mesh_Generation)

It is also fairly easy to generate random or pseudo random levels using blueprint (Unreal's visual scripting API) or C++

If you have any other questions I'd be happy to help.

<comment><USER16> Thanks for your response, great info.

The reason I ask is that I have a large dataset that I would like to visualize.

The ideal situation would be: I could use python to create a giant flat plane, and then iterate over the dataset to generate the level (which initially would just be quads on the 2d plane).

The dataset would be too large to do in-engine in realtime, and the quads are static - I would like to generate the level as a 1 off process, and then have Unreal do any optimisations that it can by "compiling" the level (e.g. frustum culling, quad tree calculation etc) I might have some of those optimizations wrong since my knowledge of engines is still stuck in the Quake 1 era, but you get the gist :)

Do you think this is possible? If this tech doesn't exist, I would pay someone to develop it for me....

<comment><USER4> anything is possible :)

How many total datapoints are you trying to represent at once, ballpark? My first instinct is that it would probably be most efficient to feed the data into Unreal and build the physical representation there VS importing a large monolithic 3D model, because then you could make use of instancing, distance culling, etc. This would also mean you could animate the objects separately as well, and manipulate scale, color and the like in realtime.

Again, probably depends heavily on the specifics. In _general_ though, modern gaming GPU's can handle just an insane number of polygons so either way, you should be able to represent a ton of datapoints.

As far as getting data into the engine, it supports .csv importing where you can define your own row types etc: [https://docs.unrealengine.com/latest/INT/Gameplay/DataDriven...](https://docs.unrealengine.com/latest/INT/Gameplay/DataDriven/)

I believe there's also a SQL lite support module for the engine as well, if you want to deal with really massive sets of data, though the documentation there is sparse.

My contact info is in my profile, feel free to hit me up if you have any other questions, happy to help.

</comment></comment></comment></comment><comment><USER17> This would be cool if openai would participate in it.

</comment><comment><USER18> Anyone know of something similar for Unity? Unity has a bit of a steep learning curve of you want to do stuff like importing external data and working with outside programs

<comment><USER12> The CVC lab and Virtual KITTI use Unity in their projects. They might have some internal tools. The advantage of UE4 is open-source, so that we can fully understand what the engine is doing.

</comment></comment><comment><USER11> Love it. Anybody use unreal engine with OpenAI gym yet?

</comment><comment><USER19> I, like many others most likely, thought this was gonna be a cool way to present your resume, or someone with a demo of theirs in Unreal Engine... Still, I was not disappointed by the plugin.

</comment><comment><USER20> I thought this was going to be Unreal Curriculum Vitae, and it was going to be a playable resumé. I'm slightly disappointed.</comment>


Freelance guidence - <AUTHOR>

I am looking for helpful advice and/or links, particularly about self employment and dealing with your own taxes. I am going to try to live off of my own skills this year while I go to school. <comment><USER2> A long-time HN user, Steve Jalim, has written a great book on the subject called career.fork(). Have a look: <https://leanpub.com/freelancedeveloperbook>

Self-employment and tax stuff varies by country.

</comment><comment><USER3> <http://doubleyourfreelancingrate.com/></comment>


Ask HN: How to prepare for  Google/Facebook SRE interviews? - <AUTHOR> 	I am a devops engineer at a startup and am good at my job. I do everything from automation to distributed systems, AWS, CI/CD, troubleshooting, etc.
My colleagues and founder are very happy with my performance. However, when I research on the kind of questions asked in google SRE interview, I start having self-doubts. I am not sure how much depth is expected from me in each of the areas like troubleshooting, scripting, networking etc. Ex: for troubleshooting, should I know all the syscalls and the definition and be fully comfortable with strace?
It will be great if I can get some guidance on how and how much to prepare for these areas:
Linux internals
Troubleshooting
Large System design
Scripting
Networking
I am reading the books on Linux kernel development, Operating systems, networking. How much depth should I have to ace the interviews? <comment><USER2> FB has a pre-selected list of questions about netcat, lsof, df, free, lsblk, mount, ifconfig, who, last, ulimit, grep and nohup.

I talked to them twice on two separate occasions, different people and got asked the exact same questions.

This was over one year ago and things may have changed since then.</comment>


Tech co-founder sought for ambitious consumer web startup - <AUTHOR>

Technical co-founder sought for early-stage startup solving the "It's hard to find people I'd be cool doing stuff with to do stuff with" problem.
The founder is a SF-based, hardcore business/product guy, recent grad of penn/wharton with experience leading data analysis initiatives at major consumer web startups.
The idea is called Yo You Want To. Yo You Want To will be a way for (initially) college students to find other people on campus to do stuff with. For those times that you want to get a late dinner, carve a pumpkin, see a spoken word poetry show, go weightlifting, or really whatever, and your friends aren't free, you just post: "Yo, you want to _______" where you fill in the blank with whatever you want to do. When people on campus say they're down, you guys meet up and go do it. It's a simple concept to understand, and the early success of this will rely on brilliant execution and a killer UX more than any never-before-done feats of engineering. Clever use of mobile is also key but has been largely unexplored to date.
The founder has the prototype of Yo You Want To built - it's at www.yoyouwantto.com (get in touch for a demo login).  During a test launch at Penn, hundreds of people signed up within a couple weeks, and many meetups actually happened through the site. It's clear that we've scratched the surface of a huge opportunity here and what we need more than anything is a brilliant engineer who can execute, lead and inspire. Someone who lives for elegance, speed, and innovation in social products. Someone extraordinarily entrepreneurial to be in the founding team of a company, with equity and responsibilities to match. If you're looking for an hourly rate or a salary before funding, you've missed the point.
The prototype was built with Ruby on Rails, and while we're unlikely to switch from that, your brilliance as an engineer and tech entrepreneur trumps everything.
Interested? Please contact firstname.lastname@example.com with some info about yourself - what technologies do you love? built anything cool? what are you doing at 2am on a random wednesday? what do you have going on these days? what about this opportunity interests you?
Thanks for reading. <comment><USER2> There's a popular variation on this idea that has been featured in the New York Times:

<http://www.howaboutwe.com/>

<http://www.nytimes.com/2010/07/04/fashion/04date.html>

</comment><comment><USER3> "If you're looking for an hourly rate or a salary before funding, you've missed the point."

I hope you are offering 50% equity. If not, it's not worth my time. This looks like every other business guy looking for a tech founder: you want to offer almost nothing for doing all of the hard work.

<comment><USER4> I didn't see any mention of what he was offering, so why would you say _"you want to offer almost nothing"_?

<comment><USER5> Err, seems like your splitting hairs: "If you're looking for an hourly rate or a salary before funding, you've missed the point." insinuates exactly that. I for one don't see "very much" being offered here.

Edit: ...do you?

<comment><USER4> Well I see a person looking for a co-founder, so not getting a salary or wage comes with the territory. So I don't see the need to poop on him, he's not being dishonest with his offering. He said hes a wharton graduate, which is an elite business school and gives him a bit of credibility in terms of book smarts. He has a project under way, prototyped and market tested. He needs a technical partner.

Maybe the idea will interest someone, maybe someone here was thinking about the same thing but never got it started. It is what it is. What do you find "wrong" with his posting, maybe it will help clarify your criticism?

<comment><USER3> "Maybe the idea will interest someone, maybe someone here was thinking about the same thing but never got it started. It is what it is. What do you find "wrong" with his posting, maybe it will help clarify your criticism?"

If I'm going to toil away on a project for NO PAY (which is a huge risk), it better damn well be for a huge chunk of the profits.

I have been around many business types and have seen developers put their hard work into something with little or no payoff.

My hunch is that he offers 5% or less, which is a joke.

<comment><USER6> Why put down a fair post with useless comments on a "hunch that he offers 5% or less"?

<comment><USER3> "Why put down a fair post with useless comments on a "hunch that he offers 5% or less"?"

Because he already stated that the co-founder gets no salary. He never mentioned the actual percentage, so it's a fair assumption.

<comment><USER6> Read OP: "But I 100% understand the value of a tech co-founder, and am not looking to be stingy with equity. I know I can't make it succeed on my own."

Not mentioning exact equity, how do you assume 5%!?!?

It is very common for cofounders at an early-stage startup to take no salary.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> "The Social Network" effect has begun! People think its a business model.

<comment><USER7> Holy crap. I think you're right. This could explain why I've had two different strangers corner me in the past month (and seen at least two similar pitches on HN), trying to pitch me to work on their pre-funding social network. They "have the secret sauce all figured out", they "just need need a coder to build it". Unfortunately, they "can't really get into details" until I sign an NDA. Needless to say, I still am pretty fuzzy on their secret sauce:)

It all fits though - up until now, I couldn't figure out why random shmoes that don't know anything about technology were getting excited about social networking.

I suspect Facebook/Twitter/Linked In aren't the end of the road for what we call "social networking", but I'll bet it's created by someone with at least some experience in the technology space...

<comment><USER5> Couldn't agree more, the likealittle flirting "site" is another from the same breed, although I think they might pre-date the movie.

Wouldn't it be fantastic/ironic/funny if the eventual downfall of FB came because the movie lit so many entrepreneurial fires....

</comment></comment></comment><comment><USER8> Feel free to feature your startup on the Startup List. Submit your info via <http://startupli.st> or on Twitter and metion @TheStartupList

</comment><comment><USER9> clickable: <http://www.yoyouwantto.com/>

</comment><comment><USER10> _If you're looking for an hourly rate or a salary before funding, you've missed the point._

Pretty sure you have that backwards.

</comment><comment><AUTHOR> I guess I should have been more clear. From my post, I'm looking for "someone extraordinarily entrepreneurial to be in the founding team of a company, with _equity_ and responsibilities to match."

I am fully prepared to give a way a very large chunk of my company. Will it be 50%? I don't know. I've invested a lot into this, and other people have some equity now too. But I 100% understand the value of a tech co-founder, and am not looking to be stingy with equity. I know I can't make it succeed on my own.

<comment><USER11> If you need someone with you for you to succeed, sounds like they should have the same % as you.

<comment><USER12> Splitting equity 50-50 is almost never a good idea. When hard decisions need to be made, someone needs to have the final say.</comment></comment></comment>


CloudFlare is not working - <AUTHOR> https://www.cloudflare.com/system-status

<comment><USER2> Am I missing something? I just see a whole load of "online".</comment>


How to receive a million packets per second (2015) - <AUTHOR> https://blog.cloudflare.com/how-to-receive-a-million-packets/ <comment><USER2> A thousand nanoseconds per packet? Luxury!

At 10Gbps, and hundred-odd byte packets as are common on finance multicast feeds, I get 73ns per packet. The amount that can be done in that time is limited. Typically, you just dump them in a big-ass ring buffer and let processes on a bunch of other cores pick out interesting ones to spend more time on.

The way to get the packets is usually with some proprietary kernel-bypass library (some have success with DPDK), and a very carefully isolated core that the kernel is kept from interrupting, and just spins moving packets. NIC hardware-level ring buffers are usually pretty small, so pauses make for drops.

Many NICs can be persuaded to filter packets out to multiple ring buffers, and then you can get multiple cores picking them off; that is necessary when you get to 40 and 100 Gbps. Multiple rings even with one core multiplies headroom, making occasional housekeeping without drops possible.

<comment><USER3> Excuse my naïveté, but I'd love to know what kind of feeds you connect to to pull in 40gbps of financial data. Are these basically feeds of all available securities trading everywhere in the world?

Background is that I did some work connecting to a few smaller exchanges, but we never ever thought we'd ever need to go past "just" 20mbps.

<comment><USER4> Don't get too impressed here by finance. Almost everything we're talking about here Linux Kernel Developments, SolarFlare (or Mellonx), core isolation (cpu set) was invented elsewhere. Finance merely buys it; it's a great client of all this tech. The only vaguely created-in-finance thing he/she references but then not even by name is [https://martinfowler.com/articles/lmax.html](https://martinfowler.com/articles/lmax.html) which comes from ring buffers coming from CS EE networking itself.

<comment><USER2> Never heard of LMAX; looks like a Java thing.

Nobody said finance invented anything. But high performance demands produce similar architectures everywhere.

<comment><USER4> LMAX is a java thing. It's high speed, low latency messaging system used a betting facility in the UK somewhere where in certain betting situations, the number of messages drives up explosively high, and need to be well handled. Thus high, high speed and low latency with reliably low latency were key.

</comment></comment></comment></comment></comment><comment><USER5> > Don't expect performance like that for any practical application without a > lot more work.

5 years later I have servers that can do 2 million (multi process) without messing with Numa or nic settings.

<comment><USER6> At least for TX it's _much_ easier to get that performance today due to relatively recent kernel changes. Unsure about RX though.

A few years ago I worked on the upload service within the YouTube Android app. Shortly after YouTube started using QUIC for video streaming I decided it would be interesting to experiment using QUIC for mobile uploads; my thesis being that its BBR implementation might lead to better throughput on lossy wireless connections and therefore faster mobile uploads. I discovered though when tethered to my dev workstation I wasn't able to break ~100mbit while the native TCP stack was able to saturate the USB-C gigabit ethernet adapter I was using. This problem got the attention of the QUIC folks, which lead to a more detailed investigation than I was capable of and eventually to kernel patches described here: [http://vger.kernel.org/lpc_net2018_talks/willemdebruijn- lpc2...](http://vger.kernel.org/lpc_net2018_talks/willemdebruijn- lpc2018-udpgso-paper-DRAFT-1.pdf)

I'd like to think my little project was the major motivator for that work but I'm pretty sure the server efficiency gains described in that paper were a much stronger motivation. :)

<comment><USER7> That paper is a real disappointment with the places it _doesnt_ go.

Specifically, if QUIC is to have high performance and high power efficiency, just batching the packets isn't enough. Think of a usecase - a CDN wants to be able to send a block of data from disk to a client. It should be able to direct that data from the SSD to the network card without the CPU touching the data at all.

On the receive side, a mobile device should be able to download a big file to ram or flash with the main CPU asleep most of the time.

That means the network device should be encrypting the packets, doing the pacing, doing the retries, etc.

It may be that a quic specific set of options or syscalls is best for that, or it might be that the application should provide a "this is how to send my data" set of bytecode which the network hardware executes repeatedly.

But just batching UDP packets is a half way measure which isn't in the direction of the final goal.

<comment><USER6> One step at a time.

Right, that paper doesn't introduce life-shattering innovation but rather just fixes a long-standing issue with UDP performance in the Linux kernel. That ain't nothing.

A large motivator to develop and deploy QUIC is that by implementing much of the stack as a userspace library the protocol can be iterated on at a pace impossible with TCP. BBR congestion control experiments are a great example of that; there's enormous value in quick iteration, even at the cost of compute efficiency.

I agree with your sentiment though; I have to imagine eventually the protocol will settle down and hardware acceleration will become a thing - but imho not to the degree you propose.

</comment></comment></comment></comment><comment><USER8> Curious, does anyone know how things look on Windows?

</comment><comment><USER4> OP has Solarflare NICs. So surely > 1MM pps is possible on that HW? But as you opened with 1 million per second on the Linux stack, I gather that kernel by- pass and such were by definition out of scope. The goal here was to always pass through the kernel. But if so, why use Solarflare? Why not see if one can do + 1MM pps on a nice, even beefy but not Solarflare/Mellonx NICs whose motivation includes kernel by-pass.

</comment><comment><USER9> 1 Gbps, unoptimized, commodity home network between an ancient thinkpad t520 and gen 4 i7 workstation using ZeroMQ throughput perf test gives more than 1M messages per second:

        $ ./local_thr tcp://192.168.1.123:5678 100 1000000
        message size: 100 [B]
        message count: 1000000
        mean throughput: 1143708 [msg/s]
        mean throughput: 914.966 [Mb/s]

<comment><USER10> Assuming ZeroMQ batches messages like other message queue technologies, you're comparing apples to oranges. The article is about receiving 1M _network packets_ per second.</comment></comment>


I want to learn Lisp; any books or site to learn from? - <AUTHOR>

I want to learn LISP any books or site to learn from? <comment><USER2> Depending on what sort of lisp you're wanting to learn, theres the Structure and Interpretation of Computer Programs.

It uses scheme (a varient of lisp) and though a bit . . . challenging, is excellent:

<http://mitpress.mit.edu/sicp/>

</comment><comment><USER3> I wonder if you googled about this even a bit! There are quite good number of pages devoted exclusively to review & discussion of LISP books.

For starters: <http://bc.tech.coop/lisp-books.htm>

</comment><comment><USER4> Practical Common Lisp - <http://www.gigamonkeys.com/book/>

The Little Schemer

The Reasoned Schemer

The Seasoned Schemer

</comment><comment><USER5> ANSI Common LISP by Paul GRAHAM -> <http://www.paulgraham.com/acl.html>

<comment><USER6> Depending on how much programming background he has, pg's On Lisp might be a good option as well, and it's free: <http://www.paulgraham.com/onlisp.html>

</comment></comment><comment><USER7> <http://landoflisp.com/></comment>


Show HN: 10 free to use high-res illustrated wallpaper images every 5 days - <AUTHOR> http://animoot.com <comment><AUTHOR> Hi guys and gals!

As an aspiring entrepreneur and junior web developer, I despised the fact that I could not find any illustrated wallpaper images (think of cartoon-style, origami-style, paint, animated, etc.) that I could use for the background of my websites. I am myself a big fan of Unsplash and I believe a similar community should exist for this type of images.

I hope you like the idea, and feel free to give feedback on it. I am all ears!

Cheers,

Valentijn</comment>


No, don't enable revocation checking - <AUTHOR> https://www.imperialviolet.org/2014/04/19/revchecking.html <comment><USER3> For those, like me, wondering who the author might be, it appears to be this guy: "Adam Langley works on both Google's HTTPS serving infrastructure and Google Chrome's network stack. From the point of view of a browser, Langley has seen many HTTPS sites getting it dreadfully wrong and, from the point of view of a server, he's part of what is probably the largest HTTPS serving system in the world - See more at: [http://www.rsaconference.com/speakers/adam- langley#sthash.HM...](http://www.rsaconference.com/speakers/adam- langley#sthash.HMdxPRI7.dpuf")

<comment><USER4> On the other hand we found that as of Friday, Chrome DID NOT recognize that one of our wildcard certs for Efficito had been revoked. We sent out an email to our customers saying to enable cert revocation checking.

Revovation isn't perfect and I would not suggest the current status quo is OK but the intermediary approach Chrome takes cannot be trusted as they have now shown.

If Chome will not show our cert as revoked what is the point of revoking the cert? The author has points but the approach Google ie taking is a cure worse than the disease...

<comment><USER5> Honestly, I don't see a point in certificate revocations anymore, i.e., your implicit conclusion seems to be correct. And I don't blame Google for our broken revocation system – especially because the even the best revocation system couldn't fix the current certification system that is broken in its core.

<comment><USER4> Google's problem is they decide which revocations are worth passing on to the browser. That's at least as broken by design.....

Believe me I am aware of the limits of soft-fail, but the answer cannot be even in the short-run to let a browser vendor tell us which revocations are worth knowing about.

<comment><USER6> Soft fail doesn't work at all. CRLset works for the certificates that it covers (some 25k of them, btw).

Which approach is worse?

</comment></comment></comment></comment></comment><comment><USER7> So this complete infrastructure is crap. OpenSSL, a software half the internet uses but no one cares about because it's crap. CA's not revoking keys even though they know they're compromised. Revocation being worthless because it's too much of a hassle for anyone to bother.

Great. Maybe now, when half the internet is already compromised and all our certificates are not worth the bytes they're made of ... maybe we should try to come up with something better.

edit: Actually, this whole heartbleed affair has been quite eyeopening for me, so I'm thankful for that. But it certainly didn't help with the paranoia I feel the last couple of years while using services on the internet.

<comment><USER8> Yes! Now it's time for us to generate a whole new broken infrastructure! I'm sure if we just rewrite all the Internet's crypto in Rust, everything will be great 10 years from now. No way will a radically different new transport cryptosystem grant researchers 100 new bugs to play with; after all, we'll have option types.

<comment><USER9> You're right to mock the attitude people have that the _only_ thing wrong with OpenSSL is the language it's written in, but memory unsafety has nevertheless been a factor in many security flaws.

</comment></comment></comment><comment><USER10> Still seeing lots of explanation about why the current system sucks, and not much about how a more robust system might be created and promptly adopted. Langley (the author) mentions short-lived certificates (either rapid expiration or via a 'must staple')... how soon can we enforce that? How short can that make the danger-period where the CA, and Google, and the "connected web" all know that a certificate is invalid, but a user-at-risk does not?

Why not other ways to rapid-broadcast invalidity in censorship-proof ways, so that a browser encircled by an enemy can quickly figure out something's wrong? (Or, why can't security professionals get around interdiction as effectively as copyright pirates do?)

<comment><USER11> _how a more robust system might be created and promptly adopted_

I'm quite fond of how the SSH host key system works.

Prompt me the first time I see a new key, provide me with supporting evidence (e.g. show me how many people have previously accepted this fingerprint for this domain) and alert me the same way in the future if the key ever changes.

If the 'supporting evidence' was plugin-based then this system could quickly become more user-friendly _and_ trustworthy than the current centralised system can ever be.

There could be plugins to automatically trigger a SMS challenge on first contact with particularly sensitive sites. Multiple competing P2P web-of-trust plugins, plugins that let you follow trust-lists from third parties, etc.

In the current system you rely on a single, very questionable opinion on the trustworthiness of a given certificate. In the new system you'd be presented with a trust-score compiled from a whole range of opinions. The sources of which _you_ chose before-hand.

Of course this approach doesn't include a license to print money for corrupt CA organisations and is not going to happen for that reason alone.

<comment><USER4> Supporting evidence, to be worthwhile, must come from a trusted source, right?

<comment><USER11> As said it should be pluggable. Yes you will still need seed fingerprints for a few independent plugins (like the CA list of today), and you still need to trust or vet the browser itself (also like today).

The point is that once this seed trust has been established, which ideally would need to happen only once in your lifetime (given proper sync/backup facilities), you gain actual control over who you want to delegate your trust to, if at all. On a site-per-site basis.

If an american, a chinese _and_ an EU database independently agree on a fingerprint for a site then that would be an actual trust indicator. Very much unlike the perpetually compromised zoo of certificate authorities of today.

And obviously once there's a market for plugins you'd quickly see plugins going far beyond what we get to know today (read: essentially nothing). There could be subscription-based plugins providing detailed information about the remote party, down to credit ratings, company history, you name it.

</comment></comment></comment></comment><comment><USER12> My quick write-up on this from few days ago, [http://www.ahtik.com/blog/startssl-revocation-fees-will- not-...](http://www.ahtik.com/blog/startssl-revocation-fees-will-not-matter- and-ssl-certs-are-funky_u1g8E/)

Yes, revoke is broken by design, especially with mobile and Chrome browser. I'd say it's broken everywhere except Firefox with OCSP Hard Fail enabled.

Thanks to this flaw StartSSL business model has become somewhat outdated IMHO with the free certs and paid revocations.

I'm dreaming that we can fix the revocations issue with 24hour valid certificates. Suggested at the end of my post.

But I must be naive on this as it's too simple, just haven't found the flaw in this myself. Yes, it needs technical orchestration, but at least it does not add extra layer of single point of failure for every session.

EDIT: Just finished the OP post and it does indeed also mention "short-lived certificates" in the end as a potential solution.

<comment><USER13> Indeed, short-lived certificates do seem like a solution to this problem. One downside might be the fact that (anecdotally) many users have inaccurate clocks. I read somewhere recently that a large web site has to back-date their new certificates, because, otherwise, certificate rotation/revocation causes a large spike in support tickets.

Short-lived certificates were explored in Towards Short-Lived Certificates [http://crypto.stanford.edu/~dabo/pubs/abstracts/ssl- shortliv...](http://crypto.stanford.edu/~dabo/pubs/abstracts/ssl- shortlived.html)

</comment></comment><comment><USER14> I think the author is a little disingenuous with the term "security theatre". Basically he argues that OCSP doesn't work because hard fail might cause DOS -- but fails to conclude that without OCSP SSL/TLS _is useless_. It's a long argument for saying that the CA system is broken (you can only trust the white-list chrome provides) -- and the sensible conclusion is that _you cannot trust any other certificate chains_ (without OCSP) is left out.

Without certificates, SSL/TLS falls apart.

Perhaps a better use of CAs would be to always delegate authority to the domain owner -- we'd only need OCSP for the CAs, and domain owners could issue hour/day-valid certs via a cert infrastructure. That would push a lot of complexity down to domain owners, it would probably lead to a lot of errors in implementation -- but those errors would only affect the domains -- not the main CA trust chain as such.

I'm not sure if that would be an improvement or not -- but at least you could know that if a domain was run correctly, a valid certificate could actually be trusted...

<comment><USER15> The "better use of CAs" you describe is essentially DNSSEC.

<comment><USER8> Sure, if "making the DNS totally unreliable", "baking 1990s crypto into the core of the Internet", and "conceding the CA PKI to world governments" is your idea of "better use of CAs".

<comment><USER16>
      > "conceding the CA PKI to world governments" is your idea of "better use of CAs".

How much different is this than the current CA situation? Just recently a subordinate CA of ANSSI (the French Network and Information Security Agency) issued a wildcard cert that could MITM just about anything.[^1] Firefox's list of trusted CAs includes:[^2]

      China Internet Network Information Center (CNNIC)
      Government of France
      Government of Hong Kong (SAR), Hongkong Post
      Government of Japan, Ministry of Internal Affairs and Communications
      Government of Spain, Autoritat de Certificació de la Comunitat Valenciana (ACCV)
      Government of Spain (CAV), Izenpe S.A.
      Government of The Netherlands, PKIoverheid
      Government of Taiwan, Government Root Certification Authority (GRCA)
      Government of Turkey, Kamu Sertifikasyon Merkezi (Kamu SM)
      Hong Kong

Firefox's list of pending CAs includes additional government CAs.[^3] Things are no different in Redmond. There are at least 56 government CAs (56 of the certs start with government probably others with less obvious names) in Microsoft's Root Certificate Program.[^4]

[^1]: [https://blog.mozilla.org/security/2013/12/09/revoking- trust-...](https://blog.mozilla.org/security/2013/12/09/revoking-trust-in-one- anssi-certificate/)

[^2]: [https://www.mozilla.org/en- US/about/governance/policies/secu...](https://www.mozilla.org/en- US/about/governance/policies/security-group/certs/included/)

[^3]: [https://www.mozilla.org/en- US/about/governance/policies/secu...](https://www.mozilla.org/en- US/about/governance/policies/security-group/certs/pending/)

[^4]: [https://social.technet.microsoft.com/wiki/contents/articles/...](https://social.technet.microsoft.com/wiki/contents/articles/14217.windows- and-windows-phone-8-ssl-root-certificate-program-april-2012-e-g.aspx)

<comment><USER8> It's not different from the current CA situation. That's my point.

<comment><USER15> That was my point, too: the theorized system already exists. I'm definitely not advocating its use.

Ultimately, the URL bar needs to go away. More fundamentally, the asymmetric relationship between very large organizations that authenticate their identity with browser CA certs, and individuals who authenticate their identity with passwords needs to change.

Cryptographically generated addressing schemes like Telehash can do the automate-able stuff better than the current CA situation. The problem (and solution) I'm struggling to articulate involves the fact that granular authorization systems and trust databases need better UI before we can really fix this.

I suspect cheaper hardware tokens will play a significant role.

</comment></comment></comment></comment></comment></comment><comment><USER17> Could we use bloom filters on the CRLs, before checking with OCSP? Maybe I should go crunch the numbers on the viability of that.

<comment><USER18> I just did and if you were okay with a 0.001 probability of false positive you could list all 500,000 (possibly way off) certificates potentially exposed through heartbleed in only 877.5KB of space. The current Chrome CRL contains 24,161 serial numbers and takes up 305.3KB of space. While it isn't a perfect fix for the revocation problem it would certainly be much better than the status quo.

One problem might be that the 0.1% of sites hit by the false positive effectively couldn't use OCSP stapling but Chrome could just first call back to Google as a CRL proxy to avoid making an OCSP request when the site stapled a valid but potentially revoked OCSP response. Then just store that response from Google for current version of CRL in the cache. End result is that the unlucky false positive sites don't have tons of unnecessary (unnecessary as far as the OCSP spec is concerned) OCSP requests going to the CAs and the only thing they would notice is that a new visitior takes 100ms longer to make the first page load.

And through the magic of bloom filters if you wanted to bump the false positive rate down to 1 in 10,000 it only bloats the list to 1.14MB. Furthermore, there are methods to make the bloom filter scale-able such that a client doesn't have to necessarily download the whole bloom filter again if a bunch of elements are added to it and instead just download a portion of the data required for a full update.

The more I think about it the more I wonder why this isn't already in Chrome in some form or another. The only downside is weird networks where OCSP might be filtered, but not https, and access to Google is filtered.

Edit: One thing I feel stupid for overlooking is that Bloom filters aren't cryptographically secure so an attacker could theoretically find a serial number for some CA that would cause a site to always be a false positive but I don't think any CAs are still giving out serial numbers in a predictable way after the MD5 debacle and even if they were it would seem to be impractical to me. The fix would just be do a SHA256 hash of the serial instead of the serial itself.

<comment><USER17> Oh, I wasn't thinking about having this be a perfect oracle; just a better and smaller first pass _(edit:)_ for CRL, not for OCSP.

I got the idea from Squid and the network of caches.[1] That body of experience may be helpful.

For shrinking the size, RLE might work (most entries would be 0), and rsync may reduce bandwidth. It looks like the Squid network just used http requests for refreshes. There's probably a sweet spot for bandwidth, and I'd guess that 90-99% would work fine; you're balancing the size of the continually updated bloom filter vs. the requests for certificates that match it. I didn't worry about false positives, because it could just send an OCSP query in that case.

Your numbers for revocations sounded _very_ low, but I just used crlset- tools[2] and checked, and it's about right. Which is weird, because someone else[3] mentioned a size of "4.107Kb" at version 1567, but that's somehow different - compression, perhaps. I thought I'd heard about CRLs megabytes long, but Google Chrome seems heavily curated re: CRLs.

I'd hash over _signatures_ instead of the oft-predictable serial numbers, as you noted.

[1] [http://wiki.squid-cache.org/SquidFaq/CacheDigests](http://wiki.squid- cache.org/SquidFaq/CacheDigests)

[2] [https://github.com/agl/crlset-tools](https://github.com/agl/crlset-tools)

[3] [https://scotthelme.co.uk/certificate-revocation-google- chrom...](https://scotthelme.co.uk/certificate-revocation-google-chrome/)

</comment></comment></comment><comment><USER19> Just make sure you still check revocation of code signing certificates. Otherwise you will end up running malware that is signed with a legit key they got off my stolen Windows laptop.

</comment><comment><USER20> This argument only holds if the attacker controls every internet connection you use. If you're on a portable device or you're otherwise connecting through various networks, only a subset of which are compromised, revocations are still useful.

<comment><USER21> Exactly. If I'm on my trusted network at home and receive a big revocation list, and a few weeks later go to, say, Egypt, and someone tries to MITM me there with a stolen certificate, then it would show up as invalid.

</comment></comment><comment><USER22> When I hear these arguments, I always look for what is wrong with OCSP Must Staple. The author says that at the bottom it might be a solution with short lived certs, but I dont see the need for super short lived certs, only short lived OCSP staples. The author presents this as the problem:

> if the attacker still has control of the site, they can hop from CA to CA > getting certificates. (And they will have the full OCSP validity period to > use after each revocation.)

The solution here is to not allow OCSP stapling to request a new certificate and use a full OCSP check to verify that the cert wasnt revoked.

</comment><comment><USER23> I'm honestly kind of surprised how little action there has been to assist with a migration away from the CA model. The technology is there, but people just don't seem interested enough to leverage it.

Systems like Namecoin could serve this purpose marvelously. Powerful devices have direct access to the entire cryptographically authenticated DNS and certificate database. Weak devices can specify whom they trust to provide them with DNS/certificate data, and even those devices get some cryptographic security guarantees thanks to technologies like SPV.

<comment><USER24> Why have a single entity at all? Moxie Marlinspike proposed Convergence ([https://www.youtube.com/watch?v=Z7Wl2FW2TcA](https://www.youtube.com/watch?v=Z7Wl2FW2TcA)) as a solution - I think that something like that has far more potential wheels to travel than a Namecoin based system.

I should be able to choose who I trust, a notary system would allow me to do just that. No central CA systems.

The biggest concern I can see is Identity management, but, as mentioned by Moxie, most of these CA don't do anything close to proper Identity management any more - I have a number of certificates bought from quite a few different CA's all made out to my rabbit, at no fixed address.

Notaries can, of course, do additional verification - they could even advertise this as a premium.

I don't see why this can't be extended to DNS lookup's either. I trust X notaries and pin the results I get, I can choose to trust a majority, or be hyper paranoid and require everyone to agree. No need to run a power hungry blockchain, no single point of technology failure.

Technically, all of that is feasible today. And I imagine we will see a number of different technologies combined to form a proper, decentralised, system.

<comment><USER25> The project seems to have lost support, the last github commit was over 2 years ago.

Do you know if there was a specific reason or were people just not interested/none of the browsers jumped onboard?

<comment><USER26> From Moxie:

 _" Convergence is blocking on TACK, which is blocking on browser vendors."_

[https://twitter.com/moxie/status/451020203099299840](https://twitter.com/moxie/status/451020203099299840)

<comment><USER13> There should be a clear statement about the status of Convergence on the web site. IIRC, the Firefox extension has been broken for more than a year now. Why? If Mozilla broke their APIs and made it impossible for the extension to work, then we should know about that. Otherwise, what's the excuse for the extension being broken for so long?

Convergence had the momentum, and there was a small but vocal group of people willing to support it. But, due to project mismanagement and lack of communication, that momentum has been lost.

<comment><USER27> There are some more active forks like [https://github.com/mk- fg/convergence/](https://github.com/mk-fg/convergence/) but they too seam to not really work in current versions of Firefox.

</comment></comment></comment></comment></comment></comment><comment><USER28> I've wondered many times why OCSP isn't distributed as DNS is. When we talk about websites, surely there's no more than one certificate per hostname (or less, i.e. wildcards). I don't think we're talking here of something impossible to do or not feasible with our current technology and computing power.

Also, certificate "whitelisting" could be a part of the DNS protocol itself (return the IP address of the requested hostname and the hash of its current, valid certificate).

<comment><USER28> Just to clarify: OCSP is distributed, but I can't ask my local ISP OCSP server about your certificates. I have to ask your OCSP server about your certificates.

</comment></comment><comment><USER29> It seems the only problem with hard-fail is the risk of DoS attacks by targeting OCSP servers. However, if you include OCSP stapling you won't be affected. So a solution may be to encourage all users to enable revocation checking with hard-fail, and all servers to support OCSP stapling.

</comment><comment><USER30> It sounds like the internet is broken Without CRL/OSCP we cannot truly trust that we are securely communicating.

Something has to give. We need to abolish SSL/TLS and migrate to something that isn't broken by design

<comment><USER31> > _the internet is broken_

It's not the Internet, just the CA system. There are better systems for handling trust out there, for example, people have been signing each other's PGP keys at key signing parties for decades.

<comment><USER4> > It's not the Internet, just the CA system.

Ok, so it is just the portion of the internet that involves purchasing things with credit cards and requiring passwords to access sites. The rest of the internet is just fine.

Great. I thought for a moment that the commercial basis of the internet might be in danger. Now to determine what percentage of the internet is not dependent on the CA system.....

</comment></comment></comment><comment><USER32> I am thinking that a HSTS option enabling hard-fail OCSP plus OCSP stapling is probably a good idea, though probably less secure than putting it in the certificate.

</comment><comment><USER33> Let me get this straight. Sites across the internet are (hopefully) revoking their CAs and issuing new ones to address Heartbleed but Mr. Langley is suggesting that we shouldn't check for revoked CAs because it might not do anything and it's slow?

Sorry, but after the last few weeks I'll happily accept a little slowness for the security revocation checking provides in the cases where it does work, even if it's not 100% of the cases.

<comment><USER34> I guess you didn't read the article? He's saying there _are_ no cases where it works. Making it completely pointless.

<comment><USER33> I did read the article.

> "In order to end on a positive note, I'll mention a case where online > revocation checking does work..."

<comment><USER34> That's even more proof you didn't read it. How does enabling it in chrome make any difference to code signing?

<comment><USER33> Does enabling revocation checking make me less safe?

<comment><USER8> Yes! It involves you reporting all the sites you visit to a CA!

<comment><USER33> I guess I knew that but hadn't grasped the security problem this presents. You've changed my mind. Thank you.

</comment></comment></comment></comment></comment></comment></comment><comment><USER35> Can't the replay attack mentioned in the article be mitigated by using nonces? Why doesn't anyone do this? I'm genuinely confused by this.

</comment><comment><USER36> The article gives two reasons for why 'soft-fail' is required: Captive- portals, and OCSP server failure.

To deal with captive portals: have an SSL signed 'subdomain.google.com/you_are_on_the_internet' site/page that Google Chrome can use to check to see if it's captive or not. If it's captive, enable soft- fail. If internet access is available, set to hard-fail.

Websites these days are complex, with many (digital) moving parts - <USER2> database server(s), the static image server(s), dynamic response server(s), gateway server, probably a memcache server or something similar. If any one of those goes down, the site is unusable. Why then, should the OCSP server going down be considered any differently? Is a black-hat rented bot-net running a DDoS going to care if it's the main gateway server or the OCSP server?

But let's say we do consider disabled OCSP servers to be a client-side issue. Google could query and cache the OCSP server status, either with OCSP stapling or via some side-channel they build into Google Chrome.

The combination of both would allow hard-fail to be an option in Google Chrome.

</comment><comment><USER37> Theres one thing that I do not understand - why not download the full revocation list?

<comment><USER38> Potentially too much data to download, imagine, all revoked certificates in the entire internet.

</comment></comment><comment><USER39> Why not hard-fail by default and give the user the option to ignore/override it? Similar to the way other certificate warnings are shown to the end-user.

<comment><USER40> answered in adam's blog; see the paragraph beginning with

       Everyone does soft-fail

<comment><USER39> I guess that's true when a hard-fail causes the connection to be refused immediately by the client with no user input. In that case a DoS on the OCSP servers breaks things badly.

However what I meant to suggest is a third option. Something like hard-fail with a latch. The client should opt to fail but give the user the choice to proceed.

This would seem more desirable than the current soft-fail implementations when seem to be entirely silent to the end user.

<comment><USER41> Users make terrible security decisions. ~95% of users click through certificate failure pages, ~99% of users don't notice if a website transparently downgrades to HTTP. Delegating the choice, which would be borderline impossible to explain to the user is another way of saying 'Always say yes to proceed'.

</comment></comment></comment></comment><comment><USER42> Why are we not using OCSP Must Staple right now?

</comment><comment><USER43> The author appears to entirely ignore attack vectors where the malicious party can record but not modify/block traffic.

Edit: I get it, I missed that for sites where the key has been changed the stolen key no longer allows such eavesdropping. Thank you to yuhong for helping point this out rather than just laughing at my ignorance while pushing me down the page.

<comment><USER32> Then a SSL MITM attack would not be possible at all.

</comment></comment><comment><USER15> I'm having a lot of trouble getting past: "Certificates bind a public key and an identity (commonly a DNS name) together."

X.509 certificates bind a public key and a human recognizable string (a "common name") together to _create_ a verifiable digital identity. Over- simplified, X.509 is about solving the "I'm Spartacus" problem.

CRLs solve the "He was Spartacus" problem. I agree with the broad conclusion that CRLs aren't effective for _human_ trust, but they are perfectly reasonable for _machine_ trust.

Why didn't the author mention Kerberos? The default lifetime of a Kerberos ticket is designed around humans: roughly the length of a work shift in front of a computer terminal.

final edit: meta-moderation is hard

<comment><USER44> For HTTPS to a web site, the common name is the site domain name. We're not talking about anything else here.

<comment><USER15> Aha. That perhaps explains all the downvoting. But this was my objection:

Name != Identity

It's nearly impossible to have a meaningful English conversation about these problems without getting that straight.</comment></comment></comment>


Bionic Beaver 18.04 LTS to use Xorg by default - <AUTHOR> https://insights.ubuntu.com/2018/01/26/bionic-beaver-18-04-lts-to-use-xorg-by-default/ <comment><USER2> I just recently had a problem with Wayland where qemu with VGA would not start at all, complaining about SDL not being able to find a video device. I eventually tried just logging out and then into Xorg and then it started working again. EDIT: had to run qemu with root which may have been a factor

</comment><comment><USER3> Only 15 points after 22 hours... For Linux desktop users (well, the technical ones) this story should be interesting news. Looks like nobody bothers to get excited anymore after so many years of hopes and disappointments.

</comment><comment><USER4> I don't blame them. I had to switch to Xorg as well after a while. Wayland wouldn't allow launching gui applications which required sudo. Had to manually run xhost beforehand. Drag and drop was broken and a few other issues.

<comment><USER5> I've run into similar issues and had to switch back to Xorg. Aside from the major issues, the small things really killed my enthusiasm for Wayland. The one that really forced me back to Xorg is an extension for Gnome called "No Title Bar" no longer working. It's such a small thing that I don't expect anyone else to be excited about it, but for me, Gnome just doesn't feel right anymore without it.

I wonder what percentage of 17.10 users have similar nitpicks and switched back. I like the promised enhanced security and performance of Wayland, but it's gonna hurt when they do force the switch.</comment></comment>


Google-An update to our search algorithms - Copyright Related - <AUTHOR> http://insidesearch.blogspot.com/2012/08/an-update-to-our-search-algorithms.html <comment><USER2> This is wrong. Google should aim to provide the best possible result for the search. If that is an copyrighted videostream on a non-authorized site (e.g. a stream of all of the episodes of the series i'm googling for), this should be exactly the first result. And not a lame but authorized preview of 30 seconds.

Already found myself using duckduckgo in such cases, imagining the results were better.

<comment><USER3> Does this mean Google will never surface YouTube in its search results? That is probably the largest and most known source of copyright infringing material

<comment><USER4> Heh yeah, that is amusingly unaddressed

<comment><USER5> YouTube follows the DMCA (and does ContentID or whatever). Google search probably doesn't get very many requests to remove links to YouTube.

This policy shouldn't have much impact at all on any large site that discourages copyright violations and responds promptly to DMCA takedowns, because it isn't worth bothering Google Search about that content.

<comment><USER6> I agree this is probably what's happening but I'm worried this might change in the future. Right now I think all that happens is that the link is removed from the search results but if the site itself is also penalized, I'm afraid content owners can now see the value of sending both a DMCA and copyright takedown notice simultaneously.

</comment></comment></comment></comment></comment><comment><USER7> There are a lot of interesting signals for web spam and poor web content. I don't know if this one is very good. In particular I see this scenario:

1) Google algorithmically genreates take down notices on youtube, they move this over to search results.

2) Algorithms 'err on the side of caution' with regard to fair use.

3) Every page that quotes or excerpts a copyrighted work gets flagged.

4) So Google 'manually' makes exceptions for sites they strongly believe won't engage in poor behavior (NY Times, Techcrunch, whatever)

5) No new web site that reviews products or provides critical analysis ever makes it to Google's front page.

That might be a stretch, the auto-flag stuff on YouTube is out of control, and then trying to use that as a signal in results is just inviting abuse. Black hat SEOs are malicious enough without dealing with 'Copyright joejobs'

[ Full disclosure: I work at a search engine blekko.com ]

<comment><USER8> This has nothing to do with search quality. This is just Google bowing down to big copyright. They have done it before with the autocomplete feature, censoring many suggested terms.

They have been asking for this for a long time now. "De-ranking" is number one on this proposal: [http://www.scribd.com/doc/79607883/Proposals-to-Search- Engin...](http://www.scribd.com/doc/79607883/Proposals-to-Search-Engines)

It seems Google has cushioned this in sweet words, as to not make it blatantly obvious.

These sites were very high on the rankings because people click them (and rarely return to search further). Google simply aligned its search results with peoples needs.

Google is enabling media companies in their delusions, when they could be destroying them. They might end up as a casualty.

<comment><USER7> I think you and I have different working definitions of 'search quality'. My definition is that the sites you find by searching for something give you the information you want.

I think 'bowing to big copyright' is no different than 'bowing to Chinese censors' or 'bowing to Administration staffers' or 'bowing to conservative thinkers'. All of them affect what results you get vs what you asked for. That those web pages have information that some other folks don't want you to see, is a discussion between page provider and complainer, not a fight the search engine should get into.

Using blekko.com as an example. You could create a slashtag with all the best torrent sites you can find on the web, and then you and anyone else who uses your slashtag could be searching all those sites for what is useful information to them. As a copyright holder they could search those sites too (and presumably they do) and attempt to take action against the sites if they chose to. The search engine is a window between two spaces, and while sure as a choke point it provides an easy target for those would would want to control the greater masses, it doesn't improve the 'quality' of the service when that occurs. It is of demonstrably lower quality so some folks.

<comment><USER8> I think we have the same definitions. I was just being unclear. My intent was to express that their change was not motivated by improving search quality.

</comment></comment></comment></comment><comment><USER9> I wonder what the secondary incentive effects of this will be. I see three possibilities:

1\. Encourage copyright owners to send more takedowns to Google Search, since those takedowns will be "more powerful".

2\. Encourage targeted sites to send more counter-notifications since a Google Search takedown has an effect on an entire site's ranking, not just a particular link.

3\. Encourage shared sites (hosting providers, etc.) to shut down particular users or areas that get a lot of Google Search takedowns, to stop the spillover effect on the rest of the site.

I don't know which of these effects will matter the most, but I suspect at least one of them will be behind a significant unexpected consequence of this change.

<comment><USER10> Number 1 and 2 are most likely to happen...

</comment></comment><comment><USER11> Speaking of deteriorating Google search results, my latest peeve is when a search shows dozens of results from the same domain. There used to be the much more sensible "show more results from this domain" link. I have a hard time imagining why Google thinks the new way is better, my guess is some designers wanted to remove the link because they thought it was "cleaner".

More signs that designers, lawyers and MBAs are moving to the fore forcing engineers out of the way at Google Search.

<comment><USER12> Yes, this is a sad regression. In case anyone from inside Google is reading this and looking for a good example, try this search: [https://www.google.com/search?q=scansnap+%22s1300i%22+%22s15...](https://www.google.com/search?q=scansnap+%22s1300i%22+%22s1500%22+difference)

The first 70 results are all from the same site!

It's a decent site, but most of the pages are barely relevant. I was searching multiple viewpoints that might explain why I would prefer one model of a scanner over another, and getting page after page from the same single source is much worse than the previous behaviour.

<comment><USER13> weird, I see 5 different domains (in incognito mode in chrome and private browsing in Firefox). The first three are all from one site, but after that there's fujitsu, amazon, lawyerist, and some kind of "snapsnapcommunity". There are more than a few decent looking reviews on the second page, too, which I'm not used to seeing in review searches...

There are quite a few from that one site, though.

<comment><USER12> Thanks for the detailed results. I get many more than 3, so your info inspired me to try figure out what's happening. I'm now more convinced that it's a bug rather than desired behaviour.

I first tried signing out, in case personalization was breaking things: no change. Then I tried "verbatim": no change. Then I tried changing the number of results from 100 to 10, and I see what you do: 3 from Documentsnap, 5 from various Fujitsu sites, then Amazon and Snapscancommunity.

Switch the number of results back to 100, and I'm back to getting more overwhelmed by Documentsnap. I counted again, and this time got 65 Documentsnap followed by 10 Fujitsu, then finally some variety.

</comment></comment></comment></comment><comment><USER14> I wonder if this has anything to do with Google wanting to server more and more paid content through Google Play and Youtube. Will Google be on our side or their side when the next SOPA bill appears?

<comment><USER15> I have no idea what "our side" is supposed to stand for in this context. Is "our side" the side that wants unlimited copyright infringement? I support that side as little as I support SOPA-like bills.

<comment> muuh-gnu "our side" is the pro-freedom, pro-copying, information-wants-to-be-free, consumer side. "their side" is the pro-restriction, pro-enforcement, pro-1984 content producer side.

> I support that side as little as I support SOPA-like bills.

Theire is no "undecided" side. Either you support copyright enforcement against people who want to freely share information without a north-korea- style censorship body over their shoulders, or you dont. There is no middle ground. You cant support "a little bit" of censorship.

<comment><USER15> Haha. Well in that case no I _strongly_ do not support what you so generously call "our side". But thanks for throwing my hat into your ring.

I support environments that enable and encourage the production of new creative works. I believe that copyright in some form is necessary to support such a system.

"A distinguishing characteristic of intellectual property is its "public good" aspect. While the cost of creating a work subject to copyright protection—for example, a book, movie, song, ballet, lithograph, map, business directory, or computer software program—is often high, the cost of reproducing the work, whether by the creator or by those to whom he has made it available, is often low. And once copies are available to others, it is often inexpensive for these users to make additional copies. If the copies made by the creator of the work are priced at or close to marginal cost, others may be discouraged from making copies, but the creator's total revenues may not be sufficient to cover the cost of creating the work. Copyright protection—the right of the copyright's owner to prevent others from making copies—trades off the costs of limiting access to a work against the benefits of providing incentives to create the work in the first place. Striking the correct balance between access and incentives is the central problem in copyright law. For copyright law to promote economic efficiency, its principal legal doctrines must, at least approximately, maximize the benefits from creating additional works minus both the losses from limiting access and the costs of administering copyright protection." - Posner, 1989, An Economic Analysis of Copyright Law

<comment><USER16> While that is certainly a nice and well regarded model, it does not say much about what actually happens. It really only amounts to a proposition, and cannot be read as indicating a necessity. As Landes and Posner themselves later say:

"Economic analysis has come up short of providing either theoretical or empirical grounds for assessing the overall effect of intellectual property law on economic welfare."

\-- 'The economic structure of intellectual property law'; Landes, Posner; 2003. Conclusion, p422, s3.

That is, to spell it out, we do not know if intellectual monopoly law is doing any good at all. This is easy to see for oneself: there must be, and obviously are, costs -- legal, enforcement, search engine 'fixing' -- yet we have no idea how much the gain is. If there is a negative, but the positive is unknown, it is quite possible the sum is itself negative.

</comment></comment></comment></comment></comment><comment><USER17> How does this work for countries with different copyright laws?

</comment><comment><USER18> So if you want to decrease a competitor's search engine placement, would you just have to file C&D notices on them?

I don't see how the amount of C&D notices correlates with value to users. Google does not appear to be benefiting its users in doing this.

<comment><USER19> You'd probably have to file thousands of takedowns and hope the site doesn't challenge them.

<comment><USER20> Also, Google outright rejects some dmca requests and Google also releases data about the actions of copyright owners and DMCA activity on <http://www.google.com/transparencyreport/removals/copyright/>. So even if not all abuses were caught they would at least be published for public scrutiny.

</comment></comment></comment><comment><USER21> My first reaction on hearing this is the by _demoting_ unauthorized content they will end up indirectly _promoting_ authorized content.

And I don't know how I feel about this. From what we've seen with Youtube where unwarranted claims have taken down videos very much under fair use, I don't want Google automatically demoting search results because of a claim made by another party.

I foresee this causing no small amount of contention.

</comment><comment><USER22> Google has not shown that it is able to competently handle copyright complaints. YouTube copyright take down notices happen all the time for legitimate content, chances are a similar ratio of false positives for site content.

</comment><comment><USER23> Sounds like the path to breaking Google's "monopoly" has just been revealed...

Seriously, unless it's just the faintest hint of influence I can see this blowing up in their face. Takedown notices are too ubiquitous and too often bogus.

</comment><comment><USER24> <http://dmcarank.staticloud.com>

DMCARank is a Google Custom search that only displays results from the top 50 domains with a high number of DMCA takedown requests ([http://www.google.com/transparencyreport/removals/copyright/...](http://www.google.com/transparencyreport/removals/copyright/domains/?r=all- time)).

</comment><comment><USER25> In addition to this being a dubious signal for measuring result quality, this could have a chilling effect for sites concerned about SEO. We've seen the bizarre rain-dance that businesses are already willing to do to try to improve their position in Google results, and now we can add "avoid fair use" to that dance.

</comment><comment><USER26> So I guess YouTube won't show up in search results anymore. #doublestandard

<comment><USER27> right?? But obviously, this wont happen, and I don't think that's fair.

</comment></comment><comment><USER28> Won't this impact YouTube dramatically?

<comment> B-Con FTA:

> Starting next week, we will begin taking into account a new signal in our > rankings: the number of valid copyright removal notices we receive for any > given site.

Following the link, you get to this page of stats ( [http://www.google.com/transparencyreport/removals/copyright/...](http://www.google.com/transparencyreport/removals/copyright/domains/?r=last- month) ) on take-down requests. If that page is what they make their basis on, then YouTube doesn't crack the top pages, so I would guess that it won't be hurt too badly.

Although, I have no idea if their count for YouTube is accurate or not. It's not like they wouldn't have an interest in under-reporting the numbers.

<comment><USER20> read the FAQ for the site (or the "Whats included" box on overview page).

"The data below consists of the copyright removal requests we've received through our web form for Google Search. It is a partial historical record that includes more than 95% of the copyright removal requests that we have received for Google Search since July 2011."

In other words youtube data is not included. This is only for websearch

<comment><USER13> That's the point. If they are going to rank links to youtube the same way as they link any other sites (as gp is using as the basis of the question), they would only use search takedown requests as the signal, not requests that youtube receives directly.

</comment></comment></comment></comment><comment><USER29> Copyright wars begin in earnest...

<comment><USER30> And a power vacuum in search created.

<comment><USER31> Exactly what I was thinking. Not sure how Bing and DuckDuckGo stack up but it might create an opening for a new player also - granted search is probably still the toughest game to get into.

</comment></comment></comment><comment><USER32> EFF's take: [https://www.eff.org/deeplinks/2012/08/googles-opaque-new- pol...](https://www.eff.org/deeplinks/2012/08/googles-opaque-new-policy-lets- rightsholders-dictate-search-results)

</comment><comment><USER33> This can be disastrous for user generated content websites.

</comment><comment><USER10> Oh man, they should've never started with copyright removals in the first place (although I get that it's near impossible to do). This new filter has the same abuse potential as "negative SEO", where competitors posted so many spam links to good sites that they were dropped from the first page (with the competitor's site ranking higher as a result).

</comment><comment><USER34> What happens if you have two pages, each listing the takedown notices applied to the other? Seed the pages periodically and wouldn't you get a pretty good measure of the popularity of your taken-down links by ranking them based on how often they are taken down?

</comment><comment><USER24> This will mean ore traffic for <http://www.filecrop.com> and <http://www.filestube.com>.

</comment><comment><USER35> Sounds like this might make it harder for companies to actually find websites posting 'stolen' content. Just saying.

</comment><comment><USER36> With Google increasing it's content footprint over the years, it's interesting to see how the search group treats other Google groups. It's entirely possible to spend a day on the internet without leaving Google web properties (except hn, of course)</comment>


What Geeks Don't Get: The Limits of Moneyball - <AUTHOR> http://thepaintedarea.blogspot.com/2010/03/slaon-conference-report-what-geeks-dont.html

<comment><USER2> I don't think the title of this submission is accurate. More accurate would be "summary of sports business conference, ft. Michael Lewis and Mark Cuban".

There was little analysis to suggest these limits, really.

Further, Moneyball was NOT (as is commonly believed) about walks in baseball. It was about finding and exploiting market inefficiencies. And you need quantifiable, reliable, data in order to do that.</comment>