The citizens of Detroit are building their own Internet - <AUTHOR> http://www.popularmechanics.com/technology/infrastructure/a13819941/the-citizens-of-detriot-are-building-their-own-internet/ <comment><USER2> in 1996, we created the Seattle Peoples' Internet by co-locating our equipment at an upstart internet cafe in Belltown called Speakeasy.

The deal was that we paid for their most expensive monthly line item: the T1 circuit. We covered 100% of the T1 yet were allocated only 50% of the bandwidth, if I recall. We recruited enough co-op members to cover that fee before beginning and had a wait-list on Day One.

Our PortMaster was configured to monitor monthly bandwidth consumption per co- op member. We accommodated spikes in traffic of individual members, but if the aggregate went above some threshold, the co-op would pay more to Speakeasy. The intent here was that our overage fees would help pay for a second T1.

We also had some co-location hosts in their rack, which was around the corner from the coffee machine in those days.

Member fee schedule was proportional to type/speed of connection used and anticipated usage. Beyond that, membership was at cost.

Internally, co-op members would resolve financial details among ourselves, and we sent one payment check per month.

The agreement was fairly balanced. It allowed our members physical access to reboot a piece of co-located equipment, but any maintenance required pulling and removing the machine. (Again, rack was in tight quarters of a coffee shop.)

Since this was mid-1990's, all members supplied their own pair of modems for each side which were mostly 28.8 back then. I think one person was going with ISDN, and maybe another had Telebit Trailblazer 56k.

I'd like to think that upon seeing how this was a benefit to the cafe's financial statements that it gave Mike, Gretchen, Chris, et al the idea to move into the larger ISP space for which they then became known.

<comment><USER3> Awesome. Did you guys host gave servers? Or perhaps co-op members? For some reason I remember a lot of game servers with "Speakeasy" in the name – I think this would have been Q2/Q3A/CS1.6 days

<comment><USER2> SPI did not directly have game servers, but yes, at least one of our members ran one. I don't recall if it was a MUD or something else.

Speakeasy also had other co-location hosts in the rack. If I recall, one was for a design firm somehow connected to one of the owners. Maybe there was a game server there, too.

For a bit more historical context on the Seattle Peoples' Internet, I slapped this together: [https://play.org/articles/seattle-peoples- internet](https://play.org/articles/seattle-peoples-internet)

</comment></comment></comment><comment><USER4> I'm glad that more people are now thinking of mesh, or radio-delivered, internet.

I think there is a lot we can do (as programmers) to start supporting mesh- network style Internets. We're at a critical point right now where distributed systems research has turned up some decent consensus algorithms, where there's now a (finally understood) need for a computer network run by the people who use it, and where consumer radio/copper technology is fast enough to start building out reasonably fast networks.

<comment><USER5> "We're at a critical point right now where distributed systems research has turned up some decent consensus algorithms,"

Not by a long shot. We have proof of work and systems that rely on all parties being known and identifiable, and not much else. Proof of work does not scale and the need to identify all parties requires at least some centralization. Beyond that, we also live in an age where the bulk of Internet traffic is video, and thanks to the decades-long effort of the RIAA and MPAA, most video distribution is centralized, so people demand high throughput; and the widespread use of TCP also requires low latency. Mesh networks are still not up to the task despite all the progress.

You want to support mesh networks? Great! I'm with you, but let's not oversell the state of the art. Mesh networks are still experimental and without some centralized coordination (which is a de facto ISP) the performance and reliability really start to become questionable.

<comment><USER4> High throughput is 'easy' to obtain if you don't mind high latency. The only thing the internet is great at is reducing latency. Luckily for human scale actions this is not a big problem.

Mesh networking technology is old at this point. Application layer software that takes advantage of it's strong suits has yet to be developed. Though there are many ideas of where to start.

As for what you have brought up. In my opinion many-replica content networks, when combined with modern video compression technologies, could solve the issue of video streaming from a no-master and high-latency system.

</comment></comment></comment><comment><USER6> I helped set up a wireless mesh network in St Louis (though others were much more instrumental than I was) - combined with building and giving out free linux computers, we got hundreds of people online that wouldn't have been able to otherwise (pre smartphones)

[http://gowasabi.net/](http://gowasabi.net/)

<comment><USER7> Awesome. Whether or not net neutrality goes under (more than it already has), I hope we start seeing a lot more of this.

<comment><USER8> I'd argue that this sort of project would solve the same problem that net neutrality tries to.

The root problem is that if an ISP starts throttling content, a lot of customers won't have another ISP they can switch to. Something like a mesh network (high-latency, high-bandwidth) would be perfect for what most people want (Netflix, Facebook, etc.). Mesh networks being widely available would force ISPs to provide actual value to their customers.

</comment></comment></comment><comment><USER9> There's an apartment tower near my house, and I've toyed with the idea of renting space on their roof to set up a small local cooperative ISP. The demise of net neutrality has caused this idea to bubble back up to the surface. Of course I need to compute the business case -- how many neighbors do I need to sign up, in order to make it attractive.

<comment><USER10> I've been doing stuff like this for a long time. I've thought about trying to build a web app to help with estimating the costs of doing exactly this. Would that be useful to you? I wouldn't charge for it, but it would help me get motivated to do it if I knew someone might use it!

<comment><USER9> That would certainly be interesting. At present, I don't know squat about any of this, except to imagine that I'd have to pay for some sort of super high speed cable to the building, run it up to the roof, build some sort of radio station up there.

Maintenance would be a chore, as mentioned in one of the other posts.

But an app that lets people begin to think about how this stuff works -- for instance what the pieces consist of, might hit pay dirt if the whole net neutrality thing gets nasty.

I've also thought about the idea of what people could do if they set up some kind of mesh network in a higher density urban area -- for instance if they could use it to bypass the ISP's while maybe also improving their security and privacy.

<comment><USER10> Well y'all have me convinced! I'll get started and be sure to post here when I have something presentable. In the meantime anyone interested drop me a line at my email in my profile and I'll keep you posted. Would love to have some outside perspective on what's important to include.

<comment><USER11> eager to see how it's doing, best of luck to all

</comment></comment></comment></comment></comment><comment><USER12> Comcast or Verizon, I forget which, wanted $10k to add service to the building I was working in a year or two ago.

We ended up going with NetBlazr ([http://www.netblazr.com](http://www.netblazr.com)). Worked pretty well.

I remain hopefully optimistic that other potential disruptors, like Starry or municipal broadband, will unseat the incumbents.

</comment><comment><USER13> If it gets in the way of the local cable company/ISP, I imagine Michigan will vote this is illegal. Money > citizens.

<comment><USER14> Yes, MI is where GOP state appointed emergency managers override city and county governments (mostly to the detriment of said localities)

<comment><USER15> This is common in many states when local governments go bankrupt.

</comment></comment></comment><comment><USER16> In France, there are some associative ISP working with volunteers in almost every region, although they are quite small.

<comment><USER17> Could you provide a name/assos for north paris area ? I wish to provide a couple of hours of my time

<comment><USER16> Not sure if there are in your area, you can check on the FFDN site, which is like a meta association : [https://www.ffdn.org/en/members-fdn- federation](https://www.ffdn.org/en/members-fdn-federation)

The most famous is FDN ([http://www.fdn.fr/](http://www.fdn.fr/)) which is covering the whole territory, and in the Paris area you have Franciliens.net ([http://www.franciliens.net/](http://www.franciliens.net/)).

</comment></comment></comment><comment><USER18> This has a lot of of similarities with the german Freifunk[0] initiative. Although the philosophy is different. (They work as WISP and let people sign up for the service. Freifunk distributes APs that anyone can plug into their internet connection and provide Freifunk connectivity in their street. Anyone can connect to it).

[0] - [https://en.wikipedia.org/wiki/Freifunk](https://en.wikipedia.org/wiki/Freifunk)

</comment><comment><USER19> The original article that this one refers to, and which doesn't ask you to accept malware[0].

[0] [https://video.vice.com/en_us/video/motherboard-dear- future-p...](https://video.vice.com/en_us/video/motherboard-dear-future- people-building-their-own-internet-detroit/59cebd5795073d0905939aeb)

</comment><comment><USER20> Now this is fantastic! I started a company to build a platform specifically for local intranets like this to run social software. We should get in touch and help their Apps program.

[https://qbix.com/communities](https://qbix.com/communities)

I am very happy to see communities building their own intranet software!

</comment><comment><USER21> What kind of equipment are they using for this? Is this something that is affordable for testing out? I was thinking of trying something like this at my cabin.

</comment><comment><USER22> Didn't know you could be a citizen of detroit...

Why not just say detroiters?

<comment><USER23> > Didn't know you could be a citizen of detroit...

What did you think 'citizen' means?

[https://www.merriam-webster.com/dictionary/citizen](https://www.merriam- webster.com/dictionary/citizen)

> Why not just say detroiters?

Like many people I would guess, I know what a 'citizen' is, and I know what 'Detroit' is, but I wouldn't know what a 'detroiter' was.

For example if I wrote an article about 'Wollybacks building their own internet' would you know what that meant? That's a name for people from where I live.

<comment><USER22> > What did you think 'citizen' means?

A member of a state/nation. For example, I am an american citizen, not a new york citizen. Citizen has historical etymological roots to ancient greece and their city-state but the word has changed in meaning in modern times.

> Like many people I would guess, I know what a 'citizen' is

Apparently you don't if you think someone is a citizen of detroit. At least in the US, we are citizens of the nation, not a city. I don't need to get visa or permission to move from NYC to Detroit. I don't have to change citizenship if I move from NYC to Detroit.

> but I wouldn't know what a 'detroiter' was.

So you don't know what a new yorker is?

> That's a name for people from where I live.

Detroit is one of the major cities of the US. Pretty sure everyone knows what detroit is.

I'm not sure if you are being sarcastic here.

<comment><USER24> > So you don't know what a new yorker is?

I don't know what a "yorker" is, never mind a new one.

To be fair though, I'm guess you meant one of these two; either a "New Yorker" (citizen of the state of New York), or a "New Yorker" (citizen of the city of New York, New York).

Or would that second one be a "New York, New Yorker?"

\</comment></comment></comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>-

For example:

The new new New Yorker was new to New York, New York. New New York is known for it's saturation with new New Yorkers. Now I know that would be a bit nuanced, but nevertheless I knew you as a new New Yorker of "The New New York" (New York) would find it relevant. I just didn't know if you knew that yet.

\</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>-

Related...?

"Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo"

"... is a grammatically correct sentence in American English, often presented as an example of how homonyms and homophones can be used to create complicated linguistic constructs through lexical ambiguity and the usage of homophony and homonymy."

"More easily decoded, though semantically equivalent, would be:

Buffalo from Buffalo... whom other buffalo from Buffalo bully... [themselves] bully buffalo from Buffalo."

([https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...](https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo))

<comment><USER25> Or a new citizen of York, England.

<comment><USER26> Or a citizen of one of the newer cities named York! [https://en.wikipedia.org/wiki/York_(disambiguation)#Places](https://en.wikipedia.org/wiki/York_\(disambiguation\)#Places)</comment></comment></comment>


www.canadiens.it: quando vedi l'inukshuk sulla tua strada, sai di essere arrivato - <AUTHOR> http://www.kleisranking.it/comunicati-stampa/abbigliamento-e-moda/www.canadiens.it-quando-vedi-linukshuk-sulla-tua-strada-sai-di-essere-arrivato.html <comment><USER2> Can the moron who posted this translate it for those of us who do not understand Italian.

The profile of the guy who posted this was created an hour ago and matches the domain of the post. This looks like blatant but unintelligible self promotion.

<comment><USER3> I think there's a simpler word for it: 'spam'.

Those who monitor this site are usually pretty good about axing this kind of thing, although it often takes them a bit.</comment></comment>


CIS 194: Introduction to Haskell (Spring 2013) - <AUTHOR> https://www.seas.upenn.edu/~cis194/spring13/ <comment><AUTHOR> After much internet searching I found this course mentioned as one of the best ways to learn the basics of Haskell. I've only worked through the first couple weeks so far, but I'm having a lot of fun with it. Has anyone else here given this course a go? And does anyone have any other suggestions on learning Haskell?</comment>


Is the Saudi Government Plotting Against Another U.S.-Based Critic? - <AUTHOR> https://www.newyorker.com/news/news-desk/is-the-saudi-government-plotting-against-another-us-based-critic <comment><USER2> If there was ever a reason to reduce fossil fuel dependency ...

</comment><comment><USER3> They believed that technology and democracy would save them until the age of techno-tyranny began - thus erasing all of their progress.

</comment><comment><USER4> He better don't step on any Saudi embassy.

</comment><comment><USER5> the not so secret reality of all of this is that saudi money is everywhere. and they only very recently allowed women to drive a car.

there's no question that the valley is owned / driven by this influence. the question is: at what cost?

but, I digress, by all means, please, continue on the start-up-y discourse...

</comment><comment><USER6> >After leaving the Bureau, in 2005, he started the Soufan Group, a security- advisory firm in New York. His company runs a training academy for police and intelligence forces in Qatar, a neighbor and bitter rival of Saudi Arabia.

The beginning of the article would have you think that this related to Soufan's anti-terrorist work. However, reading more it seems that this is related to his employment by Qatar. There is somewhat of a cold war going on the the Middle East between Saudi Arabia and Iran and this seems to be related.

Soufan is basically a mercenary working for a foreign government(Qatar) that another foreign government(Saudi) wants to do in. Even the article does not talk about any intention to kill him in the US, but in Qatar.

This is very different from what happened to Jamal Kashogi who was a journalist and not involved in military or security affairs.

<comment><USER7> Khashoggi was a long time Saudi intelligence operative who operated in Afghanistan during the Afghan Russian war, then was posted in the US embassy for several years as an intelligence officer, officially "advisor to the ambassador". He turned journalist and was very loyalist to the Saudi royalty. The ascension of the current monarch and his son turned him after power swapping that happened in Saudi, so he left and turned "activist". He held so much dirty laundry on the Saudi govt from his previous work that they saw him as a threat. I doubt Ali Soufan or any other Saudi dissidents are seen as such a threat as Khashoggi.

<comment><USER8> Thank you for that informative background. Did you gather all this from public articles ?

</comment></comment></comment><comment><USER9> I have an off-topic observation ...

So I happen to be working in an old VM, and it has an old version of Firefox. And yes, I know that I ought to migrate. Anyway, this old Firefox just won't load some sites, including The New Yorker. But there's a simple workaround:

       user@host:~$ w3m -dump https://www.foo.bar/baz/ | less

<comment><USER10> I hate to be rude but: No one cares. How is your very specific linux/firefox setup any relevant to this conversation.... ?

It is so common/cliche for the first comment to be: it wont load, too slow, i hate the fonts, etc.. etc...

It doesn't add anything relevant to the convo/topic.

<comment><USER9> YMMV, but w3m is a _very_ useful tool.

Also, given your comment history, it's disingenuous to say that you "hate to be rude".</comment></comment></comment>


Why Google Stores Billions of Lines of Code in a Single Repository (2016) [pdf] - <AUTHOR> https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext <comment><USER2> Previous discussion:

[https://news.ycombinator.com/item?id=11991479](https://news.ycombinator.com/item?id=11991479)

[https://news.ycombinator.com/item?id=15889148](https://news.ycombinator.com/item?id=15889148)

[https://news.ycombinator.com/item?id=17605371](https://news.ycombinator.com/item?id=17605371)

[https://news.ycombinator.com/item?id=22019827](https://news.ycombinator.com/item?id=22019827)</comment>


Ask YC: Where to start with creating a distributed crawler - <AUTHOR> Hi there. We're just starting out and want to create a crawler that will sit on EC2.  Any advice appreciated. Here's what we're thinking of:
1. Using Beautiful Soup for the actual parsing of pages
2. we're not sure what to use for the crawl itself :( We use Python and love it, but don't know if we need to create our own crawler or what the best route would be. Any advice on this would be good
3. I'd like to create a distributed crawler where we can replicate the crawler over EC2 instances, but not sure how to do this
Apologies if I should ask this elsewhere. I love this community and have passively read many of the articles and comments on here for a couple of months now
Any help or pointing in the right direction would be appreciated
John <comment><USER2> How you build it will depend on the service you host it on. So with AWS, you'll use EC2, S3 and perhaps most importantly, SQS.

Fundamentally for a crawler, you will need the following:

1\. A list of URLs to crawl, perhaps even ranked in priority of crawling. This is a database of sorts.

2\. A set of crawlers that figure out the most important URL on the list and fetch it.

3\. A parser and HTML storage service. The parser will also feed new URLs into the list.

Each of the above pieces are easy to do on their own. The trick is how you glue them together. I would suggest something like the following as a starter for using AWS for crawling:

1\. A MySQL list of URLs with some kind of priority ranking. This can be a cluster of EC2 instances that store and prioritize the links. Early on, you can ignore the prioritization aspects.

2\. The URL cluster will dispatch queue messages of URLs to crawl in the desired crawling order.

3\. A cluster of EC2 instances check the SQS queue for crawling messages and fetch the URL specified in each message. As the message is being processed, it's locked so others can move on.

You can make the whole thing dynamic by adding crawling instances if the queue gets too long. You can also have instances that determine the crawling priority for the next time (one metric is number of backlinks to a page). Another set of instances might be parsers or do the actual analysis of the crawled pages.

Which language to code it in? If you're going for maximal speed, perhaps you should consider a compiled language. If not, Python or PHP or Perl would do just fine. Personally I'd do it in a scripting language to begin with and invest the time into a faster crawler later if warranted.

And good luck!

<comment><USER3> Creating the list of URLs and prioritizing them is the hardest thing about building a crawler! That is, a good, web-scale one. A replacement for wget might be sort of fun, but the real way to make a fast crawler is to be choosy about which pages get updated frequently, which are likely to contain good content (by computing a pagerank-like stat on the fly), etc.

It is far from my area of expertise, but the Wikipedia page about this looks very useful. It cites a bunch of wicked smart people. <http://en.wikipedia.org/wiki/Web_crawler>

If you just want to suck down a bunch of pages, then there's nothing wrong with wget.

</comment></comment><comment><USER4> Check out Nutch, not sure if its exactly what you want though. It's in Java not Python, but it works with Hadoop quite nicely

<comment><USER5> I concur with the Nutch vote; but more specifically, take a look at the crawler code written in the src trunk for use with Hadoop. That is probably a good place to start. Also worth a look is Heritrix (crawler for archive.org). <http://sourceforge.net/projects/archive-crawler> Sadly, this too is written in Java.

The only Python one I am aware of for which code is available is: <http://sourceforge.net/projects/ruya/>

Edit: You might also want to take a look at <http://wiki.apache.org/hadoop/AmazonEC2>

Edit2: Polybot is another Python based crawler, but no code. However, the paper has some interesting ideas:

Design and Implementation of a High-Performance Distributed Web Crawler. V. Shkapenyuk and T. Suel. IEEE International Conference on Data Engineering, February 2002. <http://cis.poly.edu/westlab/polybot/>

<comment><USER6> Good response. We've created a basic crawler in Python, but are looking for something more powerful too. Heritrix above looks good

</comment></comment></comment><comment><USER7> BeautifulSoup is not foolproof, meaning it does not always even approximate the way a browser would parse the HTML. One important failure is that it fails to recognize when HTML tags are inside of JavaScript strings (and so should not be considered). Whether this is important or not depends on your application.

</comment><comment><USER8> 1.) Don't do it yourself. Use Amazon's Alexa Web Search service (aws.amazon.com). Through that you can access Alexa's 10 billion page index, complete with all the pages, run complex queries etc. Plays nicely with EC2.

2.) If you must do it yourself, Heritrix is the most sophisticated crawler out there (crawler.archive.org).

3.) Nutch is an option, but nowhere near as powerful as Heritrix.

Don't try to reinvent the wheel, writing a robust crawler is a lot of work as there are endless edge cases to take care of (if you are looking into a general purpose web crawler)

<comment><USER9> Nutch is good and I would second it, but I would suggest to NOT build a crawler - its not trivial and inadvised in a startup, that is unless your startup is just about building a crawler.

</comment></comment><comment><USER10> I have recently written a Beautiful Soup / Twisted crawler. To make it distributed, presumably we'd use Amazon's queue service.

Feel free to get in touch if you're interested in the details.

<comment><AUTHOR> Thanks John. I'll do that. Your resume looks awesome mind. Not sure how much we'd be able to help you!

</comment></comment><comment><USER11> Thanks for the previous positive comments about Heritrix, which is my project at the Internet Archive. If anyone has questions, please send them my way.

Heritrix was designed for archival projects, which has meant an emphasis on having a "true record" (including non-text resources) and high configurability for inclusion/exclusion. Any text indexing or link-graph-analysis is completely external; we've used Nutch (without their crawler) for that.

Whole-web multi-billion-page crawls have not been the focus yet, though we've tried one and have heard of outside groups successfully using Heritrix for 2+ and 4+ billion page crawls.

Our distribution story is spotty; we provide some options that help you split the URL-space you want to crawl across crawlers, and remote-control crawlers from other programs, but syncing their launch and other steps is left to an expert operator's own devices. We've run coordinated Heritrix crawls on groups of 4-8 machines (dual-opteron, 4G+ RAM, 4x500GB+ HDs) and understand others have used up to 12.

</comment><comment><USER12> Given the way that Google is heading ( <http://news.ycombinator.com/item?id=149894> ), you should have started at least six months ago. Regardless:

1\. 10 years ago, at least 99% of web pages failed validation. Nowadays, the majority still fail. You could validate and then fall through to tag soup processing.

2\. 10 years ago, the conventional wisdom ( [http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTO...](http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTOC) ) was to use a compiled language, such as C, for spidering ( <http://www.tbray.org/ongoing/When/200x/2003/12/03/Robots> ). Given that memory increases faster than processing power which increases faster than bandwidth, this may not be the case nowadays.

3\. That's the meta problem. Solve that and you may find that a search engine is easier.

</comment><comment><USER13> I recently built a crawler of my own in Ruby after trying, and ultimately deciding against, Nutch. Depending on what you want to do with your crawl there is a very good change that you'll be able to write a small crawler that is much easier to extend on your own, and you'll probably be able to write it in the time it would take you to install, setup, and configure Nutch.

As I said I used Ruby and specifically Hpricot for the page parsing. I'm starting to run into problems with Hpricot right now though and I may actually try a python version with Beautiful Soup very soon. Let me know how it goes for you and maybe we can share some code.

<comment><AUTHOR> Hi there. I think I've decided we need to build our own, mainly as what we're wanting to do is quite specific - monitoring of sites for keywords and really we should understand,to a degree, this technology. Happy to share if you end up heading down the Python route. If there is anyone else on here who is doing crawling/data mining then maybe we could share ideas, help each other somehow :) My email is in my profile

<comment><USER13> Have you seen this article yet? Looks like it would be useful to you. [http://blog.ianbicking.org/2008/03/30/python-html-parser- per...](http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/)

</comment></comment></comment><comment><USER14> +10 for Python +1 for Nutch +1 for Hadoop +1 for Amazon EC2

I think you have bundled two things (crawler and parser (or may be scraper)) into one term : crawler.

Beautiful soup is ok. Give html5lib a try (on google code) - but at some point you are going to have to hack the parser, but that depends on what kind of parsing you want to do.

<comment><AUTHOR> Thanks for that suggestion. Will have a good look at it.

</comment></comment><comment><USER6> Look at Harvestman. Quite useful

<comment><AUTHOR> I looked, but this is not distributed, so currently (whilst good) its limited to one server</comment></comment>


Get pictures of your days mail - <AUTHOR> https://informeddelivery.usps.com/ <comment><USER2> I use this every day and it saves us a trip to our PO box unless we have something important to pick up. Snapping a picture of packages would be a huge upgrade.</comment>


You waste a lot of time at work - <AUTHOR> http://www.atlassian.com/time-wasting-at-work-infographic

<comment> cam- Ungraciously it could be argued that atlassian's tools fall in the time waster category as well.

<comment><USER2> Jira is the bane of my existence.

</comment></comment><comment><USER3> Currently wasting time trying to get Crowd to work with eDirectory.</comment>


Goodhart's Law: When a measure becomes a target, it ceases to be a good measure - <AUTHOR> https://en.wikipedia.org/wiki/Goodhart%27s_law <comment><USER2> This is probably where I should mention my two longer articles that discuss this in depth; [https://www.ribbonfarm.com/2016/06/09/goodharts-law-and- why-...](https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why- measurement-is-hard/) [https://www.ribbonfarm.com/2016/09/29/soft-bias-of- underspec...](https://www.ribbonfarm.com/2016/09/29/soft-bias-of- underspecified-goals/)

As well as my more recent paper categorizing the different ways it occurs; [https://arxiv.org/abs/1803.04585](https://arxiv.org/abs/1803.04585)

<comment><USER3> Since you're here, what does the following property stand for in your second paper?

s ∈ AifM(s)/gec

<comment><USER4> Not OP but that should probably render as: s ∈ A _if_ M(s) >= c

So: s (system state) is a member of A (permissible region) _if_ M(s) (proxy measure of s since we can't directly observe it) is greater than or equal to c (threshold)

</comment></comment></comment><comment><USER5> "You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

<comment><USER6> When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales."

I was too taken aback to follow up asking why # of sales can't be gamed too.

<comment><USER7> > interviewer replied "that's a dumb answer

a bit ot, but was that the actual response?

personally, stuff like this would have me ending the interview early.

<comment><USER8> And what is your gentle phrasing for that action?

<comment><USER9> It's an interview, not a competition. The interviewer would've been much better of asking: "Why?"

<comment><USER8> I meant for excusing yorself.

I'd tend to go to "I'm sorry, but I'm afraid I'm wasting your time," for example.

</comment></comment></comment></comment></comment></comment><comment><USER10> And as soon as a measure becomes known, it becomes the target.

IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure."

"I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks."

"What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."

<comment><USER11> Gamification increases as understanding improves.

Information is exploited.

<comment><USER12> [Targeting Intensifies]

</comment></comment></comment><comment><USER13> The breaking of this law when one uses profit as the measure of a successful company or business is why I think that a well set up market system actually works. People and companies can set up making a profit as their target and it is still a good measure of success, because it is actually what you want.

The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not. A hard thing to get right but forcing people to inform you of the ingredients of food for sale, is an example of a good regulation I think.

<comment><USER14> Even disregarding the potential for negative externalities etc., profit alone can still be a terrible KPI.

I worked for a large engineering consultancy that was very profit motivated. They figured out a brilliant way to maximise profit: get rid of business development! Only pay for work when clients are paying you to do it. This was GREAT for quarterly profit margins, at the expense of the project pipeline.

As the pipeline dried up, they figured out a new way to maximize profit: get rid of engineers! The customers are big, stupid organisations, and it'll take them a long time to notice. You can _totally_ maximise quarterly profit margins until they do.

And that's how you can simultaneously be highly profitable and in a corporate death spiral.

Moral of the story: reality is a fundamentally complex place. You need multiple KPIs to get a handle on it -- and ideally those KPIs should be mutually contradictory enough to force you to engage with that complexity head-on, rather than boiling things down to game-able metrics which ultimately will defeat themselves because they are such a poor proxy for reality.

<comment><USER13> I would say the complexity is in balancing profit over time. Here is your contradictory force. In the end that large engineering consultancy was making zero profit and was worth nothing. Whether running a business into the ground will give you more overall profit than just selling it and putting the money into government bonds is something one can consider, but in your case was the consultancy owned by the people making these short-term profit maximizing decisions? The agent/principal problem for companies has not been solved.

It is probably not a coincidence that the top 6 US companies by market cap (Apple, Amazon, Alphabet, Microsoft, Facebook, Berkshire Hathaway) are still run by their founders or were run by their founders for most of the companies existence before a recent step-down (Microsoft) or death (Apple). Most public companies aren't and they struggle to grow or even survive over time.

<comment><USER14> > in your case was the consultancy owned by the people making these short-term > profit maximizing decisions[?]

It was a publicly-listed company, so that'd be a yes.

Profit is obviously over time a _necessary_ condition for making a sustainable business. But it's definitely not a _sufficient_ condition. I think there are a huge number of people who don't understand this.

Eg.: when I see people clambering for TSLA to make a profit... I think they're wrong. Or, at least, mostly wrong. TSLA has been unprofitable in the service of dramatically (and rather effectively) expanding both capacity and market share. Those are excellent reasons to be unprofitable. And TSLA still obviously has a lot more room to grow in both respects, so in this situation, trading profitability for growth is a good decision. (exhibit A: Amazon)

<comment><USER13> >> in your case was the consultancy owned by the people making these short- term profit maximizing decisions[?] >It was a publicly-listed company, so that'd be a yes.

I'm a bit confused. I stated that public companies are not often owned by the people who mange them and that is a real problem. You are asserting that all publicly-listed companies are managed by their owners. That is just wrong. I own both TSLA and AMZN and of course there are hundreds of things that need to be done to make a sustainable business. Just talking past each other at this point, I think.

<comment><USER14> Ah, sorry, you're correct. The company was owned by people who _wanted_ short- term profit maximising decisions (its shareholders). The people actually _making_ the decisions also owned shares, but nowhere near a controlling interest.

Anyhow, I agree with your point about companies that are controlled by founders (whether public or not). A (sufficiently pragmatic) visionary with a controlling interest can achieve things that no amount shareholder-focused governance can. Of course they can also drive a company into the ground faster than anyone -- but the heights are reserved for them.

</comment></comment></comment></comment></comment></comment><comment><USER15> Indeed this is the whole idea behind OKR. The Objectives are _supposed_ to be grand, audacious, and only achievable under extreme stretching. Furthermore, they are _supposed_ to not be used as a KPI for performance reviews.

</comment><comment><USER16> I wish more people knew and understood this concept.

It's important to know the impact that you have on a system, especially at large scale. Decades after Goodhart made this law, we still have shoddy credit risk measures that put tens of trillions of dollars at risk just in the US alone. Hell, the last recession alone was mostly caused by the government's choice of credit worthiness metrics.

<comment><USER17> Can you explain how the recession was caused by "government's choice of credit worthiness metrics", instead of rollbacks of regulation?

<comment><USER16> I'm not sure why you think these things are mutually exclusive. And as far as Goodhart's Law is concerned in this case, they're intertwined concepts.

The government's choice to allow securities rated by S&P/Moody's to be treated as gospel allowed/encouraged banks, insurance companies, GSE's and pension funds to take far more risk than was systemically reasonable.

</comment></comment></comment><comment><USER18> Aren't there are exceptions? For example, a measure of health might be life expectancy. If this becomes known, is it no longer a good measure? I think it's still a good measure. I mean, how can you get to a life expectancy of 100 without a healthy population?

<comment><USER19> Life expectancy is actually a really great example of the opposite. People routinely claim our health system is bad compared to europe because our average life expectancy is lower.

It turns out african americans skew our metrics because they have

1) high infant mortality 2) high murder rates of young men

The high infant mortality is irrespective of wealth and has a disproportionate impact on average life expectancy because the death of a 0 age person has a huge impact on the average. When you look at the rest of the population, our whites are on par with europe and our asians are on par with asian countries.

One way this is gamed is that some countries have different standards for reporting infant mortality.

<<The infant mortality rate is defined as the number of deaths of children under one year of age, expressed per 1 000 live births. Some of the international variation in infant mortality rates is due to variations among countries in registering practices for premature infants. The United States and Canada are two countries which register a much higher proportion of babies weighing less than 500g, with low odds of survival, resulting in higher reported infant mortality. In Europe, several countries apply a minimum gestational age of 22 weeks (or a birth weight threshold of 500g) for babies to be registered as live births. This indicator is measured in terms of deaths per 1 000 live births.>>

In the US it is gamed in reverse in the sense that the average hides the true problem is with subpopulations and not the entire US medical system as a whole. Our (gamed) poor performance in this statistic is used as a measure to support moving to a substantially different system.

<comment><USER20> It's not like Europe doesn't have marginalized populations. Your argument is basically "if you exclude the parts of the US population that are worst treated by our healthcare system, then ours is the best!" There is something meaningful in comparing the "median" or "modal" health experience in different countries, but "we treat black people badly compared to white people in the USA so you have to only look at white statistics" is a terrible defense of the system.

Your point about infant mortality statistics being gamed is on point, though (and I pointed it out in a sibling commment).

</comment></comment></comment><comment><USER21> Another article on the front page decries Harvard's holistic approach to admission - with arguments that admission should be based on a single measure of academic performance (like standardized test scores, GPA, etc.).

<comment><USER22> Yes, but holistic approaches have -also- become a target. Think of the over- achieving high schoolers doing as many extra-curriculars as possible just for their college application.

</comment></comment><comment><USER12> During the Vietnam War, it was body count. That turned out to be a great way of incentivizing the most murderous troops.

<comment><USER23> Probably more of an incentive for NCO's and Officers to overclaim the results, so they look good to their superiors.

<comment><USER12> That seems to happen in all wars. Aircraft downed and tanks destroyed in WWII, for example.

There were occasions in the Vietnam War where regions were declared to be free fire zones, and everyone within was to be killed, down to the last child and granny.

<comment><USER24> I'm sorry but this is a serious (and arguably offensive) falsehood.

A 'free fire' zone is an area wherein there is no requirement for soldiers to coordinate heavy arms fire with other units via HQ etc.. This is a very reasonable policy in jungle fights where units can come under ambush at any time, i.e. "We're in an ambush and we're about to die, can you ring up HQ to see if we can shoot back?"

But the notion that US forces would just designate an area where everyone including civilians are to be killed is an outright and offensive lie.

Though there are some situations in which Army Units did actively and knowingly fire on citizens, these are known and well documented tragedies of soldiers acting with fury in the moment agains villagers who were supporting insurgent forces - this was not and never was any kind of US policy, and the perpetrators faced Court Martial.

The Vietnam war is so completely misinterpreted in pop culture; even to this day, it's disturbing.

<comment><USER12> _I 'm sorry but this is a serious (and arguably offensive) falsehood._

I'm sorry, but I will show you to be mistaken below.

 _But the notion that US forces would just designate an area where everyone including civilians are to be killed is an outright and offensive lie._

Then you should have an issue with the Ken Burns documentary. I think I got the terminology wrong; "free fire zone" means something else. However, the practice I am referring -- that of designating everyone in a certain place as an enemy, then killing everyone within -- did happen.

[https://en.wikipedia.org/wiki/My_Lai_Massacre](https://en.wikipedia.org/wiki/My_Lai_Massacre)

From Ken Burn's "The Vietnam War" Part 5 narration: (00:44:43,393 --> 00:44:47,428)

 _In the summer of 1967, Tiger Force was sent to the fertile Song Ve Valley. The entire population had already been herded from their homes and crowded into a refugee camp. But some had come back to resume the farming they had always done. The valley had officially been declared a free-fire zone, and Tiger Force 's officers took that literally. "There are no friendlies," one lieutenant told his men. "Shoot anything that moves." Over a seven-month period, they killed scores of unarmed civilians. Among their victims were two blind brothers; an elderly Buddhist monk; women, children, and old people hiding in underground shelters; and three farmers trying to plant rice. All were reported as "enemy... killed in action."_

<comment><USER24> No, you have misinterpreted.

The US does not declare that US forces should just go and kill anyone, rather, 'anyone' can be considered a target, which is perfectly reasonable policy while fighting an insurgency.

In normal rules of engagement, the women and children running out of the village are obviously out of bounds. They are civilians. You can't fire at them for any reason.

In a 'free fire' zone, anyone can be considered an enemy combatant if the situation indicates as such. The woman running out of the village with an AK-47 - is 'in bounds' as an enemy combatant in an area where villagers are considered on the side of the insurgents. Obviously, random civilians are not.

The misinterpretation of 'free fire' or the 'loose application' of it are considered 'war crimes' by the US and it was absolutely never intended that the policy be used by soldiers to just shoot up people - your implication that it was used this way is completely false.

The My Lai massacre was a war crime, not an intentional act sanctioned by US forces.

Also: war is dirty. There are no wars without war crimes. Once the bad genie comes out ... it spills all around, the best we can do is contain it.

FYI that's one of the hardest things to do, i.e. make those kinds of calls in the field - often it's not obvious.

Go and watch one of the documentaries that delve into US forces wrangling over a decision to drop munitions, and consider how tricky those decisions are ... so many factors.

<comment><USER12> > The US does not declare that US forces should just go and kill anyone, > rather, 'anyone' can be considered a target

 _" There are no friendlies," one lieutenant told his men. "Shoot anything that moves." Over a seven-month period, they killed scores of unarmed civilians. Among their victims were two blind brothers; an elderly Buddhist monk; women, children, and old people hiding in underground shelters; and three farmers trying to plant rice. All were reported as "enemy... killed in action."_

> In normal rules of engagement, the women and children running out of the > village are obviously out of bounds. They are civilians. You can't fire at > them for any reason.

During the Korean War, my grandmother had her life saved by a wad of cash (large, because of wartime inflation) hidden in a secret pocket in her skirt. It stopped a piece of shrapnel produced by strafing from a US Douglas A-1 Skyraider. Was she "in bounds?"

> The misinterpretation of 'free fire' or the 'loose application' of it are > considered 'war crimes' by the US and it was absolutely never intended that > the policy be used by soldiers to just shoot up people

Can you see how using Body Count as a metric would tend to produce "targeting" of the metric which would result in the targeting of people to turn into corpses? That is my whole point here.

<comment><USER24> "It stopped a piece of shrapnel produced by strafing from a US Douglas A-1 Skyraider. Was she "in bounds?""

Civilian casualties are generally avoided if possible. I'm happy that your grandmother was saved, but it's not relevant.

"Can you see how using Body Count as a metric would tend to produce "targeting" of the metric which would result in the targeting of people to turn into corpses?"

No. 'Body count' is used in every war and has been since the dawn of time, and we continue to use it today as one of many metrics.

Using this metric will not encourage professional soldiers to arbitrarily murder civilians.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> We hold people accountable for actions instead of outcomes, since outcomes (even profit) can be gamed at the overall detriment of the organization.

We talk about the outcome we want, look at the levers that will get the outcome, and then agree to the actions that we are going to take.

If the actions dont result in the forecasted outcome then the underlying assumptions are wrong. The assumptions are wrong if 1) people arent doing the tasks properly 2) the relationship between the task and the outcome are different than what we thought. If the issue is 1, we provide training. If the issue is 2, we use the revised assumption for future forecasts.

We dont use the word target anymore, we use the word forecast.

</comment><comment><USER25> This is one reason why the grading system in schools is absurd and incompetent.

</comment><comment><USER26> Without dates and metrics you can't measure progress. Like mile markers along the road (back before GPS / Smartphones). But without continuous, high trust, collaborative conversation you can't actually make progress. That's the gasoline powering you down the road. The goal should be Disneyland (shouldn't it always be Disneyland?), not the next mile marker.

</comment><comment><USER27> That's why you know your company is in trouble when shareholder becomes the customer and the stock price becomes the product..

</comment><comment><USER28> This is also demonstrated quite well in machine learning with the test set. The target is test error, but you don't let algorithms optimize that directly and instead they optimize against training error. Goodhart's law is a manifestation of over-fitting in social systems.

</comment><comment><USER29> And thus for performance measures, "security through obscurity" can make sense.

</comment><comment><USER12> The same principle applies to attributes and qualities which mark successful entrepreneurs. The same principle applies to attributes and qualities which mark good developers. The same principle applies to attributes and qualities which mark good employees. Good people. Good politicians. Activists. Good pundits. Intellectuals. Top tier school degrees.

Hacker News karma.

Shouldn't we all be looking out for the "targeting" and be trying to find the next measure?

</comment><comment><USER30> Reminds me of all those computer architecture papers that optimize for IPC to the detriment of overall performance. Or the CPU frequency wars which led to ever deeper pipelines which were completely obliterated with branch mispredicts. Or the current trend of increasing core count without actually being able to do any useful work on them...

</comment><comment><USER31> yep, seen several idiots whose work habits are destructive be promoted for gaming stats

seen stats make people discourage/delay income/work to avoid anomalies/spikes in stats because if there's a spike it makes it look like you are slacking the next month, or makes a manager think you should be able to constantly hit the spike

</comment><comment><USER32> Serious question - it may no longer be a good measure, but does that _necessarily_ mean it is a poor target?

Assuming targets are necessary, do they not have to based on done measure?

</comment><comment><USER33> I've seen this a lot. Even people who intellectually know that correlation doesn't equal causality make this mistake all the time.

<comment><USER34> It's not even that correlation doesn't equal causality, it's that the correlation may also completely reverse.

</comment></comment><comment><USER35> People respond more or less rationally to their incentives. Who knew?

</comment><comment><USER36> I've seen similar topics come up on HN front page every once in a while.

Here are some of my thoughts on the topic, through the prism of metrics for a company.

[http://dimitarsimeonov.com/2018/03/23/the-vanity-metric- para...](http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-paradox)

TLDR of my post: The vanity metric paradox - every metric, sufficiently optimized becomes a vanity metric, as it stops being the make-or-break qualifier for a company.

</comment><comment><USER37> (cough! cough! "scrum")

(cough! cough! "agile")

(cough! cough! "daily stand-ups")

<comment><USER38> I always felt that agile processes were a means to give non-answers to performance measurement in order to allay the consequences of that measurement and allow developers to get on with the business of making software.

How big is the task; 10 story points

How long will that take; depends on our velocity

Can it be done by this date; we're not allowed to plan that far ahead. You can put it in the 'icebox'. We might get to it eventually.

<comment><USER39> Yes, definitely true.

If (1) everyone is highly effective in getting tasks done, and (2) working on tasks that are reasonably well prioritized by both technical risks and business value, why exactly do you need to know more?

There are legitimate reasons that a business may need a date. But making sure people are working harder than hard by beating them over the head with a date coughed out by a gantt chart is not one of the legitimate reasons.

</comment></comment></comment><comment><USER40> So we should minimize the fraction of measures which have become targets then?

<comment><USER34> You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them.

That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something that ceases to improve their life quality.

<comment><USER41> Exactly. Don't reduce everything down to one factor.

In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors.

Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things.

Focusing on only one thing while neglecting others doesn't work.

<comment><USER40> But then your "one factor" just becomes a combination of various other factors. I don't think the point of Goodhart's Law is related to "over- optimizing for just one thing," rather it's a statement about whether _any_ metric (even a broad, diverse metric that takes into account multiple different things) can retain informativeness once it is set up as an optimization target.

<comment><USER41> I think you could set up an optimization target for a company that says "be profitable and grow for the next 30 years". You would see quickly that this target is a complex beast with a lot of different factors at play that need to be balanced against each other. I don't think Goodhart's law would apply here.</comment></comment></comment></comment></comment>


Apprenticing to many masters - <AUTHOR> http://apprentice.kfitz.me/2009/10/01/many-masters/

<comment><USER2> Uhh so if you can't learn by "sitting at the feet of" a "master", you learn what you can by yourself and touch base occasionally with the good people you do know and soak up what you can. Hardly insightful.

You can just say "try to work with programmers better than you" without all this medieval framing.

I personally think all this talk of the "Way" and guild inspired talk of "masters" and "journeymen" and so on is pretentious and inappropriate for programming in the 21st century but that's just me. In the olden days you _had_ to join a guild and apprentice yourself to a "master" to learn a craft.

In programming, while it would be great to work with really good people(say, the kind of people interviewed in Coders at Work), the default for almost everyone is to learn without the benefit of a "master programmer". Pete McBreen wrote this really terrible book and from then we have all these pious mumblings about "apprentices" and "journeymen".

</comment><comment><USER3> Did this guy's URL (apprentice.kfitz.me) make anybody else think he might have the Talent? (Robin Hobb...)

</comment><comment><USER4> Where did the things I need to know to do my job come from?

      Mom, Dad, & family         10%
      kindergarden                5%
      elementary school           1%
      middle school               1%
      high school                 1%
      college (B.S. math)         1%
      college fraternity          5%
      business school (MBA)       1%
      my first mentor             5%
      my second mentor           10%
      my users over the years    10%
      my employers & customers   10%
      reading                     5%
      other programmers           5%
      doing on my own            30%
                               -----
                                100%</comment>


How big cities that restrict new housing harm the economy - <AUTHOR> https://www.washingtonpost.com/news/wonk/wp/2014/07/25/how-big-cities-that-restrict-new-housing-harm-the-economy/ <comment><USER2> Are there any restrictions which don't 'harm the economy'?

Why can't I interpret the analysis to mean the population of the city is willing to pay an extra (made up number of) $300/month per capita in order to preserve the current system?

> New housing supply benefits the entire city, even if that new supply takes > the form of luxury condos. Tenant activists have regularly objected to new > luxury developments as just another sign of gentrification. But even if that > housing is intended for high-income tech workers, it will take some pressure > off the existing units that those tech workers have occupied at the expense > of middle- and lower-income residents. In effect, this means that affordable > housing advocates who want to block new high-end developments are simply > making the city more expensive.

Let's consider this more closely. Would a restriction on luxury condos harm the economy, using the same analysis factors? I assume the answer is yes.

Would a restriction on luxury condos, but increasing the number of apartments, make the city more affordable? I assume the answer is yes.

If both are yes, then it means that actions to make the city more affordable can end up harming the economy. Which then means that 'simply making the city more expensive' cannot be used by itself to justify having luxury apartments.

If one or both is 'no', why?

</comment><comment> norea-armozel I feel like the question of whether or not St. Louis (one example city) couldn't be as productive as San Francisco is left hanging. For certain kinds of industries like manufacturing, location is everything but for software development and most non-face-to-face service industries you don't need the person to be in the same city or country to be productive. If anything, the article is built on the assumption productivity is the same across all industries in as much as the methods to achieve it. An office in St. Louis is just as good as an office in San Francisco or San Jose. Equally, a person in their own private office is just as productive as someone in their cubicle following similar practices of work. Frankly, the idea that everyone has to be in SV to make the same software of the same quality and at the same productivity level is at best naive.</comment>


New JavaScript features in ES2018 - <AUTHOR> https://twitter.com/mathias/status/956970099063189505 <comment><AUTHOR> There's also this tweet which shows the support in v8 6.4 and chrome 64 for all these features. [https://twitter.com/v8js/status/956976954074267648](https://twitter.com/v8js/status/956976954074267648)</comment>


Sylentmite - silent dynamite - <AUTHOR> http://www.youtube.com/watch?v=99tNW0Z3cac

<comment><AUTHOR> Their website is also worth a visit

<http://www.sylentmite.com/index0.html>

Especially if you fancy clickmaps, animated gifs and bright yellow background. The stuff is really cool, though.</comment>


Social Login and Social Plugins Increase Page Views, Time Spent On Site - <AUTHOR> http://blog.gigya.com/social-login-and-social-plugins-increase-page-views-time-spent-on-site/

<comment><USER2> I think that social vs non social logins probably segments your users into 2 different types of people. I consider myself to be a very savy Internet user - I don't trust social login. I don't think any good can come from connecting my accounts like that and I know that some site is going to do something inappropriate on my behalf. As users become more savy (read: cynical) I think you'll find the develop the same view point.

Additionally we found on our site that social login increased conversions for signups but those people are generally of lower value to our freemium product in the long run because they're less likely to want to pay.

</comment><comment><USER3> I second the "absolutely not" view.

In fact I "deleted" my Facebook existence and if you want to force me into a social login, any social login, you just lost me.

Your mileage may vary, of course.

</comment><comment><USER4> a. This is a plug: these guys have a stake in this kind of "social" This is no proof, no comprehensive data but just pretty charts.

b. _post hoc ergo propter hoc_ fallacy committed

c. I trust Mozilla more <https://news.ycombinator.com/item?id=5408735>

</comment><comment><USER5> As a user, I can say that I _do not_ like social login. Also, page views is meaningless as is number of mins spent on your site.

</comment><comment><USER6> It seems like a false dichotomy to say that users either like or don't like social login - in my own thought process it completely depends on the site and the social login offerings available. I am often annoyed at sites which essentially block content if you don't sign in socially (f6s.com for example), but I am quite happy to avoid the hassle of creating a new username and password if 1) I don't really care about the site I'm visiting or 2) if I can sign in with a social identity which doesn't compromise what I am getting out of the site. I have several different twitter accounts for various projects and interests, so I might sometimes just use one of those that isn't my personal account to take a few steps out of a login process. The fact that the OP (gigya) allows dozens of social network login options (arguably too many ;) allows easy social login with secondary accounts.

</comment><comment><USER7> This is dumb. Show me pageviews and I'll show you a lot more "non-social login" people. I bet there are just as many (if not more) people spending the same amount of time on site as those logged in via social accounts. You've just got millions of bounces and quick looks bringing down the average for non-social.

</comment><comment><USER8> I think the real answer is that users like choices and freedom. You will gain the maximum amount of users if you allow them to use social logins if they so desire, or allow them to create a stand alone username/password for your site. By doing it this way you don't lose the users who rebel at social logins, but you still retain the users who prefer social login. The concept of "why not both?" isn't that hard to grasp. I understand that implementing/linking accounts from numerous social logins as well stand alone accounts is more difficult for the developer, but that's the price of maximum growth.

</comment><comment><USER9> We're working on bringing more social experiences to my employer's website, but the 3rd party interactions will eventually be only for the social login. Commenting, reviews, etc are all going to be done in-house this year, moving away from Gigya's tools. Their moderation tools are... subpar. And we need to have the comments locally anyway since we want to include them in the page's HTML for SEO purposes. Social login providers like Gigya certainly have their place - who wants to keep up with the constant stream of changes to FB, twitter, etc login APIs - but so many of them want to be the primary data source. This data is going to be the core of the platform going forward, so it's super important to own it all IMHO.

</comment><comment><USER10> The article is a little biased. Indeed, users who use social login spend more time on their website. It doesn't involve a causal link between time on website and social login: perhaps users are not the same, with different habits of navigation.

And users who use social login will perhaps spend more time if it's not provided.

</comment><comment><USER11> Interesting, but I would be more interested in causal relationships (I.e. growth rate or engagement delta). I think it's quite possible that social merely segments users by their already existing commitment levels: anonymous users have no commitment, social users want to make their commitment painless.

</comment><comment><USER12> People doesn't like to create new accounts to login. That´s all.

</comment><comment><USER13> The "social login" defeats the idea of net neutrality. It limits your privacy and gives corporations more power over your accounts. Not really my cup of tea.

</comment><comment><AUTHOR> When we implemented social login into our app. It did boost our conversions. Retention is another story but at least one goal is met right after.

</comment><comment><USER14> No they don't. I've run a ton of usability tests for clients walking through this very issue. Almost no one will even consider using it.

</comment><comment><USER15> Not this user.

<comment><USER16> They work for us so far for users' engagement to register. 96% of registration done through social APIs.

<comment><USER17> That is massively out of proportion with everyone else. I suggest you look into what you broke on your normal registration process.

<comment><USER18> It really depends on the application. For a casual use app, the social logins are the smallest barrier to entry for the type of user they are targeted at.

Any "friction", like having to enter a username, email and password is three more steps than a social login requires, costing these casual users.

So, I can believe it is highly effective for specific applications.

<comment><USER17> That's a lovely theory and all, but literally _nothing_ has a ratio of social:real registration anywhere approaching 24:1. Those casual use apps you are talking about get 70% social if they have really bad registration processes. Under 50% if they have decent registration.

</comment></comment></comment></comment></comment><comment><USER19> Do not vote this up. The title submitted to HN is everything that is wrong here.</comment>


My adventures in 3D printing: Prime number sieve - <AUTHOR> https://aperiodical.com/2019/11/my-adventures-in-3d-printing-prime-number-sieve/ <comment><USER2> I'll print this when I get home. It's hard to beat this print, though: [https://www.thingiverse.com/thing:1068443](https://www.thingiverse.com/thing:1068443) (a digital sundial with no moving parts).</comment>