What every software engineer should know about search - <AUTHOR> https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d <comment><USER2> Ex-Google search engineer here, now using hosted ElasticSearch extensively in my startup.

This is a really good overview. If there's one part I want to highlight, it's that you should _expect to spend a lot of time fine-tuning your ranking function for your particular product & corpus_. The default ElasticSearch ranking function kinda sucks. It was changed in ES 5.0 to Okapi BM25, which is the current academic state-of-the-art in non-machine-learned ranking functions. However, search is one field where the current academic state-of- the-art is at least a couple decades behind where things are in industry. When you use a service with good search that just works, chances are that there's been a lot of engineer hours devoted to identifying exactly which signals are most useful in your corpus, how they relate to subjective evaluations of relevance, and how to clean them up so that noise doesn't dominate the signal.

These signals are tightly tied to the corpus you're working with. One slightly unpleasant surprise striking out on my own was just how little the stuff I learned about websearch is relevant to working with different corpora (though I guess I should've predicted this, having seen how many different teams across Google Search operated and how all their signals differed from core websearch). I'll reiterate the article's point about _process_ and _method_ being more important than any specific function or algorithm; the most useful stuff I learned at Google was actually a process for taking a vague domain where I'm not sure what's out there, and then learning how to turn that into a system for getting useful information out of it.

And then once you've got users, you probably want to feed usage data back into it through some sort of machine learning. In that order, though; with all the hotness around AI, it's really tempting to think "Oh, I'll just read the latest LtR papers, Mechanical Turk a bunch of query evaluations, and feed the data into the algorithm", but without a deeper understanding of the particular corpus you're working with and how users want to access it, it's unlikely an initial LtR system will perform well enough to attract enough users to bootstrap the system.

<comment><USER3> > When you use a service with good search that just works, chances are that > there's been a lot of engineer hours devoted to identifying exactly which > signals are most useful in your corpus

I imagine at a place like Google, your "corpus" is just about everything under the sun, as are your queries. (i.e. less chance for a subject-specific tuning)

What happens then?

<comment><USER4> > I imagine at a place like Google, your "corpus" is just about everything > under the sun, as are your queries. > What happens then?

You accept the fact that a good search engine is inevitably going to be a multi-layered beast that requires constant effort to improve, with heuristics, machine learning, magic constants, special lists of words, and regular expressions.

After that, it's just a matter of picking a commercially useful objective function ("instant search", "query autocomplete", "maximize user satisfaction") and pointing large teams of well-resourced PhDs at it. Probably a whole weekend at least.

</comment></comment></comment><comment><USER5> I think somewhere along the way work on search just devolved into handling millions of queries at a time. It could have been so much better.

If you ask a "search expert" today what's he is trying to fix, he will say something related to scaling.

If you asked an expert in the 80's or 90's they would talk about query complexity and NLP i.e. Who were the four semi-finalists of last years Wimbledon? And you would get back 4 names.

Today you will get back a result saying 50 million pages were found in 1 second. The page may or may not contain the 4 names if you wade through 17 popups. Nobody questions how brainless this is.

<comment><USER6> Well, I hear a lot of people complaining that the results on DuckDuckGo are still worse than on Google, even though both search-engines produce results within a second. And these are people that really want to quit using Google for privacy reasons. I never hear people complaining that a search is slow. So I do think that _search-quality_ is where the competition is happening.

Edit: But, I agree, we don't often see any good HN posts or papers about search-quality. Perhaps a case of trade secrets?

<comment><USER7> The quality/ranking code and algorithms are definitely one of the "holy grails" at Google. That code and documentation is locked down tight, compared to the 99.9% of the rest of the codebase that's open to any engineer.

</comment></comment></comment><comment><USER8> I used to work on textual Information Retrieval (which is quite related to search). One of the key contributions of my PhD thesis was a statistical method to extract concepts from texts [0].

Surprisingly enough, with those concepts, Tf-Idf [1] was quite good to extract keywords from documents which allowed us to build document descriptor tables which can eventually be used for document search [2]. We also built a small prototype that allowed us to retrieve results of specific areas of documents where the searched concepts were more "valuable" than on other parts of the document [3].

It is a very interesting subarea of AI/NLP which unfortunately doesn't seem to attract much interest.

Since the article also talks a bit about Wikipedia dumps datasets, here's a tool that I've created to build textual corpora: [https://github.com/joaoventura/WikiCorpusExtractor](https://github.com/joaoventura/WikiCorpusExtractor)

[0] - [http://www.sciencedirect.com/science/article/pii/S1877050912...](http://www.sciencedirect.com/science/article/pii/S1877050912001251)

[1] - [https://en.wikipedia.org/wiki/Tf%E2%80%93idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)

[2] - [https://link.springer.com/chapter/10.1007/978-3-642-40669-0_...](https://link.springer.com/chapter/10.1007/978-3-642-40669-0_42)

[3] - [https://link.springer.com/chapter/10.1007/978-3-642-40669-0_...](https://link.springer.com/chapter/10.1007/978-3-642-40669-0_44)

<comment><USER9> How does your research compare to word2vec? That seems to be getting some attention

<comment><USER8> Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space.

This seems quite similar to LSA (Latent Semantic Analysis). In LSA, a word is represented by a vector (list / array) of values which represent the number of occurrences of the word in a document. For instance, the list [0, 5, 9, 10] for the word "aeroplane" means than "aeroplane" occurs zero times in D1 (document #1), 5 times in D2, etc. Unlike LSA, word2vec seems to have the vectors values implicit in the neural network weights (?).

The interesting thing on these kind of approaches is that words that share similar contexts, tend to occur in the same documents. In other words, the vectors tend to have approximate values. For instance, it is natural that "aeroplane" tends to occur in the same set of documents as "aeroport".. Tf-Idf [2] does something similar, so these vectors are mostly a data representation of single words.

As for my research, the basis of it was to be able to quantify how "specific" a word or multiword is. The idea is that the more a word/multiword occurs in different contexts (in my case, near different words) the less specific it is. Being able to identify the specificity of words allowed me to know which words/multiwords were probably concepts and which ones weren't. Then, with these concepts I was able to use Tf-Idf to extract document keywords and later I was able to infer broad relationships between concepts (like in if two words tend to co-occur in the same documents and same parts of documents, they are somewhat related).

So, how does word2vec compare to my research?

\- I prefer simple statistical approaches like the one I did (which is just counting frequencies and dividing numbers, etc.) than the black-box approach of neural networks. I can tell you why my research provides good results (explained above) but if you read a follow-up paper on word2vec [3] this is what you can find in section 4: "Why does this produce good word representations? Good question. We don't really know".

\- Also, bag-of-box approaches like word2vec and LSA don't tend to represent multiwords (such as "President of the United States of America", which leaves out lots of information on texts.

\- Finally, I think there's more possible applications after you know which words/multiwords are informative. With word2vec and LSA you are only representing something similar to frequencies of words in documents, which mostly allows for Information Retrieval-like applications. On the other hand, one of the applications I did with the extracted concepts was to implement a prototype which allowed us to find the definition of a concept on the corpus. The idea was that the definition of a concept was the paragraph where that concept occurred many times but also used other concepts to help the first one to be defined. I don't know how you could do something like that using something like word2vec.

But as I said above, I am not that knowledgeable about word2vec, so I may miss something..

[0] - [https://en.wikipedia.org/wiki/Word2vec](https://en.wikipedia.org/wiki/Word2vec)

[1] - [https://arxiv.org/abs/1301.3781](https://arxiv.org/abs/1301.3781)

[2] - [https://en.wikipedia.org/wiki/Tf%E2%80%93idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)

[3] - [https://arxiv.org/pdf/1402.3722.pdf](https://arxiv.org/pdf/1402.3722.pdf)

</comment></comment></comment><comment><USER10> Doug Turnbull here, author of Relevant Search. I'm in love with this post. Thought I'd point out I added a few pointers that stream-of-conscioused into my head as I read this!

[https://medium.com/@softwaredoug/this-is-a-fantastic- post-e9...](https://medium.com/@softwaredoug/this-is-a-fantastic- post-e9caae910334)

</comment><comment><USER11> I have some experience in developing IR/search software ( [https://github.com/phaistos-networks/Trinity](https://github.com/phaistos- networks/Trinity) ) and, the way I see it, it all comes down to accepting the following premises:

\- It's all about the relevance models. Specifically, BM25 and TF-IDF which are widely used by e.g Lucene and variants just won't do. Specifically, they only word for large enough documents anyway.

\- Indexing and Search algorithms and practices haven't changed much in decades (though some novel ideas have been introduced not long ago). Lucene's index encoding is compact and facilitates fast access, but even compared to the one made available by Google which is arguable simpler in design, doesn't result in more than around 5% reduction in index size and postings list access time (according to my measurements that is). Posting lists intersections, unions and other such operations implementations are pretty much common across IR systems as well, with little room for improvement.

To get great results, you need really great relevance models(1), a great query rewrite system(2), and, because query rewrites usually expand a query to include multiple disjunctions (OR terms and phrases), your search engine needs to be particularly efficient at handling those(3).

You also need to care for spelling suggestions and personalisation/content biases and factors, but those are secondary concerns.

</comment><comment><USER12> Very interesting article. However I really hate the floating menu and footer on the site. I browse on a small laptop and I really don't like the obscured viewing

<comment><USER13> Seemed OK with Reader View in Firefox.

<comment><USER12> It is OK with Reader View , it's not really the point though is it?

</comment></comment></comment><comment><USER14> What's with the random unicode characters spread throughout this article? Several headings and sentences are prefixed oddly with an exclamation mark ("\u2757\ufe0f"), or diamond ("\U0001f537").

<comment><USER15> There's an "Emoji Legend" near the top of the article.

It's a nice idea, flagging part of the text with coarse semantic meaning. However, I find I don't trust the writer's judgement to decide what is important for me; I must read the whole thing anyway.

<comment><USER14> I've never encountered an "Emoji Legend" before. I instinctively skip over the word "emoji" so I missed that section.

</comment></comment></comment><comment><USER16> This was a great read and incredibly helpful. As someone that has been working on a search system for three years, it is incredibly difficult. There is a ton of variables and the cost of being wrong early costs you dearly later. There is a lot in this article that resonates with me. Some stuff I know, some stuff I wish I had known, and some stuff that was new. Great read!

</comment><comment><USER9> I think search problems are a ton of fun, because there are a lot of opportunities for creative solutions to problems.

One thing I'd add is a book called "Relevant Search" \- I have a hobby project to search talks, and the book helped me out a ton ([https://www.findlectures.com](https://www.findlectures.com)).

<comment><USER10> Thanks for sharing my book ;) I actually just replied to this (fantastic) post with some extra pointers I've discovered on my search projects! [https://medium.com/@softwaredoug/this-is-a-fantastic- post-e9...](https://medium.com/@softwaredoug/this-is-a-fantastic- post-e9caae910334)

</comment></comment><comment><USER17> Great post. I'm working on another open source search engine on top of Redis ([http://redisearch.io](http://redisearch.io)), mostly focused on index building and serving, and real-time updates of the data. The part about queries being highly varying is extremely challenging. You have to deal with simple "foo bar" queries and complex queries with intricate filtering and crazy stuff (I have a user doing an AND intersection of 17 OR unions, each of 32 terms, while checking the ordering of term offsets!). It's super fun to work on this stuff.

Another recommended book that was not mentioned in the post: Search Engines: Information Retrieval in Practice [https://www.amazon.com/Search-Engines- Information-Retrieval-...](https://www.amazon.com/Search-Engines-Information- Retrieval-Practice/dp/0136072240)

<comment><USER18> If you allow users to execute arbitrary queries, how do you protect your server resources from being totally consumed by a small subset of your users?

<comment><USER17> In terms of redis being single threaded, we put a lot of work into making the search concurrent, see here for details: [https://redislabs.com/blog/making- redis-concurrent-with-modu...](https://redislabs.com/blog/making-redis- concurrent-with-modules/). Basically we rotate between long queries and short queries, making long queries run slower but not block short ones. Right now rate limiting is left to the user and not solved at the engine level.

<comment><USER18> This is about the complexity of queries (which, without knowing your system, may be unbound) - rate limiting will not help.

<comment><USER17> Yeah, I was also thinking about adding execution cap to a single query, that's not hard to add as there is a simple scheduler that checks their progress constantly. So just terminating a long running query is simple.

</comment></comment></comment></comment></comment><comment><USER19> I was also going through the similar problem of search. How the open source has multiple products for various stuff, but for search and index creation, we just have lucene and tools on top of that like solr or elasticsearch. Not sure why we are not innovating in the search space with new algorithms, systems, open source systems/tools and so on.. Almost all search engines are on top of lucene and inverted index. Why is no one able to reverse engineer google/bing capabilities ?

<comment><USER2> Separate out the concepts of "search infrastructure" (how documents and posting lists are stored in terms of bits on disk & RAM) and "ranking functions" (how queries are matched to documents).

The former is basically a solved problem. Lucene/ElasticSearch and Google are using basically the same techniques, and you can read about them in _Managing Gigabytes_ [1], which was first published over 2 decades ago. Google may be a generation or so ahead - they were working on a new system to take full advantage of SSDs (which turn out to be _very_ good for search, because it's a very read-heavy workload) when I left, and I don't really know the details of it. But ElasticSearch is a perfectly adequate retrieval system, and it does basically the same stuff that Google's systems did circa 2013, and even does some stuff better than Google.

The real interesting work in search is in ranking functions, and this is where nobody comes close to Google. Some of this, as other commenters note, is because Google has more data than anyone else. Some of it is just because there've been more man-hours poured into it. IMHO, it's pretty doubtful that an open-source project could attract that sort of focused knowledge-work (trust me; it's pretty laborious) when Google will pay half a mil per year for skilled information-retrieval Ph.Ds.

[1] [https://www.amazon.com/Managing-Gigabytes-Compressing- Multim...](https://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia- Information/dp/1558605703)

<comment><USER20> > The former is basically a solved problem.

That's a bit of a stretch :) The high-level architecture is quite mature and stable, but there's still a lot of research, both in academia and industry, on the data structures to represent indexes, on query execution (see all the work on top-k retrieval), and distributed search systems (for example query- dependent load balancing, novel sharding methods).

<comment><USER11> It's true that different index codecs are being designed with different tradeoffs (index size vs postings lists access time and cost), but the work is very incremental IMHO, no huge advances to speak of. Also, can you talk about the top-k challenges you mentioned? Priority queues are not optimal enough ?

</comment></comment></comment></comment><comment><USER21> TFA mentions case when documents are coming from potentially adversarial sources. Any tips for handling relevance when users are trying to game the system?

</comment><comment><USER22> There is a 100% chance I'll end up referencing this article in the future.

For most needs personally after having learned the ins and outs I still have a soft spot for sphinx so was happy to see honorable mention in there. It can scale really cheap and is a tank that never has downtime. It is closer to metal but if you look at how Craigslist does it you can do the fancy scaling things still

</comment><comment><USER23> This is indeed a great article about building a practical search system. However the title is a bit aggrandizing. Not every software engineer actually needs to know this stuff, just people who are building products with search capabilities.

</comment><comment><USER24> Just. Great.</comment>


New Jersey School Eases Pressure on Students, Baring an Ethnic Divide (2015) - <AUTHOR> http://www.nytimes.com/2015/12/26/nyregion/reforms-to-ease-students-stress-divide-a-new-jersey-school-district.html <comment><AUTHOR> TLDR The school district, was facing a crisis. Its students were overburdened and stressed out, juggling too much work and too many demands.

Superintendent urged parents to join him in advocating a holistic, "whole child" approach to schooling that respects "social-emotional development" and "deep and meaningful learning" over academics alone.

Superintendent suggestion revealed a divide roughly along racial lines:

On one side are white parents who has come to see the district's increasingly pressured atmosphere as antithetical to learning. "My son was in fourth grade and told me, 'I'm not going to amount to anything because I have nothing to put on my résumé,' " one parent said.

On the other side are parents like Mike Jia, one of the thousands of Asian- American professionals, who said Superintendent reforms would amount to a "dumbing down" of his children's education.</comment>


How to request for unban a banned domain in hackers news - <AUTHOR>

Hi Guys, I have been using hacker news from long time and I like the links here and even benefited from some exposure in past, but due to my ignorance I got my blog http://javarevisited.blogspot.com banned by over submitting post. Its more than a year or so but still its banned and due to which no one can share my post in hackers news. Is there any way I can request moderator to unban a banned domain ? I know the rule now and ensure that I will not submit any post myself.<p>Thanks Javin <comment><AUTHOR> no reply :(</comment>


Ask YC: Anyone done any work with Processing? - <AUTHOR>

I'm about to jump in and do some experimenting. Anyone have anything to share? <comment><USER2> There's a new Oreilly book out on it called 'visualizing data'.

I'm only a few chapters in, but it's quite good so far. Covers the basics, a few good sized 'real' projects, and quite a bit on integrating it with java.

</comment><comment><USER3> I did a couple of things with it; it's fun, like turtle graphics for grownups.

Don't rush out and buy the book right away; the website has plenty of content to get a feel for it: <http://processing.org/learning/basics/index.html>

</comment><comment><USER4> I've only poked at it, but some pals are knee deep into it. They produce truly beautiful interactions with it.

Check out <http://www.processingblogs.org/>. It's a blogblender of active work on processing.

</comment><comment><USER5> I've had great experiences using processing. It's been sweet for prototyping multimedia stuff - live video processing, physical computing, data visualization, toys. The libraries make it quick and painless to get graphics floating around.

</comment><comment><USER6> I've been involved with the project off and on for a few years now. (I wrote the Interfascia GUI library for Processing.) As an interface designer, I find that it makes a great tool for rapid prototyping.

</comment><comment><USER7> I don't have anything specific to share, but my past experiences were pretty good. It's definitely great for doing toy types of things (I wrote a 3d clock, for instance, that changed color based on where the hands were). There's also great library support and a pretty big community. If I happen to dig up that clock I'll post it.

<comment><USER7> Easier than I thought to find it: <http://apgwoz.com/processing/ColorMixClock3D/applet/>

Source is [http://apgwoz.com/processing/ColorMixClock3D/ColorMixClock3D...](http://apgwoz.com/processing/ColorMixClock3D/ColorMixClock3D.pde)

</comment></comment><comment><USER8> I have nothing to share, but check out this site it has a bunch of different processing stuff, <http://createdigitalmotion.com/tag/processingorg/> the best site is their site and the forums, just cruise them and you'll find all sorts of good stuff

</comment><comment><USER9> Great stuff. It's very easy to dive in. The editor is clunky, though.

I'm currently working on a series of programs that process webcam input and render "magic mirrors" made out of words, wood, swimming fish, etc.

</comment><comment><USER10> I've messed around with it for some classes that had projects in it at GaTech, but that's about it.</comment>


Beta Flash Player 'Gala' supports H264 video hardware decoding on Mac OSX 10.6.3 - <AUTHOR> http://labs.adobe.com/technologies/flashplayer10/gala/

<comment><USER2> _To date, support for hardware accelerated H.264 video playback in Flash Player has only been possible on Windows PCs and x86-based netbooks_

I love how they try to make it look like they support two different platforms when they're both exactly one and the same..... and they forgot to add "that run Windows" at the end of the sentence, because netbooks running Linux or OS X are _not_ hardware-accelerated.

Shame, really.

<comment><USER3> I don't think they're making it sound like they're supporting two platforms, they're just trying to exclude non-x86-based netbooks..

<comment><USER4> So couldn't they have taken their cue from the PC comment and said:

"To date, support for hardware accelerated H.264 video playback in Flash Player has only been possible on Windows PCs and _Windows_ -based netbooks"

<comment><USER2> Or simply "On Windows-based PCs and netbooks"

</comment></comment></comment></comment><comment><USER5> From the release notes: "This framework is only supported by Apple on Mac computers equipped with the NVIDIA GeForce 9400M, GeForce 320M or GeForce GT 330M graphics chips."

Unfortunately my three year old MBP is too old :( If someone with a newer Mac wants to report back on how well this works, here is a good test video: <http://www.youtube.com/watch?v=4N2YWRJ-ppo>

</comment><comment><USER6> OBS: The hardware acceleration is not part of the 10.1 release. It will only be provided in an update _after_ the 10.1 release.

<comment><USER7> Right -- 'Gala' _is not_ 10.1. It's a separate preview release.

</comment></comment><comment><USER8> You can already play H.264 with hardware support using QuickTime on the Mac so there is no reason to use a Flash player. The only reason to use a Flash video player on all platforms is laziness.

<comment><USER9> Given that there are Flash players which simply wrap around H.264 files, there's no reason not to use <video> on browsers which support it, and fall back to Flash on browsers that don't.

</comment></comment><comment><USER10> Question is 'Does it actually work?'

And if it does, What's the actual improvement in performance?

EDIT: I'm not being sarcastic, I'm really asking... DOES IT WORK?

<comment><USER11> I'm still getting 60% (of one core) CPU utilisation playing HD youtube videos. Playing HTML5 video tag version of the same video, its only about 20% utilisation. As far as I can remember, 60% is about what it was at before, so I'm not really seeing any improvement.

One of the videos I used to test: <http://www.youtube.com/watch?v=rQ_z-_4fF4Y>

I'm using Safari, and also ClickToFlash for switching between Flash and HTML5 HD video on Youtube. Hardware is a 2009 Macbook Pro 2.53ghz.

<comment><AUTHOR> Do you see the white square?

Some of youtube's videos are FLVs, not H264. This is for H264 only.

<comment><USER11> Correct me if I'm wrong, but aren't all Youtube HD videos either H264 (.mp4) or H264 in an .flv container? I don't see why a particular container format would prevent hardware decoding of the H264 video stream.

</comment></comment></comment></comment><comment><USER12> I would rather have proper 64bit support for linux....

<comment><USER13> It's funny. Every blog post Adobe makes about improving Apple support has complaints like this in it. But every one about improving Linux support has complaints about Apple support.

(Although lately the Apple complaints have changed from "make it suck less" to "screw you we don't need you".)

</comment></comment><comment><USER14> not on my slightly older imac's graphics card.</comment>


Beyond Klout: Better Ways To Measure Social Media Influence  - <AUTHOR> http://readwrite.com/2012/10/24/beyond-klout-better-ways-to-measure-social-media-influence

<comment><USER2> _"But perhaps Topsy's biggest selling point is what it doesn't do: the service doesn't reveal user scores. That simple, easy to read score - Klout's calling card and biggest selling point - is also its biggest weakness. Obssessing over that number can quicly become addictive - and separated from geniune social influence."_

Absolutely. This has been known for a long time:

<http://buildingreputation.com/writings/2010/02/on_karma.html>

</comment><comment><USER3> Do we actually need metrics for 'social media influence?' How is this any different from an e-popularity contest? How does it change online communication for the better? It seems like it is more a play for advertising.

I grew up in the age of the Internet being this nerdy and niche thing where what we said online didn't quite have the levity of offline speech. That doesn't mean it wasn't important; but instead, the disconnected and semi- anonymous nature of the Internet made it such that the boring power hierarchy was not as much of a factor in communication as it would be in real life.

Now, it seems like we want to re-create that using questionable metrics. Remember, if there's one thing humans always get right, it is using metrics properly!

</comment><comment><USER4> Last week, a friend got himself onto the top 30 Klout influencers list for some conference he wasn't even at. He simply started tweeting lame general platitudes about business with the appropriate hashtag, and by 10 AM he was a top influencer at the conference. I doubt he was even dressed by 10 AM.

The only thing Klout measures is how hard somebody is willing to work to increase their Klout score.</comment>


Steve Jobs Explains The Blue Box - <AUTHOR> http://devour.com/video/steve-jobs-explains-the-blue-box/

<comment><AUTHOR> "The fact that two teenagers could build this box for $100 worth of parts and control hundreds of billions of dollars of infrastructure, in the entire telephone network, from Los Altos and Cupertino California... that was _magical_. And experiences like that taught us the _power of ideas_."

"If we hadn't have made blue boxes, there would have been no Apple."</comment>


Adsuck – a small DNS server that spoofs blacklisted addresses - <AUTHOR> https://opensource.conformal.com/wiki/adsuck <comment><USER2> I use `dnsmasq` and configured it to return `NXDOMAIN` for a giant list of ad domains I found. It doesn't have feature-parity with this, but the software was readily available for my platform, which was nice.

<comment><USER3> Tool to automate it: [https://github.com/jakeogh/dnsgate](https://github.com/jakeogh/dnsgate)

<comment><USER4> dnsmasq can read from /etc/hosts so you can automate it just using `curl` (or other standard UNIX tools):

       # the first command is a one time only job
       cp -vip /etc/hosts /etc/hosts.default

       # the following is the automation.
       curl http://someonewhocares.org/hosts/hosts --silent > /etc/hosts.sowc
       cat /etc/hosts.default /etc/hosts.sowc >> /etc/hosts

The other benefit of this is you can then just do a `dnsmasq reload` (since the hosts file is an external configuration file) rather than the full `dnsmasq restart` that dnsgate performs.

<comment><USER5> Silently putting data fetched over unauthenticated HTTP into your hosts file doesn't sound like a great idea.

<comment><USER4> Good point. Sadly that's pretty much how all adblockers keep up-to-date (albeit at least with TLS). Though pragmatically even if the update was manual, you still need to place some trust in the site when dealing with files of that size. Granted it would be better if those resources served TLS, but that would only prevent MITM and not the file being dodgy from source (eg site hacked).

Thinking a little more about the problem you raised, I guess you could grep it for IPs that aren't localhost (ie exit the shell script before writing the new hosts if `grep` returns a zero exit code). Aside that, I'd genuinely be interested in any ideas you have for keeping the file updated in a secure way (if at all possible).

<comment><USER6> [https://gist.github.com/Holzhaus/ed4ac1675a57f11c3057](https://gist.github.com/Holzhaus/ed4ac1675a57f11c3057) contains a simple reverse grep for doing this.

You should still have a modicum of trust towards the host you're pulling from, as they can also insert something to potentially break the parsing code, which opens up a whole 'nother host of issues.

</comment></comment></comment></comment></comment></comment><comment><USER4> This is a nice idea, but it's really easy to replicate using dnsmasq and a custom hosts file (I use "Someone Who Cares"[1], but others exist as well, [2][3])

dnsmasq can read from the hosts file and the whole thing is automatically updated once a week with the latest bad domains (a simple `curl` request is all that's needed)

This method also ports much easier to other *nixes and even Windows since dnsmasq already has wider platform support.

[1] [http://someonewhocares.org/hosts/](http://someonewhocares.org/hosts/)

[2] [http://winhelp2002.mvps.org/hosts.htm](http://winhelp2002.mvps.org/hosts.htm)

[3] [https://github.com/StevenBlack/hosts](https://github.com/StevenBlack/hosts)

<comment><USER7> It's a lot less resource intensive not to use a hosts file. This might not be a concern for people with modern machines, but having a hosts file with 12,000 lines in it does take a certain amount of processing.

If I might mention my own site for a minute, I maintain a list of ad server (and tracking server) hostnames: [http://pgl.yoyo.org/adservers/](http://pgl.yoyo.org/adservers/)

You can view the list as a dnsmasq config file, a BIND config file, and a bunch of other formats.

<comment><AUTHOR> Also, running your own DNS server means you can do this for _every_ device, even those for which you can't easily access the HOSTS file or perhaps don't even have such a facility (locked-down mobile devices, embedded systems, etc) and not have to worry about keeping multiple copies of HOSTS files in sync and updated.

Although I don't have such devices in my network, I've heard that others do this to their "smart" TV/media box/creepy home surveillance gadgets.

<comment><USER7> Yes, absolutely! It's way easier to configure your wifi router (or cable modem, or whatever acts as the DHCP server on your network) to use a local DNS server that blocks ads. Then ad blocking just _works_ , whatever you connect to your network.

<comment><USER8> is the best way to do this to run something like DD-WRT? Or is there a way to accomplish this with stock router firmware? I ask because I recently tried to install DD-WRT but ran into issue and had to revert to my stock firmware.

<comment><USER4> Personally I run dnsmasq on my file server (separate FreeBSD jail), but before then I was running it on my Asus router with pretty much the stock firmware. So I don't think there is a "best" approach specifically - just whatever works for you.

What's your router model?

<comment><USER8> Very late in seeing this comment! It's a Netgear WNDR3700v4. DD-WRT __is __supported on it, and i did successfully get it installed, but I had no connection to the internet with it. There was a wiki page for my specific router model with instructions which I followed. I went back & forth with people on their forums and they were pretty much telling me "oh yeah, don't install _that_ version that the instructions tell you to!" and "well yeah, this whole project is beta, so what do you expect?" All of that combined kind of turned me off to this project from now on :/

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Take a look at PowerDNS Recursor or Unbound, both of which have scripting capabilities than enable this type of behavior and much more (Lua for PowerDNS, Python for Unbound).

I'd say a script for any of these would be a better choice.

<comment><USER7> I just yesterday added an Unbound format for my list! [http://pgl.yoyo.org/as/#unbound](http://pgl.yoyo.org/as/#unbound)

<comment><USER10> I used just 'local-zone' entry. Unbound answers NXDOMAIN for those domains, and it prevents next HTTP request to 127.0.0.1. I'm not sure what approach is better.

<comment><USER7> To be honest, I don't use Unbound myself - I was just going off what was sent to me as an example of the format to use.

</comment></comment></comment></comment><comment><USER11> Why do you need to spoof as opposed to block domains?

<comment><USER4> In the case of this HN submission, they're the same thing. From what I can gather, Adsuck is essentially just a DNS forwarder that sends NX DOMAIN for blacklisted domains but forwards the DNS requests for all other DNS lookups. So "spoof" is a little misleading since it's actually doing the job of a normal DNS forwarder - albeit tuned with privacy in mind. So I think it's fair to say this method could more accurately be defined as "block[ing] domains" rather than "spoofing DNS". However I'd welcome a correction if I'm wrong.

</comment></comment><comment><USER12> I wrote a simple python script to blacklist bad domains for unbound [https://github.com/khainebot/DNS-Unbound-Blocklist- Downloade...](https://github.com/khainebot/DNS-Unbound-Blocklist-Downloader)

</comment><comment><USER13> Similar service [http://ba.net/adblock](http://ba.net/adblock)

</comment><comment><USER14> Is there any upside of blocking using a [browser] plugin compared to this ?

<comment><USER15> If you're Joe User, it's much easier to install a browser plugin or hosts file than it is to set up a DNS server of your own, route your machines to it (even internally), configure it properly...

<comment><USER16> Personally, it's also less annoying to shut off uBlock/etc per site than to go into hosts, commenting out a domain, then uncommenting when I'm done when visiting sites that break when you cut off google analytics or doubleclick

</comment></comment></comment><comment><USER17> the only issue with all these dns-based blacklists is that, you normally can use its IP directly to fully bypass them.

<comment><USER17> plus it applies to all the domain, i.e. it treats all the machines the same, sometimes you may not want that.

</comment></comment><comment><USER18> I like these ideas, but i don't like setting up a complete operating system just for DNS purposes.

I know i could add it to other existing servers, but i'd really love a really small independent virtual machine which sole purpose would be DNS.

<comment><USER19> Take a look at [https://github.com/longsleep/adblockrouter](https://github.com/longsleep/adblockrouter) \- i hacked this together some months ago and use it since to provide DNS blacklisting to dnsmasq running on OpenWRT. If your OpenWRTC router has wget and ssl, it even runs directly on it.</comment></comment>


How Web mail providers leave door open for NSA surveillance - <AUTHOR> http://news.cnet.com/8301-13578_3-57590389-38/how-web-mail-providers-leave-door-open-for-nsa-surveillance/

<comment><USER2> The general consensus around here seemed to be that "most" webmail providers did use encrypted SMTP, interesting to see that refuted. The "fat pipes being safe" rationale has been thrown around for a long time, definitely cast in a new light post-Room 641A, fiber tapping leaks, etc. We really do make it easy, don't we?

</comment><comment><USER3> factoid - fastmail.fm uses SMTP-TLS</comment>


Tesla to cut its workforce by 7 percent to ramp up Model 3 production - <AUTHOR> https://www.cnbc.com/2019/01/18/tesla-to-cut-its-workforce-by-around-7-percent.html <comment><USER2> Layoffs are always the sign of a happy, healthy, and sustainable work environment! Mr. Welch told me to embrace competition amongst the lower ranks, because clearly management can never do anything wrong.

/s</comment>


Uruk – Turkish as a Model for the Black Speech? - <AUTHOR> http://forums.theonering.com/viewtopic.php?t=82210 <comment><USER2> At a guess, I would say no, Turkish was not a model. At least if we limit ourselves to language.

It is telling that the linked thread throws up three completely different near-eastern languages as candidates. Hittite a (distantly) Indo-European language, Turkish - a turkic language and Hurrian which is something completely different.

Even if there is some "non-genetic" similarity among all these near eastern languages, then consider that _all_ of Tolkien's non-elvish languages sound vaguely near-eastern to my untrained ear. This includes even the Common Speech, which sounds completely different from the English that it was translated into.

But if we broaden our view from language to culture, then yes, I suspect that the Ottomans loomed large in the minds of Christian Europeans for so many centuries that they (or rather the western caricature of them) has influenced many evil empires of modern fantasy.

<comment><USER3> As a Turk - The Yörüks are not necessarily obscure but not really very well known or oft-mentioned either like other ethnic groups like the Laz, Kurds, Circassians, Turkmens, etc. So it would be surprising to hear that many foreigners ever even heard of them at the time, let alone Tolkien. Then again, I guess I wouldn't put it past him to have encyclopedic knowledge of mythologies and histories.

More: [https://en.wikipedia.org/wiki/Y%C3%B6r%C3%BCks](https://en.wikipedia.org/wiki/Y%C3%B6r%C3%BCks)

<comment><USER4> I was going to say that, to my ears, the Black Speech of the inscription on the One Ring sounds a lot like Turkish, but I'm not a Turkish speaker [1]. So I looked for a few examples of (modern) Turkish and I have to say that- maybe not, after all?

Compare:

Ash nazg durbatulûk, ash nazg gimbatul, Ash nazg thrakatulûk agh burzum-ishi krimpatul. [2]

With:

Bütün insanlar hür, haysiyet ve haklar bakımından eşit doğarlar. Akıl ve vicdana sahiptirler ve birbirlerine karşı kardeşlik zihniyeti ile hareket etmelidirler.

Not even close.

But of course, there's all this stuff about all the southerners teaming up with Sauron and so on, so I always thought it was a fair assumption that there was a bit of north/south good/evil divide in old J.R.R's writings.

__________

[1] Except of course for the many Turkish loan words in my language, Greek, which interestingly are very often food-related: baklava, kataifi, halva, pastourma (pastirma), peinirli, yaourti, soropi, karpouzi etc etc.

[2] I even thought the "û" was supposed to sound like "ü".

<comment><USER5> Try an older Turkic text like Uyghur inscription:

nama but : nama darm : namo [sang] altun önglüg y(a)ruk yaltrık -lığ kopda kôtrülmiş nom iligi atl(ı)ğ nom bitigde kılınç adartmakın ôçürmek atl(ı)ğ bişinç bölük üçü[nç tegz]

<comment><USER3> This is more like it! Still, that inscription seems to have vowel harmony, and the uruk-hai text doesn't ("gimbatul", "krimpatul"). Oh well, it's fantasy after all.

Also, re: food: "dolmades"! :-) The interesting thing about words like "karpouzi" is that again, the o and i don't harmonize because the former is a low vowel, and the latter is a high vowel, and that's a pretty strong clue that it's been changed, because there are very few if any words in Turkish that don't follow vowel harmony. We say "karpuz". Then again, who knows who had it first and who changed it after! :P

<comment><USER4> Aw, dolmades, how did I leave that out :D

I didn't know about vowel harmony, but it seems we have many, many words from Turkish in Greek that are just the Turkish word with an "-i" stuck at the end: pasoumi, ati, ginati, gieri ("gut", from cigeri), kapaki, kaloupi etc etc. I guess that just brings the sound of the word more in line with what we're used to in the rest of the Greek language.

In Greek we don't have the long-"u" that Turkish has (incidentally, that long "u" sounds very similar to the French "u" to me, French being my second language) so we basically change it to the shorter, rounder "ou", hence "karpouzi" instead of "karpuz". Sometimes we just shorten and round-out an "o" sound also: we say "boudroumi", for "dungeon", coming from "bodrum" I reckon (the old joke is that "Boudroumia & Baboules" is the Greek version of "Dungeons & Dragons"; I don't know if "baboulas" comes from Turkish- it means "bogeyman").

It's funny because not speaking Turkish I have no idea what some of those words really mean, or how they 're pronounced in Turkish, or even sometimes if they're actually Turkish at all. I know that, to my ears they sound not quite Greek, and I can't really figure out their meaning, so they're probably loan words and therefore most likely Turkish (although there have to be plenty that come from Albanian).

I quite like that btw, hope you don't want them all back now :P

</comment></comment></comment></comment></comment></comment><comment><USER6> From the precious little I know about Mongolian language, I'd say that Black Speech shares a number of phonetic similarities with it.

You can read and listen to samples of speech about 3/4 down this page: [http://www.omniglot.com/writing/mongolian.htm](http://www.omniglot.com/writing/mongolian.htm) (Search for "Khün bür törzh mendlekhee erkh čölöötei")

</comment><comment><USER7> Seems like a pretty straightforward 'no', given what Tolkien said about it himself:

Quoted in [https://en.wikipedia.org/wiki/Black_Speech](https://en.wikipedia.org/wiki/Black_Speech)

"The Black Speech was not intentionally modeled on any style, but was meant to be self consistent, very different from Elvish, yet organized and expressive, as would be expected of a device of Sauron before his complete corruption. It was evidently an agglutinative language."

Turkish is agglutinative but so are zillions of other unrelated languages. Zillions of languages might have a 'uruk'-like word in them.

</comment><comment><USER8> Uruk is a derogatory word in Russian language commonly used to discribe immigrants from Muslim countries.

</comment><comment><USER9> The fourth comment in that thread has an extensive quote from JRRT which lays the whole thing to rest but don't let that stop you ...</comment>


Stay Strong - <AUTHOR> I don't know who needs to hear this in the world right now, I know this isn't the place to post this, I know some of you are up to solving some big problems, but if someone here needs to hear this today – Stay Strong! I have never seen the world battle something like COVID-19 and never have I ever been so scared of what is ahead.<p>I know science will work this out, but till then take care of yourself and your family. <comment><USER2> Thank you. I'm a scientist (work on some AI tech that isn't directly applicable to the covid fight) and it makes me realize how critical research is to our society and species. I am trying to up my personal game professionally.

I'm also trying to develop acceptance. I have seen things over the last few days that I never expected to see in my lifetime (seeing all food items, thermometers sold out, etc.). As technology professionals, we are so lucky. I realize this so clearly now and won't take it for granted.</comment>


PowerShell DSC for Linux is now available - <AUTHOR> http://blogs.msdn.com/b/powershell/archive/2015/05/06/powershell-dsc-for-linux-is-now-available.aspx <comment><USER2> Keep in mind that this is "PowerShell DSC" which is a state management tool that leverages PowerShell, _not_ PowerShell itself. [0]

If you're using only Linux, it's difficult to see how this could provide much value add over more mature tools (Puppet, Chef, Ansible, etc etc). However, if you have a mostly Windows centric environment and need to manage a few Linux nodes, this may be become a good option.

At this stage PowerShell DSC has a vaguely "side project" feel to it. It's difficult to opt for something that's relatively unknown and immature to manage your infrastructure, but it seems to me that with enough attention this could be a very useful tool.

[0] [http://blogs.technet.com/b/privatecloud/archive/2013/08/30/i...](http://blogs.technet.com/b/privatecloud/archive/2013/08/30/introducing- powershell-desired-state-configuration-dsc.aspx)

</comment><comment><USER3> This is great. I just wish PowerShell itself becomes available on Linux as well. Although there's a Mono port already (Pash [0]), it's not identical.

[0] [https://pash-project.github.io/](https://pash-project.github.io/)

</comment><comment><USER4> Why ? Bash and Fish are far better that powershell</comment>


Consumers can scan bank deposits at home - <AUTHOR> http://news.yahoo.com/s/ap/20080207/ap_on_hi_te/scanning_checks;_ylt=AqyraYYd_M_TJ8KwPRHBs8QjtBAF

<comment><USER2> My bank (USAA) has offered this for about a year. It's really convenient and awesome, more banks should do it.</comment>


Click on a few dots and our program will guess your age - <AUTHOR> http://kgajos.eecs.harvard.edu/ag/

<comment><USER2> My guess - this experiment has nothing to do with the clicks, but rather is trying to measure something else. Something like the amount of time people take on breaks, plotted against the amount of time people spent reading the instructions. Or something like that.

<comment><USER3> It could, for example, be used to demonstrate that people are more willing to reveal some private information if you make them jump through a few hoops to gain this "privilege".

<comment><USER4> Interesting idea, but if this was the case I'm sure they wouldn't bother writing javascript which ajaxes timestapms of your clicks to their server.

<comment><USER3> Why not? Placebos also look like the real thing, don't they?

Anyhow, in cases like this it's always nice to save all the data you can get. The correlation you'd like to demonstrate might not exist at all. However, if there are many parameters, you might be able to find something else.

</comment></comment></comment></comment><comment><USER5> Does that thing just throw random numbers between 28 and 33 ? Looks like that if I read the comments :D .

<comment><USER6> Probably has more to do with the age of your average HN reader.

<comment><USER4> For me it also said 30 although I'm few years younger actually.

<comment><USER7> For a friend that's 29, it said he was 10.

</comment></comment></comment></comment><comment><USER8> I'm 36, it says I'm 31. I feel like an atomic superman. Also my fingers hurt.

</comment><comment><USER9> Not very close. Said 32. I'm 47. From the other comments, seems like a pretty narrow range of guesses.

</comment><comment><USER10> I'm 35. It gave me 55 on my 1st attempt using the trackpad on my brand new laptop which I'm still getting used to (I turned on momentum yesterday and am not used to it yet). I then plugged in a mouse and got 30. The sensitivity on the mouse wasn't what I was used to, I felt hampered. I'll try it again at home on the machine + mouse set up I use when playing starcraft 2 and hope that I get at least a few years younger.

</comment><comment><USER11> return 30 if referer == '<https://news.ycombinator.com/>

Is it close?

</comment><comment><USER7> Kinda shocked that at the end it didn't give a debriefing like most study tests do.

</comment><comment><USER12> Age 29 the first with regular speed of clicks. Age 53 the second time taking lots of time between clicks and being more careful to click the center of the circle.

</comment><comment><USER13> Said 40, I'm 21, really off!

</comment><comment><USER14> Just put up a video on my youtube channel asking my subscribers to try this out. Looks like it does predict the 29-31 range much more then others. Anyway check out the comments over there for results of a much different demographic then here: <http://www.youtube.com/watch?v=_K6bSVKao5I>

</comment><comment><USER15> It looks like it even transmits mouse move events on the canvas via ajax to the server. Too bad we don't know the calculations over there.

I used a Thinkpad tracking point which is extremely accurate and fast (Lenovo will hopefully not remove them from the next ThinkPad after the T431 :-( ), yet the page guessed I was 10.

</comment><comment><USER16> It thought I was 2 years younger, so not bad. Gamers will probably get rated much younger than they really are.

</comment><comment><USER17> Not even close - guessed 44 and I'm 25.

<comment><USER18> Thank god I'm not the only person that it guessed _older_.

</comment></comment><comment><USER19> I used a MS trackball (thumb operated) and it read me as 9 years younger than my actual age. I expected the trackball to be a significant handicap, but maybe it's not? It certainly felt like I was struggling to hit the targets, with plenty of overshoot.

<comment><USER4> I also used MS TBO but in my case it guessed few years too much. I'd say it's interesting if there was a single person here who didn't get answer between 25 and 35.

And TBO definitively _is_ a handicap, try playing Quake with it if in doubt.

</comment></comment><comment><USER20> Despite being about 10 years off my actual age, I think this is pretty cool. I wonder if there's any other information they're taking apart from speed between clicks? Geo-location might help to narrow down the range of possible ages.

<comment><USER21> I wonder if the duration of a "quick break" is taken into account for their calculations?

</comment></comment><comment><USER22> Broken for me. The question page at the end didn't have a submit button (just oddly quoted: Preparing results..."), and when I submitted by pressing <enter> in an input box I got a "connection reset" error from Firefox.

</comment><comment><USER23> And just when I finally get it to load and to my test:

        The following error was encountered:

        Connection to 18.104.22.168 Failed
        The system returned:

            (60) Connection timed out

</comment><comment><USER24> I'm almost 35, it said 32. Close enough. I wonder if it just uses time to click or if there's something involving extraneous clicks or noticing the patterns. Interesting in any case.

</comment><comment><USER25> It said I was 28, I'm 19.

</comment><comment><USER26> Off by only 1 year for me.

BTW the site seems to be under high load, it timed out, and had to reload the results page quite a few times, but eventually I got the result.

</comment><comment><USER27> It's probably a privacy test to see if they can get personal information out of test takers online simply by making you feel it's relevant to the test.

</comment><comment><USER28> Is this just a clever way to collect survey data? Based on the guesses being in a pretty narrow range I suspect that's the real goal.

</comment><comment><USER29> Not even close. Said I was 29 and I am 48.

</comment><comment><USER30> I got only the "preparing results"... I think I took the site down! Now none of my three browsers can connect.

</comment><comment><USER31> Not accurate for me. I'm 51 and it guessed 32. I'd be interested in what the key factors are.

</comment><comment><USER32> It was off by a good decade for me.

</comment><comment><USER33> That is amazing. It was spot on.

<comment><USER34> How old are you?

<comment><USER33> 45

<comment><USER34> Hmmm. That's pretty cool it guessed correctly then.

</comment></comment></comment></comment><comment><USER35> Most of the numbers are around 30~ I don't think there is anything accurate about it.

</comment><comment><USER36> Calls me 52. I'm 43. Using (badly) a touchpad.

I have terrible co-ordination.

</comment><comment><USER37> Firefox: I am 10 years old Chrome: I am 30 years old IE: crashed in the end

</comment><comment><USER38> It guessed that I was 33. Divide that in half and it would pretty spot on.

</comment><comment><USER39> >> Our best guess is that you are 10. Is it close?

Nope, I'm 28.

</comment><comment><USER40> Says i'm 37, but really i'm 906. That's weird.

<comment><USER41> What country were you born in?

<comment> qu4z-2 I'm gonna guess he put down Gallifrey.

</comment></comment></comment><comment><USER42> 31 ... I'm over 50. And I don't feel particularly dextrous.

</comment><comment><USER43> It thinks I'm 42.

I'm 21. Something must be wrong with me.

<comment><USER44> You/are are almost directly inverted in terms of age/score.

</comment></comment><comment><USER18> Guessed 31. I'm 26. Crap. I must have awful motor skills.

<comment><USER41> Guessed 30 (I'm on regular mouse/desktop). I'm 38.

</comment></comment><comment><USER45> not even close, but now I do feel like bragging, so thanks for that ;) Actually the next position of the dot was pretty predictable in most cases.

</comment><comment><USER46> It said 33, I am 40.

<comment><USER47> It told me I was 33 also and I'm 25.

The survey at the end was fairly short. I wonder what factors they might be missing. I type most of the day in a terminal and rarely use my touchpad except for scrolling and occasional clicks on websites. I also don't play video games often. It seems like both of those facts could have affected my speed.

</comment></comment><comment><USER6> It said 29 and I don't turn 30 for a few more weeks.

<comment><USER48> I also got 29, but I'm 33. You click like a 33 year old! ;)

</comment></comment><comment><USER49> Not far off, but not amazing - 29 vs 33.

</comment><comment><USER50> Got it right on the money. Spooky.

</comment><comment><USER51> I'm 46, and it said 46. Uncanny.

</comment><comment><USER52> It said 31. I'm 27. Who knows.

</comment><comment><USER53> this has very deep and far reaching implications for pro-level starcraft players!!! :p

</comment><comment><USER34> It said I'm 36. It guessed 30.

</comment><comment><USER54> I click like a younger model.

</comment><comment><USER55> Way off. Like, by half.

</comment><comment><USER56> It got me within 4 years

</comment><comment><USER57> Off by a year. Not bad.

</comment><comment><USER58> Spot on for me too. Wow.

</comment><comment><USER59> Said 29 for me, I'm 24.

</comment><comment><USER60> I am 29 and it said 29.

<comment><USER7> 30/30 here

</comment></comment><comment><USER61> I'm 30, it guessed 10.

<comment><USER62> I'm 33, also got 10.

</comment></comment><comment><USER63> said 31, I'm 41. yay?

</comment><comment><USER64> Guessed 30, I am 28.

</comment><comment><USER65> I got 39, I am 35

</comment><comment><USER66> Bang on for me.

</comment><comment><USER67> said 33, im 30.

</comment><comment><USER68> right on the number for me. 33.

</comment><comment><USER69> 28 for 33</comment>


Orwell's Animal Farm is a new game from the dev behind the Reigns series - <AUTHOR> https://www.androidpolice.com/2020/08/17/orwells-animal-farm-android/ <comment><USER2> I play games to get away from life not simulate it.</comment>


OCEN: A single API to power billion loans for India - <AUTHOR> https://the-other-side.blog/accelerating-financial-inclusion-using-ocen/ <comment><USER2> This API was made by the government or a private company ?

<comment><AUTHOR> The API specs were written by iSPIRT and open sourced. You can see it here - [https://github.com/iSPIRT/lsp-lender-protocol- specification](https://github.com/iSPIRT/lsp-lender-protocol-specification)</comment></comment>


Is there a resource where one can find sample 'Terms of Agreement' between partners? - <AUTHOR>

<comment><USER2> Hi Ulfstein, here is a sample document: <http://www.podnikanivusa.com/wp- content/uploads/2007/02/partnership-agreement-sample.doc> ... but always consult with lawyer :)

</comment><comment><AUTHOR> I'm setting up a loose partnership in order to explore an idea. Having proper contracts drawn up at this stage is a bit pricey so we're thinking of using Terms of Agreement for now.</comment>


Judge awards iPhone user $850 in throttling case - <AUTHOR> http://news.yahoo.com/judge-awards-iphone-user-850-throttling-case-195042925.html

<comment><USER2> The main reason why they don't allow you to tether your phone is not because they just want you to pay twice, although I'm sure that has some play into it as well, but most people won't pay for it anyway, but because they don't want people to actually use the data amount they were given.

Right now part of their business model is that most people do _not_ use the whole 2 or 3 GB of data that they are given. Allowing them to tether, even though it would bring them more money if the user goes over the limit, means a lot of people will start using all the data they received.

<comment><USER3> That's a tactic many businesses use. Oversubscribe services and profit from the limited use of the product. Punish the few who actually try to make full use of the service.

See any cheap, shared hosting service. They'll promise unlimited so long as you don't try to take them up on that offer.

<comment><USER4> Certainly not a new thing. Circa 1994 my dial-up ISP (with "unlimited" internet, the first one in the area) was oversubscribed like every dial-up ISP, they had an order of magnitude less connections available than paying customers. When I actually set the connection to stay up 24/7, they shut the account down within a week.

<comment><USER5> This is called "modem math".

</comment></comment></comment></comment><comment><USER6> > "AT&T forbids them from consolidating their claims into a class action or > taking them to a jury trial."

How is that even legal in the US? Is customer protection really that bad that companies can take away these rights?

<comment><USER7> I think it was a Supreme Court case related to arbitation.

<comment><USER8> To be fair, it was the Supreme Court ruling on the (big-business backed) Federal Arbitration Act[1]. The Supreme Court didn't just decide out of thin air "hey, we should really fuck everyone over and make it so that all your grievances have to go to a secret court where the judge is paid by your opponent".

[1] <http://en.wikipedia.org/wiki/Federal_Arbitration_Act>

(Though I would be interested to hear how such an act isn't unconstitutional as it seems to me to remove the right of a person to petition the government for a redress of grievances - isn't that a guaranteed right?)

<comment><USER9> I've always wondered what prevents a court from saying "Your agreement not to sue is between you and the person, we're not a part of it, i.e. take your agreement and shove it".

If I sign a piece of paper that says I won't sue you, I don't see anything stopping a court from ignoring that outright.

<comment><USER10> _If I sign a piece of paper that says I won't sue you, I don't see anything stopping a court from ignoring that outright._

If courts ignored such pieces of paper, then settling lawsuits out of court would be impossible.

<comment><USER11> If someone did, say, agree to an out of court settlement for something, then one party didn't perform as agreed I imagine it could easily end up in court again.

Here wasn't it the same contract that mandated the arbitration that was not fulfilled by AT&T?

</comment></comment></comment></comment></comment></comment><comment><USER12> I was really disappointed when I got a warning in January when I was only at around 2GB for the month on my AT&T iPhone "unlimited" plan. On the one hand, I like unlimited because in theory it means I never have to think about my data usage, but the warning and apparent immediately subsequent throttling I experienced for the rest of the month (~7 days) was painful enough for me to consider switching my $30 unlimited to $30 3GB (and freeing me up to change providers when the next upgrade cycle rolls around–previously I wasn't willing to give up my grandfathered unlimited plan, but I never expected "unlimited" to mean less data for the same money [and the throttling is so severe that there would have been no way for me to reasonably reach 3GB data usage]).

I'm not going to go to small claims court over this or anything, but I had goodwill towards AT&T even when the rest of the country was bashing them (coverage has been reasonably good in areas I lived while on AT&T) and that's gone now.

<comment><USER13> _I'm not going to go to small claims court over this or anything_

Are you surprised that AT&T thinks they can treat their customers like this when almost none of them will do anything about it?

<comment><USER14> In all fairness even the guy who did something got $850 for his months of work. Not exactly bloodying up anyone's nose with that kind of a settlement.

<comment><USER15> I'm not sure about that. If AT&T sends a lawyer to every one of these things, it will quickly become very expensive. If they don't, they risk it becoming very easy to get awarded this money. Plus with precedent set, it will be considerably easier for someone else to do this.

If the settlement dollar values hold up and this becomes easy enough, it could become far more costly to AT&T than any class action suit would have been.

<comment><USER16> It was small claims court, AT&T can't actually send a lawyer there.

A lawyer could potentially do all the prep work (and bill for it) but they can't argue in front of the judge.

<comment><USER11> So who represents AT&T at the hearing?

Anyone the company designates as long as they're _not_ a lawyer?

I think AT&T should be able to send a lawyer in such a case _as long as he is wearing a clown suit including the red rubber nose_.

</comment></comment></comment></comment></comment></comment><comment><USER17> A bit of background on why (IMO) it's good to see these small claims cases succeed for non-trivial amounts:

AT&T is essentially class action proof post- _AT &T v. Concepcion_. AT&T had always put mandatory arbitration and class action waiver provisions in its contracts but states (most notably CA) usually ignored them and allowed class actions to proceed. In _Concepcion_ , the Supreme Court said that the Federal Arbitration Act (an act that basically says promoting arbitration is so desirable any contract that calls for arbitration must be honored) preempts state law that would ignore those provisions.

In other words, the formula for becoming class action proof is now 1) insert mandatory arbitration provision; 2) insert class action waiver; 3) dare consumers to sue you one at a time.

Virtually the only way I see to overcome _Concepcion_ right now is mass, coordinated (but individual), small claims suits or arbitrations. If AT&T had to face several million suits where it was actually relatively easy to win $850, they'd be begging for the old class action system.

<comment><USER18> To what extent is a third party allowed to assist a claimant in small claims court? If someone were to start a business that made it as easy as possible for people to take AT&T to court...

</comment></comment><comment><USER3> I would love for this ruling to stand but ultimately AT&T can't let it or there will be a flood of people hitting them in small claims court. They will appeal.

</comment><comment><USER19> Shouldn't the title be about an "AT&T Subscriber"?

</comment><comment><USER20> That title is somewhat framing the issue; a better one would be "Judge awards AT&T customer $850 in throttling case"

</comment><comment><USER21> Glad to hear that AT&T is getting some heat regarding this issue.

<comment><USER22> Yes. I was hoping for a class action lawsuit. Perhaps this is on the horizon?

<comment><USER23> The AT&T subscriber agreement requires you settle claims through binding arbitration. You give up the right to enter a class action suit by subscribing. The Supreme Court has upheld these clauses in contracts as valid. This was in the article, above the fold.

It's unlikely there will be a class action suit.

<comment><USER22> Thanks for the info. I guess I missed that.</comment></comment></comment></comment>


Mono 5.0 released with C#7 support and open sourced Microsoft tooling - <AUTHOR> http://www.mono-project.com/docs/about-mono/releases/5.0.0/ <comment><USER2> Fantastic - now if only Unity could get on a less paleolithic version of Mono.

I'm not sure why Microsoft hasn't bought them yet.</comment>


A Tech Perspective: Title II Isn't the Answer for Internet - <AUTHOR> http://www.realclearpolitics.com/articles/2015/02/24/a_tech_perspective_title_ii_isnt_the_answer_for_internet_125706.html

<comment><USER2> "There are currently no major issues with how Internet Service Providers offer broadband service, nor is there any groundswell of dissatisfaction from the millions of Americans who are generally able to consume as much bandwidth as they desire."

I'm not even sure what to say about that; it's so plainly false it's not even laughable. It conflicts with every bit of evidence and every anecdote that everyone else is reporting.</comment>


Coronavirus effective drug found, claim Bangladeshi doctors - <AUTHOR> https://africanewspay.com/news/273859416/ <comment><USER2> This was "discovered" in March in the US, and then "discovered" again in April by Monash University in Australia. The Ivermectin plus Doxycycline was proposed almost immediately as an alternative to chloroquine.

Doctors in the West are almost completely pre-occupied with treating the severely ill, at which point chloroquine and other treatments appear to have little effect (outside of anecdotal cases).

Now that we have testing in place and available, doctors really should think about trying these treatments on the not severely ill, i.e., try to prevent the progression of the disease. Even now almost all studies of Chloroquine and Ivermectin focus on the severely ill.

I get why you don't want to give Chloroquine and Azithromycin to people early as it can cause heart problems, but Ivermectin and Doxycycline are even more widely prescribed than Chloroquine and have fewer side effects. People could be safety taking these drugs to prevent the progression of the disease, or at least start testing pre-severe treatments more aggressively.</comment>


Colemak keyboard layout - <AUTHOR> http://colemak.com/ <comment><USER2> I'm not sure how relevant this is to anyone, but this post is as good excuse as any to talk about my transition to Colemak.

In January I moved from Qwerty to Colemak, and also from a staggered to a columnar keyboard layout (on a Kinesis keyboard[1]). My Qwerty speed was only 75 wpm and my accuracy was terrible. Also I couldn't touch type - I never learnt to type properly with Qwerty; I just started typing when I was young and developed lots of bad habits. By the time someone tried to teach me how to type properly my habits were too ingrained to change. (Or maybe I was too stubborn -- I also have a strange way of holding a pen.)

Anyway, I decided to change to Colemak, and to learn to touch type properly. I also liked the idea of a columnar layout so I changed to a Kinesis. What's interesting is that because of the physically different keyboards, I actually cannot type in Qwerty on my Kinesis, and I also cannot type in Colemak on my laptop keyboard. This means I have maintained my Qwerty skills.

My current maximum speed is roughly 70 wpm in Colemak, so I am almost at my Qwerty speed after 6 months, although now I can touch type so it's more useful. When I type with Qwerty now, it feels like my hands are flying all over the place. I also feel like I have a lot of room to get faster than I currently am with Colemak.

So even though I have maintained my Qwerty skills, the only reason I need them is because I can't get a laptop with a columnar keyboard. I may end up just biting the bullet and switching to Colemak on that laptop keyboard anyway, though missing the thumb cluster is killer.

One other interesting thing I learnt is that using Colemak on a smartphone completely sucks. Having commonly used keys on the home row on a small touchscreen means that it's much more error prone, as you're hitting the same area on the screen consecutively.

[1]: [http://www.kinesis-ergo.com/advantage.htm](http://www.kinesis- ergo.com/advantage.htm)

<comment><USER3> I am very jealous of you -- my inability to maintain QWERTY after I switched to Colemak was what forced me to grudgingly switch back.

I just couldn't do it; after I became acclimated to Colemak, my QWERTY was just destroyed. I had to _think_ about where the keys were in QWERTY, and I realized I don't even really know, because typing is a doing-without-thinking activity. So I ended up having to _look_ , which is extremely inefficient and finally I abandoned Colemak. (I couldn't avoid QWERTY because part of my job these days entails showing people how to do things on their own computers, which aren't using Colemak.)

The same thing happened when I learned Japanese; I used to be conversant in Spanish, and after I got to roughly the same level in Japanese (from living in Japan) one day I noticed that the Spanish wasn't there anymore -- I would have to think about what I wanted to say in my native English and convert it to Spanish.

Seems like a way different thing, but it felt the same, because it was like an unconscious activity being supplanted by another, and thus becoming something I had to _think_ about. Which broke it.

It made me sad though, because now typing QWERTY feels idiotic. Kludgey and stupid. Colemak didn't make me a significantly faster typist (maybe it would have had I done it for more than 9 months) but it was way more _comfortable_.

<comment><USER4> I use Dvorak, and I have almost no problems typing QWERTY. But this only came after I became completely fluent in Dvorak. When I still had to think to type Dvorak, I couldn't switch to thinking about QWERTY without getting all confused. Now I just peek a little at the keyboard with QWERTY, especially for non-alphanumeric symbols, but it's no problem.

<comment><USER5> I had to abandon Dvorak because it killed my ability to do QWERTY. It was also an _almost_ traumatic experience because when I was going through it I felt like I was going crazy or that I was just plain stupid. How could my brain possibly discard 20+ years of QWERTY after just a few weeks of Dvorak.

But having said that I recently learned that other people have had the same experience and so now I don't feel so bad and I want to change again.

/lifestory

</comment></comment></comment></comment><comment><USER6> I tried to switch to Colemak once, but it was too big of a change. I did, however, successfully switch to an alternative of the Carpalx QWKRFY layout: [http://mkweb.bcgsc.ca/carpalx/?partial_optimization](http://mkweb.bcgsc.ca/carpalx/?partial_optimization)

Instead of 5 swaps, I only do the first three (K/E, J/O, F/T). According to their scoring function, it gets you most of the way there anyways. (Whatever that's worth).

Their rankings of popular layouts might also be of interest: [http://mkweb.bcgsc.ca/carpalx/?popular_alternatives](http://mkweb.bcgsc.ca/carpalx/?popular_alternatives)

</comment><comment><USER7> I am cautious to change my keyboard layout because I use a lot of different machines. Has anyone tried this or the Programmer Dvorak layout?

Ref: [http://www.kaufmann.no/roland/dvorak/](http://www.kaufmann.no/roland/dvorak/)

<comment><USER8> I use the standard Dvorak layout. It hasn't been that big a deal for me.

Things have gone something like this:

1) Learn Dvorak at a time when I only used my own machines. Stay that way for ~3 years.

During this time, after my typing speed and error rate had reached QWERTY levels, the occasional use of QWERTY on someone else's PC was highly annoying.

2) Spend 2-3 years in jobs involving near-daily use of machines I couldn't switch off QWERTY, but still did most of my work on Dvorak.

A few weeks into this, I was able to use QWERTY effectively again, still typing faster than most people ever can (though slower and more error-prone than I was with Dvorak).

3) Go back to exclusively using my own machines for ~2 years.

After a while, annoying to use QWERTY, but not nearly as bad as during period #1.

4) Spend ~9 months in similar situation as #2 again.

After a while, same as 2, a few weeks in things were pretty OK.

5) Now (for ~2 years), using only my own machines.

Occasional QWERTY is annoying, but even better than during #3.

I only speak English, but I suspect switching between Dvorak and QWERTY is kind of similar to switching between a language you use a lot and one you use a little. Degree of fluency in the little-used language goes up and down based on how much practice you're getting.

<comment><USER9> I switched to Dvorak 11+ years when I was in college and had a lot of time to kill. I think your spoken language fluency analogy is pretty accurate; I imagine I'll always retain a bit of my 'native' qwerty muscle-memory, but the more I practice it, the easier it is to make the mental 'switch' between languages.

That said, if I had to do it all over again, I wouldn't switch to Dvorak. The efficiency gained on my keyboard layout does not offset the efficiency lost on shared computers.

<comment><USER8> I definitely wouldn't suggest anyone switch for performance reasons. I switched because of pain. It helped that a lot. Made virtually no long-term difference in typing speed.

(Then again, my typing speed has been 90+ wpm since I was about 9, so I'm an outlier in various ways.)

</comment></comment></comment></comment><comment><USER10> I've used Colemak for a year and a half now. The common shortcut keys are in the same place still (ctrl-c, ctrl-v, etc.). I use Vim and keep the default bindings and it works fine (though movement is odd.. up and down are inverted). I used to type at around 100 wpm with QWERTY and can reach that or higher now. I can still use QWERTY when I need to although I have to look down a bit.

All in all I like it. It feels good and efficient to type. I initially switched because I wanted to learn to touch type on QWERTY.. Not only was it a struggle to change habits but QWERTY felt just.. Awful and slow. So I wanted to learn a new layout. Colemak seemed less radical than Dvorak and also more efficient (less finger movement). Learning a whole new layout was quite hard (especially in the middle of a semester), but I think it did make learning to touch type easier. The hardest part was 's' being moved a letter to the right.. I confused 'r' and 's' for a while now. So yeah, if anyone is thinking about trying it, go for it. Don't rearrange the keys, just print out the layout and look at that for reference so you learn the locations by feel.

<comment><USER11> I'm a Vim user as well, and when I tried Colemak for a few months some years back, Vim was what made me go back to QWERTY. It was just too awkward as I would like to be able to work with both layouts.

Creating new bindings didn't work for me at all (especially since I didn't want to lose the ability to use someone else's Vim, or a vanilla Vim). I think the key might be to focus on what shortcut key you're actually pressing instead of where it is on the keyboard. This turned out to be pretty difficult for me with the hjkl keys, which are, as you say, fairly odd.

Interestingly enough though, today I can still write Colemak fluently whenever I accidentally switch to it, and it's a lot of fun! So I might give this whole thing with Colemak a new try, knowing that someone went the same route as I (keeping the default Vim shortcut bindings).

<comment><USER10> It also helps that I wasn't too proficient with Vim, if at all, when I started using Colemak, so I didn't have to deal with any burned in habits with Vim either.

</comment></comment></comment><comment><USER12> The author of that site should site stats comparing Colemak with Dvorak because that's really their 'competition' so to speak.

<comment><USER10> There's a link right on the front page ('Compare') to an applet which compares QWERTY, Dvorak, and Colemak..

<comment><USER13> Yeah but the author doesn't provide any summary stats or analysis like he/she does vs. qwerty. I would potentially be switching from dvorak - on average is there much difference?

I ended up pasting some posts from this thread into the compare tool. Colemak and dvorak seem pretty similar, with colemak consistently having slightly less same-finger usage (i.e. two keys in a row being pressed by the same finger). Both are about the same improvement over qwerty. Doesn't look like it's worth switching from dvorak.

</comment></comment></comment><comment><USER14> I observed something quite strange that readers might find interesting. When I first tried to learn Dvorak two years ago I found that it completely destroy ability to type qwerty. I simply couldn't type either layouts! I ended up giving up early and regaining my qwerty powers. But then just last year I tried to learn it again and found that it had genuinely no effect on my qwerty ability. I could switch between them like a fluent multi-linguist. Naturally I gave up again :P until last week when I decided on giving Colemak a proper go. I have been on Colemak for 3 days without changing back until right now when I wrote the second half of this with switching back to qwerty, and observe no difficulty in typing, except for backspace where I keep occidentally hitting the CAPS key which is what I use as backspace for Colemak. Not sure why my brain seems to treat backspace differently such that that happens... I also have no idea how my strange experience happened, but for those concerned about forgetting their qwerty ability, know that you don't need to even if at first your mind turns to spaghetti and you can't even type with any layout

</comment><comment><USER15> I switched to Colemak in December last year so I'm 6/7 months in. Anecdotes and tips within.

At the same time I picked up KeyboardRemap4Macbook (not just macbooks, but osx) which has some other interesting features. One important one I use daily is using held down spacebar to enable arrow navigation by UNEI, and "return" on O. Related is the terribly named but essential PCKeyboardHack which allows one to remap the CAPSLOCK key to backspace, giving me a more accessible backspace key. I suspect this is a double edge sword though, lowering the cost of an error.

When I started with Colemak I was slow. I forced myself to write even urgent mail in it, else I wouldn't succeed. I ended up writing emails as if English were my 2nd language but after a week I was over it.

Changing my phone to Colemak was a very helpful aid. Avoid writing in Qwerty.

I also learnt for a while on a keyboard without the letters on the right keys. This screwed me up big time at first (i didn't touch type) but I pushed through. Now I work daily on a keyboard with Qwerty layout but don't give it a second thought and it just serves to mess with the minds of my colleagues.

I still have to look for numbers because I just don't type them frequently.

I haven't had a massive speed increase because my speed is limited by my ability to decide what I'm going to write which is dreadfully slow. I make fewer errors.

Going back to qwerty is relatively easy if I free my mind. I typed cowboy Qwerty with my hands flying all over the keyboard for so long that it is possible for me to call it back. I am sure this is made easier in part by the two layouts being typed in completely different styles, Colemak formally and disciplined by correctness and touch, Qwerty as an untempered and free dance of my fingers and hands.

</comment><comment><USER16> What matters most when programming is all the non alpha numeric symbols.

You likely type symbols such as ', ", (, and ) more often than all numbers, and more often than half the letters.

If these symbols are accessible only by using the shift or alt gr keys, your keyboard layout sucks for programing.

Colemak sucks as a programing layout (just like virtually all keyboard layours actually).

<comment><AUTHOR> Remapping my AltGr layout has done me wonders. I can press AltGr with my right thumb, and then I get {}[]()_= as the no-motion home row. Colemak has nothing to say about AltGr.

</comment></comment><comment><USER17> I type about 125 wpm with QWERTY. I've been touch typing since I was 6 though (Dad was a writer and bought me my first typewriter when I was 5).

Interestingly, I get about a 5-10 wpm bump when I smoke marijuana.

I tried switching to DVORAK to see what the hub bub was about. Spent about two weeks with it but never attained the same speeds. I'd try Colemak but I'm not sure what I'd gain.

Keyboard makes a huge difference for me too. Apple's keyboards suck and can put a serious dent in my speed. I've been using the Moshi Luna and it's like playing a piano, though it's more cheaply made then it appears (I've been through two).

I'm fastest on an IBM Selectric typewriter though. When the fingers are blazing there is definitely a rhythm that you fall into that maintains that speed, so long as your brain doesn't jam up.

<comment><USER18> You might like a nice mechanical keyboard. This is a good source: [http://www.elitekeyboards.com/](http://www.elitekeyboards.com/)

A good starter keyboard is the Leopold with Cherry brown switches. There's a slight feedback bump partway down, and that's as far as you have to press to register the stroke. Your fingers can just float across the keys, very comfortable and, after a little practice, very fast.

I recently bought that one, and another with Cherry red switches, which lack the feedback bump. Now I'm completely hooked. And as a bonus, it costs about the same as your Luna, is rated for 50 million keystrokes, and if you spill something on it you can just soak it in water and let it dry out a couple days.

Lots of good discussion at geekhack.org.

</comment></comment><comment><USER19> Lefthanders like me need an inverse Colemak to remain effective, given that many of the most used letters are on a QWERTY keyboard on my left hand (while with Colemak e.g. the E has moved to the right). Does an inverse Colemak exist?

<comment><USER20> I don't think this exists. Using the keyboard layout analyzer [1], it looks like your perceived hand usage might not match up with actual usage. At least for typing out _Alice 's Adventures in Wonderland_, Qwerty slightly favors the right hand (52% vs 48%). Colemak is only slightly worse for a lefty (56% vs 44%). Dvorak greatly favors the right hand (65% vs 35%).

[1] [http://patorjk.com/keyboard-layout- analyzer/](http://patorjk.com/keyboard-layout-analyzer/)

</comment></comment><comment><USER21> I wonder if switching to a new keyboard layout is easier for people who are already used to switching between layouts (I use both Azerty and Qwerty), like it's easier for someone bilingual to learn a new language.

</comment><comment><USER22> I'm not sure why every new layout decides to remove the CAPS LOCK KEY. I imagine coders aren't the inventors of these layouts.

<comment><USER23> May I infer from your post that you use Caps Lock at least semi-regularly?

Myself, I abhor it. I've always just held Shift, even for long runs of capital letters. As far as my usage patterns go, Caps Lock serves only to foil my attempts to type my password. :/

</comment></comment><comment><USER24> The FPS shooter fans would be delighted - where previously WASD were used now they read WARS!

<comment><AUTHOR> Just remap it within the FPS (don't they all have this option?), or press Alt+Shift when you enter the game.

</comment></comment><comment><USER25> vim config?</comment>


Global heating: London to have climate similar to Barcelona by 2050 - <AUTHOR> https://www.theguardian.com/environment/2019/jul/10/global-heating-london-similar-climate-barcelona-2050 <comment><USER2> [https://crowtherlab.pageflow.io/cities-of-the-future- visuali...](https://crowtherlab.pageflow.io/cities-of-the-future-visualizing- climate-change-to-inspire-action#213121)

That page is the link to the interactive site.

<comment><USER3> The "full map" at [https://hooge104.shinyapps.io/future_cities_app/](https://hooge104.shinyapps.io/future_cities_app/) seems to be 503 right now. :/</comment></comment>


Which Country Would Win in the Programming Olympics? - <AUTHOR> http://blog.hackerrank.com/which-country-would-win-in-the-programming-olympics/ <comment><USER2> This article claims to rank nation's Olympic programming ability, but uses the average score from online tests.

The olympics are scored by the best a nation can do, not by its average.

</comment><comment><USER3> The conclusion I would draw is China, Russia, and Poland perform the best in programming challenges. The article even shows this in a quote about the greatness of Chinese programmers:

> In universities and colleges, education resources are relatively fewer in > comparison with many other countries, so students have less choices in their > paths to programming. Many great students end up obsessed with competitive > programming since it's one of the few paths.

The headline is somewhat click-baity.

</comment><comment><USER4> Well, Hackerrank is a recruiting company. If you are from China or Russia, and you want to leave the country and find a job elsewhere, having a high score on Hackerrank is going to improve your chances. If you are from the UK or US, you probably wouldn't bother.

</comment><comment><USER5> I never heard of hackerrank until now. Maybe the site has a location bias? Interesting bit, when I used to have ad-sense for my tumblr app certain regions just visited more of that site than others

<comment><USER6> That's the purpose of the whole article.

</comment></comment><comment><USER7> Or may be good programmers in US don't feel the need to prove themselves on such sites, instead they can just go and work for good company like Facebook, Apple, Google, etc.

</comment><comment><USER8> Are those tests representative of what we have to do at work? I did a couple of tests on another site and they were almost artificial, something you do only at university exams but never at work.

If I were an interviewer I'd use them only to see how somebody starts thinking about solutions but we can use real world problems for that.

</comment><comment><USER9> Is the raw data for this calculation available anywhere? I am guessing the sample size for some countries in top 10 is too low to be meaningful.

</comment><comment><USER10> The ranking is a joke. Look at taobao, the most famous website in China. Its front page is a mess. As an Chinese I can tell you those Chinese copycat software or websites are often patched up Frankensteins.

<comment><USER11> What does UI/UX have to do with HackerRank? I don't think HackerRank even measures more "pragmatic" qualities of software (e.g., code organization, clarity, etc). In any case, the "busy UI" or cluttered look that is almost universal across Chinese and east Asian websites and apps is often intentional because the chaotic UX is preferred by consumers. It reflects the experience of shopping in a Chinese market. It packs lots of assuring information into the page. It has historical and technical precedent, and breaking the trend would mean "sticking out". Sleek, simple designs may even be viewed with suspicion. Another factor is that Chinese text will generally appear more chaotic to Westerners who cannot read Chinese and are accustomed to simpler Latin characters and Western typographic conventions.</comment></comment>