Mobile financial services would increase emerging economies' GDP by $3.7T - <AUTHOR> http://qz.com/787137/developing-nations-would-be-3-7-trillion-richer-if-more-people-could-handle-their-finances-from-their-mobile-phones-instead-of-going-to-the-bank/ <comment><USER2> I am trying to envisage the different possible ways new banking services can emerge. Can we ever expect a crypto currency to work? Or are we forever reliant on trusting third parties (ie banks)?

Having mobile banking services is possible of course but actual banking services are more than tracking my latte spend. The guy who founded Bank Of America walked into San Francisco the day after 1906 earthquake with a wheelbarrow of cash and started lending to the shop and business owners who needed to rebuild immediately.

That's the kind of banking service Africa will need in the next few decades, and New York or London or SV are not planning on an app that can do that.

So the consumer banking apps, they will come, but a banking infrastructure, funnelling loans into real businesses and infrastructure - that's a different kind of banking. One it seems we forget.

</comment><comment><USER3> It was my understanding that Google banned non-Google in-app payments from the Play store. This has certainly stifled a lot of innovation, and basically leaves pre-loading and side-loading as valid distribution methods, neither of which are ideal. Historically I think that was partly to remove carrier billing as an option (Google and device manufacturers wanted to 'own the customer' instead of letting the carrier do it, which is traditional in the US). I know this as I am one of the few people in the world who has written an from-scratch in-app payment solution for Android that was actually widely deployed (US carrier-billing for T-Mobile and AT&T via a pre-loaded app on intial generation Samsung Galaxy series devices). Here in China, Google Play is blocked, there are at least five different app markets that come pre- installed on various vendors' phones, and we mostly all use WeChat payment and just routinely download APKs from the browser. It is normal for browser- downloaded APKs not to request permissions though. whereas fintech stuff well written may need quite a few permissions. I suspect Google has already made this hard.

</comment><comment><USER4> ... if more people could handle their finances from their mobile phones

is the rest of the title.

Also, the bank I work for allows mobile banking for 11 countries in Africa. 11/54 is not bad if you ask me.

<comment><USER5> Reminds me of the Onion headline, "New Study Finds Blacks More Likely" [http://www.theonion.com/article/new-study-finds-blacks- more-...](http://www.theonion.com/article/new-study-finds-blacks-more- likely-18552)

</comment></comment><comment><USER6> I'm one of the organisers of the FinTech For Good Summit in London (more info: [http://fintechforgoodsummit.com/](http://fintechforgoodsummit.com/)).

If this topic is interesting to you and it is a discussion you want to get involved in, please do email me on firstname.lastname@example.com. All welcome!

We are building a powerful group of people to advocate for the transformative impact of FinTech in developing economies, and we'd love to start a conversation with you too.</comment>


Cardiologist-Level Arrhythmia Detection Using a Deep Neural Network - <AUTHOR> https://www.nature.com/articles/s41591-018-0268-3 <comment><USER2> The 12-lead ECG is the standard for cardiac diagnosis in a clinic or an emergency department. This uses numerous points of contact on the skin. Each lead (whether directly measured, or virtually measured as a function of the directly measured leads) then is plotted in a standardized way. The X-axis represents time, the Y-axis represents voltage. When the voltage is toward the lead, the value is positive; it can of course be neutral, or pointed away from the lead (negative).

In contrast, when someone needs to be monitored while at home, a several-lead device is cumbersome. So, people have devised single-lead devices that can remain attached for days-to-weeks (e.g., Zio patch), or which can be activated when desired (e.g., Apple Watch, AliveCor).

A single lead ECG can be more difficult to read, in part because you simply get much less information. (Though, generally, what you're looking for [arrhythmia] is a subset of what you look for in a 12-lead ECG.)

Obviously, there are automated algorithms that attempt to read these already. However, deep learning approaches are appealing here, and at a glance this looks like it performed well and is using a reasonable set of comparisons. (Will have to read the methods section to know for sure; heading to bed so won't get a chance to look for awhile.)

<comment><USER3> The "classical" I,II,III leads are balanced. aVR,aVL,aVF are derived from I,II,III (through voltage dividers). V1-6 are singled ended with a common "ground" lead.

</comment></comment><comment><USER4> I always wondered how precise were the computerized ECG interpretations from 24-48h Holter results. Considering you have to take notes and report date and time whenever you feel "something" it might need a manual check.

Edit: Code is open source and found it here [https://github.com/awni/ecg](https://github.com/awni/ecg)

<comment><USER5> They're pretty shit. Probably half of the essentially normal EKGs I see come with the automated interpretation of "Abnormal EKG," and no meaningful information. You absolutely 100% cannot rely on them, and all doctors are trained - from the get-go - not to.

<comment><USER6> Yup. We pretty much just ignore what the machine says.

</comment></comment></comment><comment><USER7> link to the paper: [https://arxiv.org/pdf/1707.01836.pdf](https://arxiv.org/pdf/1707.01836.pdf)

<comment><USER8> Pretty impressive.

From the Figure in on page 2 [https://arxiv.org/pdf/1707.01836.pdf#page=2](https://arxiv.org/pdf/1707.01836.pdf#page=2)

> _Figure 2. Deep Neural Network architecture. Our deep neural network > consisted of 33 convolutional layers followed by a linear output layer into > a softmax. The network accepts raw ECG data as input (sampled at 200 Hz, or > 200 samples per second), and outputs a prediction of one out of 12 possible > rhythm classes every 256 input samples. The first and last layer are > special-cased due to the pre-activation residual blocks._

^^^ 33 convolutional layers --- wow that's pretty deep for 1-D input.

<comment><USER9> I find it interesting they don't cite any framework. They presumably wrote all the DNN code from scratch, which can certainly be done, but seems like a lot of effort unless you plan to commercialize, which clearly they are.

<comment><USER10> Their published code ([https://github.com/awni/ecg/tree/master/ecg](https://github.com/awni/ecg/tree/master/ecg)) uses Keras. It's a pity they didn't cite it though.

</comment></comment></comment></comment><comment><USER11> What is "single lead" device? Is that a single electrical contact patch? How can that detect anything? I've used a single patch with 4 electrical contacts for a 30 day Holter test, plus the standard 24 hour Holter monitor with a bunch of patches and wires.

<comment><USER12> What device have you used for 30 day test?

<comment><USER13> There are various options, but for long periods single lead devices are often used, can't really walk around for 30 days with a 12 lead holter strapped to you...would make bathing rather difficult.

It's possible to implant small loop recorders under the skin on the chest. These can record for months. Those are mostly useful in cases where symptoms occur infrequently.

</comment></comment></comment><comment><USER14> For those who think this will revolutionize medicine, it won't. ECG analysis is a tiny part of diagnostics, and can only give a very limited amount of information for most people, and for the small number of people who absolutely need ECG for diagnosis, a clinician is still required and the ECG is a part of the information needed for determining treatment.

Further, the bar is set fairly low: as good as a panel of cardiologists at a very specific task, that can also be done by non-cardiologists depending on level of training and desire for mastery.

Lastly, this technology will never be a panacea where you walk into a Walmart and the ECG analysis spits out a report of your overall "health".

</comment><comment><USER15> I would have thought that you could get pretty damn close with just some simple stats in frequency space normalised to the base frequency, i.e. Fourier transform divided by the fundamental frequency.

<comment><USER13> Not likely. Heart rhythm isn't so much about the rate at which the whole thing beats, it's rather a little more complicated than that. It's about the synchronization between different parts of the heart, about the way the signal propagates from the Sinus Node, about how it relates to the body's requested demand.

Many things can go wrong. Many things do go wrong even in healthy hearts, for instance most people get ectopic beats every now and again, often without their knowledge. A simplistic FT approach would highlight stuff like that and no one would benefit from it.

<comment><USER15> Thats... not quite what I meant, perhaps I worded that a little bit weird. I meant frequency invariant, not amplitude normalised. As in divide the frequencies by the fundamental frequency (i.e. subtract their logs on the 'scope), not divide the amplitudes by the amplitude on the fundamental.

Any deviation from the profile of the FT irrespective of its base frequency, from the normal sinus rhythm, would easily show. There might be a few problems dealing with cardiac load transitions, and I suppose if you are trying to differentiate the exact class of arrhythmia (as in the paper) then perhaps not, but differentiating normal (sinus), noise, abnormal it should work fine.

</comment></comment></comment><comment><USER16> The dataset is here [https://physionet.org/physiobank/database/mitdb/](https://physionet.org/physiobank/database/mitdb/). Only a handful of files...

</comment><comment><USER17> For those of you concerned about heart disease, whether you have risk factors or a family history, consider getting a calcium score. It's the gold standard for detecting heart disease. It's usually not covered by insurance but it's very inexpensive (About $200).

Involves a CAT scan of the chest so it's not a blood test and there are some X-Rays involved. But it's totally worth it.

</comment><comment><USER18> The study contains no comparison with competing algorithms, so I don't see what supports the claim that

 _If confirmed in clinical settings, this approach could reduce the rate of misdiagnosed computerized ECG interpretations and improve the efficiency of expert human ECG interpretation by accurately triaging or prioritizing the most urgent conditions._

</comment><comment><USER19> sidenote: that there are 24 leads ecg

[https://www.ncbi.nlm.nih.gov/pubmed/18433615](https://www.ncbi.nlm.nih.gov/pubmed/18433615)

[https://www.ncbi.nlm.nih.gov/pubmed/24880763](https://www.ncbi.nlm.nih.gov/pubmed/24880763)</comment>


Apple's capitulation to China undermines Obama's tough talk on snooping - <AUTHOR> http://qz.com/356233/apples-capitulation-to-china-undermines-obamas-tough-talk-on-snooping/

<comment><USER2> It's not tough talk on snooping until something is done about the NSA within OUR OWN BORDERS.</comment>


Mike Turner's Do-It-Yourself 95 MPG AeroCivic (Slideshow) - <AUTHOR> http://www.treehugger.com/files/2009/02/mike-turner-aerocivic-95-mpg-honda-civic-diy-modified-photos.php

<comment><USER2> It should be possible to make a fiberglass modification kit for one or two gas-efficient models, and sell the kit and installation service.</comment>


Stack Overflow Hits 10M Uniques - <AUTHOR> http://techcrunch.com/2010/11/25/stack-overflow-hits-10m-uniques-boldly-goes-where-no-qa-site-has-gone-before/

<comment><USER2> Just to be clear: 10 million uniques is measured using analytics cookies, quantcast in this case. The exact definition varies by analytics provider, but regardles, it has nothing to do the number of user accounts.

Of course it's an estimate as there are complications in calculating it like users deleting cookies, one user with multiple browsers (ahem, programmers), etc.

It's still an awesome milestone.`

<comment><USER3> Just for reference there are roughly in the region of 15-30 million professional developers in the world. However this doesn't include a fairly large chunk of people (i.e students) who are probably the group most likely to be using StackOverflow. So I think Joel might be pushing it a bit to say most software developers are using it (although they have a fairly respectable chunk of the market).

<comment><USER4> Where did you get the 15-30 million number? We believe the number is closer to 9 million.

For the United States, there are 1,336,300 programmers, according to the Bureau of Labor Statistics.

The United Kingdom has 333,000 "software professionals," according to the Office for National Statistics.

In Canada there are 387,000 people working in IT according to Statistics Canada.

I haven't dug down to other countries but it's unlikely you could get to 15-30 million. Most development tool vendors report that about 40% of their sales are in the US.

<comment><USER3> I did something similar to you pick up numbers from individual countries and summing them, I don't have the underlying data to hand (I did this about a year ago). But I'll do it again as I need the data anyway and post an update.

IDC put the figure at around 12million in 2002, and at 15million in 2008 though. I assume they did a more thorough job than either of us :-)

</comment></comment></comment></comment><comment><USER5> This is a classic case for "Be so good that they cannot ignore you".

For such a long time TC have deliberately refrained from mentioning/promoting StackOverflow.

Thank you guys (the SO team and the contributors), for providing such a valuable website for programmers.

<comment><USER6> There was never a deliberate effort from TC to ignore StackOverflow, I linked to it in some of my posts there.

IIRC we simply were not in the loop on their news, announcements or tips.

A lot of the TC'ers use Quora which is why it is mentioned a lot, but in TC dev we were definitely big fans of StackOverflow, its just that the full-time writers didn't use it or notice it as much.

If you look at Techmeme, you can see that StackOverflow hasn't been getting much attention from the usual tech blogs:

[http://techmeme.com/search/query?q=stackoverflow&wm=fals...](http://techmeme.com/search/query?q=stackoverflow&wm=false)

I figure it is for the same reason - they weren't actively reaching out to the tech bloggers and instead using their own blogs for PR. Still worked out well for them :)

<comment><USER3> Quora has become hugely significant in the startup community, it's probably now the largest online community of startup people.

<comment><USER7> How can you tell how many users there are?

<comment><USER3> I'm not saying that with any scientific basis, but rather on the basis that a much larger percentage of the startup community seems to be active there than on other sites.

Often when you see someone ask about a random startup you'll get the startup's founder or an investor answering, something which you rarely see elsewhere.

</comment></comment></comment></comment></comment><comment><AUTHOR> Stack overflow has probably been the best Q&A site around, content-wise, user- experience wise and probably every other criteria a user would care about. Good work !

<comment><USER8> Agreed. I've found it especially valuable as a bit of a teaching tool lately. I spent three years in a mind-numbing COBOL-based job, and now I'm building a PHP application to try to showcase the fact that I am more than the sum of my career, and to try to find a decent job.

Since I'm doing this in isolation, I'm trying to use StackOverflow as a surrogate mentor. When I have a question about how to do things, mostly design issues, I can usually find an existing discussion (yes Jeff, _discussion_ ) on the topic. It sucks up an awful amount of my time, but I feel like I've grown a lot professionally because of it.

<comment><USER9> About the only drawback is that it's now so popular that new questions only show up on the front page for a few minutes. So finding questions to answer - or getting your question answered is becoming harder - unless you are prepared to do a very narrow subject specific search.

One of the nice things about the early SO was it was so broad, you could see interesting questions that weren't directly in your language and read those as well.

</comment></comment></comment><comment><USER10> Still a noob programmer, but I'm a huge fan of SO. I've yet to find a thread with any real bickering or pretentiousness, and 9 out of 10 times find an answer.

It seems like they dominate search results also, which is why I'm surprised that Quora, which I've never heard of, seems to get more respect?

<comment><USER11> Quora has a much smaller community mostly composed of people interested in start-ups in the Silicon Valley. They get coverage on sites like TechCrunch and HN because of that and the founders left Facebook (most likely because Zuckerberg replaced one of them as CTO with Brett Taylor).

I talked to them and they believe they can beat the competition simply by piggybacking off Facebook's social system and by using AI filtering to prevent people from being overloaded since they allow every topic unlike Stackoverflow.</comment></comment>


Four Degrees of Separation - <AUTHOR> http://vigna.dsi.unimi.it/ftp/papers/FourDegreesOfSeparation.pdf <comment><USER2> cool!</comment>


Visa is under DDoS attack - <AUTHOR>

Getting "An error occurred while processing your request." error at visa.com right now. I wonder if their payment gateways will suffer like mastercard's did. <comment><USER2> This is a confirmed target of "Operation Payback," a 4chan/Anonymous attack against those who have opposed Wikileaks.

The primary targets include MasterCard (which was hit earlier today) and Visa, who refused to process donations to Wikileaks. The Swedish Prosecution authority is apparently also on the list.

Although these types of DDoS attacks (they are using the Low Orbit Ion Cannon tool available at <https://github.com/NewEraCracker/LOIC/> ) do not affect cardholder data, it will render legitimate users completely unable to login to their merchant accounts.

</comment><comment><USER3> Keep trying, I'm sure it will load eventually...

</comment><comment><USER4> The site went down almost as soon as the attack begun... <http://isitup.org/www.visa.com>

</comment><comment><USER5> Will this have any impact on customers wanting to use their Visa card? For example, if I head down to my local dinner, will my Visa/Mastercard transaction get processed?

I'd wager that the Visa network engineers would segregate all customer transactions. Processing would take place outside their general www/office/etc traffic.

</comment><comment><USER6> [http://latimesblogs.latimes.com/technology/2010/12/operation...](http://latimesblogs.latimes.com/technology/2010/12/operation- payback-announces-on-twitter-planned-web-attack-on-visa-after-wikileaks- mastercard.html)

</comment><comment><USER7> I really, really don't understand how can this happen if www.visa.com is Akamaized.

<comment><USER5> I'm not saying this is the case today but... in general Akamai works be finding your location and directing you to the closest servers. The closest server might be overloaded but the network could be functioning fairly well for everyone outside your service zone.

</comment></comment><comment><USER8> What does Anonymous hope to prove?

What, do they expect people to say, "Oh why you're right. Wikileaks is a good thing and we should totally support it. Thank you for opening my eyes by horrendously disrupting my business for the day." That's asinine!

<comment><USER9> Is that a serious question? They're trying to show Visa that there are quite a lot of people who are very upset with them for booting Wikileaks.

It may be asinine, but many political demonstrations are.

<comment><USER8> A botnet can be run by a single person, though; how does DDoSing say that a lot of people are upset when just a few people with a lot of bots can perform the act?

<comment><USER9> Sorry, I'm not saying it's not misguided--just that they're trying to create the impression in Visa's mind that a lot of people are upset.

<comment><USER8> Apologies. In the CNN article now posted on Hacker News, [http://www.economist.com/blogs/babbage/2010/12/more_wikileak...](http://www.economist.com/blogs/babbage/2010/12/more_wikileaks) , the Low Orbit Ion Canon is volunteer. It would seem that there are that many who are agreeing with the assault, or at least not removing their agreement with Anonymous' general decisions.

</comment></comment></comment></comment></comment><comment><USER10> I had an error yesterday evening. Worked the second try, so I thought nothing of it. Knowing that CC sites are under attack now...

</comment><comment><USER11> Yep. <http://twitter.com/anonops/status/12613087077072896>

</comment><comment><USER12> Yes they completely busted them see this: <http://www.youtube.com/watch?v=-4TgCU1CWYo></comment>


Apple Watch users struggle to find a compelling use - <AUTHOR> http://nypost.com/2015/07/08/apple-watch-users-struggle-to-find-a-compelling-use/

<comment><USER3> I'm a bit of a fanboy for all the shiny Apple products, but wasn't entirely excited about the watch. Still, I pre-ordered the sport version and waited.

Then, the stores had personal fittings. I went in to take a look, and came out changing my preorder to the stainless watch with the link bracelet. It's a nice watch, looks great, and I figured it'd be kinda cool to play with.

Fast forward a month, and it's become a part of my everyday activity. I don't miss messages anymore because of the haptic taps --- I used to miss messages all the time because I frequently don't feel my phone's vibration in my pocket.

I used it to check in last time I took a flight. Barcode goes to screen via Passbook. Just like the Starbucks app. And ApplePay is great where you can use it. I have an iPhone 5S, so it opened that world to me.

Every day, I use the activity monitor to keep me motivated. I use a bluetooth heartrate monitor as a more accurate HR measure while working out, and pair it to the watch. Works great. (The watch is not a reliable HR monitor when you're moving around a lot -- it's not designed to be).

I control my music via the watch, use bluetooth headphones all day now.

When driving, the watch guides me -- no more holding my phone to see the map...the haptic + just in time directions are great.

I've taken the occassional call through the watch (you use it as a speakerphone).

It's a winner, and will only get better from here. It's a great fitness companion that lets you keep your phone in your pocket for all kinds of things, which feels remarkably more convenient.

No one NEEDS one. It's useful jewelry. That's compelling enough for me.

<comment><USER4> Took the words right out of my mouth

My personal uses:

At the beach - it allows me to keep my phone away from sand while still being able to control music / get messages / get email.

Answer my phone while washing dishes or working in the yard and my hands are wet or dirty

The activity monitor works great and is a nice addition when the wife and I go on bike rides

My wife was a skeptic also (as most people are) - I told her I wanted her to have one (why should I be the only one to have all the fancy toys) - <USER2> resisted, but said ok. 3 weeks later, she's attached to it as much as I am. Though, the first thing out of her mouth was "can i browse the internet with this?" A valid question, but clearly a question from a non technical person, haha

</comment></comment><comment><USER5> Great insight!</comment>


Tmux, dotfiles, and the joy of text - <AUTHOR> http://thechangelog.com/post/17827235767/episode-0-7-3-tmux-with-brian-hogan-and-josh-clayton <comment><USER2> My workflow with GNU Screen is something like the following:

      * start my computer
      * fire up a terminal
      * run `screen`
      * ssh into another machine
      * run `screen -e^vv` on the remote machine

At this point I can control my local Screen with ctrl+a and my remote Screen with ctrl+v. Is there an equivalent workflow with tmux? Is there an alternative? The podcast hints at "tmux inception" and everyone hints that they avoid tmux within tmux.

</comment><comment><USER3> Interesting Episode. But I really don't get it why I should trade the benefits of a real GUI-Editor like MacVIM with crippling myself in the terminal. What's the gain in switching editor- and shell-windows with tmux-Shortcut against using Command-Tab and the (customizable) shortcut for switching tabs in iterm?

Feels a little bit like nerdery for the reason of nerdery.

<comment><USER4> Because command-tab takes away your focus. There are other things in that menu as well that you might switch to. With tmux, I can split my windows into regions and turn the terminal into an IDE.

And then I can detach from it and leave it running as I switch to another configuration.

And I can invite you in to pair program with me over a low-bandwith solution.

Finally, when I switch from OSX to Linux I can have the exact same environment. For example, I use this on servers.

So it's the same reason you'd use vim over Eclipse. Sounds like nerdery for the sake of nerdery, until it works for you.

<comment><USER3> That's a valid point. But a lot can be done with conditional configurations om vimrc and gvimrc, my vim runs fine and nearly identical on OSX, Linux-Fluxbox- Sessions and on servers. IMHO we must stay flexible enough to use "the best" of the platform we currently working on, otherwise we are left with the lowest common denominator of a tmux in a terminal and must do some strange things to even use the clipboard in the right way(as heard in the podcast).

For example, I have Command-T opening FuzzyFinder in ssh-Sessions and in Gvim, but use the lovely Peepopen in Macvim.

<comment><USER5> At the end of the day, it's whatever works best for you. I've been using tmux for a year and a half and it's just as important to my workflow as vim. I've had people complain to me that I'm too fast with vim + tmux (either people I'm pairing with, giving a workshop to, or just demoing some code). It works for me. Vim looks great for me. I'm able to think less about what program I'm working in and more about how I get things done faster and with less context- switching.

If you're using GVim or MacVim or some other GUI and it works, wonderful! If you're content with the speed that you're able to code, that's awesome. That said, back when I was using TextMate and Terminal, I thought I was really _fast_ ; and I'm probably 30-40% faster in vim than I ever was (or could be) in TextMate.

Clipboard usage is a bit odd; that said, how often do you spend using the clipboard every day? If the time spent mucking with clipboard is less than the time you save doing everything else... then you have your answer as to whether switching will be worth it to you. It absolutely, positively, with out a shred of doubt, is worth it to me.

<comment><USER3> Of course it's always whatever works best for you. But it's interesting to see what you win and what you lose before adopting cool things you hear from hero programmers in a podcast. ;-)

Although I don't see me using nothing more than a tmux session exclusively in the future, I already adopted some things for my server work and I'm looking forward to the pragprog tmux book.

</comment></comment></comment></comment></comment><comment><USER6> Great podcast. I made a short screencast on Pair Programming with Tmux the same day. I'd love to know what you think: [http://readncode.com/blog/pair- programming-with-tmux-screenc...](http://readncode.com/blog/pair-programming- with-tmux-screencast/)

</comment><comment><AUTHOR> On sharing dotfiles, Josh around the 29:55 mark: "If you don't think your dotfiles are the best out there, you're doing it wrong."

</comment><comment><USER7> I hate to say this but... is there a tmux for Windows :0</comment>


7,500 Faceless Coders Paid in Bitcoin Built a Hedge Fund's Brain - <AUTHOR> https://www.wired.com/2016/12/7500-faceless-coders-paid-bitcoin-built-hedge-funds-brain/ <comment><USER3> The company the article refers to is [https://numer.ai/about](https://numer.ai/about)

From their homepage:

"February 27th 2016, an artificial intelligence named NCVSAI joined Numerai. . . He uses an untraceable email address. . . He is completely anonymous. His strongest prediction: buy Salmar ASA — a Norwegian salmon company. . . Numerai's hedge fund went long Salmar ASA."

Ok, so how is this anything but an anonymous proxy to arbitrage insider information?

Wrap confidential, non-public information under the guise of developing "AI" for trading?

I am serious, I don't necessarily understand the need for the data scientists to maintain anonymity, seems like the only functional reason is to let them break the law?

<comment><USER4> I've been participating in Numerai for a few months now. (I've only made some beer money from it, nothing serious.) When you get the data, you have no idea what it is. It's just a file with ~70,000 data points, each of which has 21 features. Each feature is uniformly distributed between 0 and 1. All you have to do is make a binary classification of 0 or 1 (or, more accurately, the probability that the data point is in class 0 or 1). They don't tell you what the 21 features represent.

As far as you know, these predictions could be used to make currency trades, or stock predictions, or real estate purchase, or something more exotic. You really have no idea. And since you don't know what these data points represent you can't use any insider knowledge about anything to help you.

<comment><USER5> EDIT: This is total conjecture.

only 1 or at most a few of those 21 features represent real data. The real data represents similar information to the insider information which they wish to act upon.

Example data prep:

1\. Insider source says that a contract is falling through, a patent is being filed for, quarterly numbers have been missed/surpassed etc.

2\. Similar information is gathered from historic performance data of the company, similar companies or market segments.

3\. The information is correlated with whatever metric they wish to move along and encoded in one of the 21 feature classes.

4\. Repeat for whatever relevant information that can be linked to the insider source - i.e. competing companies, re-encoding separately for long and short positions etc.

5\. Fill in remainder of 21 feature classes with noise.

6\. Profit.

<comment><USER6> What would be the point of all this? Step 1 is the illegal part, and the other steps don't erase that.

<comment><USER7> Well of course it is illegal, you just don't want to get caught, and this allows you to have plausible deniability.

</comment></comment></comment></comment></comment><comment><USER8> I went to school with Richard and so clearly remember him explaining this to me over coffee less than a year ago when the website had an under-construction style landing page and it's been amazing to watch this grow so fast.

The platform is great and I'd strongly recommend anyone wanting to get machine learning experience or who has played with Kaggle to check out Numerai!

The homomorphic encryption piece is fascinating and I think it'll be an important piece in balancing the privacy vs. utility of personal data as machine learning seeps deeper into the fabric of our lives.

<comment><USER9> > The trouble with homomorphic encryption is that it can significantly slow > down data analysis tasks. "Homomorphic encryption requires a tremendous > about of computation time," says Ameesh Divatia, the CEO of Baffle, a > company that's building encryption similar to what Craib describes.

> According to Raphael Bost, a visiting scientist at MIT's Computer Science > and Artificial Intelligence Laboratory who has explored the use of machine > learning with encrypted data, Numerai is likely using a method similar to > the one described by Microsoft, where the data is encrypted but not in a > completely secure way.

Doesn't this imply that homomorphic encryption isn't being used, but something like it instead?

<comment><USER10>
        """
        https://arxiv.org/abs/1508.06574
        "An encryption scheme is said to be homomorphic
        if certain mathematical operations can be applied
        directly to the cipher text in such a way that
        decrypting the result renders the same answer as
        applying the function to the original unencrypted
        data."
        The function = GradientBoostingRegressor
        the cipher text = X_encrypted
        original data = X
        same answer = mean absolute error
        """
        import numpy as np
        from sklearn.metrics import mean_absolute_error
        from sklearn.ensemble import GradientBoostingRegressor

        # Replicability
        np.random.seed(0)

        # Create a data set with 1000 samples and 3 features
        X = np.random.randint(0, 60, (1000,3))

        # Create ground truth (the product of the three
        # features - 100) / 11
        y = (np.prod(X, axis=1) - 100) / 11.

        # Encrypt y
        y_encrypted = y + 20

        # Encrypt X
        X_encrypted = X * -0.5

        # Init our model
        rgr = GradientBoostingRegressor(random_state=42)

        # Fit model on first 500 unencrypted features
        rgr.fit(X[:500], y[:500])

        # Predict the remaining 500 features
        preds = rgr.predict(X[500:])

        # Fit model on first 500 encrypted features
        rgr.fit(X_encrypted[:500], y[:500])

        # Predict the remaining encrypted features and decrypt
        preds_decrypted = rgr.predict(X_encrypted[500:]) - <USER2>

        # Evaluate both functions
        print(mean_absolute_error(preds, y[500:]))
        print(mean_absolute_error(preds_decrypted, y[500:]))

        #>>> 323.09
        #>>> 323.72

<comment><USER11> The encryption here is being done by "adding 20" / "multiplying by -0.5"?

Given this "encrypted" X , y dataset, I could easily find the unencrypted version... (even if I don't know 20 or -0.5, this still reveals so much of the structure that I don't believe it provides any real protection against anything except the most lazy attackers)

<comment><USER10> It is a toy example to show that a form of homomorphic encryption is possible, without going Fully Homomorphic Encryption.

And simple linear transforms on already anonymized features are not so easy to reverse engineer as you may think. Just try it on a few datasets from UCI.

<comment><USER11> Ah ok, sure. I wouldn't call something like a linear transform on anonymized features "encryption" (more like obfuscation?), but I guess it's good marketing in that it lets them associate with the "recent advances in [real] homomorphic encryption"

<comment><USER10> If you desire something more one-way, consider PCA, random projections, feature expansions (with something like Random Bits Regression), hashing, or the last hidden layer activations of your best in-house neural net. Then combine these approaches for good measure.

Agreed on the clever marketing, but at least they put their money (expensive dataset) where their mouth is (release it to reverse engineers the world over).

Fully Homomorphic Encryption challenges would be interesting, but it would disqualify our current state-of-the-art algorithms, and reduce the playing field to a handful of people who know how to write algo's that work with Fully Homomorphic Encryption (if any competitor at all is allowed to work on this, and not too busy working for the NSA).

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> A while ago I wrote about how AI might be trained on a data set and then used to form the basis of an alternative cryptocurrency for data mining. I.E. maybe you train the AI to recognize images of mountains and then use that as your proof-of-work algorithm to reward "miners" for finding related images on the Internet.

What I never imagined is that you might outsource the process of building that AI itself as a separate entity (maybe even a separate blockchain.) You could do the entire thing with commitments:

* Build a blockchain that is about rewarding data scientists for predictive models on data sets.

* Commit to a hash of a data set (the challenge to the network.)

* Hash enters chain.

* Release data set.

* Give them a dead line for a solution after which they commit to a hash of their solution / predictive model along with an ECDSA pub key for a reward.

* Solutions are released after N blocks.

* Top N solutions automatically receive rewards in this new cryptoasset.

* Zero trust would be required since it operate on outcomes that anyone can check.

* (This could also be done as an Ethereum smart contract instead of a blockchain.)

Scaling that would be quite hard though and you would need to use standard proof-of-work to avoid attacks.

IMO: What I find the most interesting about this article is that they're using masked data as the input which I think is the kind of futuristic cipherpunk vision a lot of people had in mind for Ethereum when it launched. Ethereum + AI + crypto + game theory is a match made in heaven and we probably haven't even scratched the surface of what's possible in this space. I can't wait to see the kinds of things people come up with in the future.

Edit: formatting.

<comment><USER13> great idea! although sharing data or model with blockchain is very costly I think?

<comment> surrey-fringe That would make it a _bad_ idea.

</comment></comment></comment><comment><USER11> Anyone want to speculate about how the data is being "encrypted"? It seems like they don't want to say, which immediately sets off red flags in my head...

I am pretty sure homomorphic encryption is not being used. I think if they are doing anything rigorous, _maybe_ they are using order-preserving encryption ([http://www.cc.gatech.edu/~aboldyre/papers/bclo.pdf](http://www.cc.gatech.edu/~aboldyre/papers/bclo.pdf)). This would mean that the only valid operations on the ciphertexts are comparison operations. I can't seem to find anywhere where numer.ai actually says how to interact with the "encrypted" data. I think it's a little strange that they would suggest that they are using homomorphic encryption yet have only comparison operations actually make sense on their "encrypted" data.

A second hypothesis would be that no encryption is being used at all, and this is just unlabeled features that have been renormalized within [0,1].

A third would be that order-preserving encryption is being used, but in an ineffective way which is basically just resulting in the second scenario. (understanding the security guarantees of order-preserving encryption is practice is very complicated)

</comment><comment><USER14> I might be wrong but this look like Bernard Madoff in disguise.

"Give me your money and trust my big black box that nobody understand!"

Then just to be more credible add some meaningless data point "We have 7 500 anonymous data scientists" because it is known, the less you now about people who write abstract model that manage your money the more you should be reassured.

"My son told me about anonymous they are a hell of good hackers!"

<comment><USER15> Unless they require some sort of financial contribution from the 7500, it seems like the biggest risk (apart from wasting your time) is that if you do come up with something useful you have no way to know the value of it; you basically just have to trust them.

<comment><USER16> I think parent's concern is scamming of investors, not programmers.

<comment><USER17> Why not both?

<comment><USER16> Sure. I was just responding to the original comment that spoke exclusively to the investor risk.

</comment></comment></comment></comment></comment><comment><USER18> That all sounds awesome and impressive, but I have the same problem with it that I have with any of the previous neural network hedge funds or, for that matter any of the 24-hour business news stations:

 __* If you truly had the ability to predict the price direction of ANY stock /fund/option in ANY consistent time frame (24-hours/week/month/quarter/year) ahead of time, with ANY accuracy greater than a coin flip, it would be _ridiculously_ easy to turn this information into piles of cash.

So, if you are NOT talking to me from atop your piles of cash, I have to assume your accuracy is no better than the flip of the quarter in my right pocket.

<comment><USER19> The article says: "Numerai's fund has been trading stocks for a year. Though he declines to say just how successful it has been, due to government regulations around the release of such information, he does say it's making money. And an increasingly large number of big-name investors have pumped money into the company"

So perhaps they are sitting on piles of cash?

<comment><USER20> This screams 'Ponzi scheme'. If you are thinking of putting money into this, don't walk away - run away.

Above-the-board trading institutions don't have problems with nebulous 'government regulations', when talking about how profitable their funds are.

<comment><USER21> You've no idea what you're talking about. Hedge funds (legally) can't advertise to retail investors (if they do, they're no longer hedge funds, which substantially increases their regulatory burden and narrows the range of strategies they can trade). Having said that, even if they could, they probably wouldn't want retail investors - if they're good, they can get rich investors (e.g. $10M+), reducing their administrative costs and allowing them to trade with longer horizons.

</comment></comment></comment></comment><comment><USER22> The idea that data science is some kind of numerical alchemy where you can just anonymize the data so much that the people doing the modeling don't even know what the problem domain is just irks me. It's utter nonsense.

</comment><comment><USER23> I wonder if in the end there will be just a single model, and a single faceless coder who actually makes the best predictions, and the whole of Numerai will be just an elaborate way of finding that particular person and his model to make the market.

<comment><USER10> With meta-modeling you can build an ensemble (single model) that beats any individual model. This should also have less variance (See Breiman's Bagging Predictors). I'm saying that even if there is a super talent on Numerai, one should still use, say, 0.95 to 0.05 weights, to "hedge your bets" and improve accuracy. With enough competitors, it becomes near impossible for a single agent to beat all the others combined.

Kaggle competitions also see this. [https://www.boozallen.com/content/dam/boozallen/documents/20...](https://www.boozallen.com/content/dam/boozallen/documents/2015/12/2015-FIeld- Guide-To-Data-Science.pdf) [page 93] shows a graph where a simple average of the top models in a competition gives an ensemble model with higher accuracy than any of the individual competitors.

I think that is largely the beauty of Numerai. Using adversarial agents to build a "collaborative" model. Disclaimer: Won a few bitcoin on Numerai.

</comment></comment><comment><USER24> This remembers me of the "neural network black box based forex trading" companies 10 years ago, based in countries where finance fraud did not count es a crime.

</comment><comment><USER25> This is about 2 (big) steps from the plot of "Daemon".

<comment><USER26> This one? [http://a.co/6lbILZm](http://a.co/6lbILZm)

<comment><USER27> Yep! It's a good read. Highly recommend it.

It has a follow up called Freedom(TM). Both books by Daniel Suarez.

<comment><USER26> Thanks! I placed a hold for it at my library.

</comment></comment></comment></comment><comment><USER28> >> " If you are a US taxpayer and have tournament winnings, you will be required to submit to Numerai a Form W-9 with your Taxpayer Identification Number, and you will receive from Numerai a Form 1099."

Bit of a stretch to say they do not know any of the names of coders.

Anyone know how many valid W-9s they've received? Seems like if the user was connecting from an IP in the US that the user would need to prove they're not a US taxpayer.

<comment><USER29> You have the option to self declare to be not a US resident.

<comment><USER30> If you actually are a US resident and declare that, you would be breaking the law.

And if you receive payment in a US bank account and declare yourself a non- resident, expect questions to be asked.

<comment><USER29> Payment is in bitcoin.

<comment><USER29> It's much more difficult to track. You can convert your bitcoin to cold hard cash at an ATM anonymously.

</comment></comment></comment></comment></comment><comment><USER31> How does this compare to Quantopian?

<comment><USER9> > Write your algorithm in your browser. Then backtest it, for free, over 14 > years of minute-level US equities data, and soon, US futures. (source: > Quantopian website)

Numerai encrypts the data they give you, so you can't just take your algorithm and start your own hedge fund (because you won't have any data to improve with).

</comment></comment><comment><USER32> So it could be 1 dev who supplied 7500 different models.

</comment><comment><USER33> No sarcasm intended here: Does it matter in terms of social utility whether these types of predictive models are making predictions of human decisions to buy or sell a stock versus predictions of AI based decisions to buy or sell a stock?

<comment><USER34> A model that we understand seems more useful than one that we don't. If the hedge fund is saying "buy company X but not company Y because company Y leases its equipment and that will cost them in the long term" (say), that's notionally more valuable than "buy company X because the computer says so", because other companies can learn the lesson of this and not lease their equipment.

In practice given that hedge funds tend to be secretive anyway maybe it makes very little difference.

<comment><USER35> It's reasonably easy (not trivial, not impossible) to track the trades of hedge funds, since they do have to disclose their transactions. And some guys will even tell you! [1] [2] [3]

 _Why_ they make their decisions is another question.

[1]: [https://www.bloomberg.com/gadfly/articles/2016-09-14/herbali...](https://www.bloomberg.com/gadfly/articles/2016-09-14/herbalife- leave-it-for-icahn-and-ackman)

[2]: [http://etfdb.com/etfdb-category/hedge-fund/](http://etfdb.com/etfdb- category/hedge-fund/)

[3]: [https://whalewisdom.com/](https://whalewisdom.com/)

</comment></comment></comment><comment> Paul-ish Without knowing much how they obfuscate the data, if anyone could figure out a way to unblind the data, they could reasonably predict what trades Numerai will make, right? Even more important, they could influence those trades.

<comment><USER36> No, because the predictions are not public. If they were, everyone could just copy the top predictions.

<comment> Paul-ish It may be possible to anticipate the predictions of others working from the same data if you know what techniques people tend to apply.

</comment></comment></comment><comment><USER9> Doesn't this assume that market is composed of completely rational actors? On the same note, why not just do sentiment analysis on the people trading and buying stocks?

</comment><comment><USER37> I'm pretty sure they're using Order Preserving Encryption and not true Fully Homomorphic Encryption. Marketing is great though - I love the look and messaging.

</comment><comment><USER38> Investing like this it is like implementing Texas Holdem Poker AI without knowing history of previous hands and your opponents attitude. Just won't work, all marketing around it is just a bubble.</comment>


Show HN: Applying the Unix philosophy to neural networks - <AUTHOR> https://github.com/cloudkj/layer <comment><USER3> I don't see any claims about performance, but I would be very surprised if it was anything better than abysmal. In a modern neural network pipeline, just sending data to the CPU memory is treated as a ridiculously expensive operation, let alone serializing to a delimited text string.

Come to think of it, this is also a problem with the Unix philosophy in general, in that it requires trading off performance (user productivity) for flexibility (developer productivity), and that trade-off isn't always worth it. I would love to see a low overhead version of this that can keep data as packed arrays on a GPU during intermediate steps, but I'm not sure it's possible with Unix interfaces available today.

Maybe there's a use case with very small networks and CPU evaluation, but so much of the power of modern neural networks comes from scale and performance that I'm skeptical it is very large.

<comment><USER4> > I don't see any claims about performance, but I would be very surprised if > it was anything better than abysmal. In a modern neural network pipeline, > just sending data to the CPU memory

Notice that the bulk of data does not necessarily go through the pipeline (and thus by the cpu). You may only send a "token", than the program downstream uses to connect to and deal with the actual data that never left the gpu.

<comment><USER5> Pretty sure you can't do this since the GPU's memory is per-process isolated as well.

<comment><USER6> Unless you share it between processes.

</comment></comment></comment></comment><comment><USER7> Excerpt: "layer is a program for doing neural network inference the Unix way. Many modern neural network operations can be represented as sequential, unidirectional streams of data processed by pipelines of filters. The computations at each layer in these neural networks are equivalent to an invocation of the layer program, and multiple invocations can be chained together to represent the entirety of such networks."

Another poster commented that performance might not be that great, but I don't care about performance, I care about the essence of the idea, and the essence of this idea is brilliant, absolutely brilliant!

Now, that being said, there is one minor question I have, and that is, how would backpropagation apply to this apparently one-way model?

But, that also being said... I'm sure there's a way to do it... maybe there should be a higher-level command which can run each layer in turn, and then backpropagate to the previous layer, if/when there is a need to do so...

But, all in all, a brilliant, brilliant idea!!!

<comment><USER8> > Now, that being said, there is one minor question I have, and that is, how > would backpropagation apply to this apparently one-way model?

The author mentioned that this is only for inference of neural networks (not training), so this does not support backpropagation.

<comment><USER9> This kind of misses the point of the Unix philosophy of being able to dynamically reconfigure things - realistically, to get decent results, you'll need to do inference with the exact same connections as you trained (or at least finetuned) them, so there's no good reason to split the model in smaller parts.

<comment><USER10> I think that there is room for this idea at a higher level of abstraction where pre-trained sub networks are exposed as command line utilities.

cat tweets.txt | layer language-embed | layer sentiment > out.txt

<comment><USER9> My point was that this is _not_ possible as the trained layers are intrinsically tightly coupled. You _can 't_ combine pre-trained sub networks in arbitrary manner without retraining. In all the standard practice of reusing pretrained networks, you would take a pretrained network or part of it, and _train_ some layers around it to match what you need, optionally fine- tuning the pretrained layers as well. If you want use a different pre-trained embedding model, you retrain the rest of the network.

In your example, the sentiment layer will work without re-training or finetuning only if preceeded by the _exact same_ language-embed layer as the one it was trained on. You can't swap in another layer there - even if you get a different layer that has the exact same dimensions, the exact same structure, the exact same training algorithm and hyperparameters, the exact same training data but a different random seed value for initialization, then it can't be a plug-in replacement. It will generate _different_ language embeddings than the previous one - i.e. the meaning of output neuron #42 being 1.0 will be completely unrelated to what your sentiment layer expects in that position, and your sentiment layer will output total nonsense. There often (but not always!) could exist a linear transformation to align them, but you'd have to explicitly calculate it somehow e.g. through training a transformation layer. In the absence of that, if you want to invoke that particular version of sentiment layer, then you have no choice about the preceeding layers, you have to invoke the exact same version as was done during the training.

Solving that dependency problem requires strong API contracts about the structure and meaning of the data being passed between the layers. It might be done, but that's not how we commonly do it nowadays, and that would be a much larger task than this project. Alternatively, what could be useful is that if you want to pipe the tweets to sentiment_model_v123 then a system could automatically look up in the metadata of that model that it needs to transform the text by transformation_A followed by fasttext_embeddings_french_v32 - <USER2> there's no reasonable choice anyway.

<comment><USER10> Yes. I understand how neural networks work. In my example language-embed and sentiment are provided by layer. This allows layer to provide compatible modules. If two modules which are incompatible are used together they might provide junk output. That is true for any combination of command line utitilies. If I cat a .jpg I'm going to have a hard time using that output with sed.

</comment></comment></comment></comment></comment></comment><comment><USER11> What's wonderful about this concept (and unix concept in general) is that the flexibility it gives you is amazing. You can for example pipe it over the network and distribute the inference across machines. You can tee the output and save each layers output to a file. The possibilities are endless here.

</comment><comment><USER12> Great concept. Would like to see more of this idea applied to neural network processing and configuration in general (which in my experience can sometimes be a tedious, hard-coded affair).

</comment><comment><USER13> I've been thinking about something like this for a long time, but could never quite wrap my head around a good way to do it (especially since I kept getting stuck on making it full featured, i.e. more than inference), so thank you for putting it together! I love the concept, and I'll be playing with this all day!

</comment><comment><USER14> This might not be a great way to build neural networks (as other commenters have said regarding performance). But, it could be a great way to learn about neural networks. I always find the command line a great way to understand a pipeline of information.

</comment><comment><USER15> Great idea but however equally great caveat - it's just for (forward) inference. Unix pipelines are fundamentally one way and this approach won't work for back propagation.

<comment><USER16> I don't see any reason you couldn't just spit out the output and the derivative of the layer output with respect to the weights, then multiply and carry these all the way down. Then if you have a loss function at the end you have the gradient. Probably this project is for fun and not scale so it's fine. But then you need to think about changing the weights on every layer based on the optimization

</comment></comment><comment><USER17> Sounds like the kind of thing John Carmack would enjoy hacking on.

<comment><USER18> How's that? Isn't he more of a C/game developer? Is he a unix guy?

<comment><USER19> [https://news.ycombinator.com/item?id=9810342](https://news.ycombinator.com/item?id=9810342)

 _John Carmack working on Scheme as a VR scripting language_

</comment></comment></comment><comment><USER20> See also Trevor Darrell's group's work on neural module networks:

[https://bair.berkeley.edu/blog/2017/06/20/learning-to- reason...](https://bair.berkeley.edu/blog/2017/06/20/learning-to-reason-with- neural-module-networks/)

</comment><comment><USER21> Wonderful idea and the Chicken Scheme implementation looks nice also.

I wrote some Racket Scheme code that reads Keras trained models and does inferencing but this is much better: I used Racket's native array/linear algebra support but this implementation uses BLAS which should be a lot faster.

</comment><comment><USER22> [https://www.jwz.org/blog/2019/01/we-are-now-closer-to- the-y2...](https://www.jwz.org/blog/2019/01/we-are-now-closer-to- the-y2038-bug-than-the-y2k-bug/#comment-194745)

<comment><USER23> Linking to jwz directly from hackernews will show a nutsack in your browser and not anything useful.

<comment><USER24> What an absolutely childish thing to do.

<comment><USER22> the problem is that every time somebody posts a link to his site, a bunch of HN folks go and say "hur hur" in the comments. Jamie doesn't have patience for fools.

</comment></comment></comment></comment><comment><USER25> Well - I will say I like the general concept. I just wish it wasn't implemented in Scheme (only because I am not familiar with the language; looking at the source, though - I'm not sure I want to go there - it looks like a mashup of Pascal, Lisp, and RPN).

It seems like today - and maybe I am wrong - but that data science and deep learning in general has pretty much "blessed" Python and C++ as the languages for such tasks. Had this been implemented in either, it might receive a wider audience.

But maybe the concept itself is more important than the implementation? I can see that as possibly being the case...

Great job in creating it; the end-tool by itself looks fun and promising!

<comment><USER26> Ironically, I was only mildly interested in this before I read your comment and learned that it's in Scheme. Now I'm eager to check it out, haha.

<comment><USER23> Me too! I was like holy smokes, it's in Scheme?! Make AI Lispy again!

</comment></comment></comment><comment><USER27> Reading the title, I can't help but think of the 'Unix haters handbook' and groan, why would you want to apply the unix philosophy to nets??

<comment> F-0X >I can't help but think of the 'Unix haters handbook' and groan

The one true gripe they outlined is that the user interface requires some explanation before one can bootstrap their own knowledge.

I'll defend the unix philosophy till I die, probably. Why _wouldn't_ you apply it here?

<comment><USER23> Because all the time spent marshalling and unmarshalling data structures into bags of textual bytes takes up CPU and does nothing but hasten the heat death of the universe. We have _better_ models for that sort of thing. See: PowerShell.

</comment></comment></comment><comment><USER28> I love you</comment>


Big Tech marshals a right-leaning army of allies for antitrust fight - <AUTHOR> https://www.axios.com/big-tech-marshals-a-right-leaning-army-of-allies-for-antitrust-fight-155d5760-86d4-4ce0-9cda-734dbc95e252.html <comment><USER2> "You either die a hero, or you live long enough to see yourself become the villain." Being in the good graces of the left isn't something you can count on. The fact that you got rich by making products people want and need and made peoples' live better is something that will be forgotten as soon as its politically convenient. Given which way the winds are blowing, "Big Tech" isn't making a mistake by trying to make some "right-leaning" friends.

<comment><USER3> In the early days, they were altruistic and doing things to make people's lives easier. And then they turned into advertising company who sells personal info that was gathered under a pretense of good faith and respect of privacy. Disliking google isn't a political whim; google's practices are a capitalistic whim. When those practices went from "do no evil" to "profit however and don't look to closely at the ethics", that's when they lost the support of this "leftist".

"It's better to die on your feet, than live on your knees"

<comment><USER4> Counter argument, it's wildly more important that people be able to monetize their data and most of the people against it are failing to appreciate how economically progressive data monetization is. People used to /pay/ for software, not just professional software but software in general. And when they didn't pay for that software it was often because there simply wasn't software of that kind for their use case. Being able to sell data is the primary reason consumers have access to the plethora of free and quality services that they have today. There has never been this many good free games, this much free music, nor any other similar quanitity of free and useful services like office software in history and a large amount of that is due to data monetization. It's nearly impossible for me to care about what the data privacy advocates have to say when the movement has historically handwaved away the social benefits of data monetization while clinging to dubious icons like Snowden and Manning.

</comment></comment></comment><comment><USER5> > Big Tech marshals a right-leaning army of allies for antitrust fight

"Status quo elites marshal an army from the faction defined by defending the power of status quo elites for fight that might threaten their power" has to be, at least very nearly, the most dog-bites-purpose-built-dog-chew-toy story in the history of news.

The unusual aspect of this is that the antitrust _threat_ comes from the far right largely as retaliation for platforms being insufficiently accommodating to the far right, not that the _defenders_ are the classic center-right-to- solid-right defenders of entrenched corporate power.

<comment><USER6> But will the employees revolt? Or will they think the ends justify the means?

<comment><USER7> Google routinely funds the campaigns of far right US politicians and pays lip service to social justice issues. The latter has historically been sufficient to appease the employees.

</comment></comment></comment><comment><USER8> The "right-leaning army" is just toll-paying to try and make this go away. Aside from Warren, the right is where the anti-trust charges are coming from.

It's like getting pulled over in a small town -- you pay a lot into the local economy to make the ticket disappear.

<comment><USER2> No, "breaking up big tech" fits squarely into the left's anti-corporate narrative: [https://www.reuters.com/article/us-usa-election-tech- factbox...](https://www.reuters.com/article/us-usa-election-tech- factbox/where-us-presidential-candidates-stand-on-breaking-up-big-tech- idUSKBN1ZN16C)

> Biden slammed big tech companies in a January interview with the New York > Times, saying he had "never been a fan of Facebook" and arguing that online > platforms should not be allowed immunity for content posted by users.

> Warren is leading the charge to break up big tech companies on the grounds > they hold outsized influence and stifle competition.

> Sanders, who frequently criticizes corporate influence, has also called for > the breakup of big tech companies such as Facebook and Amazon.

Make no mistake, this isn't the Clintons' party anymore (or it won't be for much longer). The young blood hates business and corporations and is pulling the whole party in that direction.

<comment><USER9> > "breaking up big tech" fits squarely into the left's anti-corporate > narrative

True but, (anecdotally) a growing number of my center and right-leaning acquaintances are warming to the idea of breaking up and/or (at least partly) nationalizing parts of the economy - particularly big tech - because we appear to be in a cold - potentially hot - war with China.

imo this idea is going to be coming from the right more frequently in the near future.

<comment><USER2> That's fair. The social conservatives and Tucker Carlson nationalists are getting wobbly on that front for sure.

</comment></comment></comment></comment><comment><USER10> The key sign of a sector's maturity is its presence on both/all sides of the aisle.

I never go into another country thinking "wait did you guys just elect extremists because if so that makes me not want to do business here" so why would I treat America differently?

When people use the term "virtue signaling" as a criticism of an organization's actions, it is based on the observation that the same organizations doing it are involved in plenty of other countries and didn't consider the political outcomes at all, only the convenience of what is possible in that country. Ireland, Cayman Islands, Netherlands are part of the standard cocktail, and when the head of state is rolling out a red carpet, I do not care how they got there.

</comment><comment><USER11> Considering the utter alienation of right-leaning opinions on Big Tech platforms under the guise of not being "advertiser friendly", they have a bloody tough fight on their hands.

</comment><comment><USER12> I think about this all the time:

Would it be possible to change the anti-trust laws so that companies can not merge or be acquired if the sum of their assets and revenue exceeds a set dollar figure adjusted for inflation?

If A(assets + revenue) + B(assets + revenue) >= $, then not allowed to merge.

This will promote competition from upstarts and increase the efficiency of these companies by preventing them from gobbling up every new market.

</comment><comment><USER13> I will be very curious when the Google Anti-Trust suit drops to see how the tech world reacts. We've seen a lot of consolidation in the past few years- Google seems to be gobbling up any tech company that collects a lot of personal information without consequence.

There are a lot of Googlers on HN who blindly follow Google, and more people who worship the ground Google walks on; i'll be curious to see what the discussion is like when Google's evil is laid out for the world to see. Lot of the stuff Google (and other companies like Facebook) is indefensible when you look at it from an anti-trust point of view. Stuff like this article, where Google is working with people who are against what a lot of Googler's stand for will be an interesting show.

<comment><USER14> > _We 've seen a lot of consolidation in the past few years- Google seems to > be gobbling up any tech company that collects a lot of personal information > without consequence._

Like what? I can only remember what seemed like some bad vr acquisitions and companies that were basically already Google Cloud products.

<comment><USER15> Since January, 2015, they've acquired Launchpad Toys, Odysee, Softcard, Red Hot Labs, Thrive Audio, Skillman & Hackett, Timeful, Pulse.io, Pixate, Oyster, Jibe Mobile, Agawi, Digisfera, Fly Labs, bebop, BandPage, Pie, Synergyse, Webpass, Moodstocks, Anvato, Kifi, LaunchKit, Orbitera, Apigee, Urban Engines, API.AI, FameBit, Eyefluence, LeapDroid, Qwiklabs, Cronologics, Limes Audio, Fabric, Kaggle, AppBridge, Owlchemy Labs, Halli Labs, AIMatter, HTC (portions), Bitium, Relay Media, 60db, Redux, Tenor, Velostrata, Cask, GraphicsFuzz, Senosis, Onward, Workbench Education, Sigmoid Labs, DevOps Research and Assessment, Superpod, Alooma, Nightcorn, Looker, Elastifile, Socratic, CloudSimple, Typhoon Studios, Pointy, North, and, last but not least, Fitbit.

Prior to that, Google acquired Dejavue, Outride, Pyra Labs, Neotonic Software, Applied Semantics, Kaltix, Sprinks, Genius Labs, Ignite Logic, Picasa, ZipDash, Where2, Keyhole, Urchin Software Corporation, Dodgeball, Akwan Information Technologies, Reqwireless, Android, Skia, Phatbits, allPAY, bruNET, dMarc Broadcasting, Measure Map, Upstartle, @Last Software, Orion, 2Web Technologies, Neven Vision Germany, YouTube, JotSpot, Endoxon, Adscape, Trendalyzer, Crusix, Tonic Systems, Marratech, DoubleClick, GreenBorder, Panoramio, FeedBurner, PeakStream, Zenter, GrandCentral, ImageAmerica, Postini, Zingku, Jaiku, Omnisio, TNC, On2, reCAPTCHA, AdMob, Gizmo5, Teracent, AppJet, Aardvark, reMail, Picnik, DocVerse, Episodic, Plink, Agnilux, LabPixies, BumpTop, Global IP Solutions, Simplify Media, Ruba.com, Invite Media, Metaweb, Zetawire, Instantiations, Slide.com, Jambool, Like.com, Angstro, SocialDeck, Quiksee, Plannr, BlindType, Phonetic Arts, Widevine Technologies, eBook Technologies, SayNow, Zynamics, BeatThatQuote.com, Next New Networks, Green Parrot Pictures, PushLife, ITA Software, TalkBin, Sparkbuy, PostRank, Admeld, SageTV, Punchd, Fridge, PittPatt, Dealmap, Motorola Mobility, Zave Networks, Zagat, DailyDeal, SocialGrapple, Apture, Katango, RightsFlow, Clever Sense, Milk, TxVia, Meebo, Quickoffice, Sparrow, WIMM Labs, Wildfire Interactive, VirusTotal.com, Nik Software, Viewdle, Incentive Targeting, BufferBox, Channel Intelligence, DNNresearch, Talaria Technologies, Behavio, Wavii, Makani Power, Waze, Bump, Flutter, FlexyCore, Schaft, Industrial Perception, Redwood Robotics, Meka Robotics, Holomni, Bot & Dolly, Autofuss, Bitspin, Nest Labs, Impermium, DeepMind Technologies, SlickLogin, spider.io, GreenThrottle, Titan Aerospace, Rangespan, Adometry, Appetas, Stackdriver, MyEnergy, Quest Visual, Divide, Skybox Imaging, mDialog, Alpental Technologies, Dropcam, Appurify, Songza, drawElements, Emu, Director, Jetpac, Gecko Design, Zync Render, Lift Labs, Polar, Firebase, Dark Blue Labs & Vision Factory, Revolv, RelativeWave, and Vidmaker.

</comment></comment></comment><comment><USER16> Although various individuals from the conservative with a small 'c' world plus libertarians are cited in this article, the arguments put forth here are for me another example of a misunderstanding of what 'right leaning' might mean in this era. Proponents and builders of surveillance capitalism and associated global data gathering for corporate and state uses tend to lean 'left', as in neo liberal left. The older tradition of conservatism tends to be much more interested in personal privacy and respect for the individual. Example: [https://www.theamericanconservative.com/articles/facebook- an...](https://www.theamericanconservative.com/articles/facebook-and-google- must-be-regulated-now/)

Neo conservatives and neoliberals (the UK Guardian is a good example of globalist neo liberal ideas) are largely the same thing with some nuances. Just as there are various flavors of 'left' this is also true of the 'right'.

<comment><USER17> _" Proponents and builders of surveillance capitalism and associated global data gathering for corporate and state uses tend to lean 'left', as in neo liberal left."_

Neoliberalism is a right wing ideology, not a left wing one.

 _" "Neoliberalism" is contemporarily used to refer to market-oriented reform policies such as "eliminating price controls, deregulating capital markets, lowering trade barriers" and reducing state influence in the economy, especially through privatization and austerity."_[1]

This is pretty much the _opposite_ from what those on the left aim for.

Those on the left want nationalization, not privitazation, regulation of capital markets, not deregulation, price controls, not their elimination, and more state influence in the economy, not less.

[1] - [https://en.wikipedia.org/wiki/Neoliberal](https://en.wikipedia.org/wiki/Neoliberal)

<comment><USER18> It depends on which "left" you're talking about - it's not quite so simple. Clinton's 90's administration is frequently (and I would say accurately) described as neoliberal. The democratic party has become much more business and corporate friendly over the past 40 years. I think you'd be right in describing modern-day progressives as the "left" that you are defining here in opposition to neoliberalism. Does that make neoliberalism itself right-wing? Are establishment democrats right-wing if they have adopted neoliberal policies, as many have? Is Clinton now considered to have been a right-wing president by that definition?

I think these are interesting questions to ask now as the left, broadly speaking, doesn't have a strong political identity, with many progressives further left of the democratic party and other disaffected voters leaning libertarian.

<comment><USER17> _" Is Clinton now considered to have been a right-wing president by that definition?"_

To the extent that Clinton embraced neoliberal policies, he would be considered to be pursuing a right-wing economic agenda.</comment></comment></comment></comment>


Testing their skills on California roads, self-driving cars have had 4 accidents - <AUTHOR> http://www.startribune.com/self-driving-cars-getting-dinged-in-california-4-have-had-accidents/303248161/

<comment><USER2> >Self-driving cars have had 4 accidents

Two of which came while humans were driving and the others were not the fault of the car.

<comment><USER3> wonder if they drive through South LA

<comment><USER4> If so, that has to be some sort of record.</comment></comment></comment>


Ask HN: Harnessing the collective abilities of the HN community - conrad-mac Have there ever been any attempts to harness the collective knowledge and skills of the HN community in to a single project for social good?<p>What would this look like? What might be some barriers to effective implementation of such a project? <comment><AUTHOR> I don't know about social good, but we once tried to get the HN community to help pick startups for YC to fund. This failed, not just in its top-level outcomes but also in most of its details along the way. A community like this doesn't respond well to "harnessing". Trying to bring it into any purpose- driven order seems to have a dumbing-down effect. Genuine intelligence, to the extent that it shows up, does so in the small, randomly, and without a purpose. If you try to make it happen in any predictable way, that discourages it. There are a lot of counterintuitive effects like that.

Edit: but I should say something positive too. If you do find a way to join forces with others in a project for social good, or anything else really, you can often get surprisingly interesting feedback from the HN community about it. It will just be sporadic.

One of these years I want to work on ways to support HN users to form more real-world connections with each other. It seems like that could lead to projects and collaborations that otherwise might never get started. Whether these were startups, efforts for social good, or whimsical side projects, it seems like people's lives could be enriched, and maybe there would be good effects beyond that.

</comment><comment><USER2> Using HN as a cluster of people with different knowledge and ideas seems like a good basis, although it might be a little too big.

About idea sharing, I've seen some reluctant commenters on a "Ask HN" post about side incomes when asked to share their ideas. I hope it's not a pullback force strong enough to stop your initiative.

<comment> conrad-mac Yes, my initial thought was that the community is quite large which might be somewhat of a barrier to getting something going.

Re: sharing ideas. I agree, although I sometimes find myself with a similar mentality when I have a new idea so I can somewhat understand.

</comment></comment><comment> seph-reed Would love to work on something together. Perhaps education?</comment>


Anonymous release appears: 33 GB of Aaron Swartz's JSTOR files - <AUTHOR> http://www.artificialeyes.tv/node/1002

<comment><USER2> It's an older torrent which you can download here: [https://tpb.ipredator.se/torrent/8014584/R.I.P._Aaron_Swartz...](https://tpb.ipredator.se/torrent/8014584/R.I.P._Aaron_Swartz_- _JSTOR_archive_35GB)

</comment><comment><USER3> Not to be a buzz killer, but didn't JSTOR provide free access [1] to over 1,200 journals to the public days before Aaron committed suicide? You've got to register to get access, but that Anon release seems noobish.

[1] <http://about.jstor.org/news/free-read>

EDIT: And it's also a copy of an old torrent, pointed out by several others. Nothing to see here.

</comment><comment><USER4> The size of this is suspiciously equal to Greg Maxwell's release of the JSTOR archive of the Philosophical Transactions of the Royal Society. Yes, the PDF scans of just those 18K documents are 33GB. I suspect Aaron's archive was far, far larger - and I similarly suspect that it really was destroyed, no matter how much Anonymous wants to count coup.

<comment><USER4> Ooh, weirdly the earlier torrent available of JSTOR data was JSTOR's free- access stuff older than 1923 - which is just about the same size.

How coincidental!</comment></comment>


Ask HN: Whatever happened to the 3-strike ISP piracy scheme? - <AUTHOR>

I remember the start date for the 3-strike anti-piracy program that ISPs were supposedly going to implement. (maybe it was 4, 5, or more strikes?)  That date went by and I haven't heard anything about it since.  Did the ISPs come to realize it did nothing to help them and that is would only upset their customers? <comment><USER2> It's six strikes and I guess you don't read mainstream tech news. [http://arstechnica.com/tech-policy/2013/02/six-strikes- enfor...](http://arstechnica.com/tech-policy/2013/02/six-strikes-enforcement- policy-debuts/) [http://arstechnica.com/tech-policy/2013/02/heres-what-an- act...](http://arstechnica.com/tech-policy/2013/02/heres-what-an-actual-six- strikes-copyright-alert-looks-like/) [http://arstechnica.com/tech- policy/2014/05/isps-sent-1-3m-co...](http://arstechnica.com/tech- policy/2014/05/isps-sent-1-3m-copyright-infringement-notices-to-us-customers- last-year/)</comment>


Ice Balls - <AUTHOR> http://the-tusk.com/2015/01/09/ice-balls/ <comment><USER3> What a wonderful piece. Brave, thoughtful, funny. One of the things I most enjoy about my gender-nonconforming friends is how they can make some causal, offhand observation about how people act that will crack my head open. All my life, I'll have just taken some aspect of a socially constructed gender binary as The Truth. And they'll point out something that is at this point utterly mundane to them because they've seen it from multiple perspectives. Often something that they'd rather not even deal with, but everybody keeps pushing on them.

Thanks for posting it.

(And also, I am darkly pleased to find out that club is even more awful than I expected. The "stylized power ritual" paragraph is perfect.)

<comment><USER4> I took a totally different message away from the story. It was a story -- one you don't often hear -- of someone who discovered their trans tendency was a result/coping mechanism borne of early trauma.

I don't think the club sounded any more/less douchey than a place like HN (the owner notwithstanding). It sounds like a place where a certain group of people complain about the issues that affect them. Like all specialized groups, their complaints sound ridiculous to non-members. For comparison, consider this complaint: "We just can't afford a house in Palo Alto, all we can spend is $800k and anything halfway decent is at least $2MM" \-- that conversation probably happens 100x daily in the Bay, but it sounds ridiculous and hopelessly disconnected from reality to anyone not living here. Perspective.

<comment><USER5> I'm not sure if the message you took from the story was intended, but you do raise an interesting point.

Because of a number of reasons I've always spent a lot of time around people who struggled with their gender/sexuality in society, and there were more than a few individuals, mostly gay men, where I strongly suspected that they were gay, at least in part, by choice.

Now just to be clear: I think it can be dangerous, damaging, offensive and ultimately pointless to make such a judgment, but as a (private) observation it fascinated to me.

(On the other hand, I've also met a number straight men of whom I suspected that they could be comfortable gay or bisexual, but 'chose' the more orthodox, straight identity.)

</comment></comment></comment><comment><USER6> I don't know about the rest, but the physics behind the ice ball does not seem right to me. The ice absorbs heat from the drink and cools it. The ice melts at a rate governed by how fast it absorbs the heat.

If the ice melts more slowly, it absorbs less heat, so it cools the drink less. You can get the ice to melt more slowly by reducing its surface area, but your drink will also be warmer.

<comment><USER7> Here's a video of the ball in action [https://www.youtube.com/watch?v=uVwhVz_y8m0](https://www.youtube.com/watch?v=uVwhVz_y8m0)

The ice press sells for $700 to $1,100, depending on the size. [http://www.williams-sonoma.com/products/cirrus-ice-ball- pres...](http://www.williams-sonoma.com/products/cirrus-ice-ball-press-kit/)

<comment><USER8> Whats wrong with just using a round mold [http://www.amazon.com/Ice-Ball- Maker-Premium-Silicone/dp/B00...](http://www.amazon.com/Ice-Ball-Maker- Premium-Silicone/dp/B00CKH8ITQ) ?

<comment><USER9> Quoting from bartender Jeffrey Morgenthaler's recent book:

> My experience using these has resulted in more than a few cracked ice balls, > whose deep fissures essentially increase the surface area of the ice ball, > negating any benefits in that department.

I had pretty much the same experience with my molds (I was using the Tovolo brand, rather than the one you linked to, though).

<comment><USER10> They work better with filtered and double/triple boiled water.

Or you could go for a pricier option such as [http://www.wintersmiths.com/](http://www.wintersmiths.com/) (which uses directional freezing).

</comment></comment></comment></comment></comment><comment><USER11> I think there are bad parts of both genders- would this story be much different if it was a trans woman working in a female super upscale department store, and coming to the same conclusion, but that she'd rather stay a man?

<comment><USER5> I think the difference is that it would be less interesting to me.

What I like about this article is that it's (for me) a rare case of an article that doesn't highlight the advantages of being a man, or the disadvantages of being a woman.

</comment></comment><comment><USER12> Am I the only one that thinks this is kind of surprising? I was taught that transgenderness wasn't something you could choose, but apparently this woman _can_ and did choose her gender. She didn't "deep down feel like a man" but still was identified with gender identification disorder.

<comment><USER13> There are multiple possible answers. The simplest being, that she was originally mistaken in her interpretation of her depression.

It doesn't strike me like the author chose to be a man or stay a women like how the rest of us chooses between vanilla and chocolate ice-cream. It was a profound realization that where she was going in life would not make her happier, only more sad.

Another explanation is that she identifies 50% male and 50% female and both are equally right or wrong for her. But she still needs to choose to be either or.

</comment></comment><comment><USER14> That was strangely moving and beautiful. Normally I'd have abandoned such a piece early as an overly dramatic representation of one person's demons - <USER2> I've got plenty of other people's demons on my mind already. This author was different. I think it was the pace with which she revealed her situation and the process that brought her to where she is today - deliberate and very careful.

</comment><comment><USER15> I believe the establishment that the author references is Wingtip, a pretentious clothing store + private club in FiDi.

[http://wingtip.com/club](http://wingtip.com/club)

Not that it really matters for the (excellent) piece. Just thought some other readers might be curious.

<comment><USER16> No need to add dox'ing to a perfectly wonderful story.

</comment></comment><comment><USER17> > _I learned mansplaining was not a thing men do to women, but rather a > dynamic some men do to all people at all times._

Interesting.

<comment><USER18> At that point, is it a sexist microagression anymore?

Further, what if the mansplainer's friends don't mind being "mansplained" to? At what point do accusations of mansplaining simply become nasty ridicule of those who take joy in knowledge of the world around them?

<comment><USER3> Mansplaining isn't about joy in knowledge. Read, as an example, the Solnit piece which led to the coining of the term:

[http://articles.latimes.com/2008/apr/13/opinion/op- solnit13](http://articles.latimes.com/2008/apr/13/opinion/op-solnit13)

That wasn't about joy in knowledge. He interrupted her to expound about a book he _hadn 't even read_, while energetically ignoring explicit statements that she not only the author of a book on the same topic, but the author of the very book he was mansplaining about. And then, when he finally realized that an expert on the topic was in front of him, he still wouldn't shut up.

I know a lot of smart people who take joy in knowledge. Who take joy in _sharing_ knowledge. But every one of them can also shut up and listen.

Mansplaining is about social dominance, not joy.

</comment></comment></comment><comment><USER19> Ice is a poor conductor of heat. This means that thermal energy of the liquid medium will be exchanged between the ice near the surface of the ice body/bodies. Thus, the rate at which the ice accepts energy from the alcohol is determined by the surface area of the bodies of ice. A single sphere of ice produces the lowest possible surface area. On a graph of drink temperature vs time, crushed ice (highest surface area) would produce a large dip and then level out. The orb would create a broad valley of lower temperature, though never as cold as the crushed ice.

</comment><comment><USER20> I'm sure I'm not alone in finding this piece incredibly obnoxious. Complaining about folks being judgmental and condescending while yourself being judgmental and condescending isn't profound.

</comment><comment><USER21> Entertaining, but I feel sorry for her/him for having such a misguided impression of the world. Generalizing about men from the experience in a silly club is just wrong.

Also, if she wants to see condescending women, she should go to a meeting of mothers comparing the progress of their kids. Although as a trans she'll probably be spared.

</comment><comment><USER22> Wonderful read!

Entertainingly, the author uses "condescending about ice balls" as a symbol of all things she hates about the testosterone-driven world. In the comments here: people arguing about ice-ball physics. Metaphor complete!

<comment><USER23> Wouldn't it be ironic if the people arguing about ice-ball physics were doing so simply because they enjoyed the mildly entertaining physics problem, while the people attributing the ice-ball arguments to a need for condescension were only doing so because they themselves needed a mechanism to condescend towards the people arguing about ice-ball physics?

<comment><USER24> Wouldn't it be even more ironic of both groups of people discussed ice balls because they mildly enjoyed the physics problem, and the perception of condensation was one entirely fabricated in the mind of some more sensitive observers?

</comment></comment></comment><comment><USER25> oh god i'm a trans woman and i want this so much

<comment><USER21> what exactly?

</comment></comment><comment><USER26> Close call.

</comment><comment><USER27> I don't get what's so beautiful or brave about this blog post. I personally felt it was more cynical and attention seeking.

<comment><USER28> It's brave because it's talking very honestly about something very personal.

I don't know whether I'd say it's beautiful, but it's definitely well written.

It's cynical, but that's also part of why it's funny.

And of course it's attention seeking. All writing is?</comment></comment>


Ask HN: What would you like to search and can't? - <AUTHOR>

Here's something that keeps me up at night. My company (IndexTank) makes it possible for people to build custom search engines. I've been hacking a few pet apps with our product to search  stuff that's important to me (e.g. our irc channel logs).
There are many apps out there with poor or no search (e.g. HN), but in most cases you can make do with Google (searchyc in this particular case).
What data/apps are out there for which there are no acceptable search options? <comment><USER2> Source code and programming related searches are often problematic. For example, "-" is a search operator, and so is the word "and". Special characters are dropped by default. Also, specifying the language results must be in would be handy. There has been some work on this (koders.com) and you can work around it on google with quotes, but I'd say there's no killer appp yet. Some query examples:

foo -bar (where bar is a command line option)

foo and bar

@username

session[:user]

"rails nested routes"

One particular problem is that google has an anti-youth bias, so very often, I get old doc or results for an error or tutorial unless I'm careful. For example, "rails nested routes" returns old rails 2 route info unless I add in "rails 3"

</comment><comment><USER3> Not sure if this reply will help you, but I'd like to be able to "search" the physical stuff I own, especially the documents. Could it be possible with some sort of RFID tags?

<comment><AUTHOR> The search part would be the easiest. The hard part is keeping track of spatial coordinates for all those items in near real-time. If you could tag say 1000 items in your house and gave me that api, I could set up a web app for you to find them.

Does anyone know if such technology is readily available/cheap?

</comment></comment><comment><USER4> One of the major problems I have is that I talk to my friends across all kinds of different media: Facebook, Twitter, AIM, MSN, e-mail, forums... sometimes one of us says or links something and I'd like to be able to search everything, but I have to search each medium separately.

Privacy concerns probably make this kind of convenience impossible though.

<comment><AUTHOR> I've been thinking about that a lot. Greplin is supposedly working on that problem but it's impossible to sign up these days.

</comment></comment><comment><USER5> Web page source code (not just visible content).

<comment><USER6> What would be interesting would be an HTML !diff type search: Find all the pages that are coded similar to this [example].

This would allow comparison of code and see how others do things.

</comment></comment><comment><USER7> It's not really what I would like to search, but really how I would like to search. Adding filters or restrictions is becoming very important to me. For example, I want to search google and I don't want my search results to return any stackoverflow copy cats, or I'd like to be able to group my results by something (e.g. domain, relevance, date, etc). Also, I'd like to specify what to search (e.g. search title only, h1 only, body only, etc..)

<comment><USER8> I agree. There are no good search options for power users that want to get very specific.

</comment></comment><comment><USER9> All my reading history across various channels. I think I read news in Hacker News, Instapaper, twitter and Google Reader. I have on many occasions said "I just read this cool article, let me find it for you..." only to spend 20-30 minutes trying to find it with Google site: incantations.

</comment><comment><USER10> All of my phone/voice conversations that I have ever conducted through my iPhone.

</comment><comment><USER8> Full text search (HTML aware) on my browsing history. I'm working on it :-)

<comment><USER11> Google Chrome already features this in the Bookmarks > Show Full History view. Make sure to close the page before searching for it otherwise it won't be indexed yet.

Perhaps you mean something else by "HTML aware" and this is not what you're looking for.

<comment><USER8> "Google Chrome already features this"

This works great for searching your Chrome history on the current computer. Throughout my day I use multiple browsers on at least 3 machines.

"HTML aware" is exactly what I'm looking for. I want to be able to limit my search only to links or headers or maybe even text within elements with class="content"

</comment></comment></comment><comment><USER12> Math related search is generally not good. You can find theorems and definitions but proofs (non-famous ones) exercises are difficult to find.

</comment><comment><USER13> Wha about search for sites like posterous or Tumblr?

<comment><AUTHOR> Posterous is an interesting one. I'd build a whole-site search function if they had a firehose-like api call, but as far as I know it's only possible to access content for an individual blog.

</comment></comment><comment><USER14> The dark depths of my soul.

<comment><USER15> Are you sure you wouldn't rather search the life of the mind? :)

<comment><USER14> Look upon me!

</comment></comment></comment><comment><USER16> The female mind.

<comment><USER17> Females of what specie?

<comment><USER16> You're kinky!

</comment></comment></comment><comment><USER18> About half of all the content on the Internet.

What content is that? The part where keywords are from poor down to useless to characterize what the user wants and _means_ in their search.</comment>


How (not) to forecast an election: Analysis via a hierarchical Bayesian model - <AUTHOR> http://datart.com.br/blog/us_2016_elections_en.html <comment><USER2> 538 made an interesting point about these kinds of biases. Essentially, polling errors are correlated. You can't unbias your estimate, but you can realize that if voters are reluctant to state their Trump preference in Wisconsin, then they are also likely to be reluctant in Michigan. If you know how correlated your errors are, you can properly account it in your own uncertainty, probably leading to a prediction that is closer to 50-50.

<comment><USER3> Actually the discrepancy stems from underestimating voter turnout for Trump and over estimating it for Clinton and is not caused by the mythical timid Trump voter.

<comment><USER2> Doesn't matter. All that matters is that the errors are correlated. If voter turnout is biased in one state, then it likely is in another. The reluctance was just an example.

</comment></comment></comment><comment><USER4> "We have found that even a modestly sophisticated statistical model does very little to counter unreliable data"... so.. garbage in... garbage out?

I have an alternative explanation for this : people who buy newspapers and mainstream web sites are liberal. Pollsters and analysts who wish to make a living must get clicks and sell copy. The ones that create models that agree with the general zeitgeist of the liberal milieu get clicks and sell copy, other models are gradually ground to extinction.

Markets are not efficient mechanisms for creating insight.

<comment><USER5> Unfortunately your explanation is not correct. For example, it does not explain the extensive coverage of Donald Trump that was used to get clicks and sell copy, despite his violation of everything liberals hold dear...

Early in the campaign, Trump acknowledged his media advantage. "I've gotten so much free advertising, it's like nothing I'd have expected," he told the New York Times in September. "When you look at cable television, a lot of the programs are 100% Trump, so why would you need more Trump during the commercial breaks?"[1]

[1][http://www.marketwatch.com/story/trump-has-gotten- nearly-3-b...](http://www.marketwatch.com/story/trump-has-gotten- nearly-3-billion-in-free-advertising-2016-05-06)

[http://www.politico.com/blogs/on-media/2016/02/les- moonves-t...](http://www.politico.com/blogs/on-media/2016/02/les-moonves- trump-cbs-220001)

<comment><USER6> The coverage of Trump was entirely negative, though. It makes perfect sense that one could sell a magazine to liberals by attacking the main competitor to liberals in the Presidential race.

<comment><USER5> And the coverage of Hillary wasn't?

<comment><USER7> The coverage of Hillary tended to criticize her actions (e.g. regarding emails). The coverage of Trump tended to criticize his character. Objectively speaking, one was more of an ad hominem attack, which reflects poorly on the media.

<comment><USER8> Ad hominem is a fallacy when evaluating an _argument_. When you're evaluating a person (as a candidate), arguments are necessarily ad hominem, whether they're about temperament or conduct.

<comment><USER7> As someone who thinks elections should be decoded by positions on policy not moral character or temperment, I disagree.

<comment><USER8> You may disagree with the metric, but that doesn't make it a _fallacy_ , just a difference of opinion.

<comment><USER7> I didn't use the word fallacy. You did.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> One of the better articles I've read on why US election forecasts are inaccurate. And written by a couple of guys from Brazil.

While beyond the scope of this article, it would be great see more research done into the relative strengths of each bias, i.e. which biases are skewing polls the most, rather than just using lumped bias terms. Though I did see after Brexit forecasts got it wrong too the British Polling Council is "looking into it".

What's most interesting is there wasn't more of an effort by the media to get accurate vote forecasts before the election. Granted, it's more expensive and takes longer but it's not terribly difficult to do. It does involve a research team and statisticians spending time in person in the state and getting deeper data from people. But the time/costs aren't as outrageous as one would think because it only needs to be done in the handful of close swing states. And a sample size of just a few hundred people is enough to get significant results. Seems obvious and worthwhile, maybe in 2020 they figure it out.

</comment><comment><USER10> This is cool! But the bias slider isn't draggable for me on chrome on linux mint</comment>