Automatically Check In To (get an awesome seat on) Your Next Southwest Flight - <AUTHOR> http://checkintomyflight.com/ <comment><USER2> Southwest sues people who monetize this kind of product -- see a 2012 example at [http://www.bizjournals.com/dallas/news/2012/03/02/Southwest-...](http://www.bizjournals.com/dallas/news/2012/03/02/Southwest- sues-to-stop-website.html?page=all) or a 2007 example at <http://www.internetlibrary.com/cases/lib_case511.cfm>.

<comment><AUTHOR> I'm not making money off of it. I just built it for myself and thought I would share

<comment><USER3> Your site could also compete with their 'Early Bird Checkin' service that that they charge for:

[http://www.southwest.com/flight/early-bird-retrieve- reservat...](http://www.southwest.com/flight/early-bird-retrieve- reservation.html)

<comment><AUTHOR> I think that those are taken out even before they release the check ins to the public.

<comment><USER4> Correct. Early bird and A-List automatic checkins occur 36 hours prior to the departure of the flight. The general boarding checkin starts 24 hours prior.

<comment><AUTHOR> Awesome, thanks for the info - just curious how did you find this out?

<comment><USER4> [http://www.southwest.com/html/customer- service/faqs.html?top...](http://www.southwest.com/html/customer- service/faqs.html?topic=rapid_rewards_program_terms_and_conditions)

 _Southwest Airlines automatically reserves a boarding pass for A-List Members beginning 36 hours prior to scheduled departure of the Member's confirmed Southwest Airlines flight._

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> I'm gonna go out on a limb here and say that what this discussion needs is a "Hey - thanks for making this."

<comment><AUTHOR> Thanks!!! I was feeling pretty bummed after the comments.

<comment><USER6> Don't be bummed. Even though I personally wouldn't use this (my life is too short to fly Southwest if I don't have to) I think it's great that you made something and are sharing with a community of people who are making things.

<comment><AUTHOR> Thanks! :) Makes me feel better:)

</comment></comment></comment></comment><comment><USER7> With all the talk of you probably getting a C&D shortly, the real coup would be to release the code that does it for free. Can't imagine they can do much against that.

<comment><USER8> agreed. open source lives forever!

<comment><USER9> Yes open source this please.

or wait..why don't i spend an hour and write it myself.. ha

</comment></comment></comment><comment><USER10> Betting this will be down within 72 hours. I know a guy who did this years ago. The letter he got was not so polite (but I love it!). I don't think they ban them for the money, I think it is to make sure humans are doing the work because it shows real intent to make your flight. Their overbooking algorithms don't work so well otherwise...

</comment><comment><USER11> This is a bad idea. Not only is it against Southwest policy, it also pisses off people like myself who actually go and check-in "exactly" 24 hours before and end up with a "B-7" boarding pass!!!!!

Clearly, Southwest has their reasons for assigning boarding priority 0-24hrs before flight on a first-come-first-serve basis, or else they would just assign priority during payment (early-bird checkin $10 notwithstanding).

Possible Reasons: 1) Loyal customers can get a "one-up" on new fliers or people who don't really care when they board and where they sit. 2) People who don't change flights within 24 hrs of takeoff can also take advantage of this --the rest might be "penalized" in this model.

Sites like "checkintomyflight.com" break apart the aformentioned model.

But, this is hacker news. So, congrats on building this service.

<comment><USER12> A-List folks get auto-assigned a boarding order 36 hours before a flight starting with A-16. A1-15 are the "Business Select" seats for which a premium is charged. So, on flights with many frequent fliers, many of the "A" slots may be not be available even to those who check in early.

</comment></comment><comment><USER13> Very cool! Can you somehow make sure, I don't have to put in all this information? For eg. if I forward the confirmation email to an "automatic checkin email id", can you pick up the relevant information and automatically check me in.

<comment><AUTHOR> Thanks!! Awesome idea - totally - do you think people would be okay forwarding their itinerary?

<comment><USER14> They're giving you enough information to get the itinerary through the website no? If so then I see no reason why I wouldn't just forward the email for the sake of convenience.

<comment><AUTHOR> That's true - okay good idea - I'll see if I can whip this up. Any ideas on the best way to receive and have a system to grab and parse email?

<comment><USER15> One method would be to setup a normal email address (even something like gmail) and on the server setup a cron job that connects to it via imap and downloads the messages. Probably not the nicest solution, but it might be easiest.

<comment><AUTHOR> Cool - I'll try this

</comment></comment></comment></comment></comment></comment><comment><USER16> Normally you want to create an application that becomes more valuable as more people use it. This would be the opposite.

<comment><AUTHOR> There are hundreds to thousands of Southwest flights every day. Chances of collisions are small and even if they happen each person is still better off than if he/she had not used the site.

This isn't meant to be a "take over the world business", it's just a little app I built that I thought I should share with my friends and anyone who wants to benefit from it :)

</comment></comment><comment><USER17> Very nice! I wrote a simple python script with urllib2 and beautifulsoup to do the same thing a while back. When I was making it I noticed that the southwest servers would sometimes report slightly different times in the response headers (I'm assuming because of load balancing or something). To deal with this, I had the script start polling the check in page 5 minutes before and steadily increase the rate as time went on. I doubt it actually made much of a difference but it was pretty fun to make.

<comment><AUTHOR> tterrace - email me, something funny to tell you. Also Thanks!

</comment></comment><comment><USER18> I assume this is for those of you too cheap to spend the $10 to have this done by Southwest. Pay for service that's worth it.

<comment><USER17> I assume this is for those of you too cheap to spend the $0.25 on a phone call. Pay for a service that's worth it.

</comment></comment><comment><USER19> Simple yet brilliant. I love the simple design and clean user experience. Emailing me when I've checked in with my boarding pass is extra nice as well. Well done, Nikil!

I was just buying a Southwest ticket and was complaining about having to remember to check in when I found this site. It made my day.

</comment><comment><USER20> Now I wonder what happens if everyone on a flight signs up for this service. Who gets priority?

<comment><USER9> hey everyone..i wrote a program that automated posting to this service so you will be the first one in the sites queue.

<http://checkintocheckintomyflight.com/>

</comment></comment><comment><USER21> Will this check in both people travelling under the same confirmation number?

</comment><comment><USER22> This is awesome - I've often thought about building something like this myself. Have you considered writing a little about your implementation? Background jobs and web scraping? An API?

<comment><AUTHOR> Good idea! I literally just built this for myself so I didn't think about anything like that.

Would it be interesting?

Thanks!

</comment></comment><comment><USER23> Thanks for making this. I live in Dallas and fly out of Love Field frequently and use SW every chance I get.

Hope this stays around.

</comment><comment><USER24> Why only Southwest? Do other airlines not allow this?

Also, are you using some API or a more "primitive" method?

<comment><USER25> Southwest is one of the few (American) airlines that assigns boarding groups based on checkin time rather than some other factor (usually seat assignment). Also, since Southwest is the only major American airline that doesn't offer pre-assigned seating, it's the only one where getting in an earlier boarding group gets you a tangible advantage beyond having an easier time finding overhead luggage space.

<comment><USER7> Some other factor is usually $.

<comment><USER4> Having been a frequent flyer, I have noticed that Southwest's rates are no longer as low and competitive as they were in the past. Frontier Airlines here in Denver usually has lower rates and better amenities, like Direct TV and stretch seating. Too bad I have all my airline points with Southwest.

<comment><USER26> I used to fly SWA pretty frequently too, and now I never even bother to check their site unless I'm planning very far in advance. They are usually 30-50% more expensive than even traditional carriers on most routes originating in St. Louis.

</comment></comment></comment></comment></comment><comment><USER27> Oh my god I love you

<comment><AUTHOR> :D

<comment><USER28> I just signed up.. I fly Southwest all the time and this is a MAJOR pain point.. BRILLANT!!

If you do get shutdown.. will you honor the existing reservations made?

<comment><AUTHOR> haha thanks!!! Um... I'll try, wanna help?

<comment><USER27> even if you get shutdon you've inspired me to write a script to do it myself. any chance you would publish your code if you were shutdown?

</comment></comment></comment></comment></comment><comment><USER29> Yeeah Nikil! Nice work!

<comment><AUTHOR> Thanks ZG! :)</comment></comment>


The youngest person to be cryogenically preserved - <AUTHOR> http://www.bbc.com/news/world-asia-34311502 <comment> martin-adams I find cryogenics such a thought provoking field. If you're going to be frozen after death, you would probably want the restoration process to be fully mature before you are restored. But yet, someone has to go first (outside of shorter frozen trials).

But then again, who is to say that the restoration process will take the form of thawing out the brain. An advanced digital scanning technique could imprint the brain image onto an organic robot, thus making it possible to have many copies of the same person.

I also find the concept similar to the teleport. You may be the person who goes in, but are you the same person who comes out the other end? Indistinguishable from you, only you are not the observer of your own reality.

[Edit: A couple of really obvious grammatical errors]

<comment><USER2> Is there any name for this puzzle? I've been thinking about this for a while now.

Cloning the particles (and their states) that make you up would most likely produce a separate mind, as I think would teleporting where v1 is destroyed and v2 consisting of different particles, although in the same configuration. But what about separating said configuration, transporting them somewhere else and putting them back together? Cryogenics is in my opinion the only possible way to give a chance in preserving an original mind well beyond natural human lifespan without actually extending it.

Not only can't others tell a clone apart from the original: the hypothetic clone, as I understand it, couldn't do that either. Now what if this process, due to cellular regeneration in the brain, happens constantly? Your mind is not the same it was a minute ago: that mind is dead and gone. Would it even matter?

<comment><USER3> The thing is, unless you have certain metaphysical beliefs, the very concept of "identity" is a flawed one and doesn't really represent the reality of us being actually a complex aggregate structure of cells. The concept of "self" is in my opinion just a rough approximation or a framework that we use to quickly assess reality, much like time, as we commonly think of it, does not fully represent the complexities of relativistic time, however it is useful to manage our daily life[1]: the sentence "I'll be with you in 5 minutes" could open a whole lot of possibilities if we consider time in a relativistic scenario, whereas it is fairly unambiguous in our daily life. Similarly, the concept of "self" would be very ambiguous with regards to e.g. annelid worms (who can be split in 2 and survive as two separate individuals[2]) whereas it sums up fairly well the way we think of ourselves as individuals.

[1] See [http://www.economist.com/news/science-and- technology/2166333...](http://www.economist.com/news/science-and- technology/21663338-according-theory-relativity-time-elastic-it-can-slow-down- speed-up-and-even-reverse) [2] See [https://en.wikipedia.org/wiki/Fragmentation_(reproduction)](https://en.wikipedia.org/wiki/Fragmentation_\(reproduction\))

</comment></comment></comment><comment><USER4> Serious question: what's to salvage from the brain of a terminal brain cancer child? This strikes me as a very silly way to preserve a human being. If they really wanted to give their child a "chance" to live a full life, they should have cryogenized her sooner, no? (It might be illegal, though.)

<comment><USER5> Personally, I'm with you on this. Cryonically suspending a person that died from brain damage or debilitating disease has even more fleeting chance for success. Unless there's a chance that at the moment of death the conscience is still there (due to great redundancy in the brain), it might be futile, after all. This is one issue that I believe all cryonics companies and advocates prefer to wholly overlook.

I imagine it should be terrifically hard to let go of your child and ' _kill_ ' them preemptively, for them to have a hope of later life. Even if the parents did even consider that option.

As far as I see it, cryopreserving a person that is not legally dead (' _cryothanasia_ '?) _might_ be possible, but no cryonics company has procedures in place to arrange for it and I am not aware of anyone that has been preserved this way. At least, it is necessary to move to a country where voluntary euthanasia is legal and the associated autopsy is not mandatory, and you are on your own with this. [1] This is another issue that cryonics companies and advocates prefer to overlook.

Cryonics is still very niche as it is. People are still very reluctant to arrange for cryopreservation beforehand, as it is. Cryonics companies have their hands full with just continuing to operate and convincing people to use their services. For there to exist people that are _fully rational_ about their own or their loved ones' death, and think about it more deeply than the cryonics companies and advocates, is a whole next step entirely: I am unaware of such people yet.

[1] [https://en.wikipedia.org/wiki/Legality_of_euthanasia](https://en.wikipedia.org/wiki/Legality_of_euthanasia)

<comment><USER6> > Unless there's a chance that at the moment of death the conscience is still > there (due to great redundancy in the brain), it might be futile, after all. > This is one issue that I believe all cryonics companies and advocates prefer > to wholly overlook.

You seem to be arguing that death is a binary state, but I don't think this is particularly well established. There are all sorts of arguments over what constitutes definite proof of death [1]. It seems more likely to me that the process of dying is a transition, and that the exact point along that transition where someone is irreversibly gone depends on our current level of medical technology – which is exactly what cryonics is betting on.

As an analogy, when RAM loses power, the data on it doesn't vanish instantly, but rather degrades over some period of time [2]. Depending on how the information is stored, what you're willing to do without, and what you can piece together, you can declare the data in RAM "gone" at different points throughout that process.

[1] [https://en.wikipedia.org/wiki/Medical_definition_of_death](https://en.wikipedia.org/wiki/Medical_definition_of_death)

[2] [https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...](https://www.usenix.org/legacy/event/sec08/tech/full_papers/halderman/halderman.pdf)

<comment><USER5> I didn't mean to disagree argue that; just didn't have enough time to think this question over before typing.

Yes, death is not a binary state physically, but it is legally. This is what cryonics counts for.

Also, conscience is not binary; there is plenty of evidence that it is uneven and noncontinuous. People lose conscience and then regain it and live on all the time. Many people live in reduced states of consciousness most or all of the time. Our mind tries to maintain illusion of continuity of consciousness for our convenience. Sometimes people survive ridiculous amount of brain damage (men living with a hole in their head). All this is evidence that whatever forms our conscience is very redundant and just _might_ survive the damage of what today is considered death and future restoration. Especially with the help of whatever medical technology will be available in the future (nanotech, hi-res brain scanning, etc.); especially if it would be needed anyway to counter the damage sustained during cryopreservation.

<comment><USER7> > People lose conscience and then regain it and live on all the time.

Do you mean "consciousness" here? I assumed you actually meant conscience at first but this sentence doesn't seem to make sense with that word but then you use consciousness later which means it isn't just a spelling error. I'm not asking to be pedantic, but because I'm now getting confused about what you're trying to say in parts of your otherwise interesting comment.

</comment></comment></comment></comment></comment><comment><USER8> Has any living person ever had their heart intentionally stopped, been cryonically frozen, defrosted, and revived?

Because if not, then we don't know if it works on people who aren't yet dead. And if we can't do that, what hope do we have for reviving the actually dead?

(I'm aware in rare cases people have been massively cooled down and had their heart stopped for operations, but that's not quite the same.)

<comment><USER9> There's also some cases of people who's hearts had stopped after being caught outside in sub-zero temperatures being successfully revived with no adverse effects.

<comment><USER10> This isn't cryonics by any stretch, but doctors are seriously looking to put gunshot victims into suspended animation by drastically cooling them down so they can be worked on [0]:

 _When a shooting or stabbing victim goes into cardiac arrest due to massive bleeding, even the most heroic attempts at resuscitation fail 90 percent of the time. But a study to begin this month under the direction of Sam Tisherman and Patrick Kochanek at the University of Pittsburgh Medical Center Presbyterian Hospital will see if there 's a better way: cooling the body after the heart has stopped beating, to the point where all other functioning virtually ceases as well.

By putting patients literally into a state of suspended animation—or "emergency preservation," as Tisherman calls it—the surgeons intend to preserve brain functioning long enough to close wounds that would otherwise be fatal._

[0] [http://news.nationalgeographic.com/news/2014/04/140402-suspe...](http://news.nationalgeographic.com/news/2014/04/140402-suspended- animation-gunshot-victims-science-death/)

</comment></comment></comment><comment><USER11> Its impossible to imagine how hard it would be to lose a child, but I can imagine why that state of mind makes even the remotest glimmer of hope for reincarnation seem like a good idea.

I thought This American Life's episode on cryonics was riveting, educational and fascinating.

[http://www.thisamericanlife.org/radio- archives/episode/354/m...](http://www.thisamericanlife.org/radio- archives/episode/354/mistakes-were-made)

<comment><USER12> i agree, this is tasteless and should be forbidden in my opinion, the only one who should be able to decide if they want to be frozen and revived should be the person themselves.

<comment><USER13> It's impractical with kids any more than you can ask them if they want to be born.

</comment></comment></comment><comment><USER14> Some might call them zygotes.

<comment><USER15> I was thinking that myself. Sperm, eggs, embryos - are these not vitrified routinely?

Probably they mean youngest legal human, or youngest post-birth vitrification.

</comment></comment><comment><USER16> How much does this cost?

<comment><USER5> In this case, probably $80,000.

This is Alcor, the most established and expensive option. Their list prices are $80,000 for neuro (brain only), $200,000 for whole body preservation [1] There are various funding options. [2]

Cryonics Institute, the other US-based organization, charges $28,000 for full body [3] (they don't offer neuro)

Russian KrioRus charges $12,000 for neuro and $36,000 for full body. [4]

NB: if the cost seems high, keep in mind that most of the money is supposed to be held in trust for the cryonics patient so that income from principal can pay for long-term storage. [5] History proved that it's the only way to reliably and sustainably finance cyopreservation, potentially indefinitely.

I am not affiliated with any cryonics organization, just researching my options.

[1] Alcor FAQ: Cost [http://www.alcor.org/FAQs/faq01.html#cost](http://www.alcor.org/FAQs/faq01.html#cost)

[2] Alcor: Funding [http://www.alcor.org/BecomeMember/sdfunding.htm](http://www.alcor.org/BecomeMember/sdfunding.htm)

[3] Cryonics Institute FAQs: [http://www.cryonics.org/about- us/faqs](http://www.cryonics.org/about-us/faqs)

[4] KrioRus: Human cryopreservation [http://kriorus.ru/en/Human- cryopreservation](http://kriorus.ru/en/Human-cryopreservation)

[5] Cryonics FAQ by by Ben Best [http://www.benbest.com/cryonics/CryoFAQ.html#_IIIG_](http://www.benbest.com/cryonics/CryoFAQ.html#_IIIG_)

</comment></comment><comment><USER17> I can't imagine having to deal with this situation, loss of a child. However, this all seems a bit silly.

</comment><comment><USER18> Spoiler: two years old.</comment>


(Geekiest Of Chefs) Grant Achatz On Turning Logistical Problems Into Inspiration - <AUTHOR> http://food.theatlantic.com/back-of-the-house/food-tasting-or-art-installation.php

<comment><AUTHOR> Once you start seeing how stories like this fit into the Hacker News ethos, it's kind of hard to stop seeing it.

<comment><USER2> Chef Achatz is absolutely a hacker. The chapter on him in Michael Ruhlman's _The Reach of a Chef_ is extraordinary.

<comment><AUTHOR> Go eat there sometime. Highly recommended. =)</comment></comment></comment>


Switching to Zsh - <AUTHOR> http://blog.secondplanetanimation.com/2011/05/31 This is my blog post about finally switching to zsh and why I did. <comment><AUTHOR> This my post about zsh and why I finally made the switch.</comment>


AI competitions don't produce useful models - <AUTHOR> https://lukeoakdenrayner.wordpress.com/2019/09/19/ai-competitions-dont-produce-useful-models/ <comment><USER2> This is a bit like saying "F1 doesn't produce useful cars", isn't it?

The point of a competition is to meet specific parameters as well as possible and push the boundaries of what can be done. It's not meant to create a "daily driver".

I realize he argued against this with the coin flip test, but that is why you'd ideally want to have many of these competitions over time. If you start to see the same names popping up at the top regularly, you know there's some sort of significance to them. Teams would ultimately want to trend towards whatever wins competitions most consistently, so they'd want to rely on models they think are the most likely to perform in a real world test. They wouldn't want to simply rely on a coin flip.

And in large competitions, you have the chance of batching the top performers together and seeing what is common between them. Presumably there's a reason these models pan out over the rest in aggregate; are they worth pursuing a bit?

I think we agree at the end though, even within my analogy. A huge value of F1 racing is the publicity the teams give their sponsors. They might learn some information that can be pushed down to their consumer vehicles, but it's marginal compared to team winnings and the value of saying "See? Our engineers are the best".

<comment><USER3> The article isn't actually that trivial; it makes a very provocative and bold statement towards the end, essentially declaring that all stated progress in image classification in the last 5 years is questionable.

I don't know if I necessarily agree with that, but there is definitely a danger in evaluating models purely based on their predictive power when many models are being evaluated on a common dataset - which is exactly the mainstream practice in the world of deep learning research - and it is therefore wise to be wary, not just as individuals but as a community.

<comment><USER4> Hi, author here. I didn't actually mean to suggest that the last 5 years of performance improvement could be spurious. That clearly isn't true. I use resnets/densenets etc in my day to day work!

What the picture was trying to say is that, within a given year, the "winner" becomes less likely to be truly better than the second place team. Alexnet was clearly better than the alternative, even with Bonferroni adjusted significance thresholds. Less so by 2016/17.

I'm writing a follow up on imagenet in particular to address some of the nuance. It is very clearly not a representative example of ML competitions, but the same effects still apply to some extent (imo).

</comment></comment></comment><comment><USER5> Our company is very thankful for the models developed for a competition we held through kaggle. The model is in our current release product, and is the current known state of the art for it's task in accuracy / performance.

<comment><USER6> How sure are you that it works? I know that this sounds _really_ flippant, but I have seen a number of well regarded models that when we / I took them to bits were discovered to actually not be doing better than dominant class prediction, and in a couple of cases had actually become arbitrary.

When did someone last really pick it apart in its operational environment?

<comment><USER5> yeah, the model works. Its individual identification task there are many individuals, and we have lots of labeled data that we held out from the competition.

</comment></comment></comment><comment><USER7> [https://arxiv.org/abs/1902.10811](https://arxiv.org/abs/1902.10811) is an useful counterpoint to this article's comments on ImageNet overfitting (esp e.g. §3.3, "Few Changes in the Relative Order").

<comment><USER8> One of the authors of above paper here. There is a whole line of work that supports the notion that all progress we are making via this leaderboard (or competition) mechanism is in fact real progress -- and that we are not simply overfitting to the test set. This thread by Moritz Hardt does a good job laying a few reasons why this may the case: [https://twitter.com/mrtz/status/1134158716251516928](https://twitter.com/mrtz/status/1134158716251516928)

<comment><USER9> I'd like to signal boost this, it's a very important line of research. Congrats on a groundbreaking paper! The first time I have seen it, I was completely shocked by that perfect linear fit on "Do ImageNet Classifiers Generalize to ImageNet?" Figure 1.

To elaborate for the other commenters: vaishaalshankar's team has created a new ImageNet evaluation dataset from scratch, and they observed that the leaderboard positions of popular image recognition models didn't change much when switching to the new evaluation. The actual performance of the models decreased significantly, but without affecting the ranking.

The OP starts from a not very controversial claim: there's a good chance that the winner of a Kaggle competition is not actually better than any of the other top k contestants, for quite large values of k. But then he completely overplays his hand, and by the time he gets to talk about ImageNet, he makes claims that were actually falsified by vaishaalshankar's paper.

<comment><USER4> I don't actually think imagenet is anywhere near as susceptible to crowd based overfitting as most kaggle competitions, but I don't actually think that paper falsifies the claim that it is.

That paper shows that imagenet classifiers retain their ranking on data drawn from the same distribution as the training set. That isn't even close to the same thing as generalising to real world data. Medical AI shows us over and over again that truly out of distribution unseen data (external validation) is a completely different challenge to simply drawing multiple test sets from your home clinic.

Again, I don't actually think imagenet was as problematic as other competitions, but there is better evidence for that (not the least of which is that for the first half of imagenet's life, the differences in models were large, the number of tests was cumulatively fairly small, and the test set was huge: ie what I wrote supports imagenet as fairly reliable).

<comment><USER9> > That paper shows that imagenet classifiers retain their ranking on data > drawn from the same distribution as the training set. That isn't even close > to the same thing as generalising to real world data.

Not the same distribution, it's new data collected and processed according to the same recipe. A quite different distribution, demonstrated by the fact that the accuracy numbers drop sharply. That's why it's so surprising that the rankings do not change that much. (Okay, in principle, a possible explanation is that it is the exact same distribution, with a fixed percentage of mislabeled or impossibly hard-to-label datapoints added. Appendix B2 of the paper deals with this possibility.)

In any case, I fully agree that this kind of generalization is still much easier than generalizing to real world data.

> Again, I don't actually think imagenet was as problematic as other > competitions, but there is better evidence for that (not the least of which > is that for the first half of imagenet's life, the differences in models > were large, the number of tests was cumulatively fairly small, and the test > set was huge: ie what I wrote supports imagenet as fairly reliable).

CIFAR-10 is basically the opposite of your list of requirements. Train set small, test set small, test set public, small number of labels, grid searched to death. And yet, look at the CIFAR-10 graph from that paper. The exact same pattern as ImageNet.

</comment></comment></comment></comment></comment><comment><USER10> > Now imagine you aren't flipping coins. Imagine you are all running a model > on a competition test set. Instead of wondering if your coin is magic, you > instead are hoping that your model is the best one, about to earn you > $25,000.

> Of course, you can't submit more than one model. That would be cheating. One > of the models could perform well, the equivalent of getting 8 heads with a > fair coin, just by chance.

> Good thing there is a rule against it submitting multiple models, or any one > of the other 99 participants and their 99 models could win, just by being > lucky

I wonder what the author must think of Poker tournaments. Even assuming there is luck involved, unless all of the models are equally bad (which would be surprising) teams that produce better models should win much more than their fair share, where fair share is 1/N when N is the total number of submissions.

But lets say that the author is correct that ML competitions are mostly luck. That is a testable hypothesis - in particular, we would expect little to no correlation between the credentials of the competitors and their ranking in the competition. Is that actually the case? Do unknown individuals who just started doing Machine Learning win on their first Kaggle submission? If the author's hypothesis is correct, one would expect that to happen fairly often, and one would expect that even highly expert competitors should win approximately (only) their fair share.

<comment><USER11> I don't think the author is saying that the best team only wins 1/N of the time, where N is the total number of participants. Far from it.

What they're saying, as far as I understand it, is that the best k teams (where k << N) each win roughly 1/k of the time.

</comment></comment><comment><USER12> I guess the calculator he used is [https://select- statistics.co.uk/calculators/sample-size-calc...](https://select- statistics.co.uk/calculators/sample-size-calculator-two-proportions/). The calculator includes this caveat:

> _" If one or both of the sample proportions are close to 0 or 1 then this > approximation is not valid and you need to consider an alternative sample > size calculation method."_

0.86 is fairly close to 1. And they're not proportions but rather averages of Dice coefficients. The statistics used here seem very suspicious. I'd like to see a Monte Carlo simulation or at least the assumptions / derivation of the formula used.

<comment><USER13> You're spot on - the calculations in the post are totally wrong. For instance, a Dice coefficient is itself the average of hundreds of thousands of observations (each pixel in an image), so you can't just treat the coefficient as a single data point. However you also can't just take a standard error, since the points are highly spatially correlated.

The post also doesn't draw on any actual reported experiences of either competitors or hosts of competitions. Generally in post competition wrap-ups there's a lot of information sharing and very significant development of understanding that comes about.

To take the imagenet example that's claimed in the post as being a likely example of over-fitting: each year's imagenet results led to very significant advances in our understanding of image classification, which are today very widely used in industry and research.

<comment><USER14> Have you actually tried to use one of those networks on a decent real dataset? They're making common category errors. ImageNet roulette is an interesting fun example of what happens if you feed an overfitted network with real images. Such as misclassifying children. :)

<comment><USER15> ImageNet Roulette deliberately uses a _terrible_ categorization scheme that has long been acknowledged as so poor as to not admit meaningful results in order to make the highly political point that ML should never be applied to people. There's a reason most people scrub that whole piece of the taxonomy before training.

Good Resnet models trained on ImageNet (the good parts, not just people) tend to result in state of the art results for almost every transfer-learning domain they're applied to.

</comment></comment></comment></comment><comment><USER16> As someone who develops machine-learning models as job, I can say that there are snippets from Kaggle kernels that run in minutes, and outperform months of manual effort by a business analyst.

Winners have often developed really interesting feature engineering strategies for a domain, as well as very well organised tuning and stacking systems.

Maybe a lot of difference between 1st and 5th percentile is luck, but top end of Kaggle is a really valuable insight into building effective models, even if you might want to simplify implimentation a bit in the commercial world.

<comment><USER11> That's exactly what the author is trying to say, I think. We shouldn't put so much emphasis on who got into first place (which is mostly determined by luck) but rather investigate all techniques used by the top 5th or 10th or whatnot percentile, which is meaningfully separated from the rest.

</comment></comment><comment><USER17> Given that a handful of names occupy Gold places in kaggle competitions, I would not call it luck. Given how hard it is to stay at the top on private set, I would not call it luck.

The winning models are hardly ever used in production, but the set of skills needed to get a gold is.

</comment><comment><USER18> Kaggle CEO here.

Agree with a mild version of the author's statement: that for many competitions, the difference between top n spots is not statistically significant. However, the author's statement (as represented by this chart [https://lukeoakdenrayner.files.wordpress.com/2019/09/ai- comp...](https://lukeoakdenrayner.files.wordpress.com/2019/09/ai- competition4-1.png?w=620&h=311)) is far too strong.

The actual best model may not always win, but will typically be in the top 0.1%.

There are people on this thread who have poked holes in the author's sample size calculator (I'm not going to rehash that).

But an empirical observation: the same top ranked Kagglers consistently perform well in competition after competition.

You can see this by digging through the profiles of top ranked Kagglers ([https://www.kaggle.com/rankings](https://www.kaggle.com/rankings)). Or by looking at competition leaderboards. For example the leaderboard screenshot the author shared in the post ([https://lukeoakdenrayner.files.wordpress.com/2019/09/pneumo-...](https://lukeoakdenrayner.files.wordpress.com/2019/09/pneumo- challenge.png?w=663&h=521)) shows 11 of the 13 top performers are Masters and Grandmasters, which puts them at the top ranked 1.5K members of our community of 3.4MM data scientists (orange and gold dots under the profile pictures indicate Master and Grandmaster rank).

I actually think the author's headline is often correct: there are many cases where machine learning competitions don't produce useful models. But for a completely different reason: Competitions sometimes have leakage.

<comment><USER18> To elaborate on leakage: it's a case where something in the training or test dataset wouldn't be available in a production setting.

As a funny example: I remember we were once given a dataset to predict prostate cancer from ~300 variables. One of the variables was "had prostate cancer surgery". Turned out that was a very good predictor of prostate cancer ;). Thankfully that was an example where we caught the leakage. Unfortunately there are cases where we don't catch the leakage.

</comment></comment><comment><USER19> It seems like the author is saying two things. 1) The hold out method isn't robust enough and 2) There may not be much difference between the top performing models. Which, okay, there may be a point there. But that doesn't really support the thesis that competitions don't produce useful models. It's more like they don't produce models that can be deployed directly for clinical use. There's no reason the techniques used by the competitors can't contribute to something that is clinically useful.

Of course a title like, "Here's why you can't directly deploy ML competition models into a production environment" doesn't grab as many clicks.

</comment><comment><USER20> The headline, apart from being clickbaity, is _accepting the null hypothesis_. Which is a STAT101 no no.

The article makes several good points. But just because the testing isn't sufficient to prove that the winner didn't just get lucky, it doesn't _prove_ that the winner did just get lucky.

<comment><USER21> Yeah, we can't disprove anything, yadda yadda.

If almost every competition on Kaggle has a winner that is not significantly better than the bulk of the field, then that is proof. Chance correlations leading to you not rejecting the null can only take you so far.

<comment><USER22> I think the point is we are left with uncertainty. Your prior should be that we don't know which competitor is best, and after the competition we are still unsure.

<comment><USER21> Isn't that the same thing as "not producing useful models"? Like, sure, some of the models may work, but unless you know which ones you can't _make use_ of them.

<comment><USER22> Yes, very true, but if we're still unsure it may be worth testing them more, while if we've proven they don't work we can abandon them.

<comment><USER14> How do you test them more, with which nonbiased dataset that does not exist?

What you could do is actually describe the kind of errors the network makes. In the example of CT, false positive, false negative, wrong diagnosis. We can try to analyze what the network is detecting, rather than accept a result on some test set as real.

The millions of trials is an overstatement, but indeed few hundred thousands are needed to actually discern a winner, presuming the network did not cheat by focusing on, say, population statistics - say, certain cranium sizes being more likely to present with problems. Relying on population statistics derived from a small sample (even if representative, which it's not) is very risky...

</comment></comment></comment></comment></comment></comment><comment><USER23> There was a good discussion on Reddit on how the objective of Kaggle competitions from the competition creator's perspective isn't necessarily a _useful_ model. [https://www.reddit.com/r/MachineLearning/comments/d50lr3/d_w...](https://www.reddit.com/r/MachineLearning/comments/d50lr3/d_why_are_kaggle_prizes_so_low/)

</comment><comment><USER24> I like the diagram where it shows the improvement from "human" to "Google" and labels it as "probably overfitting". Not only does it look like the person is drawing an extra hurricane bubble to prove a point in a presidential style, but it's complete nonsense.

The earlier example says that the difference between winners in an arbitrarily picked Kaggle competition was "0.0014". Sure, I agree, seems small. But this random diagram about image classification shows that the Google model "the improvement year on year slows (the effect size decreases)". But that's not even true. The effect is exponential!

2011->2012: 38.6% "Reliable Improvement" 2014->2015: 34.9% "Probably Overfitting"

Is this really the thoughts of someone well versed in statistics? I get the feeling they are just upset they lost at a competition and decided instead to rant about it (as you can see from the aggressive use of memes). This is not a well thought out argument against ML competitions, but you might be fooled into thinking it was because it contains just enough discussion of statistics that you might not notice it doesn't hold up.

</comment><comment><USER25> Sigh this beautifully captures the tragedy of biostats and epidemiology which is held hostage by lack of systems thinking. Unlike say a clinical trials, ML competitions are not limited to data available at start of the competition in fact by having a fair measurement of performance there is strong incentive to label and share more data and run checkpointed models. Further since the goal is not "publishing" with arbitrary requirements of P values / Power an empirical strategy is likely to provide much better long term accountability and better models. Sadly the cult of bio stats is so deeply vested in publishing rather than thinking end to end (designing a system as whole) any other models is quickly rejected by the community.

<comment><USER14> You're making a critical mistake. Why make a network for a competition that won't produce great results without major modifications on real data?

That is the main problem, and the lack of systems thinking is on your side. There is a strong pressure to cheat and overfit. Sharing in fact makes this even stronger.

We've had some fun with that when trying to use ML for something as complex as music rhythm envelope extraction. (Which is easy in comparison to CT test.)

Best results were approaching 90% accuracy on the big suite, but real results were closer to 40%. A slightly worse solution did reliably 70%. (And was not a neural network even, plus possible to improve.)

General best approaches sometimes indeed work, but sometimes (often?) they are overfitted in architecture, not even dataset.

</comment></comment><comment><USER26> My comment is really similar to what's already been said, this reads a bit like a "I don't really want to waste my time so neither should you" post. I mean nothing wrong with that but a bit negative and sour then I personally would like. Plus I think the entire spirit of the competition was a bit lost on this person.

Solid case in point, the u-net came to prominence from a medical kaggle competition. Was it a "useful" model? The author might not be wrong in saying the model wouldn't work as well in the wild but I would definitely say it was useful. The unet is still a very commonly used architecture

</comment><comment><USER27> Huh? How do you "overfit" on the test set if you don't have the test set? And also to call a good ML result a "coin toss" shows a profound lack of understanding of what goes into such "coin tosses", and why solving practical problems with ML is an entirely different ballgame than training a classifier on imagenet (which is in itself pretty hard if you want SOTA results).

<comment><USER11> > How do you "overfit" on the test set if you don't have the test set?

By being lucky. And with a large enough number of solutions, at least one solution will almost surely be lucky.

<comment><USER27> I guess I'm extraordinarily "lucky" then. I don't participate in competitions, but I'm often "luckier" than entire teams of people working tirelessly for months. And I charge a lot of money for it.

<comment><USER11> I'm intentionally misunderstanding your comment now for academic reasons: it's great that you recognise the role of survival bias and luck in becoming successful at many things.

Besides that, I'm very curious about who you are and what you do. Willing to reveal more?

</comment></comment></comment></comment><comment><USER28> My comment on this would be that the winner of a particular competition might not be the best model (in fact they are likely to be sneaky, e.g.extract information about the distribution of the hold out set and tune for that), but they are a great way to get a survey of the state of the art by looking in general at the types of approaches used by the top competitors. There is definitely information there.

</comment><comment><USER29> The post talks about coin flipping, or 0/1 classification. Many competitions use different scores however - multiclasses, learning to find bounding boxes of objects, etc. It is much less likely to find "good" answers on the test set by chance. I think the points in the article are important, but with this context become a non-issue, when a random answer is unlikely to be correct.

<comment><USER30> The article is not about models being indistinguishable from random classifiers, the difference there should be very significant even on the tasks it discussed. Instead, the problem originates from the small differences in test set performance between the top N models. While that difference may very well increase when moving from binary classification to a more technically involved regression task, that is by no means guaranteed, and the main points of the article still apply.

</comment></comment><comment><USER31> Not AI related, but I had a boss who thought hackathons were great because they would spontaneously produce great things in a day.

I'm kind of an introvert so I hate hackathons but did one and had about 75 code for a day. My boss wasn't happy but I was surprised at the fun culture and connections created across teams. Five years later people mention something they learned or a technique they use or someone they met that they continue to work with.

I don't think these competitions are the best mean for solving specific problems, though think they are part of a good portfolio.

I think that they are valuable for other purposes like communication and interoperability.

</comment><comment><USER32> It feels that the article demonstrates, by and large, the law of diminishing marginal utility is still at work in research communities, which makes sense. At some point you should call it a day and do something else.

</comment><comment><USER33> [https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the- optimi...](https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s- curse-and-how-to-beat-it)

The competition and test set will still have hidden biases due to the ontology you use.

Sufficient optimization pressure always eventually overcomes your bias control metrics, in the context of actual utility rather than other metrics.

It's well known in cryptography and security that all abstractions are leaky.

</comment><comment><USER34> Well how much of AI research and development in general is actually reflecting on the human experience rather than spewing unholy algebra onto the skeleton of past discoveries?

</comment><comment><USER35> There are so many things just so plain wrong about this (I attempted to respond, then had to stop), that I feel this post is more of an attempt to instill the frustration felt when the author attempted to compete and promptly got run over by some SotA- hungry boost-junkies from countries where the p-test is not part of the curriculum in schools. I really don't know how to constructively salvage this... Talk about the role of luck in games?

</comment><comment><USER36> I think the hold-out paradigm is woefully overrated in part for similar reasons. It's always seemed odd to me to emphasize hold out samples when you know their asymptotic performance in the form of fit statistics.

I'm always for examining replicability, but the current paradigm to me seems misguided and this articulates some of the reasons very well.

</comment><comment><USER37> If the problem is getting "lucky" on a test set, then it seems like you could do several rounds of splitting test/training set, retraining from scratch each time, and then take the median performance. Not great if training takes a long time, but it would at least conclusively answer the question.

</comment><comment><USER38> They tend to produce overfitted model instead, because the such competition optimize towards that direction.</comment>


Introducing Uber Trip Experiences API - <AUTHOR> https://newsroom.uber.com/trip-experiences-api/ <comment><USER2> I've always felt Uber doesn't really have lock-in. To me the vlaue of Uber's network isn't n^2(metcalf) because each diver is the same. And wait time is a function of closest available driver not aggregate number of drivers. Also, drivers have the option to work at other networks. Also, drivers don't set pricing so that even one less way to differentiate. Now if this Trip experiences was to catch on that could be a little different. If they have something that's truly valuable and unique in these apps it could work. but how unique can you make it?

<comment><USER3> >And wait time is a function of closest available driver not aggregate number of drivers.

Which is proportional to the number of drivers.

The lock in comes from both directions. Wait time for drivers is a function of the closest available passenger which is proportional to the number of passengers on the platform.

There's a reason why investors are dumping silly amounts of money into Uber and it's not because of their innovative technology.

>Also, drivers have the option to work at other networks.

For now. I suspect that won't last long.

<comment><USER4> >>Also, drivers have the option to work at other networks. >For now. I suspect that won't last long.

That would almost certainly change the driver from an independent contractor to an employee. I don't know if the Uber business model would survive that.

<comment><USER3> [http://money.cnn.com/2014/08/04/technology/uber- lyft/index.h...](http://money.cnn.com/2014/08/04/technology/uber- lyft/index.html) \-- Uber don't seem to be aware.

<comment><USER4> Interesting. The specific IRS guidance that seems to be in conflict:

"We also want to consider the availability of services. Are the worker's services available to the market? ...Also, an independent contractor is generally free to seek out business opportunities."</comment></comment></comment></comment></comment>


Ask HN: My startup seems to be padding numbers - <AUTHOR> Startup is a marketplace.
Recently I've noticed healthy growth, hitting and exceeding VC targets. Looked into it.
Turns out a couple of orders are being processed every month for one of the co-founders other business. The niches are totally different so there is no way to claim that they're legitimate orders.
I'm struggling to find a legitimate explanation. Either, the orders are used to pad our numbers and make us look better, or something to do with the financials of the other business.
What should I do? Could there be a legitimate explanation that I'm not thinking of? Am I wrong to be concerned?
e: it's money, not just users/traffic, it's also a significant percentage. <comment><USER3> Find another job. Chances are all of management, VC's, board, everyone is in on it.

</comment><comment><USER4> I know startups that 'padded' their numbers by buying their own stuff. The justification was that people who sold their stuff through the app became regular users, so it's not so bad. Some justify it as a "marketing cost", still more effective than putting it on Facebook ads.

I don't think it's too bad, as long as they disclose it honestly, and not straight up fraud where they make up fake numbers and then selling those numbers to advertisers/investors.

But in the end, product market fit makes everything easy, so it could be a hint of much deeper problems.

</comment><comment><USER5> Best to confide in no one who could possibly leak your knowledge back to the company. Whatever you do, do not confront the perp with the evidence.

Related-party transactions are something investors would demand be disclosed. Read through any investment contract docs you have.

If the amounts are increasing, the game's on a path to blowing up.

Options:

1\. Consult a lawyer.

2\. Go to the financial authorities.

3\. Wait and hope reality improves enough to obviate the fakery.

4\. Look for another job.

<comment><AUTHOR> Don't have access to contracts without arousing suspicion.

It's been happening for a few months and amounts have increased.

</comment></comment><comment><USER6> Why do you care? If you're worried that the company isn't going to survive, go elsewhere. If you're acting out of a sense of morality, I encourage you to think about the pros and cons of making a fuss over this (you'll probably find that there is no upside and alot of downside).

</comment><comment><USER7> Just go work somewhere else

</comment><comment><USER8> Proper metrics should discount any employee-driven traffic - it's not usually real and it's sorta a hard thing to account for entirely.

We had a table of email addresses we would mark as employees and periodically refresh this manually against actual user traffic. Then this table would get joined to anything generating KPIs or real metrics so we could discount those events.

I'd point this out to the relevant teams dealing with data and KPIs - <USER2> shouldn't want to be counting this traffic, but if they insist, I'm not sure where you go from there.

<comment><AUTHOR> It's not traffic, it's money. It's also a significant percentage.

</comment></comment><comment><USER9> I'd be surprised if this doesn't happen very frequently tbh...</comment>


Ask HN: What to implement in Python as a simple OpenSource project? - <AUTHOR>

Main interests are Django, AI, Raspberry. It has a goal of practicing community building at the same time so contributing to an existing project is not feasible (but is on my bucket list:). <comment><USER2> Well, all I can suggest you is to build something you're passionate about. Try looking at a small problem that you can solve with this tools. Someone I know did an awesome thing with Python, Raspberry Pi and few sensors. He created a simple setup that would bring down the curtains of his room once the intensity of sunlight outside went beyond a certain threshold. This allowed him to have an hour of extra sleep :)

</comment><comment><USER3> Maybe start something small. I wrote a python package to interface a Kostal inverter for a photovoltaic system. I needed it myself and decided to put it out there afterwards:

[http://blog.tafkas.net/2014/08/01/pikopy-a-python-package- fo...](http://blog.tafkas.net/2014/08/01/pikopy-a-python-package-for-working- with-a-piko-inverter-from-kostal/)

It is also on Github: [https://github.com/Tafkas/KostalPikoPy](https://github.com/Tafkas/KostalPikoPy)

And on Pypi: [https://pypi.python.org/pypi/pikopy](https://pypi.python.org/pypi/pikopy)</comment>


Tesla layoffs loom and stock plunges as focus shifts from growth to survival - <AUTHOR> https://www.latimes.com/business/la-fi-tesla-cuts-20190118-story.html <comment><USER2> Tesla is using the Jack Welch tactic of clearing the bottom 10% of performers... just did it at SpaceX too. The media and Short Sellers jump on any potential opportunity of negative news.

Realistically, its incredibly healthy for a company to do this, and my best guess is that its an efficiency tactic not "survival" related. 2018 Q4 will have a modest profit, and as they open up markets/ add models, we'll see more growth from Tesla, even though it will still be a slog.

Worth mentioning, the above story won't be told though since the mainstream media has switched to mainly caring about clickbait titles as opposed to impartial reporting.

No one will click on an article that says "Tesla aiming to improve efficiency, and reduce overhead, lays off staff" its way easier to get eyeballs with "Tesla layoffs loom and stock plunges as focus shifts from growth to survival"... which is a bummer for anyone that cares about sustainable transport.

</comment><comment><USER3> What's up with all the negativity?

Tesla grew by 10,000 employees in 2018 despite laying off 9% of the workforce in June.

This is probably more of the same.</comment>


Humble Indie Font Bundle - <AUTHOR> https://www.humblebundle.com/indiefonts <comment><USER2> If this were a real thing with decent fonts I'd snap it right up. Too bad it's just a dumb April Fool's joke.

</comment><comment><USER3> Was about to comment fonts were useless...but realized it's the 1st of April.

</comment><comment><USER4> Disregarding april fools, I would actually love a humble bundle for fonts.

</comment><comment><USER5> Fonts are expensive. This would be awesome if not an April Fools joke.

</comment><comment><USER6> Aaargh, one has to be careful with the news today...

</comment><comment><USER7> At least this year you get the fonts in some form. Last year you didn't get anything from the Humble Indie Ballyhoo ([http://blog.humblebundle.com/post/115193038361/step-on-up- fo...](http://blog.humblebundle.com/post/115193038361/step-on-up-for-modern- gaming-at-its-finest-its))

</comment><comment><USER8> Well at least all the money goes to charities!

</comment><comment><USER9> Was overjoyed about this... until I clicked.

Fonts are a really valuable resource, I'd snap up a font humble bundle in a second.</comment>


An Open Letter to New Teach for America Recruits - <AUTHOR> http://atthechalkface.com/2013/06/30/an-open-letter-to-new-teach-for-america-recruits/

<comment><USER2> For the last few months I have been seriously considering applying to be a TFA teacher. But this article has definitely opened my eyes quite a bit.

The fact that even the kids themselves realize the lack of quality in the teachers should be a huge indicator that something isn't right.</comment>


Sane DotFiles manager - <AUTHOR> https://github.com/shreyanshk/sdf <comment><AUTHOR> Took a look at a lot of dotfile managers in the wild.

But:

* Ideally don't want to learn another language/tool to just manage dotfiles

* Setup a tool via a complex process to... setup my tools? Really?

* Why can't we use nice tools provided by git directly? They're all text files. Git manages text files very well.

So, SDF was born.

* No programming languages were invented.

* No extra dependencies, none! compile with go, throw binary in $PATH and you're all set.

* Straight up wraps git so you get all the git goodness.

SDF allows restoring your dotfiles as simple as:

$ sdf clone <URL to your repository>

$ sdf checkout .

Thank you.</comment>


OS X Yosemite 10.10 running slowly in vmware? - <AUTHOR>

Try this: http://www.insanelymac.com/forum/topic/302424-yosemite-on-vmware-unusable/
It worked perfectly for me - instant increase in UI speed.
I've not looked into how or why it works, but it "Disable[s] Beam sync on [the] 'Quartz Debug' utility". I've used the app "Beamoff" from the download link, appears to work fine. I've got it set to run on login. <comment><USER2> Mavericks was far faster on my Mac mini and my MacBook air. Fusion 7 and esxi 5.1.</comment>