The True Cost of Healthcare in the US - <AUTHOR> http://truecostofhealthcare.org/ <comment><USER2> We invited a dozen physicians for brunch this past weekend and they all had stories to share about how little they make per procedure compared to the amount the hospital makes.

Example 1: A procedure costs $6,000 to be done at a hospital. It prices out patients paying out of pocket. The doctor realized that he himself gets about $700 from it. So the doctor reached out to another facility who agreed to do it for $1,800 instead of $6,000 for the exact same procedure. You can imagine how happy the doctor's patients are, particularly ones paying out of pocket.

Example 2: A patient got upset a doctor's office wanted him to pay copay. The patient got the bill and felt the doctor already got paid $1,500. When the office informed the patient that out of the $1,500 the doctor only got about $300, the patient was shocked.

Really what we need is a doctor compensation to everything else index. When we get that, we will realize the sham most of the efforts to reduce healthcare cost is. Most healthcare cost efforts focus on paying the doctor a little less, say $300 instead of $400. Problem is, bulk of the cost is not the doctor's fees. It is the cost of the bed, breakfast, Tylenol etc. We need regulation on those fees.

<comment><USER3> For example 1: The reason why the hospital charges so much is the fact that it gets stiffed for a lot of emergecy patients without insurance. Many hospitals are on the brink of bankruptcy especially in poorer areas. This does not always reflect in the huge disparity in prices people pay, but is a big factor. Hospitals try to offset their losses by charging paying customers more.

The real problems here especially with emergency care are that hospitals are forced to eat the cost of treating non insured patients, and that patients have no idea what something costs when they are there or if the people treating them will be covered by their insurance. Insurance companies spend a lot of time negotiating rates and making them somewhat uniform for them, but this is totally opaque to the patient. We need to make it more clear to the people being treated what things cost, and who is "in network". There have been times that I went to an emergency room "in network" and I was "balance billed" by nearly every doctor there due to them not being in network. What is a patient to do? Sit there with a book of providers and procedures?

<comment><USER4> That's also a symptom.

One root cause is that the Federal government (ie. Medicare) must get the best deal. If you give anyone else a better deal than Medicare, you are committing an act of fraud, and DoJ will go after you. This is known as a price floor.

Layer upon this the web of contractual arrangements surrounding different insurance providers, and you get to a place where you need to mark up the MSRP of a procedure to safely operate as a business.

The issue with poor areas having bankrupt hospitals is obviously complex. Hospitals are good for emergencies, really sick people and procedures. Not so good or profitable for primary care -- they are a lowest common denominator. You cannot afford to scale up quality outpatient medical practices if you are reliant on Medicaid to pay everything, and you can't operate small medical practices effectively and deal with all of the compliance activity that medicaid comes with.

<comment><USER5> > One root cause is that the Federal government (ie. Medicare) must get the > best deal. If you give anyone else a better deal than Medicare, you are > committing an act of fraud, and DoJ will go after you. This is known as a > price floor.

Interesting; I don't see any obvious reason why other insurance companies couldn't do exactly the same thing by contract.

Can you charge Medicare for things Medicare makes you do that other insurance companies or individuals don't, such as billing them for time spent processing their red tape?

> Layer upon this the web of contractual arrangements surrounding different > insurance providers, and you get to a place where you need to mark up the > MSRP of a procedure to safely operate as a business.

Sounds a lot like selling enterprise software: corporate purchasers expect to receive a discount, so you can either antagonize them by not giving them what they expect, or mark up the price enough that you can offer "discounts" back to the real price.

</comment></comment></comment></comment><comment><USER6> After visiting the ER with a 108F temp due to pneumonia, I can attest to the fact that ER visits are a rip off. 3 unique doctor visits to the room + 1 IV + phlegm sample + 1 $60 pill + 1 prescription = $4200 for 50 minutes of office and 120 minutes of wait. Afterwards, I explained I did not have insurance. $60 pill was wrote off instantly. $4200 turned into $1200. Govt reimbursed 3/4ths of the bill instantly to the hospital. $1200 got wrote off after 3 months because it was not worth collecting...

Edit: The interesting thing was each doctor had an individual bill of $600 sent to me. These are still in collections. The phlegm sample cost $120 and come from a separate clinical laboratories bill.

My wife's birth control has a package value of $130/mo. Health insurance ($600 for wife, and kid), had a copay of $40 for this medication. I called the same clinic, requested generic and expressed my need to pay in cash. Instantly brought the price down to $8/mo.

I did not sign up for health insurance this year. It is a complete waste. I usually pay more than just asking for generic and paying with cash. The 'poor' clinics here have a higher quality of care and staff, and are more affordable than my co-pays.

<comment><USER7> Not having health care works if you don't have a catastrophic injury, and/or you are willing to let the hospital write off services and pass the bills on the the rest of us.

This isn't really a sustainable plan for the nation.

<comment><USER6> There's a hidden truth in 'writing off'.

Insurance companies wager paying the bill too. They say "We'll pay 60% of that $8000 ER visit."

Guess who covers 25% of that leftover 40%? In a majority of cases, the government.

(Source: My aunt has worked for a hospital performing these write-offs the majority of her 45 year life.)

</comment></comment></comment><comment><USER8> I haven't read the whole site yet but I jumped to the topic I know fairly well, medical malpractice. Just some additional notes to consider:

* 9 out of 10 physicians would discourage their children from becoming doctors.[3] There are many reasons for this. Malpractice insurance premiums are a factor. Not the only factor.

* California physicians have the lowest insurance premiums in the country. This is directly the result of tort reform, and caps on pain and suffering payouts. His survey of physicians he knows in the area is subject to this sampling bias. Malpractice _is_ a big issue in other states.[0] California is the model for tort reform for this reason, by not acknowledging this fact, he's missing a big aspect of the national (not just Californian) issues regarding healthcare.

* California physicians also have some of the lowest reimbursement rates in the country.

* As he indicated, internal medicine docs pay far less in premiums (as he noted) than other specialties. Surgeons pay much higher premiums. [1]

* The California Department of Insurance does a good job of keeping most premium information public. It's not a secret what your average doctor pays in insurance.[2]

[0] : [https://en.wikipedia.org/wiki/Medical_Injury_Compensation_Re...](https://en.wikipedia.org/wiki/Medical_Injury_Compensation_Reform_Act)

[1] : <http://cl.ly/image/0m390m1F2D2D>

[2] : <http://www.insurance.ca.gov/0250-insurers/0800-rate-filings/>

[3] : [http://www.thedoctors.com/ecm/groups/public/@tdc/@web/docume...](http://www.thedoctors.com/ecm/groups/public/@tdc/@web/documents/web_content/con_id_004676.pdf)

<comment><USER9> Richard Thaler just had this op-ed in the NYT on ways to reduce costs, talking some about tort reform: [http://www.nytimes.com/2013/02/24/business/overcoming- obstac...](http://www.nytimes.com/2013/02/24/business/overcoming-obstacles-to- better-health-care.html?_r=2&);

I'm skeptical about tort reform as a panacea. However, lawsuits are incredibly expensive, and so doctors take steps that we would otherwise call economically unreasonable to avoid them.

The amount of tort reform I think is really necessary is whatever would allow me to set up a private equivalent to the UK's NHS here. If someone thinks they are getting denied care that they deserve to have, the decision needs to be very cheap to adjudicate.

<comment><USER8> It's not a panacea. It's a part of the very complicated system that is providing and paying for healthcare. How important are insurance premiums relative to other factors? It depends. But there are costs to having uncapped settlements, and lack of tort reform.

Also, insurance premiums effect practices of different sizes differently. I personally think you can attribute the shift away from smaller private practice groups towards larger groups and hospitals are for other reasons. Insurance premiums are somewhat to blame, but I think the bigger driver is that many of the government mandates are too expensive (n.b. this doesn't mean it shouldn't be done) to take on for most smaller private practice groups.

I'm not sure how to respond to your comment about tort form if it get's us to an NHS like system, because I disagree that's the best way forward. But that's a topic for another day.

<comment><USER9> Most opposition to tort reform comes from the left, and fans of European-style health care also come from the left, so saying that we couldn't set up a European style because of the legal climate is intended to cause cognitive dissonance.

(I'm not advocating putting everyone into a government system run NHS-style system. Surely, though, me and a few thousand of my friends ought to be able to get together and attempt to emulate it if we wish.)

</comment></comment></comment></comment><comment><USER10> This Time article seems particularly apropos:

[http://healthland.time.com/2013/02/20/bitter-pill-why- medica...](http://healthland.time.com/2013/02/20/bitter-pill-why-medical- bills-are-killing-us/print/)

<comment><USER11> Read it this weekend: I'd love to see a response from the health care industry that wasn't hand waiving. If he's right on 50% of his facts, it is disgusting how the system works.

<comment><USER12> "If he's right on 50% of his facts, it is disgusting how the system works."

There were only a handful of actual statistics in there. The one thing he is wrong about though is that healthcare isn't "roughly 20% of the GDP", rather it's about 17.9%. The furthest out cms.gov gives yearly projections is to 2021, and even then healthcare is only supposed to be 19.6% of the GDP. And I think that might even include things like herbal supplements 'other non- durable medical supplies', I'm not positive about that though.

Source: [https://www.cms.gov/Research-Statistics-Data-and- Systems/Sta...](https://www.cms.gov/Research-Statistics-Data-and- Systems/Statistics-Trends-and- Reports/NationalHealthExpendData/Downloads/Proj2011PDF.pdf)

<comment><USER13> Facts are more than just statistics. There were numerous statements about (what seems to be) gross, gross overcharging on everything from pens to surgical gowns to diagnostic equipment. There were the exorbitant salaries and let's not forget the chargemaster.

<comment><USER11> Exactly: that's the part I'd like to see a rebutted (if there is a rational rebuttal). The ad-hoc nature of their pricing model seems crazy.

</comment></comment></comment></comment></comment><comment><USER14> Hi, My name is David Belk and I wrote the website. I noticed a lot of traffic from here this Morning. I'd just like to say thank you and, are there any questions I can answer? I'm having a busy Morning but I'll be free in about an hour.

<comment><USER15> One question : just how would you prioritize actions to turn heathcare into a real market, or as close to it as possible ?

I would go for:

1) full price transparency

2) making price discrimination illegal

3) attacking the information asymmetry by providing a free, subsidized first line triage

Whether treatment is taken or the patient decides to do without it, making sure tests and diagnosis are performed could help proper self regulation of consumption

It could also be made as a public service since it seems very close to a natural monopoly (due to the subadditivity of the cost function - see for ex <http://www.clt.astate.edu/crbrown/eleven1.htm> \- having big labs to process blood test gives economy of scales but require high fixed costs)

It could therefore be politically defensible on grounds of efficient pricing (ie pricing at the marginal cost, which requires subsidizing for natural monopolies since average cost are above marginal costs), to avoid deadweight loss.

The first line triage could then provide full price transparency (diagnosis -> probabilistic DRG), from which the patient could either decide to "wait and see", or to browse a catalog of hospital offering services (DRG -> prices) knowing the price paid would be no different with or without insurance.

At that point, it might be possible to remove all price caps and floors, and let the market work.

So basically, we could get a working market on the treatment side.

Traditional monopoly and oligopoly management (watching the HHI before allowing fusions) could then keep it that way.

I'm a bit out of idea however to have integral pure and perfect competition, at least until we get user-operable "all-in-one diagnosis devices" (like Star Trek tricoders) to remove the information asymmetry.

My last rants on this topic were posted on <http://news.ycombinator.com/item?id=5261137>

<comment><USER14> Price transparency is good but, what's also missing in health care is value transparency. In other words, not only do most people (including most physicians) not know how much anything in health care really costs, they don't know how much anything should cost! The level of blindness in this business is truly unprecedented. What's also needed are a number of consumer protection laws that we take for granted in pretty much every other industry. Those alone would go a long way toward correcting many of the problems we have in health care.

</comment></comment></comment><comment><USER16> Do other countries have employer-funded health insurance? Why doesn't my employer pay for, say, my car insurance? This system discourages people from leaving a job to start their own business because private health insurance can be very expensive.

<comment><USER17> In the UK, it's not unusual for employers to offer private health insurance (or at least, it's pretty common for profesional jobs - I can't comment for other types of jobs!). From their point of view, it's comparatively inexpensive and the cost is probably easy recouped from their employees being able to get health problems dealt with faster than they might on the NHS so they're off work for less time.

Edit - Given that we've got the NHS as well, losing your private coverage isn't such a big deal. It's nice to have, not a must have.

<comment><USER18> I had private health insurance from a previous employer, and in my experience it was almost completely pointless. I'm not inclined to rate its provision as a substantial benefit for compensation purposes.

</comment></comment></comment><comment><USER19> That's my doctor's site! The real takeaway (to me) is that single-payer health insurance would be the way to go.

<comment><USER20> I think it's a bit more complicated than that. It's easy to look at other countries and say, hey, that works there, therefore it will work here. But not all governments are created equal. If the government in the U.S. had a better track record, maybe it would be more palatable. But as is, I don't see single payer being the solution to our problems.

<comment><USER19> It doesn't solve everything but it should help to simplify matters greatly.

What would help too is ubiquitous electronic records management, e.g., any records from a visit with one doctor should be usable by the patient at another doctor.

Add to that cheaper testing and more AI / self-help and that should take it even further.

Lastly, kill the corn lobby and get some real sense into people about how their diet and lifestyle directly impact their health. What percentage of US healthcare costs are going towards diabetes and any related illnesses? And what percentage of that is directly related to obesity and a sedentary lifestyle?

<comment><USER21> Electronic Health Records are harder to implement than you'd expect. HIPAA sets really harsh penalties for violations that can occur much more easily if hospitals and doctors provide open access to other facilities. Organizations are also often reluctant to provide their data to other facilities because of a perception that it will lead to a loss of competitive advantage. Add to that the typical problems of cross-vendor interfacing between organizations.

That said, EHRs are definitely the way to go, and we'll see much more of them in the future. Once Stage 2 of Meaningful Use[1] kicks in, you'll see a bunch of hospitals scrambling to upgrade their IT infrastructure to make sure they continue to receive the bonus Medicare/Medicaid payouts they're receiving now.

1\. [http://www.healthit.gov/providers- professionals/meaningful-u...](http://www.healthit.gov/providers- professionals/meaningful-use-definition-objectives)

</comment></comment></comment></comment><comment><USER22> The author missed a very important reason why hospitals are charging such high fees. In 1986 Reagan passed the "Emergency Medical Treatment and Active Labor Act". Since that time, hospitals are required to treat anyone, regardless of whether or not they have insurance, as if they were a fully insured patient. My wife, who is a physician in a busy hospital, consistently sees about 40% of her patients have no health insurance and are receiving free healthcare from that hospital. This may vary based on location, but I cant imagine by much. Hospitals foot the bill in this situation, and that is why costs to the insured are much higher.

Everybody seems to think they are smarter than everyone else when it comes to healthcare reform, but in all of the "smoking gun" articles that are posted, none of them have ever touched on this fact.

Even the new healthcare plan that passed does not address this directly, it assumes that the people who do not have healthcare will use a state healthcare exchange to get their own. I am a little pessimistic about this, and am assuming that only a small percentage of those people will do this.

<comment><USER14> You should read my epilogue on why hospitals are going bankrupt: <http://truecostofhealthcare.org/epilogue> Or watch the video: <https://www.youtube.com/watch?v=LieVr0QvkME> No one is admitted to the hospital these days who doesn't qualify by a rather strict set of criteria. If you qualify and have nothing then, in most States, Medicaid picks up the tab. If you have anything, the hospital gets it.Hospitals give free care the way Heidi Fleiss gave free love.

</comment></comment><comment><USER23> I'm not sure if this is true, but I heard that in some parts of China you only pay for health care when you are healthy because when you are not health care failed.

Ofcourse this raises a lot of questions about responsibility. But in our society health care is making money when people are unhealthy. That is an unhealthy situation.

</comment><comment><USER24> Call me naive but why doesn't the US simply switch to a healthcare system similar to the UK's NHS? I live in Britain and I live happy knowing that if I get a scratch, I won't go bankrupt.

<comment><USER21> A significant portion of Americans don't trust their government to provide efficient services. In general, we'll trust other citizens and corporations to do a better job. On top of that, you have many people who are reasonably insured and don't have to worry about it, many people who don't want to pay for other people's healthcare, and many people who are scared of socialism. Once you combine those groups, you end up with a pretty significant chunk of the country that is opposed to single-payer for whatever reason.

That said, the US healthcare system's problems go much much deeper than just single-payer vs private-public.

<comment><USER24> At the end of the day, my guess is it all boils down to your FREEDOM. right guys?

</comment></comment></comment><comment><USER19> I pinged Dr. Belk about the dialog here and invited him to join this conversation.

</comment><comment><USER25> Back when he was alive, Milton Friedman did a study of the costs of health care in the US from 1900 to the 1970s. (I think the first study was in the 1970s, and it was updated in the 1990s)

What he found was that the funding mechanism of health care over that time had shifted from patients paying out of pocket to government paying hospitals and government incentivized (by making it a tax write off for employers) and regulated (by preventing efficient pools of insured with state-by-state regulations) insurance.

Consequently, not only did the availability of health care go down, but the costs went up 26fold.

Or put another way, despite massive improvements in technology, medicine and productivity over 70+ years, costs skyrocketed from what you could afford out of pocket to what would bankrupt you. To put it in 2013 dollars, if you hypothesize a $100,000 surgery, in 1970[1] it would cost you $100,000, and you'd need insurance or go bankrupt, but in 1900 it would have cost you $3,846 (an amount you could put on credit cards today.)

Command economies don't work, and the change in health care over the last century has been a centralization in control. I with Obamacare this has dramatically ramped up and now there are boards to decide who gets what care based on cost measures, rather than medical need.

Whenever someone talks about the need for socialized medicine because "otherwise people would go bankrupt", remember it is the socialization of medicine that caused costs to rise to the point where people would go bankrupt.

[1] The study was originally done in the 1970s, but I'm using 2013 dollars here so you can understand how cheap things would have been.

<comment><USER26> One of the ideas I've heard discussed is that the only way we will ever get the cost of health care down is if consumers are back in the drivers seat with how the dollars are spent. Right now it's disconnected. I go in to have something done, the doctor bills my insurance company, and I really don't know (or care) how much it costs, unless I happen to look at the insurance paperwork. The same goes for pharmaceuticals. Which pharmacy in your town has the lowest price on that antibiotic that was just prescribed for your son's ear infection? Who cares, right? It's a $15 co-pay no matter where you go, so you go wherever is most convenient.

One solution that's been proposed is the increased use of HSAs + high deductible policies. That way people are paying their expenses out of pocket (for the day-to-day things at least), and are therefore more likely to shop around.

I'm not hear to argue the merits of that, but it definitely seems like an approach that might work. I fail to see how anything that takes control out of the hand of the individual will fix this problem (short of regulating the prices, which I wouldn't be a fan of).

<comment><USER27> Unfortunately healthcare is not a market you often willingly participate in. Some health needs can be anticipated, and doing more to encourage consumer empowerment would definitely lower prices. Think about cosmetic surgery or LASIK. Every participant in that market is voluntary, so all the providers are motivated to do it cheaper and better.

But when you have a major health crisis, your "empowerment" is just fundamentally limited. If you get into an accident with a loved one and they're unconscious, you're not doing to be in a position to decide whether you should go to the hospital 1 mile away, that may be X% more expensive than the one 5 miles away. Your ability to price discriminate is also fundamentally limited by your own knowledge of medicine. If you have a tumor and one doctor suggests taking medicine A which costs $50,000, and medicine B which costs $5,000, is A better than B? Is it 1000% better? If it's only 2% better, unlike any other consumer product, are you trying to optimize for "value" or "not actually dying"?

There's definitely things we can do in terms of government policy to encourage more customer empowerment to bring down costs. But ultimately you have a market with unvoluntary participation and opaque pricing, which means Adam Smith can't really do his thing.

<comment><USER26> You make some very good points, and I think you're spot on for the most part. I think HSAs have more benefit for the lower end of the cost spectrum, such as routine care, or minor medical issues (sprains, fevers, and the like). But I don't know what percent of the total US healthcare spend is on those types of items, vs cancer treaments and MRIs. So maybe it won't help as much as I'd like to think.

</comment></comment></comment></comment><comment><USER28> I am one of the lucky few who has insurance provided by their employer. Unfortunately, if I am in a car accident and rack up $100,000 in bills, I am bankrupt.

<comment><USER29> > Unfortunately, if I am in a car accident and rack up $100,000 in bills, I am > bankrupt.

...or you just have to haggle with the hospital and refuse to pay their grossly inflated funny money prices.

I have a good friend who incurred 30K in bills for a 2 day stay related to a broken femur. He has no insurance. Since seeing his bill, which wasn't itemized at all, he's basically refusing to pay anything until he sees what he's being charged for. In response to this demand the hospital reduced his bill by about 12K, magically, and still providing no itemization.

He is still refusing to pay and only time will tell what the outcome will be. It's very clear, however, that he was grossly overcharged and that the hospital has no clue what they're really billing for. It's a game. Some people roll over, others fight.

If someone really ends up with 100K+ in medical bills after adjustments then it might actually make sense to take 7 years of 'bad credit' after a bankruptcy. No easy answers here, but it's pretty clear we're all being swindled.

<comment><USER30> The rule of thumb that I've heard (from a health cost sharing ministry [0]) is that you should expect to be able to negotiate a discount of at least 40% if you're a self-pay patient.

My sister shattered an ankle a while back, and was billed around $40,000. Between reductions, writeoffs, and charities the hospital connected her to, she ended up only paying about $5000.

[0] <http://www.chministries.org/whattodo.aspx> under "general information"</comment></comment></comment>


Ask HN: Is it a bad practice to keep different versions in one build? - <AUTHOR> Hey there, here is the story. I have published one Unity plugin last year and got 2000+ downloads now. I want to make some breaking changes to the plugin but it will definitely affect users who have downloaded the old version. It is a paid plugin so I cannot just put the old version on Github release for example. I am trying to minimise the effects, but I don't know if it's a good thing to put different versions inside one build? My current solution is to add a suffix in my namespace such as Company.Product.v2, but is it a right way to do it? I haven't handled this situation before, so I am looking for more opinions! <comment><USER2> In similar situations, I've left the old api as is and created a singleton to contain each version.

      Plugin.getInstance('<version_number>')

Over time move people to that construct and eventually deprecate the old api.</comment>


Patent Application from "Steven Jobs et al": Advertisement in Operating System - <AUTHOR> http://www.google.com/patents?printsec=abstract&zoom=4&pg=PA1&id=PVjJAAAAEBAJ#v=onepage&q=&f=false

<comment><USER2> Many people say that Erich Schmidt quit the Apple Board of Directors because of iPhone/Android competition, but this application suggests another reason.

Google AdSense is great at monetizing high volume, low cost content. I work at a news aggregator and our eCPM is around $.50 from AdSense. I suspect this is typical. This puts a cap on how much it can cost for us to produce content: more than $.50 for 1000 page views means we're losing money. If you want to make more expensive content, it will be hard to make money using AdSense, because AdSense does not take into account the quality of your content.

It's hard to know what Apple is planning based on a patent application, but we do know one thing: nearly a decade's work has gone into iTunes. It is one of the biggest content distribution systems on the web. Compared to YouTube and Hulu, I would bet it's more profitable also. Whatever they are planning is probably related to extending the iTunes/App Store framework: they currently distribute songs, TV shows, movies, games, and apps. Ebooks, news, magazines and blogs seems like a plausible next step.

</comment><comment><USER3> Could this be for a tablet PC that displays media content (i.e. magazine, newspapers, etc) and is ad-supported, helping it beat kindle etc on pricing?

</comment><comment><USER4> In other words, it's a patent to turn a computer into a television set. Sad.

<comment><USER5> Yes, but at least the ads are likely to be targeted and contextual rather than the blanket-cover approach of conventional TV ads.

<comment><USER6> It seems very unlikely to me that Apple would install advertising into their operating system or applications. It would be premature to guess that they would ever implement this patent.

<comment><USER7> They probably patented the concept to prevent anyone else from doing this sacrilege.

</comment></comment></comment></comment><comment><USER8> I just don't see how this doesn't have prior art. The PeoplePC, for example.

</comment><comment><USER9> This is the opening line in the abstract: "Among other disclosures, an operating system presents one or more advertisements to a user and disables one or more functions while the advertisement is being presented. At the end of the advertisement, the operating system again enables the function(s)."

To rephrase it: the operating system of a machine you paid good money for will disable your ability to use it while it force-feeds you ads.

<comment><USER10> Then to continue that paragraph it says 'The presentation of the advertisement(s) can be made as part of an approach where the user obtains a good or service, such as the operating system, for free or at a reduced cost'.

I assume their logic is somewhat like we have ad-subsidized web sites (i.e. most of them), so why not ad subsidized devices? Shown on page 4 as part of the 'Advertising Control' components is 'Buy Ad Free Version'.

My guess is that this would be tied into a reduced cost iPhone. If you look down the page though, they show plans for how it would be integrated into desktop MacOS.

</comment></comment><comment><USER11> Personally I don't want my computer to become little more than an ad delivery mechanism. I can't think of anything worse than embedding ads directly into the OS, wasting yet more of my time.

"I see that you wanted to open a terminal window, but before I can do that for you here's a special message from our sponsors..."

In the Windows world crapware on new PCs is bad enough. Most people hated popup/under ads and went to some lengths to block them, so that historical precedent would suggest to me that this isn't going to work very well as a business practice.

</comment><comment><USER12> A free or discounted ad based OS is reasonable for them to try. I would not want my OS to be ad supported but I am sure many people would be willing to use this to save a few bucks.

<comment><USER13> Is it even -possible- to buy Windows without gigs worth of crap preloaded on?

<comment><USER14> Maybe not a new machine, but how hard is it to immediately wipe to a clean installation? "Windows" doesn't come with the "gigs worth of crap", the machines typically sold with Windows preloaded do.

<comment><USER15> My HP laptop had no CD to do a clean install with. Only a built-in partition with an image on it, already pre-loaded with their crapware. And when I go in install/uninstall programs and try to get rid of it, a few uninstallers give me errors, and the other crap won't uninstall if I don't get rid of them first! I can't understand by what reasoning they arrived to the conclusion that paying a team of programmers to implement this kind of crap was a good idea. There's a password manager which works when it feels like it, plenty of useless tray icons for things such as keyboard shortcuts that never worked, an anti-virus trial, a couple firewalls (the Windows one isn't sufficient, apparently), auto-update for the HP utilities, and the rest I don't even remember because it was just too useless. And it all loaded when the machine booted, yay!

Sorry for the venting, I spent quite a few hours to get this machine to a usable state, and even now there's some leftovers that I just can't remove without a fresh, clean, pirated copy of Windows. It's frustrating.

<comment><USER16> I have one of those, too. Vista SP2 won't install. I seldom use that side of the machine, though... these HPs play nicely with Fedora out of the box. (pavilion dv)

</comment></comment></comment></comment></comment><comment><USER17> Perhaps it's for Apple TV. The images in the patent obviously look more like a traditional OSX desktop, but the patent does mention other types of devices.

</comment><comment><USER18> Maybe they just want to prevent others from doing this. Or they could be preparing some location-based ads/offers for iPhone.

</comment><comment><USER19> Advertising through _disabling features_? That's against Apple's (and Steve's) ethos. He wouldn't venture down that road. This is probably just to prevent competitors from doing that.

</comment><comment><USER20> One more reason the patent system is royally screwed up.

</comment><comment><USER21> I suppose that would be one way to ship OS X for non-Apple PC's. (Disclosure: I used to work there, but definitely not on anything like ads for OS X.)

</comment><comment><USER22> This isn't for a desktop computer, its for the iPhone or a tablet, right?

</comment><comment><USER23> It's odd that the pictures of Mac OS X are Tiger.</comment>


Useless but fun things to do with the Linux CLI - <AUTHOR> https://lunduke.com/posts/2020-02-18/ <comment><USER2> Hah, I appreciate giving the terminal an attitude.

</comment><comment><USER3> I also like ponysay, toilet (and banner)</comment>


Show HN: Ruby gem to sync your mongo dev and prod db's with capistrano - <AUTHOR> https://github.com/openlistings/capistrano-mongo-sync <comment><USER2> It's generally a bad idea to develop with a copy of your production DB. This is how emails accidentally get sent from staging servers to real people and how sensitive information finds itself in insecure places. I didn't see any part in the tests that deals with scrubbing sensitive data, so to me this is just enough feature to be really dangerous.

<comment><USER3> Yes, it's a very specific power tool. If you don't want to pull production data or clone it to your staging environment then this is not useful.

I used to work at Amazon where this was a huge deal to us because of credit card numbers, PCI compliance, etc. Still, I believe our staging environments had a copy of prod data. At Open Listings we have pretty good safeguards against mass emailing (mostly this mail interceptor: [https://gist.github.com/psugihara/3caa217486f86a848d6abf21d7...](https://gist.github.com/psugihara/3caa217486f86a848d6abf21d734b15e)) and while we handle some financial docs for proof of funds, we ask users to black out account numbers, etc. No matter what you do though, at least some engineers are always going to have the keys to the kingdom (and since we're 3 engineers, all of us do).

The most common use case (why I use it almost everyday) is for debugging something that went wrong in production. For instance we import listings from several MLS sources every few minutes and merge updates into Property objects in our system. The raw MLS response is saved on the Property object so that we can rerun the update if something changes and we have to update our data mapper. Likewise, if something gets strange with an Offer, I just pull Offer and User collections and can take a look at the actual documents in my rails console without manually touching our prod servers.

</comment></comment><comment><USER3> I use this everyday, totally recommend it if you're using mongo + capistrano! Many thanks to pharkle for making it.</comment>


Ask HN: Anyone know of a good and practical short course in Statistics? - <AUTHOR>

I haven't touched Statistics in thirty years. Just haven't had a need for it since university.  I find myself playing with a project that needs proper statistical analysis and need to get back up to speed.  Can anyone suggest a good online resource to do a Matrix-style brain download over, say, a month?  The goal isn't to go for a PhD but rather to get a practical handle on the subject.  If Python is involved all the better. <comment><USER2> [https://www.coursera.org/course/statistics](https://www.coursera.org/course/statistics)

They demonstrate using R but its not required.

<comment><AUTHOR> I'll look it up, thanks.</comment></comment>


Transitive decay in social networks - <AUTHOR> http://www.blogger.com/comment.g?blogID=19303585&postID=7325194960111151574

<comment><USER2> Mind you, 'trasitive trust' in this context is also 'expert-oriented trust'. For example:

John (who's a network engineer) doesn't recommend XYZ firewall ... Mark trusts John ... Lisa (Mark's friend) trusts 'Mark's trust' on John ... and so on

'Transitive trust' is applicable mainly to highly specialized domains.

</comment><comment><USER3> The actual article can be found here: [http://www.jgc.org/blog/2007/12/transitive-decay-in- social-n...](http://www.jgc.org/blog/2007/12/transitive-decay-in-social- networks.html)</comment>


Initially dismissed, representation theory is now central to much of mathematics - <AUTHOR> http://abstractions.nautil.us/article/571/the-useless-perspective-that-transformed-mathematics <comment><USER2> Also discussed at [https://news.ycombinator.com/item?id=23473919](https://news.ycombinator.com/item?id=23473919)

</comment><comment><USER3> I have no idea how nautil manages to so consistently talk about mathematical concepts in a way that is so hard to make heads or tails of.

Here is a simpler explanation.

A group is a collection of elements with a multiplication such that multiplication is associative, there is an identity, and every element has an inverse. Multiplication is generally NOT commutative.

The classic example is the set of all permutations of a set using function composition as multiplication. "Multiplication" is associative because (f o (g o h))(x) and ((f o g) o h)(x) works out to be f(g(h(x))) for any x. The identity permutation is "leave everything where it was". The inverse of a permutation is simply "map everything in reverse".

They spend time talking about S3. That is just the permutations of 3 objects. (Which they visualized as the corners of a triangle.)

Another classic example is the set of invertible nxn matrices using multiplication as multiplication.

A representation is just a function from a group to some set of invertible matrices such that F(x * y) = F(x) * F(y). Note that a representation does NOT have to be one-to-one - you can always just map everything to the identity matrix.

OK, so representations exist. Why would we care? The answer is that each representation of a group shows something about the structure of that group. And says it in a language that we have a lot of tools to work with. Admittedly, the identity representation doesn't say much useful. But the others do. And representations have provided ways to reduce a lot of hard problems to easier ones that we have a better chance to solve.

There are a lot of examples, but the most widely known example of a hard problem that used this idea as part of the solution was Andrew Wiles' solution of Fermat's Last Theorem.

<comment><USER4> > I have no idea how nautil manages to so consistently talk about mathematical > concepts in a way that is so hard to make heads or tails of

Because they're writing for laymen. I have a degree in math and work in a subfield of CS that relies on math a lot, so I have a lot of practice explaining concepts to some of the smartest people I know without a math background. The tradeoff for your concise expression is that it uses jargon that adds cognitive overhead for people unfamiliar with college-level or higher math, even if they've got a high quantitative intelligence. A non- exhaustive list that would trip up, say, my friend with a PhD in pathology:

1) associative and commutative are not commonly-know terms

2) "A group is a collection of elements 'with' multiplication" is underdetermined. What does it mean to be "with"? The notion that multiplication isn't commutative is unintuitive without understanding that the definition of multiplication is flexible here, defined based on the type of the element in the group.

3) Using o as a composition operator is something that a layman is guaranteed never to have come across

4) What does invertible mean? It's not trivially obvious to a layman that some matrices A have no matrix B such that AB = I.

Etc etc etc. Note that each of these gaps and confusing points (for laymen) compound; I'll occasionally read a paper in a field I'm not an expert in (like economics), and the being bogged down every other sentence by confusion is a severe impediment to understanding how these pieces are composed into something novel. Hell, even when I understand each piece, sometimes the most foreign concepts haven't marinated long enough for me to use them as building blocks yet.

Whats your complaint here? "I have a math degree, how dare nautilus target anyone but me with their articles?"

<comment><USER5> This may be very dependent on your education system, but commutativity and associativity are terms I was taught around fifth grade, while function composition, groups and matrix inverses were high school maths.

I've lost contact with many of these after finishing my university, but they are still pretty common terms around me.

<comment><USER3> Where and when I went to school, commutativity and associativity were taught in elementary school, and again in high school because we had all forgotten it. Matrix math was something I learned in university.

However that was a long time ago and standards have changed. [http://www.corestandards.org/Math/Content/HSN/VM/C/9/](http://www.corestandards.org/Math/Content/HSN/VM/C/9/) indicates that matrix math is now considered high school material across the USA.

<comment><USER5> So those should still be terms that are familiar to most people in programming today, not some arcane post-grad math stuff like category theory.

<comment><USER3> In theory, yes.

But you know about theory vs practice.

</comment></comment></comment></comment></comment></comment><comment><USER6> I'm not so sure about the historical accuracy of the claim that representation theory was initially dismissed. They quote Burnside being skeptical of representation theory in 1897 in the introduction of his book Theory of Groups of Finite Order, but soon after he this he reformulated Frobenius' work on characters in a group/representation-theoretic framework. Burnside used representation-theoretic methods to prove his eponymous theorem in 1904, and representation theory appears in the second 1911 edition of this book.

A lot of people don't realize that the concepts of abstract groups, vector spaces, linear transformations, etc. were not particularly obvious to people, and a lot of results were developed in specific contexts that were later realized to be more general. After the development of abstract algebra, these subjects morphed into something much closer to the standard algebra curriculum we see today.

</comment><comment><USER7> One thing I've been having trouble with as I'm learning math is to coalesce new concepts into a broader context. When learning things I usually assume there's a tree of knowledge that I can travel back up, but with math it seems like a tangled graph of concepts which can be grouped into many different permutations to form different theories, also it's not entirely clear at what level of abstraction one should stop digging. I'm beyond lost.

<comment><USER8> Just want to offer some help. If you are interested in math, don't get disheartened about how tangled things seem. Even if you can't find any external references to a broader context, you will find that, given enough time and effort, a broader context emerges in your mind.

Dig until you get tired or bored - because you could really keep digging forever.

If you ever feel lost, read the masters - Euler and Dirichlet are two favorites. Here is a great book: [https://www.amazon.com/Euler-Master- Dolciani-Mathematical-Ex...](https://www.amazon.com/Euler-Master-Dolciani- Mathematical-Expositions/dp/0883853280)

If you get tired of that, read _about_ the greats. MacTutor has a great archive. I especially like their biographies: [https://mathshistory.st- andrews.ac.uk/](https://mathshistory.st-andrews.ac.uk/)

The most important thing (only my opnion) is to not lose the sense of wonder at the beauty of the universe as seen mathematically.

I don't know what your particular situation is - are you in university or school and studying math or learning math as you juggle a career? Any more specific advice depends on your circumstances and your objectives, but I really hope I was able to help somewhat.

<comment><USER8> Re: masters. Also Shelah: [https://arxiv.org/abs/math/0211398](https://arxiv.org/abs/math/0211398)</comment></comment></comment>


We've got so much stuff that it's easing the slump - <AUTHOR> http://online.wsj.com/article/SB10001424052702304410504575560552064806106.html

<comment><USER2> I love this concept.

For the last several years I've actively reduced the number of things I own to the bare essentials.</comment>


Ruby 1.9 doesn't implement ISO 8601 properly - <AUTHOR> http://tommorris.org/posts/8114

<comment><USER2> The article is making two somewhat unrelated complaints:

1\. Date.iso8601 does not correctly parse some ISO 8601 formats like ordinal dates or month-only dates.

2\. The Date class does not support year-only or year/month granularity.

In response to the first, the documentation for Date.iso8601 ([http://ruby- doc.org/stdlib-1.9.3/libdoc/date/rdoc/Date.html#...](http://ruby- doc.org/stdlib-1.9.3/libdoc/date/rdoc/Date.html#method-c-iso8601)) says:

"Creates a new Date object by parsing from a string according to some typical ISO 8601 formats."

Not "all ISO 8601 formats," "some typical ISO 8601 formats." While supporting more formats is obviously desirable, the method never claimed to be a complete ISO 8601 implementation. It would be nicer if it were, but would also take more work and it appears that 100% spec support was neither prioritized nor promised.

I believe the second complaint is less justified. Asking a single class (Date) to support multiple granularities opens up a lot of semantic conundrums that have no obvious resolution. What _should_ this return?

       Date.iso8601("2012-01") < Date.iso8601("2012-01-05")

If one is truly month granularity and one is day granularity, then the two aren't directly comparable. It would make more sense to me to have the month granularity representation be a separate class altogether (YearMonth?) with easy and well-defined conversions between the two.

<comment><USER3> If you support only a subset of the standard, or if you extend the standard, that may still be OK (not without problems, but still OK in some situations).

This case seems like an outright violation, however. If the standard says that ####-## is a year-month date with month granularity, and you interpret it as an ordinal date, then that's a clear violation. It means that others communicating with you won't just get an error, they will get the wrong answer, which is much worse.

<comment><USER4> So it's not even a subset of ISO 8601, it's just the umpteenth ad-hoc date parsing method with semantics defined as "portion of a 3k line C file with no comments": [https://github.com/ruby/ruby/blob/7ea675732ac1dac72f07756498...](https://github.com/ruby/ruby/blob/7ea675732ac1dac72f07756498706678d8725719/ext/date/date_parse.c#L2600)

PS I did my undergrad thesis on Ruby; I love the language. But now that I have a job where people read my code more often than it's written, I understand why things like clear semantics matter.

And why method names should be precise since they lead developers to assume behavior. Didn't your english teacher tell you a poem's title means something? So do method names. "iso8601" implies some meaningful relationship to ISO 8601, but the only relationship here is that the coders of this method were thinking hard about ISO 8601 when they wrote it.

</comment></comment></comment><comment><USER5> Actually the article seems to be a bit confused:

> If you aren't doing ordinal dates, you aren't doing ISO 8601.

It seems ruby1.9 does implement ordinal dates just fine:

        irb(main):002:0> Date.iso8601("2012-012")
        => #<Date: 2012-01-12 ((2455939j,0s,0n),+0s,2299161j)>
        irb(main):005:0> Date.iso8601("2012-366")
        => #<Date: 2012-12-31 ((2456293j,0s,0n),+0s,2299161j)>

What it does not implement is the "YYYY-MM" format:

        irb(main):010:0> Date.iso8601("2012-12")
        => #<Date: 2012-01-12 ((2455939j,0s,0n),+0s,2299161j)>

<comment><USER6> Yeah, I wrote it in a bit of a hurry on the train. Will fix.

</comment></comment><comment><USER7> Maybe he should subclass it and make it strict, renaming it the RFC3339 method:

<http://tools.ietf.org/html/rfc3339>

Basically, RFC3339 is ISO8601 cut down to the most straightforward implementation (YYYY-MM-DDTHH:MM:SS), without the "Ordinal Date" variant that is at issue here.

</comment><comment><USER8> I think it would make the point much clearer if the author of that article would actually explain a use case for this (apparently incorrectly handled) syntax which I have never ever come in contact with :).

<comment><USER9> Well the point is that they shouldn't call the method iso8601 if it doesn't implement the whole iso8601 standard

<comment><USER5> That's a bold statement. I don't think I've ever seen a program / library which implements any standard completely and without issues. It's not uncommon to see a list of things that are not done, incomplete, or just called out as wrong and rejected from the implementation.

It's not perfect, but no implementation will ever be imho.

<comment><USER10> Not implementing part of a standard is different from implementing it differently than the standard specifies.

YYYY-MM is year-month, not year-day365. It's not very reasonable to interpret yyyy-mm as year-day365 in a function called iso8601 when the standard says that's not how to interpret it.

</comment></comment></comment></comment><comment><USER11> So do tons of other libraries and PL standard libraries. jodatime in Java, moment.js, JS' Date object, python's datetime stdlib module, python-dateutil etc. ISO-8601 is actually quite a difficult standard to program for. 9 out of 10 times, these libraries either don't support just year, year-month, or both. 9 out of 10 times, when a datetime object is formatted to a string, you see inconsistencies about that dangling Z marker at the end.

Tom Morris is right, they all suck when it comes to ISO-8601 support.

<comment><USER12> I wonder how Perl's DateTime::Format::ISO8601 [1] holds up...

[1] <https://metacpan.org/module/DateTime::Format::ISO8601>

<comment><USER13> Seems to be fine:

      $ re.pl

      >> use aliased 'DateTime::Format::ISO8601';

      >> ISO8601->parse_datetime("2012-012");
      2012-01-12T00:00:00

      >> ISO8601->parse_datetime("2012-366");
      2012-12-31T00:00:00

      >> ISO8601->parse_datetime("2012-12");
      2012-12-01T00:00:00

      >> ISO8601->parse_datetime("2012");
      2012-01-01T00:00:00

And if it isn't then there is also Date::ISO8601 [1] which is written by Zefram [2] who is renowned for being a Date/Time nut [3] :)

1 - <https://metacpan.org/module/Date::ISO8601>

2 - <https://metacpan.org/author/ZEFRAM>

3 - <https://metacpan.org/module/Date::Darian::Mars>

</comment></comment></comment><comment><USER14> Not only that, but ISO 8601 also allows you to just specify the year, but _Date.iso8601("2012")_ raises an ArgumentError "invalid date". Nor does it support "week dates", _Date.iso8601("2012-W01")_ raises an ArgumentError too.

I don't know the standard, but either Wikipedia is wrong or the Date implementation is sorely lacking.

<comment><USER15> It may well be lacking in completeness, but I bet it covers over 99% of actual real-world ISO8601 dates. When was the last time you ran across a "week date"?

<comment><USER16> Very common in Europe, where businesses often operate relative to weeks; we frequently use week numbers for that reason (an abomination in my opinion). Ruby has decent but not great support for ISO week numbers.

<comment><USER6> Yep, an example: I edit OpenStreetMap. The opening_hours property allows one to represent a wide range of opening hours. <https://wiki.openstreetmap.org/wiki/Key:opening_hours>

A lot of public parks in London are open at different times during summer and winter.

Here's an example I added personally, the churchyard of St Anne's Church in Soho. <http://www.openstreetmap.org/browse/way/40879988>

"week 1-13 Mo-Su 10:00-16:00; week 14-43 10:00-18:00; week 44-52 10:00-16:00"

</comment></comment></comment></comment><comment><USER3> Was it just me or was the wording in the article slightly confusing? The second paragraph changes from talking about the standard to talking about the ruby implementation with out saying so in either case and no transition.

If I understand correctly, the complaint is that: ISO 8601 defines ####-## to be a year and month only, and it should have the granularity of a month; but Ruby treats it as an ordinal date.

It seems like it could be said much more clearly with an example:

        irb(main):011:0> Date.iso8601("2012-12")
        => #<Date: 2012-01-12 ((2455939j,0s,0n),+0s,2299161j)>

It's treating ####-## as an ordinal date, when the standard says that it's a month-granularity date.

</comment><comment><USER17> Maybe it's rfc3339 and not iso8601? e.g. the method has been named wrong?

<comment><USER18> It is. Luckily enough that was fixed in rails recently: [https://github.com/rails/rails/commit/31f807c7aaaf12c16ea157...](https://github.com/rails/rails/commit/31f807c7aaaf12c16ea1572559c00f49d54d6f43)

</comment></comment><comment><USER19> While we're on this subject, does anyone know of a decent C or C++ date-time library? Something like Joda Time. I know Boost has a couple of options but they look a bit less capable.

<comment><USER19> Replying to my own post, in case someone stumbles on this while searching: ICU. It's rather baroque, but seems to do the right thing. I recommend writing a C++11 wrapper for it (I might actually release this at some point in the future).

</comment></comment><comment><USER20> > Ruby 1.9 doesn't [do such and such] properly

Expect to see more of these headlines now Ruby 2.0.0 is out, as Rubyists start nudging users over by painting v 1.9 as defective. Good way to avoid the Python 3 and Perl 6 conundrums I suppose!

</comment><comment><USER21> Well done you found an issue. The next step is to report it and if possible submit a patch: <http://bugs.ruby-lang.org/projects/ruby/wiki/HowtoReport>

<comment><USER22> This isn't exactly a bug; it's an API-design disagreement.

<comment><USER21> In any case if he wants to champion the change he will get more success discussing it over there. It's not that hard but yeah it's easier to moan over a blog post than actually fixing the issue.

<comment><USER6> I'm actually working with other data-folk trying to document ISO 8601 compliance across a wide range of date-time libraries and document it publicly, probably on the microformats.org wiki or W3C wiki.

</comment></comment></comment></comment><comment><USER23> Why not just fill a bug report?

<comment><USER24> That would make _too much sense_! Whining about it as though the world owes you something is the way to go.

<comment><USER6> I'm very sorry that I posted an entry on my personal blog.</comment></comment></comment>


So, what now? - <AUTHOR>

As an American Citizen, I don't think I'm _completely_ off-base when I say:
1) Voting accomplishes nothing at a large scale.
2) "Law" can not be relied upon to uphold "rights"; it has shown it can be easily manipulated with money or by those with the power to do so. 3) These "rights", as referred to, or implied by, "The Bill of Rights", are mostly non-existant (I counted...3 remaining?).
This may not be the best medium to ask this question, but...where do we go from here? <comment><USER2> The gulag.</comment>


Ask HN: How about a "Validate this idea" thread every month? - <AUTHOR> There are plenty of ideas we work on. It could be useful to have a thread once every month for 'pre-Show HN" stage projects. Just a line or two which describes the project, and maybe a link to screenshots or mockups. You could seek answers on viability, pricing etc. <comment><USER2> It has been tried a few times. Sometimes the first edition is interesting, but in the second or third the people run out of ideas. It's usually better to post and discuss each idea in a separate post.

<comment><AUTHOR> Interesting. Perhaps this will only work if directly supported/promoted by YC.

I think there's definitely value in this; could potentially save a lot of time for the makers.

<comment><USER2> Take a look at [https://news.ycombinator.com/item?id=7582482](https://news.ycombinator.com/item?id=7582482) and [https://news.ycombinator.com/item?id=7693640](https://news.ycombinator.com/item?id=7693640)

If you have a minimal interactive prototype, you can make a Show HN. If you have only a clear idea you can make a blog post (or a text post) and make a comment explaining that you are the author and want feedback.

</comment></comment></comment><comment><AUTHOR> Idea: A Google Keep clone that keeps data in a git repo. You can share todos, notes, pics etc by allowing friends to sync your repos (and by syncing from them). Data is kept in files as yaml; hence hand editable.

This is doable in the browser as a PWA (via JS git libs) or using native apps. Maybe $1.49/month.</comment>


What is BuzzFeed, really? - <AUTHOR> http://blog.naytev.com/buzzfeed-networks/ <comment><USER2> Omits one of the most interesting facts about BuzzFeed: it was started by a member of MIT's Media Lab [0]. It all made sense once I started viewing it as the result of research into content virality. Internally I think of it more like a "profitable social/cultural experiment" than "media company".

[0] [http://www.nytimes.com/2012/02/06/business/media/at- buzzfeed...](http://www.nytimes.com/2012/02/06/business/media/at-buzzfeed-the- significant-and-the-silly.html?_r=1)

<comment><USER3> The whole thing about the "science of viral content" has always seemed like fluffy nonsense to me to make the stakeholders feel like they are part of something "higher brow" and more complex than just a spammy blog.

<comment><USER4> To be fair though the "science" of anything social always seems like fluffy nonsense made to make spaghetti throwing on the wall seem like methodical, analytical research.

<comment><USER5> Related: "Is Computer Science Science?" [https://www.cs.mtu.edu/~john/jenning.pdf](https://www.cs.mtu.edu/~john/jenning.pdf)

> What is your profession?

Computer science.

> Oh? Is that a science?

Sure, it is the science of information processes and their interactions with the world.

> I'll accept that what you do is technology; but not science. Science deals > with fundamental laws of nature. Computers are manmade. Their principles > come from other fields such as physics and electronics engineering.

...

<comment><USER6> Science is manmade and man is of nature.

It's true that most programmers do not do computer science but CS as a science does exist. CS is the study of data structures and algorithms. CS can be viewed as a branch of mathematics but it is much more process oriented than most math disciplines.

<comment><USER7> If aliens ever land they're going to have the exact same sorting algorithms we have. Their history of CPUs will be the same as ours.

<comment><USER6> Unless they have quantum computers or something completely different that we have not thought of. There's a good chance that our conception of computer science, or even science for that mater, is completely unique in the universe, like a snowflake. Science is merely a model humans use to understand the world around them, with limited success.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> This article doesn't incorporate yesterday's news that BuzzFeed has missed revenue targets this year, and is forecasted to miss revenue targets next year: [http://nymag.com/following/2016/04/buzzfeed-halves- revenue-t...](http://nymag.com/following/2016/04/buzzfeed-halves-revenue- target-for-2016.html)

This is even after a _pivot_ to video-content like all the major players in the industry are doing, which was done because BuzzFeed's current model, the one praised in this post, _is not sustainable_.

The viral approach is an approach that only works once.

<comment><USER9> People are bored with it, and other clickbait-style sites.

<comment><USER8> People aren't bored, but the economics have changed in terms of ad revenue, and the notion of linkbait has been tragedy-of-the-commonsed since everyone is doing it.

<comment><USER10> According to Alexa, Buzzfeed has been dropping in daily pageviews and global ranking [1]. That's synonymous with people getting, bored, right?

[http://www.alexa.com/siteinfo/buzzfeed.com](http://www.alexa.com/siteinfo/buzzfeed.com)

<comment><USER11> Or everyone else is just stealing the playbook, and consumers really don't care where the content comes from.

Readers of clickbait presumably get to such pages from social shares aimed to grab a click, and probably have no loyalty to the site producing the clickbait.

<comment><USER12> > probably have no loyalty to the site producing the clickbait

I think that's spot on. There's more competition in clickbait and the strategy ultimately doesn't result in any sort of loyalty. They are there for the headline, quickly consume what little content there is and close the tab.

It's a process of finding a new "weird trick" and then overuse it until people get sick of seeing it. Then find a new hook and repeat the process ad infinitum.

</comment></comment></comment></comment></comment></comment><comment><USER13> I think the idea of complex algorithms is overplayed. For instance, ViralNova.com was/is a click-baity site very similar to BuzzFeed.

It was started by a single hacker like most of us here on this forum, with 2 part-time writers. Initially it was just a wordpress blog, that he later turned into a custom CMS because wordpress is so inefficient and kept crashing when he had spikes in traffic. Scott (founder) sold it in 18 months for somewhere around 80 Million $.

Source: [http://www.businessinsider.com/zealot-media-buys-scott- delon...](http://www.businessinsider.com/zealot-media-buys-scott-delongs- viralnova-for-100-million-2015-7)

> For a while, ViralNova was a one-man startup run entirely by DeLong and two > freelance writers. Together they were able to grow their website to > Buzzfeed's size and scale — about 100 million monthly readers

Doesn't this mean that BuzzFeed has too much "fat" ( in terms of employees) that cuts into their profits?

Thoughts?

<comment><USER14> First, I'll start by saying that, personally, I'd rather run a ViralNova than a Buzzfeed. However, to answer your question, the reason that a company like Buzzfeed would add that many more employees is because of the idea that the economies of scale would work in its favor. Let's make up some numbers and say with two employees, Viral Nova was doing $20MM. The idea is that with 100 employees, they could potentially grow to $100MM and the incremental cost of those employees at say a $50K average salary would only be an additional $5MM. And then you add another 200 employees and add another $100MM in revenue for only another $5MM in salaries. Add 500 more employees and you get to $500MM in revenue, etc. Of course there's other overhead costs like office space, etc. but the incremental cost of growing those aspects of the business become less and less compared to the revenue growth. Obviously in this case, the revenue growth didn't scale to the level that they hoped for, but that's the principle behind the growth in employees. There's also many other social and cultural factors as to why you would want to build a Buzzfeed versus ViralNova that could be a full essay of its own.

<comment><USER15> Which completely ignores the fact that after a certain point as you add headcount you often deteriorate productivity and communication.

<comment><USER14> Perhaps the job is to find that point, and they found it. Or perhaps they're nowhere near that point and the management team just did a shitty job scaling the business.

</comment></comment></comment></comment><comment><USER16> I fancy BuzzFeed quite a bit.Yes the sheer volume and omnipresence can be discerning and irritating.But while they use this model I enjoy the fact that they give serious readers good long form writing too[1]. So keeping apart all of the reasons for why buzzfeed might/will/will-not grow/crash/stalemate in the market.I enjoy serious journalism which they sometimes do add. I hope they someday manage to incorporate that into their business model too.

[1]:[https://www.buzzfeed.com/news](https://www.buzzfeed.com/news)

<comment><USER17> The ironic thing is that everyone who complains about BuzzFeed's fluff appears to have never read an _actual newspaper._

I mean, go pick up a copy of the _New York Times_ , which is the gold standard of print journalism. It has hard news, sure, but it also comes with a lot of... fluff. Style/trend pieces! Entertainment coverage! Movie & restaurant reviews! A crossword puzzle! And other newspapers offer even _more_ fluff. A comics page! A gossip page! A picture of a bikini model! Today's horoscope!

The thing about journalism is that _fluff is always what has paid the bills._ Even in the golden age of newspapering, a paper that was nothing but 100% long-form hard news wouldn't sell. Except for a tiny, tiny number, people never picked up a newspaper for the hard news; they picked it up to find out how the Mets did yesterday, or to read the funnies, or for some other bit of fluff that appeals to them. Fluff is what makes mass-market journalism economically possible. The money the fluff brings is what pays for the hard- news reporting, because that reporting has never paid for itself.

The major business mistake newspapers made in the modern era was forgetting this, and letting a key part of the fluff that sold papers -- the classified ads -- get stolen out from under them. They let themselves get sufficiently divorced from the realities of their market that they thought classifieds weren't important. But they were! They were what kept the lights on, not high- minded fantasies of being the next Woodward & Bernstein.

</comment></comment><comment><USER18> Buzzfeed always gets the kid gloves treatment in the media and somehow became one of "the cool kids" despite being a website built on dumb viral content like so many others.

No surprise, I guess - big names like Andreessen Horowitz invested so now we all have to pretend like this all means something. Watching something as unimpressive as Buzzfeed get treated this seriously in the press makes me wonder just how little you have to pay a tech journalist to own them.

</comment><comment><USER19> Buzzfeed is about 20% of my "Hide from Timeline" list on Facebook

<comment><USER3> I hid it a couple of years ago and I was actually recently surprised it still exists; I never see their links anywhere.

<comment><USER20> That has more to do with Facebook's content algorithm than anything.

<comment><USER21> Distribution: No one actually visits our site to read it, we are reliant on third parties. At least 50% of our users come from Facebook, which helps mask if the other sources are even profitable by themselves.

Content: We publish content on other platforms which will slowly drain our visibility if they do too well.

Audience: We were in front of the curve and road it at its peak when Facebook opened up their newsfeed to content.

Technology: When one of our spam articles does really well with organic traffic, we spend a bunch of money to pay to promote it on the most visible channels.

As Facebook gave and took away Zynga's free traffic, so goes Buzzfeed's traffic.

</comment></comment></comment></comment><comment><USER22> Everyone always complains about the cat gifs and listicles. No one ever praises the Zenefits exposé or leaking John McCain's running mate.

<comment><USER23> Build a thousands bridges...

<comment><USER24> In Buzzfeeds case it's more like "fuck a thousand goats, but do they remember the one bridge?"

<comment><USER25> Surely you mean "17 species of Bovidae we fucked, You'll never guess what happened next".

</comment></comment></comment></comment><comment><USER26> I've always explained it like the digital equivalent of those trash magazines at grocery store checkout stands. It's not for everyone, but it works for some.

<comment><USER27> great thread on this very idea of content junk food: [https://news.ycombinator.com/item?id=11475367](https://news.ycombinator.com/item?id=11475367) and [http://www.theguardian.com/media/2016/apr/10/twitter-ev- will...](http://www.theguardian.com/media/2016/apr/10/twitter-ev-williams- medium-content-fast-food)

</comment></comment><comment><USER28> BuzzFeed's biggest problem, at least as I view it, is an immense negative equity as yet another exploiter of weaponised viral clickbait crap.

I'm aware that the company produces _other_ material, but it continues to rely heavily on clickbait. And so I say: die in a fire.

</comment><comment><USER29> '"Capitalism, Peretti concluded, needs to be constantly producing identities for peoples if the system is to survive," Matthews writes. "And ten years later, he built a factory to fill that precise need.

Matthews later reached out to Peretti to ask if he felt that Buzzfeed embodies the principles outlined in Peretti's "Capitalism and Schizophrenia." Peretti simply responded "lol."'

\-- [http://www.critical-theory.com/buzzfeed-founder-responds- to-...](http://www.critical-theory.com/buzzfeed-founder-responds-to-his- marxist-roots-lol/)

</comment><comment><USER30> As someone who's worked in publishing and advertising, the rise of Buzzfeed has been fascinating to watch. There are two really amazing things about it's business:

\- Buzzfeed has no banner ads, zero—every ad comes in the form of native content.

\- Companies pay Buzzfeed to distribute Buzzfeed content through ad networks. You'll see Buzzfeed Partner videos on Facebook, these are videos that Buzzfeed has been paid to create that includes a brand in some way or another. They are then paid again to run this content on ad networks.

Creative, media buys and more—Buzzfeed is essentially an advertising agency that gets paid to create and distribute assets for it's own brand.

</comment><comment><USER31> They make social viruses, they figured out what worked and hasn't worked, hires kids for the low rates and talent and teaches them to also make social viruses.

If Facebook is social layer OS then Buzzfeed was the stuff McAfee and Norton would warn you about.

</comment><comment><USER32> I think of BuzzFeed as the company built on stolen content that sends cease & desist notices to other people that use their stolen content.

</comment><comment><USER33> [http://chainsawsuit.com/comic/2014/11/06/it-must-be- fed/](http://chainsawsuit.com/comic/2014/11/06/it-must-be-fed/)

</comment><comment><USER34> tl;dr:

"An advertising company."

</comment><comment><USER35> hell

</comment><comment><USER36> Buzzfeed is junk food

</comment><comment><USER37> It's the site that makes my browser cry in exhaustion.</comment>


Statistically controlling for confounding constructs is hard (2016) - <AUTHOR> https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719 <comment><AUTHOR> This paper discusses a major issue that I think deserves way more attention from the academically literate crowd. This is the kind of knowledge that should be taught in secondary school, at least in the modern age with papers at everyone's fingertips.

A more approachable summary is in narrative form at

[https://www.talyarkoni.org/blog/2016/06/11/the-great- minds-j...](https://www.talyarkoni.org/blog/2016/06/11/the-great-minds- journal-club-discusses-westfall-yarkoni-2016/)

> "But there's a problem: statistical control–at least the way people > typically do it–is a _measurement-level_ technique. Meaning, when you > control for the rate of alcohol use in a regression of cancer on bacon, > you're not really controlling for alcohol use. What you're actually > controlling for is just one particular _operationalization_ of alcohol > use–which probably doesn't cover the entire construct, and is also usually > measured with some error."

I strongly suggest reading the comment on that page from one of the authors as well.

</comment><comment><USER2> Causal inference has been a huge issue in statistics and its subject fields for a while, see DAG, instruments, Regression discontinuity, RCT, natural experiments...

For the main guys, the question is not that causal inference is important and traditional research design is problematic.

The issue is with all these big ego guys. For example, Pearl pretty much states that DAG are the only way and his conception is perfect and everyone who disagrees doesnt know statistics (even though DAG can not capture all models of causal inference).

Then Angrist and Pischke come and say everyone using parametrized designs, including Pearl, is not believable and its RCT logic or bust.

And Gelman fight against everyone anyway.

Much of this is less academic dispute than internet blogfighting, sadly.

The issue is that in practice, we need to do something with imperfect observational data. Its really easy to critique any study on identification and endogeneity. Its difficult to solve the issue!

<comment><USER3> I don't think it is sad, I think it is wonderful! This is philosophy of science in action. We don't know the right answer yet, it's not obvious, and so we should absolutely argue it out if only to challenge that status quo.

</comment></comment><comment><USER4> There was an amazing couple of papers recently on polygenic risk scores (now trendy in human genetics in place of GWAS, since PRS can include tiny effects from genetic variants that GWAS ignores)

Editorial on both papers here: [https://elifesciences.org/articles/45380](https://elifesciences.org/articles/45380)

In summary, when you run a GWAS, you control for population structure by using the first two or three principal components as covariates. In PRS, the model built is so complex that controlling using PCs doesn't fix all influence by population structure - which is now why some famous large-scale human genetic studies fail to replicate.

(This has large implications on a societal level - Twitter is full of 'race realists' ,i.e., racists, who are using spurious PRS to prop up their foregone conclusions)

<comment><USER5> It's not remotely that bad... You're leaving out important context here: what failed to replicate was not the GWASes, but inferred selection signals from the GWAS (which is very different); actually, 2/3rds of them replicated anyway (so they're doing much better than, say, social psychology or medicine); _and_ that 3rd one which didn't replicate didn't replicate because the check which would have caught it, the sibling comparison, which was done, turned out to have incorrectly validated it due to a Plink software bug (which is the kind of error which could happen to literally any research result these days). The situation remains precisely as it was before: GWAS results are generally trustworthy, especially when validated by sibling comparisons, and human selection is pervasive.

<comment><USER6> Can you point me to a reference re: what happened for the third study? If this was caused by an actual Plink bug instead of incorrect usage, I need to verify that the bug has been fixed...

</comment></comment></comment><comment><USER7> From 10,000 feet up, a two step solution:

(1) Get a lot of data.

(2) Do finely grained cross tabulation.

Why? Because cross tabulation is a discrete version of the most powerful foundation but for challenging questions can need a lot of data. Curve fitting is what are pushed into when don't have so much data.

Or suppose build a model of the probability of an auto accident. Okay, want to evaluate the model for 5' 2", 105 pounds, 17, blond, speaks only Swedish and Russian, and is in the US in LA driving an 18 wheel truck for the first time and talking on her cell phone with her sister in Berlin.

So, for that query, instead of a model, just have a lot of data and cross tabulate, and the cell with that person delivers the answer immediately, directly. Moreover the answer is unbiased and minimum variance ( _least squares_ ). But did I mention, need a lot of data?

<comment><USER3> This is fundamentally why we have deep neural networks. Fitting a simplified curve in a huge space.

The combination you named may never have been observed before. The curse of dimensionality strikes.

<comment><USER7> Yup. Did I mention that with cross tabulation we'd need a lot of data!!!! :-)!!!

<comment><USER8> But isn't that the whole problem? You quickly run into the case where the required sample size is larger than the entire population size.

<comment><USER7> So, you expect us to boil it all down to just F = ma, and f'get about the apple, the apple tree, the time of day, the phase of the moon, the temperature of the day, what Newton was wearing?????

That was a really good shot at causality. Let me just say, quite generally, _causality_ is super tough to find.

</comment></comment></comment></comment></comment><comment><USER9> How were those embedded charts built? They look like base R but I really like the download option. Is the library under GMT?

</comment><comment><USER10> Anyone interested in this article should also take a look at some of Pearl's work on causal inference. Here is one article: [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2836213/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2836213/)

</comment><comment><USER11> And a paper somewhat related: _How Much Data Do You Need? A Pre-asymptotic Metric for Fat-tailedness_ [1]

[1] [https://arxiv.org/pdf/1802.05495.pdf](https://arxiv.org/pdf/1802.05495.pdf)

</comment><comment><USER12> Of course it is. This is only news to people who think being able to use a couple of Python libraries makes them a "scientist" of some sort.

See also instrumental variables[1]. Oh, sorry, I forgot, who needs all of the thought that went into the development of econometrics as a discipline? We now have data mining as a career.

[1]: [https://www.nuffield.ox.ac.uk/teaching/economics/bond/instru...](https://www.nuffield.ox.ac.uk/teaching/economics/bond/instrumental%20variables1.pdf)

<comment><USER13> Translating to the language of econometrics, I think the point of this paper is that oftentimes, the instrument that you're using has errors in its measurement, or is just a proxy for the 'real' instrument that you'd like to use.

If that's the case, when IV is interpreted as a two-stage regression, you might _still_ have covariates that are correlated with the residuals in the second stage.

</comment></comment><comment><USER7> So, the article is about regression analysis: And there when I first studied the topic, I saw the interest in assigning _importance_ to variables one at a time based on the size of their regression coefficients.

Well, there is some math supporting regression analysis if want to use that math, i.e., confirm the assumptions, nearly never doable.

So, I did see that if the independent variables were all orthogonal, this one at a time work had some support. E.g., in linear algebra look at Bessel's inequality.

Otherwise, easily can get confused: So, we're trying to predict Y from U and V. U doesn't do at all well; neither does V, but together U and V predict Y essentially perfectly. How, why? U spans a vector space, and Y is not in that space. We can project Y onto that vector space and have the projection small (coefficient of U small). Same for V. But U and V together span a vector space, one that includes Y -- so U and V predict Y perfectly. Just simple vector space geometry. It can happen.

This stuff about _controlling_ can make some good sense in cross-tabulation (see my post here on that), but in regression analysis? _Controlling_? Where do they get the really strong funny stuff they've been smoking to believe in that?

</comment><comment><USER14> Mindblowing. No wonder so much science doesn't replicate.

I read scientific papers fairly often compared to the average person, and this is probably the most important I've read in years. It feels like a similar problem to Spectre in the CPU design space: a problem nobody noticed for decades, that breaks fundamental assumptions in ways that appear nearly impossible to fix. Except this is way bigger and more important than Spectre.

Here's a quick summary of the paper and its conclusions:

1\. Virtually all claims of the form "We have found X is an important factor when controlling for other factors" are wrong, because they ignore the possibility of measurement error, which has unintuitive and enormous effects.

2\. As a consequence, vast swathes of so-called findings in medical and social sciences are spurious to an even greater extent than previously realised. The paper cites Google Scholar searches to show that hundreds of thousands of papers contain trigger phrases like "after controlling for".

3\. The problem is fundamental and can't be fixed.

For (3) I should explain, because the paper does propose what initially looks like a fix: an improved form of statistical modelling that incorporates measurement error into the outcomes (SEM). This can restore statistical validity.

Unfortunately as we study this idea closer it becomes clear that there's no way scientists are going to adopt it:

• It explodes the sample sizes and therefore cost.

• Frequently we have no idea what the measurement error actually is.

• Therefore it becomes impossible to say with any confidence whether a claim is true or false.

Even very small amounts of measurement error can push the confidence level of the outcome under the already arbitrary and probably too lax 95% threshold. To detect a weak effect size of 0.1, handling realistic error rates could easily take you from needing hundreds of samples (grad student populations) to tens of thousands.

To clarify the term "measurement error" here, what the paper gets at is that in social sciences they are often trying to measure things indirectly via e.g. surveys which have some inherent noisiness due to people lying, not understanding the question, not being sure of the answer but being unwilling to admit it etc. And then on top of that they are often trying to measure something ill-defined to begin with, e.g. mood, intelligence, wealth.

But as noise rates go up, the chances of spuriously detecting correlations that don't exist reaches _100%_. The paper does various simulations to show that this can easily happen with sample sizes, p values and error rates that are standard.

The part that really nailed it for me was where they use a hypothetical example of a correlation between ice cream sales and swimming pool deaths. Common sense tells us ice cream doesn't cause people to die in pools, but rather, on hot days more people go swimming and buy ice cream: it's a confounding variable. In the case where the confounding variable is accurately measured with a thermometer, measurement errors are zero and the typical sort of regression analysis scientists use shows no correlation as expected. But if temperature is measured using a survey where people self report perceived body temperatures on the Likert scale, then you get an error rate of about 0.4. At that point a regression analysis shows a strong direct correlation between eating ice cream and dying in a pool with p < 0.001 - much lower than the p < 0.05 threshold needed to trigger publishing.

A big question is what does this mean for science and the academy? I can see nothing happening in the short run: it seems this paper wasn't picked up by journalists, or at least I've not seen any mention of it even in discussions of the replication crisis. Scientists themselves can't really pay attention to it without breaking the entire funding model of non-STEM sciences.

In the long run this could possibly lead to a major breakdown in society's notions of expertise, trustworthiness of scientists and the value of government funded academia (corporate funding of social sciences is close to zero). How many decades have people been reading stories of the form "$foo causes $bar" or "$baz (good thing) is highly correlated with $zap (bad thing)"? Many people have already clocked that this kind of science is unreliable - now we seem to know precisely why.

<comment><USER7> For measurements, the social scientists have been considering what they call _reliability_ and _validity_ for a long time. So, they do consider the accuracy of measurements.

In science we collect data then use the data. If the data has some measurement error, no big surprise. Instead just say that the _model_ is in terms of the measured values with their errors instead of some much more accurate values that don't have.

Or, if have more accurate values, then use them. Else don't feel guilty and beat up on self.

See also my earlier post in this thread on cross tabulation.

<comment><USER3> Why isn't total least squares used to control for measurement error in regressors?

<comment><USER7> By "total least squares" you mean to pick regression coefficients that minimize the squared errors between the observed values of the dependent variable and the predicted value from the regression? The predicted values are from a perpendicular projection, and we do get the Pythagorean theorem with

total sum of squares = regression sum of squares + error sum of squares

So, we minimize the error sum of squares. Does that really _control_ on measurement error? Not in any simple way I can see.

Or, if we have errors in the measurements of the independent variables, then we are facing one of the facts of life, we don't have the error-free, _true_ values. Not good but usually not as bad as having your marriage fail or losing your pet dog or cat! Like the video clip of a Heifetz master class where a student tried to play the D flat minor scale and at the end Heifetz assured the student that they were still alive!

Maybe what you are saying is that with enough mathematical assumptions, e.g., the famous _heteroscedasticity_ with independent, identically distributed mean zero Gaussian errors, as the number of observations goes to infinity the errors wash out much as in the weak/strong laws of large numbers and we get to f'get about the errors -- maybe there is such a theorem, I should get out my copy of the old

C. Radhakrishna Rao, _Linear Statistical Inference and Its Applications: Second Edition_ , ISBN 0-471-70823-2, John Wiley and Sons, New York.

and look or do some such derivations for myself.

But, to what end if we don't believe the mathematical assumptions for the mathematical theorems, e.g., _heteroscedasticity_ with independent, identically distributed mean zero Gaussian errors?

Sorry, from 50,000 feet up, it seems to me that having _control_ variables in regression is shaky stuff. And without some careful derivations, we should not be surprised at the effects of various errors.

Also, the usual derivations of the math are in the context of just some one regression _model_ where we make all those assumptions. Instead, given one dependent variable and 10 independent variables plus five more we believe are _causes_ plus 10 more we want to use as _controls_ , the dependent variable plus 25 more variables in all, last time I checked we were short on how to pick the 2^25 sets of independent variables and make sense of the different, maybe wildly different, coefficients we get.

Here's a simple view: If we have 5 independent variables and they are all orthogonal, then we can get the regression coefficients one at a time just from 5 projections, covariances, inner products (all essentially the same things except how we scale things) and have those coefficient just the same for any of the 2^5 regression analyses. That is, if we have orthogonal independent variables U, V, W, X, and Y and dependent variable Z, then we can get the coefficients one at a time and be done -- have all the regression coefficients for all 2^5 regressions. Otherwise, without othogonality, we face some possibly tricky math derivations -- maybe they are in Rao's book, it's thick enough -- and are asking a bit too much from regression analysis.

Others have seen this swamp, and a current idea from the _machine learning_ community, and going back at least to L. Breiman, is that we are not really looking for coefficients, t-tests on the coefficients, F-ratio tests on the regressions, confidence intervals on the predictions (for that might try some resampling ideas), importance of coefficients, causes, control variables, etc. but are just looking for a fit that can predict: To this end we put the data into at least two buckets, fit to one bucket and test in another one. Our main criterion is just that the _model_ predicts well for the data we have. That is, all the data in all the buckets has all the same statistical assumptions, whatever the heck they are, and we are just _fitting_ and then _testing_ (confirming) on _simple random samples_ of data all from some one big bucket. Yes, we still run into the issue of _overfitting_ , fit well in the first bucket but flop terribly testing on the second bucket. Okay, a bit crude, uncouth, vulgar, primitive, ..., etc. but maybe useful in some cases -- apparently Breiman made it useful in some cases of medical data.

<comment><USER3> I inspired a great soliloquy! Thanks for the thoughts

My point: total least squares includes X in the error minimization, not just Y and linear combination of X. There is a good introductory discussion on wiki -- essentially in standard regression we typically assume no measurement error in the independent variables.[0]

As much value as machine learning brings, there is a need for explaining as much as there is for predicting![1]

Your point on whether there is "true control" seems to agree with Pearl's main point of contention -- does the causality plot (which is testable) make sense from a theoretical, experiential, or systemic sense?

[0] [https://en.wikipedia.org/wiki/Total_least_squares#/media/Fil...](https://en.wikipedia.org/wiki/Total_least_squares#/media/File:Total_least_squares.svg)

[1] [https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf](https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf)

<comment><USER7> Okay, "total least squares" as in your [0]!!!! WOW!!! Back when I knew nothing of regression or curve fitting and was first considering the issue, the question I asked was, if we are trying to _fit_ to the given data, why not have the line as close as possible to each of the points on the scatter diagram, just as in the quite good picture at [0]!! Gee, value of ignorance!!

Again I believe we are trying to make too much out of regression.

Or, maybe, if somehow we DO have causes, we really know they are the real causes, and we have some data, good data, and the data likely satisfies the usual assumptions as in the reference I gave to Rao, THEN, maybe, on a good day, with luck, do the regression calculations, t-tests, the F-ratio, get the confidence intervals on the coefficients and the predicted values, etc. and if all that looks solid, then take it seriously.

Here, however, KNOW the independent variables, all of them, KNOW that they are the causes, and don't need _controls_ , etc. and are not fishing for the variables, we are not trying to have statistics tell us about causes, ..., then maybe okay.

But, sure, if there really are causes and if we really do have variables that do well measuring those causes, then maybe in the regression the variables that are candidates as causes will become fairly obvious.

<comment><USER3> Regression is useful because it allows us to interpolate within observed populations using relatively light assumptions. Extrapolation requires higher order theories and structure. Agreed that it can be a logical mess when one uses it bluntly, but like all tools it has its uses and misuses.

<comment><USER7> Ah, cruel, you are so cruel, how could you be so cruel; the OP was hoping for something so much better than just some interpolation!!!

Cruel or not, at least in practice, you are on solid ground.

<comment><USER3> Then you may be interested in pursuing structural models!

Thanks for the chat.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> Observational studies claiming causality are flawed compared to randomised controlled experiments.

Hopefully in time research will gravitate toward RCTs.

Distinguished Engineer at Microsoft, Ronny Kohavi has a great presentation on flawed observational studies here:

[https://www.exp- platform.com/Documents/2016-11BestRefutedCau...](https://www.exp- platform.com/Documents/2016-11BestRefutedCausalClaimsFromObservationalStudies.pdf)

<comment><USER16> But the problem is in many application domains, you cannot obtain holdout sets. For example, if you measure the causal impact of ads for a client, they usually will not agree to accept the cost of holdout inventory in order to measure the effects. They'd rather ensure all the inventory is used, and let the observational post-hoc attempt at causal inference have errors.. they knowingly prefer that trade-off.

There are actually a lot of areas like this, where you are just given a one- sided observational data set snd tasked with recovering causal effects with no option to collect a holdout set. Often nobody consciously chose it that way over an RCT, it's just how it happened.

<comment><USER17> or simple covariates like a subject's age. If we could randomly assign people to an age, then...we'd have conquered aging I guess.

</comment></comment></comment><comment><USER18> Confounding variables like destitutite candidates in the hiring process with a model trained by educated people with savings?</comment>


In Praise of Passivity (2012) [pdf] - <AUTHOR> http://studiahumana.com/pliki/wydania/In%20Praise%20of%20Passivity.pdf <comment><USER2> I once had a doctor tell me that his chief goal was to make sure he didn't take any action that would interfere with the healing of my body. My body would heal just fine if it was allowed to - he was there to (at most) remove impediments.

Too bad more politicians don't follow Calvin Coolidge's example - easily the best president of the 20th century.

However, as voters, we tend to select for people who will "do something". In opinion polls, we deride politicians who don't visit sites of natural disasters, even though their presence will disrupt recovery activities.</comment>


Show HN: xmbd - Cross Provider Media Embedding with jQuery - <AUTHOR> https://github.com/screeny05/xmbd

<comment><USER2> I like it. Nice work!

</comment><comment><USER3> Nice!

<comment><AUTHOR> Thanks! Currently it's not even alpha, but there's always some work to do.</comment></comment>


TeaHub: An open source alternative to GitHub - <AUTHOR> http://about.teahub.io/ <comment><USER2> Same people as Gitea, the Gogs fork. Also using Gitea.

</comment><comment><USER3> Is it going to be like GitLab?

<comment><USER4> You can compare it with that. It's just built up entirely transparent and open and backed by a non-profit organization.</comment></comment>