Ask HN: How do you test your Machine Learning models? - <AUTHOR> Hey all!
I'm looking to get some inputs from professional engineers in Machine Learning about how they test their models that might enter production.
I ran a survey in my own network before through a Twitter poll, and found out that over 70% of respondents were not fully confident in the abilities of their models, with 33% of all respondents saying that they are not confident in the abilities of their models at all.
Now I'd like to find out why and how to solve this. I prepared this survey and it would mean a lot if you could take 2 minutes to fill this one out.
I'm looking to write an article about ML testing methods based on those, and will be sure to share that here (and to anyone who requests it).
https://dominicmonn.typeform.com/to/n2AbR7 <comment><USER2> For production we usually just use some form of a/b testing, as we found out that the offline metrics are often misleading. When the offline metrics of your model Like f1 or whatever are good enough we just promote the model to production competing we the other models in a multiarmed bandit way.

</comment><comment><USER3> I'm interested to share our experience. Do you have a link to your Twitter poll? You can find my contact information in the profile if you don't want to share infos here, happy to chat!

<comment><AUTHOR> The survey is here: [https://dominicmonn.typeform.com/to/n2AbR7](https://dominicmonn.typeform.com/to/n2AbR7)

This is the poll that sprouted this all: [https://twitter.com/dqmonn/status/1139092798521712641](https://twitter.com/dqmonn/status/1139092798521712641)</comment></comment>


Experts on Mac vs. PC security - <AUTHOR> http://news.cnet.com/8301-27080_3-10444561-245.html?tag=rtcol;pop

<comment><USER2> This article cites repeatedly the idea that Windows is less secure than Mac because it has a larger market share than Mac and is thus a larger target for malware writers. Here are some counter-arguments which refute that idea that market share alone makes Windows a more attractive target:

\- There's also a lot more anti-malware software available for Windows. Windows users are much more likely to run anti-malware. Shouldn't this make Windows a less-attractive target for malware writers?

\- Microsoft have alleged that open-source software is less secure than closed-source because it's easier to find vulnerabilities. As much more of Mac OS is open-source, shouldn't that make it a more attractive target than Windows?

\- If Windows itself has more inherent vulnerabilities (including design traits that make it more likely users will do unsafe things) would that not be the simplest explanation as more malware writers see it as an easy target? (E.g., so many cryptic UAC warnings that users quickly learn to click "ok" without understanding them.)

<comment><USER3> "Microsoft have alleged that open-source software is less secure than closed- source because it's easier to find vulnerabilities. As much more of Mac OS is open-source, shouldn't that make it a more attractive target than Windows?"

Wait.. what?

What has Microsoft's unsubstantiated and biased opinion on OSS got to do with whether in fact Windows is more or less secure than Snow Leopard?

"If Windows itself has more inherent vulnerabilities (including design traits that make it more likely users will do unsafe things) would that not be the simplest explanation as more malware writers see it as an easy target? "

That was addressed by some of the respondents. Basically (they claim), Mac users are just as likely to mindlessly enter their user name and password when asked to install some bit of code as some random Vista or Win7 user might click past all the warnings. You get used to it for the safe cases and it becomes routine; it's just one of those steps, like agreeing to the default install folder.

Many of the security people claimed that Windows had _fewer_ inherent vulnerabilities than OSX, but that the main issue was, essentially, social engineering and poor user habits all around.

</comment></comment><comment><USER4> "In my mind, the OS question is quickly becoming moot, and will soon be replaced by the already-intense Web browser holy wars--especially with Google jumping into the fray there."</comment>


What is JSON-FP - <AUTHOR> http://www.slideshare.net/BenLue/what-is-jsonfp

<comment><USER2> What is the purpose of this when S-expressions already exist? Maybe I'm missing something, but this looks like a product of that stereotypical "everything must be JavaScript" attitude.

I'm also not entirely sure whether it's a parody.

<comment><AUTHOR> You may want to check this ([https://github.com/benlue/jsonfp](https://github.com/benlue/jsonfp)) to see why and how it's created.</comment></comment>


Would You Keep a Smart Firearm in Your House? - <AUTHOR> https://edgylabs.com/2017/01/19/mit-smart-firearm/ <comment><USER2> I would not. I also do not currently keep a regular firearm in my home, though I'm not opposed to the idea.

If I had a firearm at home, it would be for self-defense, as many are. And I'd hope to never have to use it (outside of practice). But the key thing is that if you have a firearm in your home, you are probably going to need it once, and it needs to have as close to a 100% chance of working as possible.

If you've ever dealt with fingerprint scanners, you know how often they fail to scan correctly. If it's your smart gun versus someone else's regular gun, the odds are the guy without the smart gun is going to fire first.</comment>


The Money/Time fallacy - <AUTHOR> http://thoughts.maayank.com/2012/08/the-moneytime-fallacy-or-guide-to-very.html

<comment><USER3> I refuse to peg every remaining second of my existence to the highest hourly rate _someone else_ will pay me for it. I won't say "My time is valued at $XX per hour, so it makes sense to pay $XX for someone else to do this." Valuing all time at an hourly rate is logical if a human is trying to maximize the amount of money made across their life - since you could theoretically pay for this task to be done while working, you would have more money than if you did the task yourself. Maybe you work every waking second.

But I don't. I don't want to.

I want to maximize life enjoyment across my life, not the money earned. Instead, I compare what I'm doing now to what I could be, including consequences, and decide which I'd rather do. Then I do that.

\---

Not to mention that money has limited utility - you can only trade it for things which currently exist, and which other people are willing to trade. No amount of money will buy you a Star Trek Transporter, nor will it revive dead people or let you live forever. It won't buy you food from the ground; you have to trade it for something else and put _that_ in the ground, and wait a season. Most of what money will buy is boring.

<comment><USER4> Eventually, you will come across a situation where you need to decide if you will "work" in your free time (painting your house, mowing your lawn, etc), or if you will pay someone to do this so that you can free up your own time to "maximize life enjoyment". The _time = money_ equation isn't just about always earning money, it's about cost savings & cost avoidance as well.

You have to decide, what makes you happiest? If it is freeing up your time to do something else, then you might even _pay a premium_ to have someone paint your house for you.

<comment><AUTHOR> _You have to decide, what makes you happiest? If it is freeing up your time to do something else, then you might even pay a premium to have someone paint your house for you._

Exactly, ultimately these are all pathways to the same question/answer.

<comment><USER5> So if you understand that opportunity cost is in fact about choice and trade off, why did you leave out the core of the principal? Focusing only on the money in this DIY / "pay for it" argument is what make it a fallacy. If you change your car repair example so that Don makes much less per hour than the repair cost. It still might make sense for Don to pay the mechanic to do the repair. Only Don can make that choice. Anyone one else is just giving their opinion.

<comment><AUTHOR> It's not a post about "The Only Ultimate Way to self-realization". It's a post about "given this statement I often hear, here where it works (and how to improve on it), here where it doesn't". If it doesn't work because of different reasons then:

a. Good, please contribute to the discussion

b. Different reasons resonate with different people. Therefore someone's "core of the principal" isn't the other's.

<comment><USER5> There is a clear definition for "Opportunity Cost". It's an accepted principal with a long history. From my other comment, I was under the impression that you were not aware of it. Maybe you still aren't. My point is, you are creating the fallacy in your argument by not acknowledging the entire principal. Your use of money as the only driver in the personal decision provides a false conclusion to the "statement you often hear". The conclusion you left me with is: given you're a "High Paid" engineer, you should not fix your own car or help out your community restaurant because the math doesn't work out. However, if you want to screw around with "non-billable" tasks, that's cool, you'll just pay for it.

As almost everyone here has pointed out, that's just NOT what the concept of "Times is Money" is about. BTW, the different reasons ARE the core of the principal. Of course people are going to make different decisions. But, I guess for you it only works when Billable hour > cost of item.

<comment><USER6> If I may pitch in: since you're trying so hard to throw around opportunity cost, there are, believe it or not, a couple other important points to consider here.

The first is that all valuable things can be measured in terms of other valuable things--including dollars. Opportunity cost tradeoffs of all types can be mapped onto a tradeoff for some quantity of dollars. All things can therefore be measured in dollars, even if you want to be silly and peg the value of things like "true love" as infinite--that's equivalent to infinity _dollars_. Your time is measurable in money as well.

The second point follows from something either you or someone else noted before. Eventually, paying for things that aren't worth your times adds up, and you have to work more yourself. That's the point! If you can earn more in an hour at work than you would have "earned" by spending an hour painting your house, you make money at no additional expense of time or effort. Alternatively, you could make the same amount of money as you would have gained in house/aesthetic value from painting the house, but spend less time doing it since you are more efficient at work: a time savings. Now, it isn't easy to just "work an extra hour" and get paid for it, but if you make these decisions constantly and trade more time at your job for paying other people to do stuff that isn't worth your time, you'll amass enough saved time to increase your work schedule by a reasonable block.

The key to understanding all of this is that you want to say that the concept of opportunity cost extends beyond money, which it does in a sense--but in another, since all things can be measured in terms of all other things, all opportunity costs have equivalent dollar costs. And so knowing your personal per-hour opportunity cost in dollars is a great way to get a basic quantitative idea of what things are worth your time. If spending an hour with your family is worth more than the $33.74 an hour equivalent that you make at work, it doesn't mean that spending time with your family can't be quantified in terms of dollars; it means that it's worth more than an extra $33.74 but is otherwise _difficult_ to quantify.

And curiously, let's say you reduce your working hours down to 30 a week, the bare minimum you can have to live modestly but comfortably, and spend all the extra time with your family since you love them so much. But why not reduce the hours to 29 a week? Losing that $33.74 every week would put you below the threshold that everyone is happy at. That means that now, $33.74 is worth _more_ than spending the extra hour with your family. In essence, this example shows that the fact that people go to work _at all_ demonstrates how dollar values can outweigh "immeasurable" things like sentimental values, and since they can outweigh, they can be compared, which means that sentimental values have a relative value to marginal dollar value increases or decreases. Therefore, sentimental values can be translated to dollar values. This is a specific case of the general principle noted before, that all things can be measured in dollars, and so it's no sin (and in fact quite handy) to use your dollar/hour opportunity cost liberally when making decisions on how to spend your time.

<comment><AUTHOR> Really excellent writeup. I think many people are squeamish with the idea of "personal opportunity cost" also because in their experience "personal numeric value"==easy unhealthy way to measure (seemingly) one's worth and to compare two individuals value ("I earn $10/H more so I'm better than him/He earns more so he's better than me"), which can be unhealthy. But it should be approached with as little ego (in the self-worth sense) as possible and as a tool you can use to better self-reflect on your life and your decisions.

<comment><USER6> Definitely. It's understandable why people dislike these sterile scientific considerations of very real and human problems, but they're incredibly useful. Though romantic notions of things transcending dollar valuation are well- intentioned, they shouldn't get in the way of that utility.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> I agree 100% and would even expand the idea a little further.

Let's say I make $100/hr at my job.

My car breaks down, and it's only $80/hr for a mechanic to fix it. Seems smart. Now my deck needs repainting, and it's $30/hr for someone else to paint it. Gardening? $20/hr. Washing machine broken? $50/hr. Daycare? $30/hr. Housecleaning? $30/hr... etc. etc. etc.

Because I'm a Software Engineer, pretty much every service in the entire world costs less per hour than I make, so following the logic, I should be paying someone else to do everything in my life for me. I should get someone to cook for me, drive me to work, and brush my teeth.

All of this assumes I can go to paid work an ever-increasing amount, which is obviously not true. The amount of hours I can work in a week is fixed.

UPDATE: Something else the article doesn't mention is that not only is the actual cash-in-hand dollar amount less due to tax, you should also consider the expenses required to earn that money. Transport, work clothes, work lunch, daycare and even rent/mortgage# must be paid simply to go to work. And those expenses come off after tax dollars. The amount you get in your pocket at the end of it all is a lot, lot less than your hourly rate.

#I say rent/mortgage because if you didn't have to go to work, you could live in a tiny little town somewhere paying a fraction of the rent/mortgage you are now.

<comment><USER3> _Transport, work clothes, work lunch, daycare and even rent/mortgage# must be paid simply to go to work._

I disagree on work clothes and rent, and transport to some extent. You must wear _some_ clothes and have _some_ shelter, and you _eventually will_ need to travel somewhere, even if just to the store for food.

<comment><USER7> Agree. My point is that all those costs are much higher when you go to work.

</comment></comment></comment><comment><USER8> My first question when considering outsourcing a task is this: is it a chore or a project?

A project is some usually non-recurring task in a field that I enjoy and/or would like to get better at. A chore is some usually recurring task in a field that I either don't enjoy or see no need to improve my skills in.

My advice is to outsource chores (to the extent that you can afford) and prefer a DIY approach to projects (to the extent that you have time).

Chores can usually be outsourced at a profit, since they can often take advantage of economies of scale (e.g. a lawn maintenance service can afford really nice mowers that work even on long or wet grass and that do the work of two or three push-behind mowers).

<comment><AUTHOR> It's a good guidance but I think it's more complicated. We would all like to have a zen-like efficiency at all times and do such projects whenever we have free time, but in reality it is usually far from that.

I think most people would agree they have a bulk of time where they procrastinate and it would've been more helpful (even on a strict financial level) to just do the chore yourself at that time.

</comment></comment><comment><USER9> I partially agree, but would take it a step further -- your time actually has zero value in and of itself. Because we have companies (or clients) paying us a salary that takes up a portion of our time, we tend to think in those terms, but it's not the correct way to think about it. The end goal of that work we do is not to make money. Money is not a goal in and of itself (or if it is, priorities need to be shifted). The end goal is hopefully a happier, more productive life, which money (sometimes) helps us to obtain.

I work 40 hour weeks so that when I'm not working, I am able to focus 100% on my family and self. By working a job, I can (nearly) guarantee that money will solve those other problems. Leaky roof? No worries, hire a contractor. Car broken down? Call a mechanic and get on with the more important stuff. Need to paint? Pay someone.

If you take money out of the equation, it's just bartering: I do something I'm good at so that I can trade that effort for someone else doing something they're good at, and so on. The end goal is that instead of spending our entire lives worrying about Maslow-ian needs, I can spend a portion of that time doing specialized work that allows me to not worry about those things. It's a trade of quality vs. quantity.

</comment><comment><USER10> Have to agree...

Further - it's been said before - when it comes to time -vs- money - you can always find a way to make more money if you need it - but what you can't do is make more time - time goes by whether you use it or not. If I have a choice between playing with my kids and turning down some overtime, which pays very well, I'll take spending the day with my kids. That time can't be replaced later. Odds are I can always work more. You get one life of unknown length - choose how you spend your time wisely, not by some simple formula.

Further, if I do want to do consulting work after work, it's certainly about how much I'm going to get paid, but it's also about what hte work is, how interesting it is, who it is for, and what the other network benefits might be. I won't generally work for free, some people I've charged an arm and a leg simply because I know they need it, I know I'm in a unique position to help them, and I have no vested interest in whether or not they take my offer - <USER2> personal time is valuable to me. If I had enough offers for work at high enough pay to make me really think about eating it all up, I'd quit my day job and just do that.

<comment><AUTHOR> Agreeing with what you said as well.

I think (as the user 'grecy' mentioned in another place) that as a population, programmers are in a relatively unique position. They (we) can be in a situation where there are a lot of clear opportunities to make more money given more time investment (that can't be simply said for a coroner or a florist for example) AND having it being a lot of money. Couple that with that that it's true from a relatively young age (out of college) and it gives you some pretty interesting (blessed) challenges.

</comment></comment><comment><USER5> Your Money/Time principal has long been defined. It's called Opportunity Cost. You have experienced this principal your entire life. In many cases, especially as we earn more, this principal tends to involve money. However, it's really about value and the decisions we have to make in life.

As far as how you express it, your conclusions are a bit off. The method by which you earn money (hourly, salary(taxable or not)) has no bearing on the value traded. Additionally, both time and money are finite, therefore a tradeoff must occur. This is true no matter your ability to work more or you're a billionaire. In your approach to this idea, you're assuming some kind of concrete, always balanced, equation. While this is more true(and easier to see as balanced) when opportunity cost is applied in business (outsourced factory can make widget X in bulk cheaper), this is almost always false in life situations like you describe. There are just too many variables. Skill, willingness, enjoyment, or necessity as just a few of the intangibles that factor into the decision.

</comment><comment><USER11> From an Entrepreur's perspective, time should be measured by calendar-months and be the primary optimization. Focus on efforts that will slash months from your company timeline.

Distraction is the real cost of a car breakdown: missing a meeting, finding a mechanic, bringing the car in, answering the mechanics phone calls, picking the car up according to the mechanic's hours, etc. Repair bill? Lost in the noise. If you're really optimizing, you live walking distance from the office to avoid distracting risks like this.

One reason to consider fixing the car yourself is to accomplish a fully engaging distraction as a break from work, the sort of thing that can lead to creative breakthroughs. If you can schedule it at the right time.

</comment><comment><USER12> There's a nice Quora thread that's related to this post: [http://www.quora.com/Personal-Finance/If-I-make-30-per- hour-...](http://www.quora.com/Personal-Finance/If-I-make-30-per-hour- working-a-40-hour-week-job-how-much-should-I-consider-my-time-worth-in-terms- of-deciding-whether-or-not-a-task-is-worthwhile)

</comment><comment><USER13> You can lose money and make it back, but you cannot lose time and make it back. That's the real fallacy when comparing money and time.

</comment><comment><USER14> Sadly this article about the value of your life is written by an accountant, or at least someone who sees things as a bean-counter does.

What about the intangibles that Don Draper might get by working on his car? What is the value of joy? What is the value a Zen experience? Of reconfirming once in a while that you can go beyond being a highly specialized cog in a machine fueled and greased by money? Or developing a bond with the thing that you spend much of your life in and tirelessly shuttles you around (Ever read Ursula K. LeGuin's _Left Hand of Darkness_?).

And Rory: There's a reason she was inspired with the idea, maybe something emanating from her soul. Also she eats there frequently... the place is part of her community, her village. Part of the problem with our world today is that we treat everything as business.

Sigh.

<comment><AUTHOR> _Sadly this article about the value of your life is written by an accountant, or at least someone who sees things as a bean-counter does._

This assumes he holds that exact idea to which he replies in the article and that he holds it true to all things in life.

At some point you do need to face outsourcing a chore or doing it yourself. I for example see no learning/entrainment value out of cleaning my apartment for the millionth time or to battle with transporting heavy & bulky furniture. As said in the last paragraph, if you prefer to make a non-monetary decision then sure, do it, just be conscious about it and don't blindly follow a false (or a very nuanced) principle.

<comment><USER14> [Are you pretending that you are not the author?]

But that's not what your article was about! You weren't pushing a rough awareness of opportunity cost, you were saying that it's a "fallacy" because in most cases you can't calculate it down to the penny. See my other comments.

 _I for example see no learning/entrainment value out of cleaning my apartment_

What about the value of a less stratified society? Ever see _Metropolis_ or read _Brave New World_? Ever live in a country like Brazil or India where the moneyed have servants to do everything from cooking, driving (few in India drive their own car!), and even raising their children? Yes, I want those people to have jobs, but no, I don't want to live in a world like that, or anything like _Metropolis_. And I don't want to be so elite and spoiled that cleaning my fucking toilet is beneath me.

I make well into 6 figures, though I just quit that job and will likely make much much less to contribute something better to this world than I have been, something better than what pure and cold capitalism has been telling me is valuable.

<comment><AUTHOR> _But that's not what your article was about!_

First, relax. No need for name calling (in your earlier comment), no need for getting overly excited with exclamation marks. Maybe there's a case to be made for better authorship and I welcome feedback, but as an honest suggestion please read <http://www.paulgraham.com/disagree.html>. It is very helpful as a guideline for making constructive criticism.

Second, different reasons for why the principle doesn't hold water resonate with different people. If it doesn't resonate with you because some other reasons seem to you more profound then please add to the discussion (as you've done in different comments), just be wary to do it well even when very passionate about the subject.

</comment></comment></comment></comment><comment><USER15> I have preached this for a while too. Time is worthless, it is up to you to maximize its value by making decisions to increase income or reduce expense.

<comment><USER16> How old are you, by chance? I imagine this sentiment is something that goes along with the younger of our group, as they haven't already spent much of their time to realize how little the probably have left, compared to the older of our generation that may have gone through a mid-life crisis, or are in the midst of one or are having those initial thoughts on the horizon. I imagine parents that want to spend time with their children also have a different perspective when it comes to "time is worthless"...

Unless you were trying to go for something deeper here, in which case, please expound upon that.</comment></comment>


3 invites for Pownce - <AUTHOR>

<comment><USER2> I'm curious about this thing now. thanks

mlucero_ at hotmail dot com

</comment><comment><AUTHOR> 1st 3 emails get them

</comment><comment><USER3> mattculbreth AT gmail DOT com

<comment><USER3> and thanks

</comment></comment><comment><USER4> firstname.lastname@example.com</comment>


Oregon Health exchange board puts director on notice over stalled website - <AUTHOR> http://www.oregonlive.com/health/index.ssf/2013/11/cover_oregon_board_criticizes.html

<comment><USER2> It is stunning how this past year has changed my views on government. Between this massive healthcare fuck up, the spying, the shutdown, and all the rest that doesn't need mentioning, I am basically becoming a crotchety old conservative who says formulaic things like: "Government can't do anything right"; "Governments never give up power once you let them have it". I basically went from a liberal college student to a full blown conservative/libertarian (if I pigeonhole myself into common stereotypes). Of course, I'm not even that, because I wouldn't vote for those fucking idiots either.

<comment><USER3> I'm feeling a tangible dissonance brought on by these two seemingly inconsistent truths:

* The government is incompetent at implementing good programs, like healthcare reform.

* The government is very competent at doing bad things, like domestic spying.

<comment><USER4> They're probably equally bad at both things, but being incompetent isn't a problem.

They've totally screwed up health care, and they're not giving up, they're going to try even harder to get it working.

On secret projects there's even less incentive to admit failure and kill the project because there's no public oversight. For all we know, the NSA spying programs were delayed a dozen times and came in billions over budget...

<comment><USER5> >For all we know

In at least one big case, we do in fact know it.

[http://en.wikipedia.org/wiki/Trailblazer_Project](http://en.wikipedia.org/wiki/Trailblazer_Project)

</comment></comment></comment></comment><comment><USER6> California's exchange: 28k people signed up in the first week.

Washington's exchange: 55k enrolled (mostly in Medicaid) in October, and another 40k applied for coverage.

Colorado: 37k in October.

Kentucky: 32k.

New York: 47k.

There were some glitches here in Washington. They had stability and speed problems at launch. They took the site down for a few hours, and then overnight, for maintenance. Within 48 hours, it was fixed and has been stable ever since.

One of the keys to Washington's success was to treat the exchange as essentially a startup. They sought people to build the site who thought it was a great thing to do and found it exciting and interesting, not contractors who would see it as just more 9 to 5 work to do.

Article on the above: [http://www.latimes.com/nation/la-na-washington- obamacare-201...](http://www.latimes.com/nation/la-na-washington- obamacare-20131113,0,4963580.story#axzz2kkXpE7MV)

<comment><USER7> I don't know the specifics of the Washington healthcare exchange, but I can tell you right now, the healthcare.gov didn't fail because they chose contractors to do the job, they failed because of extremely incompetent management on the government side. Had a start up been chosen to do healthcare.gov, and none of the government management were changed, even the best start up would've failed too.

The same contractors that did healthcare.gov (CGI Federal) also developed the Kentucky exchange, which has been lauded as one of the more successful state exchanges. If the contractor was the problem, you'd have expected them to fail in Kentucky too.

<comment><USER8> Not calling you wrong, but it looks like there were concerns about the competence of the contractors[0]. It was certainly paired with bad management, though.

[http://www.npr.org/blogs/alltechconsidered/2013/11/15/245399...](http://www.npr.org/blogs/alltechconsidered/2013/11/15/245399200/internal- emails-reveal-warnings-healthcare-gov-wasnt-ready)

</comment></comment></comment><comment><AUTHOR> "Oracle vice president Tom Budnar told the board that his firm had brought in additional people, including a "swat team" to ensure the site should be able to enroll people by the end of December."

This is a vice president at a software company?

<comment><USER9> I hope he didn't really mean "swat" and it was simply misinterpreted by the writer. Perhaps he meant bringing in a SWOT team [1]. In big corps I've often heard SWOT used when an emergency happens.

[1] [http://en.m.wikipedia.org/wiki/SWOT_analysis](http://en.m.wikipedia.org/wiki/SWOT_analysis)

<comment><USER10> No, he probably did mean SWAT. It's been used by management (as well as the term "tiger teams") to describe a supposed crack team of members that will just swoop in and get everything fixed.

<comment><USER11> You _obviously_ is a swat to swat bugs in your software!</comment></comment></comment></comment>


Ask HN: Can this person program their way out of a paper bag? - <AUTHOR> There are many programming interview questions available online, but all the ones I have looked at are testing language-specific expertise. When I used to teach a second year data structures course I would often start with a simple quiz: write a function in the language you learned last term that returns the sum of the integers in an integer array that it is passed. It was amazing how many of the students could not answer that simple question! Tomorrow, I am helping a family member evaluate someone he wants to hire someone to develop some Web server stuff. I want to start with some very simple screening questions to answer this fundamental question: Does this person really know how to program? I would appreciate some suggestions. <comment><USER2> Try the FizzBuzz test [https://blog.codinghorror.com/why-cant-programmers- program/](https://blog.codinghorror.com/why-cant-programmers-program/)

</comment><comment><AUTHOR> Just to clarify a little: to me there is innate skill and stuff you can learn. All the questions I have looked at so far test what you have learned. I want the other kind of question.

</comment><comment><USER3> Write a program which is passed two arrays and determines whether the contents of one is a permutation of the contents of the other.

<comment><AUTHOR> Nice idea, but this is pretty difficult as a screening question. I could simplify it by letting the candidate assume that there are no repetitions in either array.</comment></comment>


Ivan Sutherland – Sketchpad Thesis – 2003 reedition [pdf] - <AUTHOR> https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf <comment><AUTHOR> Ha, I just found out there was a previous submission through another URL [https://news.ycombinator.com/item?id=8912137](https://news.ycombinator.com/item?id=8912137) (5 month old, 20 comments)</comment>


Movie Review Aggregator Ratings Have No Relationship with Box Office Success - <AUTHOR> http://minimaxir.com/2016/01/movie-revenue-ratings/ <comment><USER2> This is a classic case of learning nothing by averaging together two different populations (as acknowledged in the article, critically-acclaimed indies and mainstream blockbusters). Its like averaging the height of an NBA team with a kindergarten class. It would probably be more informative to group movies by size of release (number of theaters they open in), as well as advertising budget. Just by eye there seems to be a correlation for well reviewed blockbusters to do better than poorly reviewed ones.

<comment><USER3> Right. For movies over $10m budget, there is a positive correlation, and because the OPs graphs are log, it isn't a minor one.

Indies form the second clump high in the rankings.

The negative correlation is caused, as the density contours show, by a knot of highly reviewed indies showing up in the data, confusing the positive correlation of the studio movies.

<comment><AUTHOR> Alright, I double-checked, with appropriate filters on data, and RT scores:

Indie cluster alone:

log-corr: -0.1243865

corr: -0.1673577

Blockbuster cluster alone:

log-corr: 0.230726

corr: 0.2570628

Maybe I should have separated the clusters, and I'll definitely do that for further analysis (even though at best, the correlation is weak in either direction, and the log doesn't change much). I did, however, want to address the general reliance on RT score for any movie, so I stand by the post.

<comment><USER3> > the general reliance on RT score for any movie

The reliance of RT score for what? What are people routinely extrapolating from the RT score that you think is not valid?

Extrapolating budget from the RT score is definitely invalid: you've shown that. But who does that?

People might go the other way: extrapolate quality from a big budget (it must be good, it's a big blockbuster), but as you say, that correlation _is_ valid (though noisy).

So not quite sure what you're standing by, sorry. I don't want to sound rude and argumentative. I'm not trying to be a jerk, I'm very happy to concede your analysis is applicable where it is, I just genuinely don't understand what you're referring to here.

<comment><AUTHOR> Very frequently I've heard the question "is this a good movie?" and the response is "what does the RT score say?" Many online stores include the RT score inline when you buy a movie too.

This avenue doesn't immediately distinguish whether a movie is a blockbuster or indie or how much money it made. But then again, some people do read movie descriptions thoroughly...

<comment><USER3> [edit: Read the article again, deleted previous response, sorry]

So you think box office is a better correlate of 'good movie' than RT score? I didn't get that from the post, but if that's what you're saying, then yes, I concede you have shown that correlation is not valid.

<comment><AUTHOR> Maybe not _better_ , but more important.

I'll update the post with a footnote tomorrow clarifying things.

</comment></comment></comment></comment></comment></comment></comment><comment><USER4> Having worked for a short time in the movie industry (at DreamWorks Animation), I thought this was obvious. And it was certainly true for DreamWorks' movies. Even people working directly on the movies were terrible at predicting how they would perform at the box office.

The reason for this is simple: movie critics are self-selecting movie aficionados. Their tastes just don't reflect the tastes of the public at large. When you aggregate critical reviews, you're only aggregating the opinions of a tiny sample of the movie-going public, and that sample largely shares the same tastes.

<comment><USER5> And specifically, I suspect that a lot of the high box office films are big, loud action films which appeal to the theater-going demographic far more than they do to movie critics.

<comment><USER6> The British film critic Mark Kermode wrote about this at length in his book _The Good, The Bad and The Multiplex_. Spend enough money on a movie and it is almost guaranteed to be profitable, because sheer extravagance turns it into a "motion picture event".

[https://www.youtube.com/watch?v=cDmluUKLbYc](https://www.youtube.com/watch?v=cDmluUKLbYc) [http://www.theguardian.com/books/2011/sep/08/good-bad- mulitp...](http://www.theguardian.com/books/2011/sep/08/good-bad-mulitplex- mark-kermode-review)

</comment></comment></comment><comment><USER7> I don't find this convincing because it feels like there must be confounding variables here. For example, high budget movies may be judged more harshly overall than indie movies, which are clearly not going to make as much money.

It would be interested to see how the picture looks when controlled for movie budget, time of year released etc.

<comment><USER8> As a dedicated reader of movie critiques I can say with absolute certainty that well known actors and directors are in general held to a higher standard.

</comment></comment><comment><USER9> I actually looked at this briefly in my Master's thesis. [https://lup.lub.lu.se/student- papers/search/publication/1626...](https://lup.lub.lu.se/student- papers/search/publication/1626974)

Rotten Tomatoes was the only one that had any hint of meaningfulness. The others (IMDB, MetaCritic had none). Twitter volume, sentiment and number of theaters it opened in were pretty good indicators though.

<comment><USER10> And ad or promotional spend? This is a missing factor, surely?

A movie will have a reputation: good, bad, or indifferent. But a movie also has to be known. Ad spend gets the movie in front of eye-balls at least insofar as it gets to be known to exist. But also word-of-mouth, which is like free ad spend. This is why there are slow burners and mega-flops.

Incidentally this is why I have refused to see Titanic, Avatar, and the latest Star Wars in the theatre. If you scream in my face I'm not going to watch your movie in the theatre. Hype turns me off seeing something, but that's just me not wanting to roll with the crowd perhaps, and not a function of hype itself. Seems like most people don't react this way though.

<comment><USER9> Didn't investigate it at all. Not really sure if it would or wouldn't be meaningful. Just thinking off the top of my head, Word of Mouth should be generated by ad spend. It's measuring ad efficacy in a way. Which might be better than ad spend (total dollars) for predicting success.

</comment></comment></comment><comment><USER11> Movie reviews are evaluations of the quality of a movie, not the marketability of a movie. I'd be surprised if there _were_ a direct relationship.

Next article: Evaluation of Paintings by Well-Regarded Contemporary Critics Has No Relationship to Future Dorm Room Poster Sales Figures.

Edit: Also, a measure of success that doesn't find a way to divide by the budget somewhere is not a good measure. A lot of flops have made 100 million dollars, they just cost 120 million to make.

<comment><USER12> Gross vs budget is huge, just look at any Blum House film

</comment></comment><comment><USER13> Not finding a significant effect when you fail to control for confounding variables is not evidence of no effect. This is ridiculously bad analysis. His contour maps basically disprove the headline. How is it on the front page?

<comment><USER13> Just real quick here. Box Office Mojo has data on 702 movies released in 2014 (I only wanted to do one year because there's no API I know of for this and I was doing it quickly, I chose 2014 instead of 2015 because there's still many 2015 releases that are still earning money and so I wanted the most recent "complete" year). The correlation between total box office gross and theaters shown in is very robust, at .79. So if you look at the top Metacritic movie from 2014, Boyhood, you can see it has a low gross, ranked 100th on Box Office Mojo's data. But if you look at the theaters shown in, it was only in 775 theaters. Mr. Turner, the second movie on Metacritic's rankings for 2014, was only shown in 120 theaters. Birdman, the Best Picture winner and the 16th ranked movie on Metacritic's list, was in 1213 theaters. Meanwhile, you have to go to the 21st movie on the top grossing list to find a film that was in fewer than 3,000 theaters. It's a classic example of Simpson's paradox[1]. You have two populations, wide-release movies and small-release pictures, and within each population, there's a positive correlation between critical approval (as measured by Metacritic) and box office gross, which is picked up by the contour maps. But because the highest Metacritic scores go to the movies released in 2000 theaters or less, without controlling for the number of theaters a movie shows in, it looks like there's a negative relationship between critical acclaim and box office totals.

1) [https://en.wikipedia.org/wiki/Simpson%27s_paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox)

<comment><AUTHOR> That's a fair counterpoint. I'll see if I can find theater showing data (that was not in my dataset, and as you note, it's hard to get), and see if I can normalize.

</comment></comment></comment><comment><USER14> ticket sales count how many people thought the movie was good enough to watch (amd that depends highly on marketing spend) not how good they thought it was.

I would like to see this analyses run on box office as a _percentage of budget_ , not just top line revenue.

</comment><comment><USER15> A few years ago I was listening to "How to listen to and Understand Great Music"[1] and the speaker made an interesting point about art.

Let's say you take a friend from another country to a baseball game. He's never seen baseball and knows nothing of the game. He can certainly enjoy himself -- there will be lots of colors, activity, and it's quite a spectacle. He may even want to come back.

As he learns more and more about baseball, however, he will have a completely different experience. It's the same game, but now you see a lot more of the complexity and nuance.

As a film fan, I use the meta-ratings almost exclusively. I'll go and watch popular films. Sometimes I'll even enjoy them. But there's a lot going on in cinema. I like understanding the nuance and detail, and I think well-mace movies make for better experiences.

1\. [http://www.thegreatcourses.com/courses/how-to-listen-to- and-...](http://www.thegreatcourses.com/courses/how-to-listen-to-and- understand-great-music-3rd-edition.html)

<comment><USER16> As someone who's not from a baseball country: I strongly doubt that friend would enjoy himself. It's not exactly action packed. I think the decisive factor isn't knowing more about the game but knowing more about the meta-game.

Sure, you will experience it differently if you know what good tactics look like but most people enjoy it because they are emotionally involved with the teams and players. This seems to be true with most sports, even for the more "knowledgeable" fans.

</comment></comment><comment><USER17> Aside from carefully constructed surveys, the best indicator of US box office success that I have found is Wikipedia page views [http://journals.plos.org/plosone/article?id=10.1371/journal....](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0071226)

Outside the US it's a harder problem [http://arxiv.org/abs/1405.5924](http://arxiv.org/abs/1405.5924)

</comment><comment><USER18> There is no effing way this is true. Everyone I know under 40 checks the RT score of a movie and factors that into their decision to see it.

<comment><USER19> I think it is safe to say that fully 1/2 of Americans would seem almost alien to you if you were to interact with them in their natural habitat(s). It's true for me, and I was raised among them, before fleeing as soon as I was old enough to drive away.

I mean no offense to anyone. But there's a lot of folks out there living a very different lifestyle than the average HN'er.

</comment></comment><comment><USER20> Marketing budget (or maybe degree of awareness of the movie's existence), theatre/screen count, and audience score predict how well a movie will do. Appeal and marketing, rather than movie quality, has always been what draws people to the movie theatre.

</comment><comment><USER21> Normally, I'm all about research, experiments and data crunching that confirm or contend ideas we pretty much all take for granted but this makes me wish there was some very polite way to say, "well duh!"

<comment><AUTHOR> Given the "There is no effing way this is true" comment in this very thread, this may not be as "well duh" as you think.

That's why data analysis is always important, even if it's dumb.

</comment></comment><comment><USER22> A lot people don't realize, but RT also shows a mean rating (out of 10) which is the same type of scoring as Metacritic. It might not be exposed in the api mentioned in the article though.

</comment><comment><USER23> Interesting! My question is: How do the p-values of your Pearson correlations look like? Correlations alone are vague. Please insert :-)

<comment><AUTHOR> I deliberately avoided using P-values since that metric is used to imply a _causal_ relationship between review scores and box office gross, which is definitely not true.

<comment><USER23> In the evaluation of correlations, it can always be informative to know the confidence interval for r, with all caution towards p-value interpretation.

Surely, correlation provides information on association rather than cause and effect (causation should rather be modeled with Granger and other regression models). Sample sizes and variances will certainly contribute to different p-value outcomes. This is because p-values reward low variance more than the magnitude of impact (Type I/II error etc.). If you have p-values, better report them and add a footnote on how to interpret them.

<comment><AUTHOR> > _However, the significance or p-value reflects the probability that the > correlation does not imply a causal relation._

Technically, in this case, a significance test would answer the question "is the Pearson correlation statistically significant from 0?" In this case, we would expect it to _fail_ since it clearly isn't, and is therefore the test is less helpful/important. (even if it passed, the conclusion would be "correlations are low in magnitude and therefore do not matter" as noted in the post anyways)

Finding the exact P-value of a Pearson correlation requires setting up bootstrapping, which is not something I have handy at the time but will work on in future posts.

Again, I'm not looking at R^2 and the P-value of a linear regression, which is different.

<comment><USER23> It's just a recommendation to improve the reporting, no general defense of p-values. Pearson does not imply to analyze a causal relationship. I see the point it's not linear (then you would have had a fitted linear reg, I assume) but still can tell you that missing p-values may cause arching eyebrows :)

In small sample sizes, correlation can easily be significant, often at the cost of low confidence. To the opposite, in large sample sizes, the magnitude of the effect may be lower but at higher confidence. In both cases, results have to be interpreted with caution. The recent p-value debate points towards a lot of issues here. For instance, there have been medical studies overestimating correlations in small sample sizes while other authors seemed to underestimate their long-term large-sample results with correlations in the ballpark of 0.15 (p<0.05).</comment></comment></comment></comment></comment>


Four in 10 pupils have had little contact with teachers during lockdown - <AUTHOR> https://www.theguardian.com/education/2020/jun/15/2m-children-in-uk-have-done-almost-no-school-work-in-lockdown <comment><USER2> At my son's school the parent involvement is high and the kids all engaged as far as I could tell.

A teacher at a rural school told me that they hadn't heard from half their parents or students since the schools switched to distancing learning despite calls and etc... not even once.

They don't know what to do as far as grades and etc.</comment>


Ask HN: How do I integrate subscription billing to my SaaS - <AUTHOR>

Hi,
I'm building a web and mobile based SaaS which has a subscription tier. It would be preferable to keep the billing outside the application (API) so that the product can stand on it's own, i.e. separation of concern. How is this managed the best? Is it even possible too keep them as separated as I wish for?
I am thinking about just adding a "Stripe customer id" field to the account model, but I was hoping to not integrate Stripe directly. The plan is to use Stripe for the payments as I understand that they have a fantastic API.
Thanks! <comment><USER2> Stripe. Just how you think you'd do it, with a CustomerID field in the user (or company) table.

Don't sweat the possibility that you'll have to change payment providers one day. You probably will, and it won't be anywhere near as much effort as you think.

I once switched a couple of my services from Amazon Payments over to Stripe in the course of an afternoon, fixing up the fields, adding a PaymentProviderID column to the table, and rewiring the backend. It was in no way painful. Certainly less effort than it would have been to build in an abstraction layer ahead of time that would have been any help at all during the conversion process.

This is a great example of "you aren't going to need it" combined with "you're just going to build the wrong thing" combined with "quit fannying about building crap you don't need and actually _ship your product_ ". To pull off the perfect abstraction layer, you'd need to know not only the system you were integrating, but also enough about the _next_ system that you wouldn't have to do all those things I describe above anyway. It'll take you just as much time to do the work up front as it would to do it when you actually need it, but in the meantime you won't have a shipped product.

In short, don't sweat it.

</comment><comment><USER3> I would avoid integrating directly with stripe, since if any trouble ever happens with your account (chargebacks, etc), and they terminate it, you need to change all your code before you can resume accepting payments. For this reason I recommend using an abstraction layer to buffer for payment gateways. One good option is Chargebee, which integrates with 12+ different gateways. This way, the same code can control different payment options, and you are not limited to one gateway.

<comment><AUTHOR> Thank you, that is great information to have in mind! As I wrote in another answer, the subscription will most likely end up as a separate service so it wouldn't be a problem for customers to access the application if I get trouble with Stripe, at least. I would imagine that having a third party gateway would add to the expenses. I am not really too interested in that in the beginning.

Could you educate me when a chargeback would occur using Stripe?

</comment></comment><comment><USER4> We create a Stripe customer at the same time the customer registers. At that same time we assign them to a "free" plan at Stripe. Then once they decide on which plan and to stay on and be a subscriber we update their plan to the new plan. Stripe then charges them for it and sets the recurrence etc.

You can also setup the plans to have a grace period before they are charged with Stripe.

As for how we actually implement it, we create a small service around subscriptions and it is called by our create steps which then hides the calls to Stripe. This way if we ever went away from Stripe it would only require some small changes in the subscription service.

Also, we store the Stripe customer id and card id in our subscription record, and we do it using a generic data structure where we list vendor, vendor customerId and vendor cardId etc. This way nothing is labeled Stripe, except the vendor value itself would be Stripe in our case.

Lastly we also have a web hook active so we can get notifications from Stripe and update records appropriately. Again, a small separate service that handles only the web hooks.

<comment><AUTHOR> Thank you for the magnificent answer. It seems like a good idea to create a separate service to handle the registration. My app is backed by a REST API (I know, I am so trendy), so it would just be to call that one and a call to Stripe's API and be done.

One thing I realized now, reading your answer, is that the customer's plans should be changed depending by the user's usage (think storage space). Should I be running a cron job once or twice a day to check the usage, compare it to their subscription plan and update it at Stripe? Of course notify them by email, and maybe give them a day to re-do and not give them unexpected charges.

<comment><USER4> So if your plans vary based on usage patterns and that carries additional charges, then yes, you'll have to run some sort of scheduled job to check for users getting close to plan thresholds. IMO you don't want to auto upgrade people, but rather warn them at say 80%, then 90% and give them the option to upgrade or be frozen at their current use level. This prevents chargebacks and complaints for auto upgrades they didn't want to have happen. Seems you already are on that path.

But on the other hand (thinking storage space), if your model is such that they pay $9.99/month with up to 5gb and for every gb over 5 they pay $0.99 say. Then you can auto charge the incremental fees and just make sure you are sending the clients an email and keeping them in the loop. I have not done this with Stripe myself so I am not sure the best way to set it up, but my initial guess would say you leave their base plan alone (unless they upgrade/downgrade it) and just add a surcharge to their account at Stripe if they go over their allotted storage space. One way I can think of off the top of my head is to change their account balance to be -$.99, or whatever their overage is, so that Stripe can bill that plus their next monthly service. But I am sure there is likely a more elegant solution.

Someone else mentioned Chargebee, which is a great service, but we chose not to go the route of like a Chargebee because paying additional fees on top of Stripe just didn't fit our model initially, and we feel our chargeback risk is very low. But if you have a higher risk business with a greater potential for chargebacks and disputes then it is best to insulate your self as best you can, and Chargebee helps you do that. In our case, we actually can process charges through Stripe or Balanced payments (although 100% is going through Stripe right now), which is why we did the vendor based model as I described above, that was our insurance in case we every had problems with one provider.

<comment><AUTHOR> This is amazing, thank you again! It seems that the way you have decided to solve this is both scalable and modular, so it's very likely that I will adopt it. I will talk with my business people about the best solution for us on handling tier upgrades, notification etc. Right now you have assured me I can let the subscription integration wait, so I'll continue focusing on the actual product and implement the subscription later.

<comment><USER4> Awesome, glad I could help some, good luck!

</comment></comment></comment></comment></comment><comment><USER5> Even though you should use stripe (we all love it), you should not directly code for stripe only. Here is how:

\- Create an abstract "Ecommerce" module in your app and let it configure different payment gateways. In your case, start with stripe as the only payment gate way to choose from. But doing it this way now lets your app easily plug into other payment APIs in future as needed.

\- For the user model, instead of having "Stripe customer id" as a field, I suggest having a user meta table with key/value pairs. So in the usermeta table, you can have {'stripe_customer_id': 'xyz'} for a particular user. This way, you can use other payment gateways and not really limiting to Stripe.

\- manage the subscriptions/cancellations/refunds in your app instead of stripe. This way, your app has all records of transactions.

<comment><AUTHOR> Thank you for the answer!

Mr. @davismwfl gave me some really good directions on my problem in his answers, do you agree on that way? Is there a reason you suggest managing the subscriptions directly in the app instead of in Stripe? And does it count as "my app" if I manage it in a separate service, as @davismwfl solution?

<comment><USER4> Codegeek is right on with this, the description I gave is in line with the same thing he is stating. Our data store is mongo so we use documents for the subscription details but if I was in an rdbms it would likely be a key/value pair as described.

The detail about storing all your subscription details locally is valid. We do it but also put the details in Stripe as that is what enables the recurring billing. That is also what I was pointing out in the web hooks as we store all the Stripe history from the web hooks. Works great.

</comment></comment></comment><comment><USER6> [https://stripe.com](https://stripe.com) is the best. You can also use BrainTree.

<comment><AUTHOR> Thank you. I'm planning to use Stripe for payment service. The thing I am looking for is the practical implementation of adding a user to my application, and creating a subscription account at the same time without mixing Stripe code into my product. Do you have any experience with that to share?

</comment></comment><comment><USER7> Braintree?</comment>


Ask HN: reasons for www. subdomain - <AUTHOR>

I purchased a domain for personal use with Google App Engine, and I found it is not possible to link an app to the so-called naked domain (without any subdomain) now. This leads me wonder what is the technique reason behind this. From usability point of view I would really like to use a naked domain primarily, with www. redirects to the naked one. Is there any technique problem with this that Google does not allow me to do so? <comment><USER2> There is no inherent technical reason why you couldn't. And this article at Google (expand the first [+] area) suggests you can use non-www domains just fine:

[http://www.google.com/support/a/bin/answer.py?hl=en&answ...](http://www.google.com/support/a/bin/answer.py?hl=en&answer=91077)

Where did you get the idea you couldn't?

(The only trick might be something specific to App Engine for generating the www -> non-www redirects, as you won't have the usual access to web server configuration options. But that'd be a better question for the App Engine forums.)

<comment><USER3> i imagine because google says you can't.

<http://code.google.com/appengine/kb/commontasks.html>

"Due to recent changes, App Engine no longer supports mapping your app to a naked domain. If your domain registrar supports URL redirects, you can redirect from <http://yourdomain.com> to e.g <http://www.yourdomain.com> or [http://appid.yourdomain.com."](http://appid.yourdomain.com.)

<comment><USER2> Aha. Guess I should have searched past finding a Google document saying it was possible!

Riobard should join the other AppEngine users pushing Google to restore this previously-available functionality, at this open AppEngine issue:

[http://code.google.com/p/googleappengine/issues/detail?id=77...](http://code.google.com/p/googleappengine/issues/detail?id=777)

Viewing the comments there, AppEngine users have had their inlinks and search rankings ruined by this change. So chalk this up as another reason to be wary of building in the Google-controlled cloud.

<comment><AUTHOR> Thanks for the suggestion! Since I just start to build the site, I am not affected by the linking and ranking stuff.

However I am interested in the following paragraph from the link you provided:

"From a GAE google groups post (posted by Google):

One question that we see a lot on the group is about setting up an app to serve from a naked domain (<http://example.com>) While this has been difficult but not impossible in the past, we've made some changes and are no longer allowing naked domains to be used for App Engine apps. The infrastructure which we use will occasionally force TCP connection resets when an app is running on a naked domain, so we're recommending that everyone switch to a subdomain (like <http://www.example.com>) and set up DNS redirects if you already have traffic going to a naked domain. "

I don't understand why using a naked domain lead to TCP connection resets. Is it a general case (which means using naked domain is a bad idea anyway), or is it just Google's problem?

Personally I think prefixing every domain with "www." is really a silly idea, but it seems now there is some technique difficulty behind it. Puzzled ...

<comment><USER2> It must be a bug in their back-end infrastructure; there's no general reason why 'naked' domains should be any more prone to TCP connection problems than any other domains. (TCP doesn't even care about domain names, and even in DNS, 'naked' 2nd-level domains are functionally equivalent to 3rd-level domains, 4th-level domains, etc.).

There is at least one legitimate reason using www (or any other subdomain) can be beneficial -- to isolate application cookies from your base domain, to assist caching. See here for details:

[http://developer.yahoo.com/performance/rules.html#cookie_fre...](http://developer.yahoo.com/performance/rules.html#cookie_free)

However, you're probably far from needing to worry about that, and also segregating your static, highly-cacheable data to another domain is another way to get the same benefit.

(I tend to prefer non-www domains, too.)

<comment><AUTHOR> Thanks for the info!

I think the cookie/cache reason explained why virtually all major websites (millions hits/day) redirect naked domain to "www." subdomain.

I also found the following info from here [http://blog.bigcurl.de/2008/08/redirect-naked-domains-to- sub...](http://blog.bigcurl.de/2008/08/redirect-naked-domains-to-subdomain- in.html)

"That is why I wrote redirector. It is a little application which redirects all the apps in your Google Apps account from a naked domain name to a subdomain.

Eg. bigcurl.de --> www.bigcurl.de

If you want it can also do this vice versa but be aware that you are not taking advantage of Googles geocaching mechanism with this approach."

So using "www." domain for GAE seems to be a better solution if one wants faster access/response.</comment></comment></comment></comment></comment></comment>


How to write better game libraries - <AUTHOR> https://handmade.network/wiki/7138-how_to_write_better_game_libraries <comment><USER2> It always baffles me when a library comes with its own arcane build code. Instructions like "Requires Python" invariably turn out to actually mean "Requires hours of debugging".

<comment><USER3> IMO the ideal way to get your library used is to generate a single .h and .c file.

<comment><USER4> Some libraries don't even ship a .c file, all the code is in the header

<comment><USER5> The downside of that is that all code is in the header. As in you will compile all the library code whenever you include the header. As third-party library code is something you probably seldom change yourself, you don't want your build to spend time building it all the time.

Sure some of those libs may have some define you need to set in one of your .c files so the implementation will go there but then whenever you change _that_ file it will result in a recompile of the third-party lib. So then you go and create a separate .c file _just_ for the lib and we're back to why the lib author didn't do that in the first place?

No, I think what sqlite does is pretty neat. If you're doing development in the sqlite code-base you have lots of files (so you can manage it) but if you just want to use sqlite, you have the amalgamation which is one .c and one .h.

</comment></comment></comment></comment><comment><USER6> This is bad advice on the c++ part. If you don't use RAII, then you could've just not done the c++ lib (and have just a c lib). Exceptions, raii and smart pointers are what makes modern c++ so pleasant, next to templates.

<comment><USER7> > modern C++

> pleasant

Nice joke. Hour-long compilations, type-unsafe templates with unreadable errors, and circular references are considered pleasant now?

<comment><USER8> \- reasonable organized code doesn't take hours. even template heavy code is quite fast these days

\- templates are type safe. it is just the error messages are not helpful a lot of times. C++20's concepts should help

\- circular references. that's a thing in C as well. as soon as you need interact with your libraries user to allocate/deallocate resources.

C++ has a large surface to interact with. Yes you can shot yourself in your foot. However, the more you know you can use the features to your advantage instead of shotting yourself in the foot. At the point you gain more than you lose.

<comment><USER9> Templates are duck typing at C++ compile time (which is their runtime). Yes, concepts should help.

'Modules via copy-and-paste' (= #include) is what I would hurl at C++ these days. Though they are working on that as well, I heard. Though [https://vector-of-bool.github.io/2019/01/27/modules-doa.html](https://vector- of-bool.github.io/2019/01/27/modules-doa.html) doesn't sound hopeful.

<comment><USER10> > Templates are duck typing at C++ compile time (which is their runtime).

That description only applies of you venture into the dark realm of template metaprogramming, and anyone who ventures into those lands is wise enough to understand that the tool is not the one to blame if one decides to abuse a feature to apply it in a way it was not (initially) designed to be used.

<comment><USER7> You must be trolling... Templates in C++ ARE for metarogramming. The fact that they work as a glorified text substitution (and they are only coming around to fix them in 2020) leaves no apology for the tool.

<comment><USER10> > You must be trolling... Templates in C++ ARE for metarogramming.

That's an ignorant statement both wrt generic programming (you know, the whole purpose of C++ templates) as well as the history of C++, in particularly how C++ template metaprogramming was discovered by accident after C++ templates were already implemented, widely used (see the STL), and standardized.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> I don't see it mentioned, but it's important to note that you don't have to write your library in the same language that you expose to your users. You can write everything in C++/Rust/Python and expose something compatible with the C ABI and people will be able to use it (provided they have your toolchain…)

<comment><USER12> Fully agree with you.

Anyone providing a library written in pure C, better be serious about security and prove that they have taken all the required steps to handle memory corruption and UB exploits.

We really need more liability on software development.

<comment><USER13> Do note where this article resides. Casey Muratori (of Handmade Hero, a game dev tutorial project) is a self-proclaimed anything-not-C hater. A community has formed around his project, and has, of course, adopted his hard-line stance.

<comment><USER12> Interesting, thanks for the hint.

<comment><USER14> The earlier comment was misinformed. Handmade Network is inspired by the series, yes, but stands on its own and embraces new languages and very different projects. See the Handmade conference[0] for examples.

[0] [https://handmade-seattle.com](https://handmade-seattle.com)

</comment></comment></comment></comment></comment><comment><USER15> The problem for game libraries in general is that game dev lives in c++ world and c++ is awful for libraries. Most c++ devs I know would rather start writing a program by defining string than by learning to handle proper library management tooling. I'm not an expert so I'm not sure if it's just a culture thing or if there are inherent features in the language that make library usage hard, but yeah. That's that.

<comment><USER16> I wish the C++ world would adopt bazel eventually as their standard tooling base.

<comment><USER10> > I wish the C++ world would adopt bazel

Why do you believe picking a specific build system is relevant wrt libraries, particularly a build system whose main selling point is build speed.

</comment></comment></comment><comment><USER17> As a warning: While C99 is a really nice language - especially over C89 - please note that the Visual Studio C compiler still does not support the full C99 specification, and it is not possible at all to compile C99 as C++ code (which sometimes is useful).

I usually recommend to write library-level code in the "subset of C99 which compiles both in C and C++ mode on GCC, clang and MSVC". This is basically a version of C that's somewhere C89 and C99 (basically a "C95").

Another quite valid option is to define a pure C-API first, but implement the "inner library code" in a simple C++ (just be careful with "modern C++", since this usually results in increased compilation time and binary size).

<comment><AUTHOR> Thanks for the comment, I might steal your "subset of C99 which compiles both in C and C++ mode on GCC, clang and MSVC" quote and put it in the article if that's ok ;)

<comment><USER18> Is there a benefit to building C code with a C++ compiler?

<comment><USER17> C++ has some stricter type checking by default, but most of this can be achieved by raising the warning level when compiled as C.

Nevertheless I received enough requests to make my C libraries 'C++ compliant' that I gave in :)

</comment></comment></comment></comment><comment><USER19> Nice article :)

Might be worth touching on error callbacks/logging as an error handling strategy.

Sometimes an error is not recoverable in the sense that the calling code can't really do anything about it, but the library should attempt to make progress anyway instead of halting the entire program.

By allowing users to specify an error callback, this means they can log errors, capture stack traces, assert, or whatever.

This isn't that helpful for smaller libraries with smaller-scoped processes, but if it's something like a renderer or interactive audio lib, those often just need to be given a bunch of frame time to do work with the complex input you've prepped and fed to it, and trying to propagate error codes up out of that simulation step would both contort the inner code and not be as helpful as an error callback.

With an error callback you can assert, set breakpoints, or do whatever. But more importantly, by default you can have it just log so when you inevitably in a bug it doesn't prevent everyone else from getting work done while it keeps asserting until you fix your shit.

This will be a less common need than the other standard error reporting mechanisms, but is important to get right if your library has these complex internal preconditions that you want to make visible to the client when violated.

</comment><comment><USER20> Honestly, for me all I would like is the ability to include/import one or more files as needed, helpfully separated by basic functionality, without needing to include the entire library, have a bare minimum of dependencies, hopefully, module independent, the ability to build painlessly for different platforms with a minimum of code changes and a painless build process for the library itself, ideally one I can include in my own build process without a lot of effort.

I've used some libraries that do some of these things, but I can't think of any that does all of them.

</comment><comment> yoz-y > Not everyone wants to use C++ (some prefer C).

This could be very well flipped over. If I have the source available, I'd much rather deal with C++ bindings.

<comment><AUTHOR> Hi, I mention in the article that "It is easier in general for a C++ user to use a C library than it is for a C user to use a C++ library." which is where the advice comes from.

<comment> yoz-y I agree. But "wants" is a term indicating personal preference. I understand that C is best suited for maximum portability and performance. But as a C++ developer I prefer libraries that come with C++ interfaces (and you do mention that it is good to provide a C++ wrapper as well).

<comment><AUTHOR> Thanks a lot for your feedback. I decided to remove that since I think you are right about indicating personal preference. I do not want this article to be a list of preferences but rather an analysis of the considerations that popular C libraries make (eg: stb, sokol, etc) and that new authors should also reason about.

</comment></comment></comment></comment><comment><USER21> Why not a C++ library with an optional C API?

<comment><USER7> Because C++ is one of the most complex languages in the world, hence a pain to interop with. C FFIs, on the other hand, are ubiquitous.

<comment><USER22> The idea (and one that worked well in several projects I've worked on) is that you use C++ as main language of the library and make outside bindings C.

It does require some discipline, but C++ has many portable constructs that make life significantly easier when writing business logic.

</comment></comment></comment><comment> Const-me Some of these recommendations are awesome, others aren't good, IMO.

> Always prefix your names to avoid name collisions

Solved by C++ namespaces.

> Use header guards instead of #pragma once.

#pragma once is supported by all mayor compilers, header guards can introduce bugs, also #pragma once builds measurably faster: [https://github.com/electronicarts/EASTL/blob/3.15.00/include...](https://github.com/electronicarts/EASTL/blob/3.15.00/include/EASTL/vector.h#L64-L66)

> Expose constants to the user using constexpr variables.

Modern C++ has strongly-typed scoped enums for such constants.

<comment><AUTHOR> Hi, thanks for your feedback. I am the original author.

Regarding prefixes, I advice that you start by writing the library in C and then wrap it in C++ for a variety of reasons that you might want to consider. In C++ you should indeed always use namespace.

Header guards have the advantage over pragma once that they are standard and you can also use them to check if a library is included. I might remove that since maybe it's not that important and people might different views.

Regarding constants, I was referring to things such as numeric constants for which you would constexpr in C++. Maybe I can be more explicit there. Thanks for the feedback.

<comment> Const-me I advise you start by writing the library in C++ and then wrap it in C if you need C ABI of the library.

It's very hard to write correct C code which does IO and supports multithreading. Take a look at Microsoft's implementation of fprintf, copy- pasted from Windows 10 SDK: [https://gist.github.com/Const- me/f1bb320969adde6c79694265ea6...](https://gist.github.com/Const- me/f1bb320969adde6c79694265ea644b9d) They use RAII to set & revert the locale, and to lock RAM buffer to avoid corruption by another threads. They use C++ lambda for exception handling. They even use C++ template to avoid code duplication between printf and wprintf.

But these C++ shenanigans are not exposed to user, user calls their `printf` (possibly in a code built by C compiler) and it just works.

<comment><USER23> > _It's very hard to write correct C code which does IO and supports > multithreading_

This is just a random unsubstantiated statement. There's nothing particularly "hard" about writing IO libraries that is language-specific. Multithreaded or not.

<comment> Const-me When you do IO and multithreading, functions often need to acquire and release stuff: mutexes or other locks, resources, locales. Even more so in videogames, e.g. OpenGL code often calls glMapBuffer / glUnmapBuffer many thousand times each frame.

The hard part is making sure you release stuff every time you acquire stuff, exactly once. C++ RAII makes it almost trivially simple, but standard C has nothing comparable. When you only targeting gcc and clang can use __attribute__(cleanup) in C, it helps but still it's more limited and more error prone compared to destructors.

<comment><USER24> > The * hard part * is making sure you release stuff every time, exactly once.

Oy vey... you can't be serious. That's rudimentary basics of using any API.

<comment> Const-me It's easy enough to correctly use a small API in a small program. It doesn't matter at all for short-living apps which clean up their resources by exiting the process.

There're also other programs in wide use, which need to reliably work for hours, sometimes weeks. Some of them have huge amount of code they built from, written by many people over many years. Combine that with large enough APIs (some peripheral devices have hundreds of writeable registers of state; or D3D11 exposes huge amount of very complicated state, only limited by VRAM amount which is measured in gigabytes) and it's very easy to make bugs in such programs.

Leaks of memory, handles, sockets, and many other resource types e.g. GPU ones. Deadlocks caused by locked mutexes, or threads which exit but forgot to release something they needed to release. Unwanted changes to global or thread state, both internal to the process and external (locales, formatting options, console colors, process and thread priorities, current directory, environment variables, CPU registers like FPU flags and interrupt masks, GPU render states) caused by some code changing stuff but not reverting the changes back. Unwanted state changes of custom peripheral devices, due to the same reason.

C++ RAII is not a silver bullet, but it does help a lot for all these things.

<comment><USER24> It's easy enough to implement correct API usage semantics in a program of any size.

If you need to rely on RAII in order not to screw things up, then it's an issue with the coding style or the application design. That's what needs fixing. Not the language choice. You got it backwards.

<comment> Const-me It's a hard problem, and language + runtime support helps. That's why C++ has RAII, C# has IDisposable / using, Java has try-with, python has enter / exit / with, golang has defer, and so on. The only reason C has nothing comparable, it's almost 50 years old now.

The problem is only tangentially related to API semantics. The problem is mutating state. The state is not necessarily managed by an API, for instance CPU registers aren't, you modify them directly. Same with other global state like I/O formatting options and locales, these things are just global variables.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER25> _Make_ is a much the lingua franca of build systems as C is for programming, so there's no reason not to provide a (simple) makefile. The alternative they suggest is worse anyway.

<comment><USER23> Windows is the reason make is not a good choice. Visual Studio is a de-facto standard for Windows development and it's not exactly makefile-oriented.

The lowest common denominator for build instructions that works everywhere is literally "set your include paths like so and compile these sources".

<comment><USER25> How strange. I have tinkered with a couple of FOSS games of Windows and none of them required VS (not to mention another couple of other non-game projects). Maybe that's because they often are multi-platform projects so Make is the best choice at the global level for them?

> The lowest common denominator for build instructions that works everywhere > is literally "set your include paths like so and compile these sources".

Yes, but that's quite inconvenient and error-prone ; a typo in a -D flag is very likely to not compile what you wanted.

 _Make_ is not a multi-megabyte program with a frack tonne of dependencies. Having your users to install it in order to build your project is far from being completely outrageous (unless for _cough_ microsoft _cough_ cults that consider the command-line is evil).

</comment></comment></comment><comment><USER26> And then the author provides example where there's Dear IMGUI which is c++.

<comment><USER17> Dear ImGui has a very C-style API though, it's just flat a function API wrapped in a namespace, and about the only other C++ feature that's used in the API is default function arguments.

For usage from C code, there's cimgui, which is an automatically generated C-API wrapper:

[https://github.com/cimgui/cimgui](https://github.com/cimgui/cimgui)

</comment></comment><comment><USER27> We just need more single-file headers like nothings' stb libraries

</comment><comment><USER28> Since this article lays so much emphasis on C, I have an honest question to everyone. What is a good way for a beginner to learn C in the current time? The minefield of undefined behavior is really overwhelming to a beginner. Are there any good resources that teach C the right way with good advice and best practices to navigate the UB minefield?

<comment><USER23> > _the UB minefield_

While UB quirks exist, they are WAY off the beaten path and it takes an effort to run into them. Doubly so if you are just starting with the language.

Just treat C as a thin convenient layer over the hardware that expects you to think and act responsibly in exchange for this nearly raw access.

<comment><USER29> In C (and C++) something as simple as forgetting to initialize a variable lands you in UB-land already, nearly every person I knew when I was learning myself (and still now) ran into these things _very quickly_.

Not to mention compilers make fun-times out of this by sometimes zeroing memory in debug and then not doing so for release builds (Hi MSVC!)..

The "thin layer over hardware" idea is a thing of the past as soon as optimizations come into the picture, and even then.

</comment></comment></comment><comment><USER30> Please add (C) to the title of the article. In this form, it is misleading because most of the advice only applies to C.

<comment><USER31> The target audience isn't C programmers, but library authors that want their library to be usable with multiple programming languages. Most of the advice applies to C libraries because the article starts by advising you to write your library in C, and gives several reasons.

</comment></comment><comment><USER32> >> C is the lingua franca of programming

I don't have the exact numbers, but this seems debatable.

<comment><USER33> It's the lingua franca in that it's the standard for interoperating. Every high level language I've used has had _at least_ one C FFI readily available.

<comment><USER32> We do have a mutual understanding on whether C foreign function interfaces are available for other programming languages, which does enable languages to send system requests (for systems which are written in C) or to create optimized C code to parse documents or calculate formulas. I am aware that the concept of making languages talk with C interfaces exist. And I made the above statement knowing this.

</comment></comment></comment><comment><USER34> _> Write the library in Standard C99_

Why not write it in Rust, and provide C99 bindings? Using stock C for the actual library doesn't sound like a good idea to me, when today there are better alternatives.

And for sure avoid using Metal lock-in directly. Use Vulkan, which Apple should have supported from the beginning. For lock-in targets, there are translation options from Vulkan.

<comment><AUTHOR> Hi, the reason for using C99 is that it is fully compatible with other languages. If you write a library in Rust you might be tempted to use Rust only feature which will then make it hard to wrap the library for other languages. And if you don't use those features from Rust in your library people will comment that your library isn't Rust enough. Same thing applies to C++. To that extent my advice is to write C in C rather than C in Rust or C++.

Also if people don't have a Rust toolchain setup, compiling the library and using it from source would be hard. In some cases integrating Rust in their toolchain could be hard.

Regarding metal and vulkan I will edit the article to mention Vulkan there too alongside metal. Thanks for your feedback.

<comment><USER34> I suppose wrapping it for other languages can be harder, but should be doable I think. The benefit of Rust though is using a much better language with rich standard library.

As for toolchain, Rust can be set up basically anywhere llvm can, which is quite a lot. There are rare cases where llvm wasn't ported yet to, but I don't think they are enough to make C a compelling option in general, and I don't think any of them are gaming related.

If you are in such case - then sure, but otherwise, I'd still prefer Rust.

<comment><AUTHOR> I think the Rust standard library is actually a liability in this case, not an advantage. The rust stdlib will essentially be an extra dependency and also the rust standard library doesn't have allocators, so unless you are very careful you might violate the principle of not doing allocations for the user. Also, even if it did have allocators, passing allocators from to Rust via a C interface would probably be awkward.

So if you do choose to make the implementation in another language, there are extra considerations that you have to take into account.</comment></comment></comment></comment>


The One Thing Facebook Announced at F8 That's Already Affecting Your News Feed - <AUTHOR> http://blog.naytev.com/instant-articles/ <comment><USER2> SYAC: It's Instant Articles. Which is wrong because WaPo ran them when they were first announced awhile ago.

Also, the "The One Thing" headline trope is usually linkbait.

<comment><AUTHOR> WaPo has been running Instant Articles for a while, as have a few dozen partners. What's changed is that with general availability, media distribution is going to change profoundly in the coming months.</comment></comment>