Why multi-armed bandit algorithm is not "better" than A/B testing - <AUTHOR> http://visualwebsiteoptimizer.com/split-testing-blog/multi-armed-bandit-algorithm/ <comment><USER2> Multi-armed bandit isn't an algorithm, it's a model of how to view the problem. Like it or not, the problem web designers face fits the multi-armed bandit model pretty well. The algorithm called "MAB" in the article is one of many that have been developed for multi-armed bandit problems. Traditionally, the "MAB" of this article is known as "epsilon-greedy".

The point of multi-armed bandit situations is that there is a trade-off to be made between gaining new knowledge and exploiting existing knowledge. This comes up in your charts - the "MAB"s always have better conversion rates, because they balance between the two modes. The "A/B testing" always gain more information quickly because they ignore exploitation and only focus on exploration.

I should say also that multi-armed bandit algorithms also aren't supposed to be run as a temporary "campaign" - they are "set it and forget it". In epsilon-greedy, you never stop exploring, even after the campaign is over. In this way, you don't need to achieve "statistical significance" because you're never taking the risk of choosing one path for all time. In traditional A/B testing, there's always the risk of picking the wrong choice.

You aren't comparing A/B testing to a multi-armed bandit algorithm because _both_ are multi-armed bandit algorithms. You're in a bandit situation either way. The strategy you were already using for your A/B tests is a different common bandit strategy called "epsilon-first" by wikipedia, and there is a bit of literature on how it compares to epsilon-greedy.

[http://en.wikipedia.org/wiki/Multi- armed_bandit#Common_bandi...](http://en.wikipedia.org/wiki/Multi- armed_bandit#Common_bandit_strategies)

<comment><USER3> This comment just sold me on MAB. You can just keep on throwing variations on a design at the system without having to make tenuous decisions. I hope all the A/B tools implement this soon.

<comment><USER4> Don't wait: <http://mynaweb.com/> [Yes, this is a shameless plug for my startup.]

<comment><USER5> It seems like most people who use these content optimization tools don't really understand the statistics involved. What are your thought on this? How do you educate your users on the merit of your approach vs a/b testing when the topic is so complex?

Also, despite this being a slightly pro a/b testing post, I have to say it's actually made me more interested in trying out Myna's approach MAB algorithm.

<comment><USER6> Same way every product from GWO and T&T on down: show a pretty graph that ignores the underlying assumption that it's even possible to use statistics to conjure certainty from uncertainty, and trust that users will never know or care about the difference.

/former AB test software dev who fought my users to try to stop them from misinterpretation results, and failed.

<comment><USER7> If it gives you comfort, if there is a significant underlying difference and the calculations are done right, with high probability they will get the right answer even though they are misunderstanding the statistics.

Acceptance of this fact has avoided a lot of potential ulcers for me.

</comment></comment></comment></comment></comment></comment><comment><USER8> I'd love to see someone who knows more explore this, but: Statistical tests and processes can not be blindly applied. They all have prerequisites before they mean anything. A common one is the assumption that the underlying distribution is Gaussian. In that case, average + standard deviation tells you something very useful. On the other hand, nothing stops you from taking the average and standard deviation of a Poisson process, but the numbers will be much less meaningful, and often quite deceptive.

I'm concerned that the assumptions that are necessary for those statistical tests are not being met. Generally the significance tests are based on an assumption that all samples are independent as far as I know, but MAB grossly violates that assumption, and I do mean grossly. MAB doesn't pass those tests because it is, itself, a statistical significance test in some sense, and it is "deliberately" not feeding the independence-base significance algorithms the data it thinks it should be getting. This is not a bug, it's a feature.

(Pardon the anthropomorphizing there. It's still sometimes the best way to say something quickly in English.)

In fact, the fact that you agree that MAB has a higher conversion rate, which in this context basically means nothing more and nothing less than _works better_ , but that there's this measure Q on which it does worse, is probably better interpreted as evidence that Q is not a useful measurement, rather than that the thing that _works better_ shouldn't be used due to its lack of Q-ness.

<comment><USER9> In this case, because you are measuring conversion vs. no-conversion, the underlying distribution would actually be binomial. With a large enough sample size, it looks similar to a Gaussian distribution, but still slightly different.

Also, I don't see the problem of lack of independence. People are still randomly assigned to one of the conditions in the MAB scheme, people in one sample don't affect people in the other sample, and each person submits one independent datapoint. The problem is just one of unequal sample sizes in your two conditions, so when you make the comparison (in a one factor ANOVA, say), you lose lots of statistical power because your effective sample size is essentially the harmonic mean of the two sample sizes (which is heavily biased towards the lower number of the two).

<comment><USER8> "People are still randomly assigned to one of the conditions in the MAB scheme"

No, they're only _partially_ randomly assigned to one of the conditions. MAB has a memory based on what the results of the previous runs are, which is basically another way of phrasing that there's a dependence.

<comment><USER9> I'm not so sure. By that logic, no experiment has random assignment. I know when I place subject X in condition Y that I do so because I need Z people in condition Y. Thus, I have some information about where I've placed other people. This is no different?

<comment><USER10> The point is that with bandit models, you're placing people in a cohort as a function of the previous conversion rate of your cohorts. That's the dependence.

By contrast, randomly allocating X% of your population to cohort C_i is not a dependence on anything but the free parameter (X).

<comment><USER11> The results will still be conditionally independent given the assignments, which is all you need to use the counts for estimation of a binomial .

<comment><USER10> _"The results will still be conditionally independent given the assignments"_

If the assignments are the source of the problem, you can't just take them as "given" and assume that things are OK.

A simpler example: if I assign people to cohorts based on their age, then the results of my experiment may be "conditionally independent" with respect to (say) eye color, but it probably won't be conditionally independent with respect to income or reading level or pant size. In other words, _the assumption of independence is violated with respect to all variables correlated with age._

With bandit optimization, our bucket allocation scheme is correlated with time, which makes it nearly impossible to apply conventional statistical tests on any metric that may also be correlated with time. And in web testing, that's nearly everything.

</comment></comment></comment></comment></comment></comment></comment><comment><USER4> When I see this "How soon statistical significance (of p-value < 0.05) was detected (in a two tailed Z-test)", I think of <http://www.evanmiller.org/how- not-to-run-an-ab-test.html>

tldr: The methodology is flawed.

<comment><AUTHOR> It's not flawed. Wouldn't you assume that whatever bias (due to "peeking" at results) was there for one algorithm would also be there for another? Also, to offset this effect somewhat, I waited till I saw statistical significance at least 10 times. (I know it's a heuristic) Moreover, I'm curious if there's a theoretical result that compares statistical power of randomization and MAB.

Though like a commenter said, comparing these two algorithms is actually like comparing apples to oranges, and that was precisely the conclusion.

<comment><USER12> All you are saying is here is that you got it wrong in the same way for all parts of the test.

Which may remove the bias (I'm not sure about this) but it doesn't inspire confidence.

<comment><AUTHOR> Yes, I agree it is not a rigorous statistical analysis but I tried doing a fair comparison (based on assumptions/heuristics clearly described in the post). The blog post simply claims what is described there.

</comment></comment></comment></comment><comment><USER13> One glaring flaw in this article is that they're comparing a multi-arm bandit with a 10% exploration rate to a simple random split test with an implicit 50% exploration rate. Of course the split test will converge faster.

In their results when they compare a MAB with a 50% exploration rate to their split test you start to see a comparable amount of time to converge. Also they only show the results of one simulation with a lot of random in it. Given we're all stats nerds it would've been handy to see a box plot for each of the styles of simulation across multiple runs.

Having said all of that, my biggest concern around MAB as it's being sold is the lack of thought around the experiments. In the end it's just data and it doesn't mean anything without human intuition and preconceptions guiding it. Example: day of week, time of year/day, new page people are getting used to still, old one that garners lots of repeat traffic, etc. When running tests there's a lot more to consider besides just "whatever the data says".

<comment><AUTHOR> We ran each simulation 25 times (with 10000 iterations each). And can share exact statistics in case you are interested. And yes, MAB with 50% exploration rate did perform similarly, but it didn't generalize when there were more than two versions.

</comment></comment><comment><USER14> Experiment and Optimization are different topics and scoring one by the criteria of the other will lead to mistaken conclusions.

Your simulation is a fiction. It only informs you of the behavior of the code you wrote. To connect statistical significance to reality requires honoring criteria you have not met.

Applying frequentest evaluation concepts to Bayesian and Markov learning methods will lead to mistaken conclusions because they follow from a different set of axioms. They both lead to numbers, but these numbers are not interchangeable and you must remember their qualifications when applying them to predicting reality.

In more frequentest terms, multi-arm bandit is searching for minimum total regret via a random walk, not rapid convergence to a significance predicate. It can do this in the face of changing conditions and weaker assumptions concerning control than frequentest significance requires, which is why such methods are now the norm in machine learning.

There are huge opportunities that you will not see with your current perspective. If you do not learn them, your competitors will.

I do not know of a particular reference for multi-arm bandit algorithms in specific, but they are a specialized case of Markov model learning described with vocabulary that predates a more modern broad and uniform understanding of these topics.

David Barber's recent book is very good.

If you want a broad understanding of common mistakes due to unstated metaphysical assumptions concerning statistics, experiment and action, read Judea Pearl's 2nd edition.

</comment><comment><USER15> Convergence to statistical signifcance is a bad metric (just like how early stopping is such an issue) but you can map it to the rigorous parameter of statistical power, perhaps. From this, it's well proven that t-tests are statistically most powerful--there exists nothing better--given that you're simulating disjoint normals.

In reality, however, tests like the Wilcoxin test are 99% as powerful and more robust to misspecification in models.

I bring this up because while statistical power and signficance is a very important metric in the theory of picking good tests, it's actually a pretty terrible one in practice. Comparing MAB, which optimizes an entirely different loss parameter, to t/z-tests on power is sort of meaningless.

MAB can produce a cleaner workflow for many kinds of websites. Underperforming classes will be underrepresented and eventually pruned. The increased power of a batch test isn't necessarily so important in this context. I'm not even actually advocating MAB over other tests, just that you shouldn't spend too much time worry about power comparisons unless you're genuinely comparing apples to apples.

<comment><AUTHOR> Yes, comparing MAB with A/B test is not accurate and that is what the conclusion (in summary) is. These algorithms optimize for different things, and our customers should know that.

</comment></comment><comment><USER16> This blog post misses to me what is the most important difference: If you are not keeping consistent assignment ratios of users into experiment buckets, then you are implicitly making the assumption that no other factors other than your A/B test can affect conversion.

A very common example would be you might have a form that converts higher on the weekends than the weekdays just because of some external factor. You fire up a test with this "multi-arm bandit method" on Friday morning with two variations that are exactly equal. By random chance variation A starts winning by the end of Friday, and then it gets shifted to 90% of impressions. Due to this external factor, that it is now the weekend and that naturally increases conversions, variation A's conversion rate is going to excel past that of variation B, maybe even significantly. However, it in fact isn't better at all.

Now this may be uncommon, but think about this: Your website is changing all the time. If you happen to make a change on your website that affects conversion rates and you do not keep consistent bucket assignment ratios, then any ongoing experiment is going to be tainted.

I appreciate that the "multi-armed bandit" method increases the conversion rates on your site during the test, but to do be honest when I am running a test I'm not worried about increasing the conversion rate during the test. I'm worried about increasing the conversion rates in the much longer future, and the best way to accurately do that is to get accurate and significant results from my test as quickly as possible.

This also bunks one of the advantages in the blog post that says you can add variations at any time. You _cannot_ add variations at anytime, because if you do you are comparing apples to oranges. The conversion rates from time period A-->C cannot be assumed to be the same as B-->C, because there likely have been changes in between A and B that could affect the "base" conversion rates (like website changes, external factors like weekend vs weekday, etc). When you add a new variation, you are creating a new A/B test that must be analyzed independently.

<comment><USER7> That was exactly my point at <http://news.ycombinator.com/item?id=4040022> and it is a valid one.

However I have been thinking about it since, and it is possible to design a multi-armed bandit approach with logarithmic regret (though higher by a constant factor than a traditional approach), that can handle the varying performance ratio. It also would allow you to add variations at any time.

There remain operational differences, but this problem is fixable.

</comment></comment><comment><USER17> Helpfully, the final paragraph of the article seems to contain its takeaway message:

"So, comparing A/B testing and multi-armed bandit algorithms head to head is wrong because they are clearly meant for different purposes. A/B testing is meant for strict experiments where focus is on statistical significance, whereas multi-armed bandit algorithms are meant for continuous optimization where focus is on maintaining higher average conversion rate."

Like any business owner, at the end of the day I care more about conversion rates than about statistical significance.

I'm not a scientist trying to advance the sum of humanity's knowledge. I'm a business owner trying to find the shortest path between what my customers need and what I can profitably offer them.

In a way, statistical significance strikes me as a bit of a fool's errand, because significant results in one context may not be generalizable to another, which means even if we know for a near certainty what worked best, it's hard to apply that knowledge reliably in the future.

Of course, with MAB we could still wait for statistical significance if we want it, before turning off variations that are performing worse. And we can certainly still try to draw conceptually useful conclusions by designing our test variations in ways that facilitate easy comparison.

But with MAB and Myna it sounds like we can pretty well count on higher conversion rates at the end of the day, as well, and that counts for a lot in a business context.

I'm grateful to the VWO team for writing up their analysis and findings, and being so frank about the relative advantages of A/B and MAB. Their summary above tells me what I need to know.

</comment><comment><USER18> Without concrete details about the experiment setup and dataset on which it was run (and ideally the source code,so we can be sure there aren't any bugs) this is essentially meaningless.

Anyone can run "simulations" to prove anything. Providing just a summary table is of little use. I am not saying that the Wingify folks are trying to mislead people - just that this article doesn't have sufficient rigor to justify its conclusions.

OTOH, many CS papers, even published ones, don't provide source code or datasets so people can replicate the results, so perhaps this is the 'new normal' ;) .

<comment><AUTHOR> Here's the code (quick-and-dirty): <http://pastie.org/4007859>

I had double-checked the code, but it is quite possible that I made an oversight somewhere.

<comment><USER18> I am not saying that your code has bugs (or that it hasn't!).

However running experiments to prove that one 'process' is superior to another in a real world situation often involves more than running a chi square test on randomly generated data ;) (which is what I understood your code to be doing on a _very_ brief glance at it- sorry if I got it wrong).

There is nothing wrong with _starting_ your exploration with something like this, but it really isn't sufficient to make the claims in the blog post about specific ways in MAB is better/worse than A/B testing.

This _seems_ to be methodologically dubious. I am _not_ a stats expert, though I use it in my work, and I could be wrong but testing _convergence to_ statistical significance doesn't seem to mean anything (mathematically/statistically)!.

I could be wrong - there are people with Stats PhD's here and I'll leave it to them to tell me if I am. But I've never heard of an experiment (in the formal sense) which ran _till_ a significance checking algorithm crossed some tripwire level.

Is the sampling random? (especially for an MAB) What are the underlying distributions (and assumptions about them)?

In your article you (imo) either need to say something like "this test isn't really valid and so the conclusions shouldn't be taken very seriously" (which would be weird thing for a company with your product to blog!) or you should do some rigorous experiment design and be able to defend your results.

Right now it _reads like_ some MBA type plugged in two black box formulae in an Excel sheet and drew conclusions from the results. (please note: I am _not_ saying you did it that way -just saying that the article _reads like_ not enough thought went into setting up a proper experiment)

(Statistical) Experiment Design is a lot of work and surprisingly hard to carry off well, and involves much subtlety - and theory! there are whole books written about it. Maybe time to buy a few? :)

<comment><AUTHOR> Yes, I agree we should put the code on the post probably so people know what exactly was done. Yes, it is a quick and dirty simulation, and not a rigorous analysis. I hope someone more knowledgeable does shed light if the arguments have any fundamental oversights. The post is mainly meant for our customers who asked if MAB is "better" than A/B testing. It clearly isn't as both are meant for answering different types of questions.

</comment></comment></comment></comment><comment><USER19> I personally do not understand the need to compare bandit with A/B.

The goal of A/B is to decide as quickly as possible for the best one.

The goal of bandit is to optimize given content.

I wouldn't use bandit to "decide" the button color but as a simple recommendation system. This seems by far more natural to me as it reacts better with changing optima.

Example: Let's say I run a larger "fun content" media webpage. I have "awesome videos", "funny images" and "goofy articles". On the bottom and right side of each content page i show follow up content. I would use bandit here to optimize the mix of images,videos,text i recommend.

To spice it up: I would create cohorts for my typical behaviour of users (e.g. registered male user) and only consider interactions of the last two weeks into my bandit calculations.

</comment><comment><USER20> "A/B testing is meant for strict experiments where focus is on statistical significance, whereas multi-armed bandit algorithms are meant for continuous optimization where focus is on maintaining higher average conversion rate."

Who cares about conversion rates, right? I'd much rather have statistical significance than more clicks on my buy button / signups for my site / etc.

<comment><USER6> The point is that with less significance you are less confidential what your conversion rate "would have been" under other courses of action. You can't say "this or that test has higher conversion" because of some simulation that has totally different structure than your real website.

Statistical about managing unknowns, not creating knowns, and everyone ignores this!

</comment></comment><comment><USER7> (For previous discussion on this topic, see <http://news.ycombinator.com/item?id=4040022.>)

Let's stop a bad meme before it gets going.

Multi-armed bandit is a general problem. There are many strategies for tackling it. An epsilon-greedy strategy is one strategy. It is not the only strategy, and it is not even a particularly good one. There are many others. In fact A/B testing is itself a solution to the multi-armed bandit approach.

In fact the standard way to classify strategies is their asymptotic average _regret_ \- how much time you spend pulling the bad lever. An epsilon-greedy strategy has linear regret - even after a version has won you keep pulling it a certain fraction of the time.

Let's consider an alternate approach. Run an A/B test, then forever after exploit it. This is _also_ a linear regret algorithm. Given two levers and the way you're going to run the test, there is a probability that you will come to the wrong conclusion and forever after do the wrong thing. Most of the time your regret is fixed, but sometimes it is not.

So they are equal? Not exactly. A/B testing converges quickly, and has a very small amount of linear regret forever after. An epsilon-greedy approach converges more slowly. Furthermore to lower the long-term regret you have to drop epsilon, which slows convergence. By the time you lower it to a level that is comparable to an A/B testing approach, convergence is so slow as to be operationally infeasible.

Thus the conclusion of this article is that A/B testing is a better operational strategy than epsilon-regret. And it is. But there are known better strategies still. The algorithms that Myna uses, for instance, have logarithmic regret. In a multi-armed bandit situation, they will soundly beat A/B testing just as A/B testing soundly beats epsilon-regret.

At that point you're left with two questions. The first is whether multi-armed bandit is a good model for website optimization. The second is whether there are any operational advantages to one approach over the other.

The answer to the first, for traditional multi-armed bandit algorithms, is a sound no. While relative conversion rates remain comparable, absolute conversion rates vary for all sorts of reasons. (Time of day, day of week, longer cycles, external website improvements, etc.) That was my big criticism the last time this discussion popped up, and there was no good answer to it.

 _But_ behind the scenes Noel (the CEO of Myna) and I have been in a discussion. I believe that I can produce a multi-armed bandit algorithm that has logarithmic regret and will work in the face of ever varying conversion rates, as long as there is a consistent relative difference between the versions.

At that point you are left with the operational differences. If you want to test at a granularity of "user", a website often has to put most of its active population into the test, before collecting enough data to know which version was good. With A/B testing you will eventually put everyone into the better version. With a multi-armed bandit, half your current users are stuck in the wrong version.

In effect you have a limited population of levers available to pull, and have to pull a good chunk of them before you start getting feedback on the rewards.

You can avoid this problem by testing at the granularity of an impression. Whether or not this is OK to do is a business question, and the answer will not always be yes. If it is not, then a multi-armed bandit approach is inappropriate to use.

<comment><USER21> > I believe that I can produce a multi-armed bandit algorithm that has > logarithmic regret and will work in the face of ever varying conversion > rates, as long as there is a consistent relative difference between the > versions.

That sounds interesting. As you undoubtedly know, there is a lot of literature on multi armed bandit problems and also on the non-stochastic multi armed bandit problem. The latter has the advantage of not assuming anything about the way in which the sequence of rewards is generated.

There is a a line of work in the non stochastic setting which sounds related to what you are describing. It's called the problem of tracking the best expert: <http://www.cse.ucsc.edu/~mark/papers/track-long.ps> The idea in this problem is to do well as compared not to the best fixed action but rather the best piecewise constant sequence of actions. There are many variations of this problem, but the basic idea is that you can have low regret as compared to a changing sequence of actions so long as that sequence doesn't change too "quickly" for some definition of quickly.

<comment><USER7> I know there is a lot of literature, which I'm unfortunately less familiar with than I'd like to be.

Here is the specific problem that I am interested in.

Suppose we have a multi-armed bandit where the probability of reward are different on every pull. The distribution of rewards are not known. It is known that there is one arm which, on every pull, has a minimum percentage chance by which it is more likely to produce a reward if pulled than any other arm. Which arm this is is unknown, as is the minimum percentage.

In other words payoff percentages change quickly, but the winner is always a winner, and your job is to find it. I have an algorithm that succeeds with only logarithmic regret. I have no idea whether anyone else has come up with the same idea, and there is so much literature (much of which I presume is locked up behind paywalls) that I cannot easily verify whether anyone else has looked at this variant.

<comment><USER21> I think what you're describing is basically the non-stochastic setting. The idea there is to assume absolutely nothing about the rewards (they can be generated by quickly changing distributions or even an adversary) but still try to do as well as the best single arm. The standard algorithm for this problem is the multiplicative weights update method, also called the exponentially weighted average method or the hedge algorithm. It's a really simple method where you keep a probability distribution over the arms and then reweight the probabilities at each time step using something proportional to the exponential of the rewards of the arms. If you run it for T time steps on a problem involving N arms, it gives you regret O(sqrt(T log N)). This is actually the best possible bound without additional assumptions. This is a good paper on it: <http://cns.bu.edu/~gsc/CN710/FreundSc95.pdf>

There are many, many different variations of the method and result. For example, you can show improved results assuming different things about the rewards. For exapmle, you can show O(log(T)) regret with certain additional assumptions. This is a really good book on different non-stochastic online learning problems: [http://www.ii.uni.wroc.pl/~lukstafi/pmwiki/uploads/AGT/Predi...](http://www.ii.uni.wroc.pl/~lukstafi/pmwiki/uploads/AGT/Prediction_Learning_and_Games.pdf)

<comment><USER7> The article is interesting, but assumed you get to pull all of the arms at once, and see all of the results. That's different from the website optimization problem.

The book looks like it would take a long time to work through, and they don't get to the multi-armed bandit problem until chapter 6. It goes into my todo list, but is likely to take a while to get off of it...

<comment><USER21> Yes, I realized after posting that this is probably a better paper to link to: <http://cseweb.ucsd.edu/~yfreund/papers/bandits.pdf> The algorithms for the partial information setting are sometimes surprisingly similar to the algorithms where you see all the results. The algorithm in the paper linked above is essentially the same algorithm but with a small exploration probability. The regret bound gets worse by a factor of sqrt(N), however.

</comment></comment></comment></comment></comment></comment><comment><USER22> From a statistics perspective, a few things are missing. Most importantly, a discussion of statistical power and a discussion of why exactly it is that they test until statistical difference is found. Every scientist knows that if you test a big enough sample, you are more than likely going to find a statistical difference, regardless of whether it actually exists. Hence, using only that as your heuristic for what makes one algorithm better than another is not very useful.

<comment><AUTHOR> Agreed that statistical power wasn't used to calculate how long test should have been run. I know statistical significance can be found earlier if you are "looking" for it, that is why I ran the simulations until it was found at least 10 times. (I know it is not the most scientific way, but I used it as a heuristic; I don't know how to use statistical power in case of MAB. Probably 'statistical power' concept is not valid at all for MAB. Need to study more.)

</comment></comment><comment><USER23> This blog post is describing epsilon-greedy exploration/exploitation as applied to A/B testing. It's not really multi-armed at all, since the multi would imply > 2\. The real beauty of exploration/exploitation is the ability to go beyond two simultaneous tests. Although they do show a test with three options, this would be far better if it emphasized the multi more.

</comment><comment><USER24> I would like to have seen more tests vs the MAB-90 algorithm as this appeared the most interesting one to me. Also the article appears to imply that A/B testing is useful in the situation where you want a cleaner statistically significant result which I believe is actually a false argument.

<comment><AUTHOR> > A/B testing is useful in the situation where you want a cleaner > statistically significant result which I believe is actually a false > argument.

Why do you say so? Can you elaborate?

<comment><USER24> Both methods can supply statistically significant results, there is no reason that A/B testing is fundamentally superior in getting a statistically significant result, although arguably A/B testing is more straight forward.

<comment><AUTHOR> True, both methods can provide statistical significance. My only point is that in general A/B testing may require less number of visitors as compared to A/B for same amount of confidence.

</comment></comment></comment></comment><comment><USER25> Talk about being too smart to solve the problem... The article is comparing apples to concrete blocks. When you're hungry, you want apples. When you're building a retaining wall, you want concrete blocks. One can't substitute for the other.

Statisticians, academics, math enthusiasts, and People Who Spend Too Much Time Building Tables and Graphs want concrete blocks. They want to build a case or a report or something that suggests a point using a mathematical model. They deal with "statistical significance" because that's how the tools they understand work. Thus A/B testing.

Web workers, businesses, advertisers, people looking for better conversion rates, etc. want to eat. They want apples. Talking about statistical significance, trial sizes, and frequency measures don't help if they don't get more apples. The article showed that MAB gets more apples than A/B and by a wide margin.

Comparing the two isn't just missing the point, it's drawing an equivalency between "statistical significance" (whatever the tools say it means) and actual _real_ _world_ significance, a.k.a. better results. If A/B testing fails to even simulate better results, the proper word for that is _worse_.

(And don't even get me started about trying to run a statistical experiment until significant results are found... someday that practice will be labeled as fraud.)

</comment><comment><USER12> My take on bandit algorithms vs A/B testing:

If you have a question where the answer can be used to inform other decisions (e.g. Which is more important for our customers, price or support level?) then an A/B test will give you a clear answer that you can apply elsewhere.

If you don't really care what the answer is and just want the best option (e.g. should the hero shot be looking up or down) then a bandit is the way to go.

</comment><comment><USER5> Reading the conclusions it seems that each approach - a/b testing vs multi- armed bandit - has its merits. Maybe Visual Website Optimizer should be method agnostic and offer customers the choice of method?

<comment><AUTHOR> Yes, we have been discussing this internally. Providing both options is great, but that assumes customers have a complete understanding of when to use which method.

</comment></comment><comment><USER26> 1\. Could you gist your Python code?

2\. You do not indicate whether you've run enough simulations to attain a high statistical significance (over the meta- result that each campaign statistical significance has been attained)

3\. Correct me if I'm wrong, but I don't think A/B testing should be run "until statistical significance has been attained" because it foils the purpose of a statistical test, that is, you should rather define before hand a number of iterations, wait until they have been completed, and then check whether it gives you statistical significance

4\. The purpose of the epsilon-greedy algorithm is that it can be run for very long duration without knowing which statistical significance is best because it will adapt the most commonly seen version of your site to this particular version (ie it is made to be run almost indefinitely, and will -- in effect -- automatically remove the non-working versions after a while!

</comment><comment><USER27> A/B testing is linear regression against a single binary variable. There are many better techniques. Multi-armed bandit is just one of them. A more direct comparison would be simply using a continuous independent variable, or at least one with more than two values. You can have more variables pretty easily, too.

Of course you get statistical significance with fewer observations if you have fewer variables to model. Although you'd get even significance with even fewer observations if you had a continuous independent variable. But more variables produces a higher r-squared. As in, you can explain more of the variation in your conversion rate.

The prevalence of A/B testing demonstrates a sad lack of statistics training. It's a problem in many science fields as well, especially biology.

</comment><comment><USER28> Was this analysis based on testing only two variations, A vs. B, using A/B testing then MAB?

If so then that ignores one of the best advantages of MAB is you can compare variations A, B, C, D, E, F, ... all simultaneously, where if using A/B testing you'd have to coordinate a tournament between all of those variations to determine the best of the bunch ... so seems it would take just as long with A/B testing (as thus a large sample population) to arrive at knowing the best variation from many choices.

</comment><comment><USER29> I'm confused: surely the right way to compare would be to say the A/B tests pick the best option 100% of the time after 'reaching statistical significance'[1] - this is at least apples to apples.

[1] Like others here, I believe testing until you see significance is Doing It Wrong.

</comment><comment><USER30> I found it very interesting that MAB-90 was outperforming RAND on both convergence speed and average conversion rate in the first experiment. This seems a very interesting result, albeit not the one you might be looking for in the article. I find it rather disturbing that this dataset was removed in later experiments. It actually seems to provide a very valuable insight namely: Multi-Armed Bandit strategies might be better than A/B testing but the exploration factor should be far higher because of the need for rapid convergence to significance.

</comment><comment><USER31> It depends on how much traffic you have. If you have enough for 10% to quickly provide statistically significant amounts of data, then there is no need to provide more traffic to that variation to determine whether it is worthwhile or not - you can split 90% to the already well performing version.

</comment><comment><USER32> This argument is wrong on so many levels.

</comment><comment><USER33> My imaginary ideal would be a genetic algorithm that mutates the HTML and CSS in random ways. You'll probably have users with completely unusable pages but that's the price you pay for finding the optimal solution as it converges over time, probably a really, really long time.

</comment><comment> loup-vaillant I cannot help but think that under a sufficiently precise set of assumptions (prior knowledge about the various one-armed-bandits, cost of losses, gain of winnings, expectations of being still in business in the future, etc), there is a provably optimal strategy.

Isn't there?

</comment><comment><USER34> From a EE's point of view, what the OP said is that there's too much noise involved in multi-armed bandit algorithm. If that's the case, a low pass loop filter is needed. It's kind of like locking a PLL to a reference clock right?

</comment><comment><USER35> Maybe a better way to do the exploitation phase would be to sample from the different choices with probabilities proportional to their expected payoff instead of just choosing the one with maximum expected payoff.

</comment><comment><USER36> ( _From reading the post quickly, I'm not left with the impression that the author understands the research they are 'refuting'. (Which I also only read quickly and cannot help understand.) This is just my impression_.)

<comment><AUTHOR> Of course, I don't claim to understand intricacies of MAB _completely_. This is on my understanding of this algorithm in last couple of days, but I'd like to be illuminated by someone more knowledgable if there's anything wrong with my understanding. In fact, that would be quite helpful for our customers as well.</comment></comment>


No full-time offer handed after internship; do I have a shot? - <AUTHOR>

This past summer, I had an internship at a medium-sized startup (about 5 full-time engineers when I was there). It was a great place, and I'd love to work there full-time when I graduate this spring. However, at the end of my internship I was not offered a position. It was the company's first year of having  interns, so I can't tell if this was because they chose to not extend an offer, if they had forgotten about it, or if they assumed I knew I would have one if I asked.
I don't think I performed astonishingly, but I was assigned relatively mundane tasks and performed them well, and I thought I fit in and was on par with the full-time employees (given a year or two of experience). I know that taking initiative to do impressive things is important, but the mundane tasks I was given were nonetheless important and time-consuming (it would have been irresponsible to shirk them to do less important, flashier things). I fear that the other interns (there were 2), who performed about as well but were given sexier tasks came off as much better full-time candidates.
I think my options are: (1) ask the other interns if they received offers (though only 1 is graduating this year), and assume that I was intentionally not given one if they did; (2) ask the company if I have an offer. Both are awkward situations, but going directly to the company seems like the right thing to do.
I'd also like to know if this is a situation that could actually happen accidentally. Basically, am I in denial for thinking that a company would actually forget to extend an offer to a candidate they want? Or is it completely paranoid of me to think any of this, given that I seemed to fit in well? If it is extremely unlikely that my employers could have forgotten and it should be obvious that I don't have an offer, is it rude of me to put them in an awkward spot by having to formally reject me?
Because I assumed this position wasn't guaranteed, I have been applying elsewhere and have offers at other companies now. But this company was kind of my ideal job, so I'd really like to return. What should I do? <comment><USER2> If you REALLY want to be considered full time you should talk to your manager and let him know that's what you want. There are many reasons why they might not have extended you an offer, but at the very least you should speak to him find out why.

</comment><comment><USER3> Have you considered saying this to your manager? I would avoid talking jobs/pay with coworkers, and tell your boss how you feel the internship went for you, how its the ideal place to work, and why, and don't forget to flatter the boss on his/her management style etc. Good luck.</comment>


Twitter clients stay signed in with pre-breach passwords - <AUTHOR> http://www.theregister.co.uk/2013/02/04/twitter_oauth_apps_logged_in_with_old_passwords/ <comment><USER2> They aren't signed in with "pre-breach passwords". They are signed in with OAuth tokens.

Not tying the authentication of third-party apps to passwords but instead to OAuth tokens is sensible precisely because of security failures like this.</comment>


How to create an OS from scratch – tutorial - <AUTHOR> https://github.com/cfenollosa/os-tutorial <comment><USER2> I've seen a number of "build an OS" tutorials pop up recently, and that's awesome! Building an OS will definitely help you understand how to write better user-space programs, and it's fun.

One thing I wish is that more of them would feature UEFI instead of BIOS, or at least come with a warning, as Intel plans to drop support for legacy BIOS in 2020.

Going through UEFI is actually easier anyway, IMHO. ... Maybe I should stop complaining and write that tutorial myself instead. Hmm.

<comment><USER3> If you are writing an OS, I highly recommend the Pure64 bootloader from Return Infinity. It's much more straightforward to use compared to GRUB which imposes a lot of conventions on how you structure things.

httpss://github.com/ReturnInfinity/Pure64

<comment><USER4> Working link: [https://github.com/ReturnInfinity/Pure64](https://github.com/ReturnInfinity/Pure64)

</comment></comment></comment><comment><USER5> "Once you pass college, excessive theory is worse than no theory because it makes things seem more difficult than they really are."

About every third minute I've changed my mind about this quote. I think that means it's a good quote.

<comment><USER6> As someone from a computer science background who's recently been getting into electronics in a really big way, this quote really struck a chord with me.

</comment></comment><comment><USER7> Writing an OS is a fun exercise and it makes you appreciate all the effort that goes into making something like Linux usable and relatively robust and performant. It's also interesting to look at how Linux (or other open source OS) implements some of these features once you've tried implementing them on your own. You'll find that much of Linux is understandable once you know what problem the code in question is trying to solve.

</comment><comment><USER8> Hi, author here. I'm humbled by the kind words. This is a project I started when I was in between jobs and unfortunately it's kind of stalled because I had difficulties implementing the file system. Really want to get back to it some time.

Anyway, feel free to leave your comments, I appreciate them a lot.

<comment><USER9> Just shot you an email to ask you to do that! Thanks for this resource. Good work :)

<comment><USER8> Glad you liked it! Thanks for the kind words

</comment></comment></comment><comment><USER10> These kinds of posts are the reason I come to HN hoping to find. Amazing work! I'm going to come back and read some chapters over the weekend.

</comment><comment><USER11> I'm currently traversing through Linux From Scratch, [http://www.linuxfromscratch.org/lfs/](http://www.linuxfromscratch.org/lfs/). Not necessarily because I wanted to build a Linux OS but because it was the best guide I could find. But at a first glance this looks promising to me, so I'd definitely check it out and try out a few "chapters" when time comes.

</comment><comment><USER12> I legit thought they wrote an OS in the scratch programming language for a second lol

</comment><comment><USER13> Can anyone recommend good resources on writing a compiler from scratch ?

<comment><USER14> I think the best first book is: James E. Hendrix, The Small-C Handbook, Reston 1984, ISBN 0-8359-7012-4

He explains every detail, goes through every routine, and you can see exactly what the compiler does. It's 'Small-C' so it doesn't have any fancy features, like floats or structs. But the basic machinery is there, and one can add features as desired. It does compile itself. The compiler outputs x86 assembly, which must be assembled and linked with standard programs.

The original Small-C was done by Ron Cain, and produced 8080 code. I like Mr. Hendrix' better because I have lots of x86 around (I specifically wanted it for my HP-200 LX palmtop, which I still have and works great!); and because the book is fantastic.

It's fully understandable in a small quantity of time. And, the back end is re-targetable, so you could, say, output to ARM or RISC-V or whatever.

I spoke with Mr. Hendrix some years ago, maybe 10 or so, and asked if he would consider putting the compiler into the public domain. He did.

<comment><USER15> Thanks for this suggestion.

In case anyone else is interested, there is a freely available ISO of the the book available from Dr Dobbs Journal. From the README in the ISO:

>"Welcome to Dr. Dobb's Small-C Resource CD-ROM, the definitive collection of Small-C related information and source code. This CD-ROM includes the full text to James Hendrix's book "A Small-C Compiler: Language, Usage, Theory, and Design," selected articles on Small-C from "Dr. Dobb's Journal" magazine, and Small-C implementations for a number of processor platforms, including the 8080, Z80, 6502, and others."

It's available here:

[http://www.drdobbs.com/developer-network-small-c-compiler- bo...](http://www.drdobbs.com/developer-network-small-c-compiler- book/184415519)

</comment></comment></comment><comment><USER16> I'll be the guy to mention Forth as it is the only system I know of that a single person can fully understand and build up a fully functional and REPL interactive system using just assembly and a microcontroller. Most Forthers are doing embedded work though and don't need everything that comes with an OS.

</comment><comment><USER17> Does any one have other resources I can look into as well. I'm still a relatively new programmer but I really want to get a deeper understanding of how OSes work. I'm assuming I need to use either C/C++ or Rust? I only Python and am currently learning C++.

<comment><USER18> Stanford CS140e - An Experimental Course on Operating Systems [1]: _students implement a simple, clean operating system (virtual memory, processes, file system) on a Raspberry Pi 3 in the Rust programming language and use the result to run a variety of devices_

MIT 6.828 - Operating System Engineering [2]: _This course studies fundamental design and implementation ideas in the engineering of operating systems. Lectures are based on a study of UNIX and research papers. Topics include virtual memory, threads, context switches, kernels, interrupts, system calls, interprocess communication, coordination, and the interaction between software and hardware. Individual laboratory assignments involve implementation of a small operating system in C, with some x86 assembly_

[1] [https://web.stanford.edu/class/cs140e/about/](https://web.stanford.edu/class/cs140e/about/)

[2] [https://pdos.csail.mit.edu/6.828/2018/schedule.html](https://pdos.csail.mit.edu/6.828/2018/schedule.html)

<comment><USER19> Thanks for these links. Very useful indeed. I've been looking for something to get me into C and Rust, so I can learn the lower level stuff, and get me away from higher level web technologies (which I play with all day, everyday.)

<comment><USER18> You are welcome.

Please note that the MIT 6.828 is really tough but will turn you into a bad- ass C hacker.

It relies on MIT 6.004 - Computation Structures [1] (has edx mooc [2,3,4]) and MIT 6.033 - Computer Systems Engineering [5] as pre-requisites given that they teach you the fundamentals of computer architecture (digital abstraction, gates, circuits, processors, assembler, memory, cashes, pipelining, virtual memory, interrupts/io, buses, what have you) and system design (basically, managing complexity); they are not programming courses, but I recommend them if you have the time and interest, they are an awesome and principled way of learning the core concepts that you really need to have down before dabbling with OSs.

[1] [https://ocw.mit.edu/courses/electrical-engineering-and- compu...](https://ocw.mit.edu/courses/electrical-engineering-and-computer- science/6-004-computation-structures-spring-2009/)

[2] [https://courses.edx.org/courses/course-v1:MITx+6.004.1x_3+3T...](https://courses.edx.org/courses/course-v1:MITx+6.004.1x_3+3T2016/course/)

[3] [https://courses.edx.org/courses/course-v1:MITx+6.004.2x+3T20...](https://courses.edx.org/courses/course-v1:MITx+6.004.2x+3T2015/course/)

[4] [https://courses.edx.org/courses/course-v1:MITx+6.004.3x+2T20...](https://courses.edx.org/courses/course-v1:MITx+6.004.3x+2T2016/course/)

[5] [https://ocw.mit.edu/courses/electrical-engineering-and- compu...](https://ocw.mit.edu/courses/electrical-engineering-and-computer- science/6-033-computer-system-engineering-spring-2009/)

<comment><USER17> Thank you so much for these resources.

</comment></comment></comment></comment></comment><comment><USER20> I've been wanting to do this but with a Pi or any SBC board with Linux from Scratch, but not sure how I would even approach LFS on a Pi.

<comment><USER21> I'm in extremely early stages of writing up some documentation for how to build a Linux distro for a Raspberry Pi 3, gauging interest and figuring out pain points. Shoot me an email [my username]@gmail if you've got ideas or requests!

<comment><USER22> That sounds interesting. I think your guide would be extra helpful if it focused on building a distro that has as much of the system loaded into memory as possible to minimize amount of reads from the SD card, and which is configured with a read-only file system and never even tries to write to the SD card. That would be very useful. SD card corruption is a pain!

</comment></comment></comment><comment><USER9> Does anyone else have a good list of similar tutorials? I'm highly interested in this type of implementation from scratch approach.</comment>


NASA headed towards giant golden asteroid that could make everyone a billionaire (2019) - <AUTHOR> https://www.foxnews.com/science/nasa-headed-towards-giant-golden-asteroid-that-could-make-everyone-on-earth-a-billionaire.html <comment><USER2> Correct URL: [https://www.foxnews.com/science/nasa-headed-towards-giant- go...](https://www.foxnews.com/science/nasa-headed-towards-giant-golden- asteroid-that-could-make-everyone-on-earth-a-billionaire)

<comment><AUTHOR> Thanks

</comment></comment><comment><USER3> > make everyone a billionaire

In much the same way everyone can be a Zimbabwean billionaire: the monetary unit is near worthless. Gold's rarity gives it value beyond its utility.</comment>


How do I start off getting involved in open source? - <AUTHOR>

I am a mostly-self taught developer, trying to develop my skills and it seems quite clear that getting involved with open-source projects is probably the best way to go about doing that, but I am never really sure where to start.  I have accounts on the major OSS hosting systems and have checked out the code for a few projects I like, but beyond that, I feel kind of lost.  Most OSS projects appear to me to be either abandoned, barely maintained, or so complex that I'm never even sure where to start.  Does anyone here have any pointers or advice for starting off with programming OSS? <comment><USER2> Decide what you like. Find OSS project about it. Work on it.

.. and follow rules in community.. prefix your questions with "Ask NH: ". =)

<comment><AUTHOR> Ach. Yeah. Oops. I was so concerned writing the working of the question, that I neglected the most important part :P

</comment></comment><comment><USER3> fedoraproject.org would be a great place to start (i'm a fan) but there are lots of distros that are alive and launching new versions very often. ubuntu.com is another...

once you have an account, you probably wanna talk to someone about the code... there mailists where developers talk about what they are doing.

hope I've helped a bit.</comment>


On Open Source, licenses and changes - <AUTHOR> https://blog.frankel.ch/on-opensource-licenses-changes/ <comment><USER2> > Yet, according to Stallman's definition [of opensource], software needs to > be free: [Free as a bird, Free as a beer]

This is totally wrong. Stallman focus on freedom to use the software, he never stated that the software need to be free of charge. Yet, Stallman hates the word opensource because is it to vague and it does not guarantee the freedom to the user. Stallman use the term free software and or libre software, and mean free as in free speech only not free beer. While Free software and opensource software's definition may overlap a bit, it is not the same, for example, some opensource license does not allow making a modified version and using it privately. Free software focus mainly on users' essential freedoms, while the opensource initiative focus on practical benefits only.

Here is a good read about opensource vs freesoftware: [https://www.gnu.org/philosophy/open-source-misses-the- point....](https://www.gnu.org/philosophy/open-source-misses-the-point.html)

Richard Stallman deploys a lot of effort trying to raise awareness about the importance of free software please STOP miscoting him.

</comment><comment><USER3> >Changing one's license is in general not a great idea. When somebody uses your software, they need to be able to trust they can use it in the future under the same terms.

IANAL, but I think this is wrong. Unless the license stipulates that the terms can be changed by the licensor, then the licensee only has to abide by the terms from when they agreed. The GPL goes out of its way to make it possible to use licenses from before what's currently available:

>If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation.

<comment><USER4> You can keep using old versions under the original terms, but you'd have to ignore newer versions of the software, or use a fork.

</comment></comment><comment><USER5> The whole OSS licensing field is a true minefield. IMHO, the Creative Commons folks (CC) have done a _much_ better job at creating a series of clearly defined licenses. With CC, you can quickly determine the freedoms and responsibilities conferred by each license.

Meanwhile, in OSS, you have to research the fine distinctions between BSD-2 clause, BSD 3-clause, MIT, etc. Or Apache vs. Eclipse, etc. Even fundamental distinctions such as between OSS and Free Software (GNU-style) are regularly misunderstood, as in this article.

I'd like to see the OSI convene the organizations that have issued the various licenses and get them to agree on a new OSS licensing scheme modeled after the system used by the Creative Commons.

I think everyone would benefit from that level of clarity.

<comment><USER6> > I'd like to see the OSI convene the organizations that have issued the > various licenses and get them to agree on a new OSS licensing scheme modeled > after the system used by the Creative Commons.

this is standards proliferate [https://xkcd.com/927/](https://xkcd.com/927/)

<comment><USER5> No, licenses are not standards that require code changes to be adopted. The OSS community has a long history of moving off old licenses to new versions and the OSI lists many retired licenses due to this. (When Apache replaced their 1.1 license with the current 2.0 license, there was no complaint about proliferation. Rather, the community adopted the new license and today 1.1 is entirely dead.)

<comment><USER6> License versions are a different thing. Changing to a completely different one in practice is a huge undertaking since every past contributor needs to agree to it

</comment></comment></comment></comment><comment><USER7> Article is good on clarity, but a little light on solutions for the "service- wrapping" problem. Maybe that was outside of the article's scope. But it does seem like a problem we have yet to solve. Many have tried, but the ones that attempt to switch their license to stop Amazon from eating their lunch then see their popularity plummet and the major distros kick the software out of their repos.

It's a big gamble, and I imagine most product developers don't wish to alienate the FOSS community unless they feel their survival is at stake, so perhaps this is where the FOSS community needs to step in and provide better guidance.

<comment><USER8> The Affero GPL is a FLOSS, OSI-certified license that by-and-large solves the "software as a service" problem, by making sure that if SaaS providers publically perform a version of the software over a computer network, they also have to make that version available under the license. This is quite consistent with the general norms of FLOSS development and does not actively introduce discrimination over private use of the software, as other licenses do.

</comment></comment><comment><USER9> This article was a light read. Well written and clearly stated, but rather short on new ideas. In particular, it would be interesting to know how good the source available licenses - e.g. Redis' RSAL license - are at addressing the cloud-provider issue. Of course, it's not technically open source, but does it work?

</comment><comment><USER10> Never understood the obsession of OSI and their insisting to not give any protection for the authors. Authors whether individuals or companies should have the right to prevent their work from being rebranded or sold by others with minor or even major modification.

How many times did it happen that somebody or organization wrote a complex piece of software and published it and found that some companies just rebranded it and even sold their work with little to none added value while the original authors don't even have the right to complain because this is how open source works.

Something like BSL [https://mariadb.com/bsl-faq- mariadb/](https://mariadb.com/bsl-faq-mariadb/) needs to get more reputation among solo developers and small companies with little to no funding that believe in the power of open source while believing also in having the right of earning a living from their own work.

<comment><USER8> > should have the right to prevent their work from being rebranded or sold by > others

Nope, the right to fork, and to rebrand concurrently with a fork, is critical to FLOSS development. Even MariaDB itself is a fork.

<comment><USER10> I am not against forking or introducing modifications, etc.... You can fork it, change it for your use case as you wish, but just don't bundle my work, rebrand it and sell it as if it were yours. I meant my work to be free and I want the legal protection to keep my work free.

<comment><USER11> That would be the NC clause of the Creative Commons licenses. This is not free / open source software anymore though because you remove the right of redistributing the work or the derivative for a fee, which is arguably desirable for free software to foster. With such a license, you might prevent business models like the one of Red Hat, which also produce a great quantity of free software. So while they make money with your work, you can benefit from this with the availability of free software they built with this money.

If you pick a copyleft license like the (A)GPL, it's right that you allow people to sell your software, but their customers will be able to require the source code under this license and will be able make it available to the world.

Now there are loopholes like the one the company behind the GRSecurity project exploits.

If anything here is not desirable, you'll probably want to use a more restrictive license than a FLOSS one, but be careful with all the possible side effect (people not wanting to use or contribute to a non-free software, licence incompatibility with libraries and the rest of the ecosystem, etc.

</comment></comment></comment></comment><comment><USER12> I've always felt like there's something fundamentally flawed with open source licenses. It seems that they are all promoting the openness of a fragment of code but at the same time twisting your hands ever so slightly when you start using them. All while different licenses are engaged in a fight as to which is more open. For many years I thought that MIT is the most open one. That is until recently when I saw wtfpl [1] and truthfully I fell in love with it. At some point I'll update everything I've ever open sourced to that and I intend to use it for all the other projects I plan on open sourcing, though I am a bit short on motivation to get to that lately(for personal reasons, not because of lack of interest or burnout or anything).

[1] [http://www.wtfpl.net/](http://www.wtfpl.net/)

<comment><USER13> The problem with the WFPL is that it may not be legal in many jurisdictions. I would encourage you to dual license with the WTFPL and a more legally vetted license, such as MIT, to allow your software to be used in a wider variety of jurisdictions.

And thank you for writing open source software!

Edit: The CC0 may be a more legally valid license that accomplishes the same thing.

<comment><USER14> I've always been a fan of the sqlite licence: [https://sqlite.org/copyright.html](https://sqlite.org/copyright.html)

Basically sqlite is public domain if your country recognizes that kind of thing. If it doesn't, then sqlite doesn't care and will never pursue you. However, if /you/ care, sqlite will sell you licence. This licence costs $6000, looks very official and shiny and whatnot, covers all your bases as if public domain did exist in your country, and sqlite uses the money to continue funding development.

I'd be inclined to do something very similar with an additional clause stating that if dedicating to the public domain doesn't free me from liability, then see $6000 licence. (I thought sqlite had this, but either they've removed it or I misremember).

</comment></comment></comment><comment><USER15> Anyone have a link on the rollback of the CoronaVirus License? In particular, I have doubts about "The license change was rollbacked as it was considered at least partially unlawful."

<comment><AUTHOR> Here it is [https://web.archive.org/web/20200319183105/https://plantuml....](https://web.archive.org/web/20200319183105/https://plantuml.com/license)

I don't find the rollback announcement again, but I distinctively remember the fact I stated.</comment></comment>


Voat: What Went Wrong - <AUTHOR> https://battlepenguin.com/tech/voat-what-went-wrong/ <comment><USER2> IMO what _really_ went wrong with Voat was that it was simply not very different from Reddit - the same issues Reddit has with the way it works and the way discussions are handled exist with Voat. The main difference between the two is the people but outside of that they're practically the same centralized, voting-based, hierarchical discussion subforums where discussions are (for all practical purposes) arbitrarily moved up and down and over time away from the viewers' eyes (unlike, e.g., a traditional forum where a thread can be active for months or even years - though that has issues of its own if you are not an active participant).

So at the end of the day it all came down to the people. When Voat first came out i made an account there to see if things will be better, but of course nobody outside those who were forced out of Reddit went to Voat and those were a tiny minority.

"Everyone" was (and is) still on Reddit, so i just stick to where "everyone" is to see new stuff and read what "everyone" things about matters. I remember being out with a friend and Voat was brought up - he also made an account and he also had the same thoughts: "nobody" was on Voat, so he stayed with Reddit.

Often when it comes to Voat and similar sites people point out the hate communities, etc, but really those only happen because most people who are on Reddit simply see no reason to move anywhere else. If everyone (or at least the large majority) moved (or also participated) on Voat then the hate groups would be the same tiny minority that they were on Reddit.

But they did not move because there was no reason to move, Voat simply didn't provide much of a difference to Reddit.

(and no i do not think that being decentralized or whatever antithesis to the comparisons i made at the beginning of this comment would be a reason for everyone on Reddit to move - i don't know what would make people do such a move. If i knew i'd build it and be a millionaire).</comment>


Where's Your Business Logic? - <AUTHOR> http://collectiveidea.com/blog/archives/2012/06/28/wheres-your-business-logic/

<comment><USER2> It is a common pattern in many "enterprisey" java applications I've been part of and was called "service layer" (coarse grained, fine grained etc). This smells like old wine in a new bottle.

EDIT: I have to give credit to the author to bring this in to the Rails apps, he's right that Rails encourages to think in terms of controllers,models and helpers.

<comment><USER3> _This smells like old wine in a new bottle._

Well, the use case for wine today is much the same as it was in ancient Rome.

Someone needs to curate all of the pitfalls of rapid dynamic development environments into one book. This would be of tremendous benefit to developers farsighted enough to learn from history.

</comment></comment><comment><USER4> I find that web frameworks focus so much on CRUD and not enough on business logic. We usually end up with either fat models or fat controllers, neither of which is ideal or practical. Symfony2 (php), however, offers something they call Services, which I've been using to fulfill the role of the OP's Interactors and so far I'm pleased.

<comment> brown9-2 What is preventing you from creating a layer of "services" or whatever-you- want-to-call-it in between the controllers and the models in those web frameworks?

<comment><USER5> I don't think there's anything preventing you from using Services in other (PHP) frameworks, it's just that Symfony2's Dependency Injection Container component makes it very simple and straight-forward. I think it'd be trivial for other PHP 5.3 compatible frameworks to adopt this component to facilitate using Services themselves.

</comment></comment></comment><comment><USER6> Perl's Catalyst Web Framework encourage you to do it slightly different:

you develop libraries for your business logic and try to make these reusable (or may be even opensourceable on CPAN for other developers to use). These libraries will include their own test, which obviously not require you to fire your main web app and need to be tested once - when installing it on local machine.

What do not fit well into standalone libraries, goes into model (aka fat models). Again, models developed as standalone piece which could be used from command oneliners, web frontend, anything else. Tests for these often are part of web application, but testing of models should not require firing web app itself.

and last piece - controllers and views. these are integral part of web app therefore these require firing web app code to test. but controller code is very thin since it just prepare and route calls into model. views are usually just an adapter, often default scaffolded, for existing templating module on CPAN.

I bet same principle can be applied to Rails framework, but it is about community around framework which encourage you to do things in a certain way.

</comment><comment><USER7> So you take your service layer, decompose it into a half-arsed version of the command pattern, and invent an entirely new term for it, just for value. Got it.

<comment><AUTHOR> It's different than the command pattern, and I didn't make up the idea or the name. The whole idea of the Interactor came straight from [http://www.amazon.com/Object-Oriented-Software- Engineering-A...](http://www.amazon.com/Object-Oriented-Software-Engineering- Approach/dp/0201544350) which Uncle Bob bases a lot of his talk on.

<comment><USER7> Yeah, I've met people calling them 'Tasks' too. If it _was_ a command object - if you were auditing logins, for example, and wanted to store the login command in your history - I'd see the need for its existence and its instantiation.

As it is, it's just the sort of thing OO programmers seem to create because they find nouns comforting.

<comment><USER7> And lordy, so many nouns. Looking at the whole pattern, with DTOs in and out of Interactors, Presenters to format the DTO for the view... it never ceases to amaze me the amount of work people will do to supposedly save themselves some work. The amount of architecture people will put in place to 'put off decisions about architecture'.

</comment></comment></comment></comment><comment><USER8> Aren't we talking the "command pattern" here? There might not be an undo function but isn't encapsulating discrete business operations its responsibility? Maybe more fine grained than a use case, but the sample code has a striking resemblance to the command pattern.

</comment><comment><USER3> _> The fundamental problem with almost every Rails project (and I'm sure in other frameworks as well), is that there is no direct codifying of the business rules and use cases of the application._

This is a damn shame. Why do I say so? Because the exact same thing happened years ago in Smalltalk projects. It's easy to start and iterate quickly in a dynamic OO language and the technical debt sneaks up on you very quietly. The next thing you know, there's a quagmire of convoluted and deeply nested conditional logic.

If it was your business to curate lots of painted canvases, then you'd have an orderly and logical filing (or search) system for keeping track of those. If it's your business to curate lots of neat functionality embodied as pieces of code, then you need an orderly and logical system of keeping track of those. It's amazing how many large projects devolve into _a big pile of stuff_ in a class for every screen.

"Enablement" or "security" or whatever you call the framework that lets particular users do particular things at particular times was one of the big wins for Smalltalk projects.

If someone asks you why user X can't do Y on screen Z, and the only way you can know this is for someone to just happen to remember or to painstakingly reverse engineer code, then you have a kind of technical debt.

Beware if your system just consists of putting in a breakpoint then reading code. There is no clear demarcation separating that from the quagmire. (The solution is to have some kind of consistent idiom/framework/something that people can learn to parse, and a way of reorganizing recurring ideas so they can be reused.)

</comment><comment><USER9> I couldn't bring myself to read past this line:

        if user && user.password == password

<comment><USER10> Why? That's how BCrypt, for one, works--you create a User#password method which returns a Password object (initialized from the contents of the User's actual #password_hash field or somesuch.) That Password class has an overridden equality comparison method, which hashes the RHS String to test against itself.

<comment><USER11> Except - if you look at the linked repo, there is no mention of bcrypt at all - [https://github.com/jasonroelofs/raidit/blob/master/Gemfile.l...](https://github.com/jasonroelofs/raidit/blob/master/Gemfile.lock)

Infact, the whole persistence mechanism is rather mystifying - an in memory store? What if I accidentally the power?

<comment><AUTHOR> It's super early stages, I'm only building the functionality I need, no persistence, no nothing special.

Isn't this a bit immaterial to the post at hand anyway?

<comment><USER9> No, because you're giving opinionated advice with weak supporting evidence. So the reader is left with 2 things to decide whether it is good advice.

1) your reputation in the community. I for one have never heard of you, so that leaves

2) code quality of the examples that you posted. a reader could infer that this may be good advice based on the fact that you appear to generate good code.

In this case your code is clearly not very well thought out, so why would the reader put trust in your unsubstantiated opinions?

<comment><AUTHOR> I'm sorry, at what point did I tell people to blindly do something? I'm offering ideas, suggestions, help to those who are and who've been in the same situations I keep finding myself in (utterly painful Rails code). It's unfortunate that you feel no-one should even attempt to contribute to the community unless they've already contributed to the community, and that you feel that every bit of code should immediately be complete and perfect.

In case you weren't aware, samples that don't require a ton of context are a particularly hard problem for informational posts like this. If you have feedback on the ideas I presented I'd be glad to hear them but I still fail to see how nit-picking an obviously early in-progress application proves anything.

<comment><USER9> At what point did I accuse you of telling people to blindly do something? I accused you of not having good support for the opinions in your article. That is all.

I, too, keep finding myself in utterly painful Rails code. Now, if I may use your metaphor: Most of it is due to the blind leading the blind, per se. Programming novices discover some new rails thing, ruby thing ("method_missing" is the worst offender), or even a new programming pattern such as Command, and think it's the most incredible thing they've ever experienced, so they write a blog post about it. Other programming novices read the blog post and then go off on a rampage, overusing that concept and writing all the shitty rails code that I am currently maintaining.

There you have the ruby community in a nutshell.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> Isn't this simply the cornerstone of functional testing?

At that level your tests should be executing business test cases. If a functional test fails there is either a corresponding unit test failure or the failure of the interaction between the methods rolled up into the business logic (functional test).

<comment><USER13> Assuming your question isn't merely rhetorical...

Functional testing is related, but it's not really the point of the article. In the context of the article, functional testing is actually one logical step above his Interactor—the functional test would end up effectively testing the interactor code (which includes all the model, etc code) plus the controller code to make sure the session is properly set (i.e. now that I'm logged in, was I properly redirected?). The latter is outside the context of the "business logic" of logging the user in.

The point of the article is in organizing the business logic of use cases into their own objects—"business logic" being defined, presumably, as anything non- web and non-persistence related. This is done in order to avoid the spaghetti that real world "thick model" applications tend to accrue; or similarly the spaghetti that some developers leave in controllers.

There have been a spate of articles on this topic in recent years. Service objects, interactors, DCI, etc, etc. One or more of them is probably good advice for large projects. :)

</comment></comment><comment><USER14> "LogUserIn" is not a class; classes are most often nouns and actions are typically verbs. A "User" class (or "UserController" class) with a "login" method is more appropriate.

See also: <http://pragprog.com/articles/tell-dont-ask>

<comment><USER15> You're actually wrong. This is a bastardisation of the command pattern. LogUserIn is a command. The verb/noun thing applies to a different class of object (your domain).

The article sucks pretty badly to be honest but the foundations are solid.

A much better solution is to use the command pattern over a rigid domain model. The rigid domain model encapsulates your business domain and logic and your commands describe different ways to mutate the domain.

This could be classified as CQS or if you use a bus to deliver your commands CQRS.

Unfortunately, and I'll probably get flamed for it, by rails is a stinking turd when it comes to architecture. I've never seen anything which isn't trivial CRUD done without causing uber spaghetti. Also to do proper domain modelling, you need a more powerful ORM such as sqlalchemy or hibernate.

<comment><USER3> _I've never seen anything which isn't trivial CRUD done without causing uber spaghetti._

It's easy to get a lot of followers by creating a framework that lets a new user create something small very quickly. Keeping it well organized as time goes on is much harder. It has been this way for decades, and still people haven't learned.

<comment><AUTHOR> And I'd like to clarify your point here. People haven't learned because there isn't enough teaching going on. It's only been the past 4 months that I've even begun to realize just how wrong I've been, and that's only because I found a mentor who pointed me in the right direction.

There's a huge problem of how to teach people good, responsible software development, and this article is a part of my attempt to help educate people towards even knowing about better software practices. People aren't going to just suddenly get it, they have to learn from somewhere.

<comment><USER15> There's plenty of teaching going on but people have stopped listening to people who know what they are talking about and have started listening to all the trendy marketing out there.

Two awesome places to learn shit:

<http://martinfowler.com/bliki/>

<http://www.eaipatterns.com/>

Both of those are available as books (EIP + PoEAA). Buy them, read them, weep.

Yes they are C# and Java but that's where most of the formalisms were developed (for a reason!).

<comment><USER3> Formalisms and learning materials available, but not for major languages that could benefit greatly from them.

<comment><USER15> Perhaps the wrong choice of language is made then?

<comment><USER3> No, I'm pointing out things that were wrong back in the late 1990's and early 2000's are _again wrong_ just for different languages. It's an opportunity, in other words.

<comment><USER15> I don't understand. What in particular was wrong? We had formalised design patterns and architectures back then.

A large portion of any industry ignores established best practices and makes a mess every generation.

<comment><USER3> _We had formalised design patterns and architectures back then._

Yes, and many of them described in books in particular languages. Not every language community that could have benefitted had the right book written using their language.

Like I said, a recurring condition and an opportunity.

<comment><USER15> The books describe portable concepts with examples in the languages of the time.

Much as SICP is applicable to a lot of languages that aren't scheme.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> Doesn't everyone do this? It's just good OO.

Although I question demonstrating the principle with a login given that those tokens are usually application specific and little to do with business rules.

<comment><AUTHOR> Heh I wish, but no, it's pretty rare to see this at least in the Rails world.

As for the login token, I feel you may be defining "business rules" differently than I am. I'm using "business rules" as "the rules in which the application functions", and one of the rules is that you need an authenticated user to do things. The term is an effort to use a word or phrase that intentionally leaves out the framework you're using, but yes it can get confusing depending on what definition you're use to using. The other phrase, "use case" can at times be too limiting, but I don't know of any better term for what I'm talking about.</comment></comment>