What's the difference between statistics and machine learning? - <AUTHOR> https://thestatsgeek.com/2019/08/08/whats-the-difference-between-statistics-and-machine-learning/ <comment><USER4> The classic explanation is Lei Breiman's 'Two Cultures' paper. He was a statistics professor who left for industry, came back, and tried to get academics to adapt industry approaches. The paper is very readable.

[http://www2.math.uu.se/~thulin/mm/breiman.pdf](http://www2.math.uu.se/~thulin/mm/breiman.pdf)

An oversimplified version may be:

Statistics focuses on fitting data to formally understandable models, whereas data science focuses on solving problems -- even if that means using techniques that aren't formally understood.

Leo Breiman is also known for pioneering random forest and bootstrap aggregation.

I think of machine learning as a subset of data science where you trick linear algebra into thinking.

<comment><USER5> This seems perfectly fair. And I think that historically there was plenty of use for statistics where people didn't care about the formal understanding, so they were doing crude machine learning before the term became widespread.

I've sat through lengthy discussions of machine learning exercises, and could not silence the voice in my head, saying: "This is just curve fitting." Fitting data to an arbitrary curve, and then extrapolating the fitting function, is as old as the hills.

<comment><USER6> Formal understanding is critical to actually know the limitations of any given system. Treating it as a black box has short lifetime, as your method of analysis will miss key features of the system or oversimplify it.

Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much translatable between these domains.

<comment><USER7> _> Treating it as a black box has short lifetime, as your method of analysis will miss key features of the system or oversimplify it._

On the other hand, it can work indefinitely if it solves a problem well enough and it is used always on the same kind of problem.

Sure, you will need to understand it better to apply it to new markets, but often that kind of research is outside the scope of a single business, and they don't need it.

<comment><USER6> How do you know it solves the problem, instead of just tricking your measure?

Do you know when it doesn't work?

Business does not care about rigor, but then terrible things happen when your face recognition system happens to not work for people with dark skin tone.

Even more terrible things happen when it detects cats as people, and worse once it's used to link data into a police database.

Even pretty bad things happen when it's used for detecting potential (treated as absolute) copyright violations.

Security holes when it's used for detecting security problems.

Loss of business when it's used for ticket prioritization.

Business world does not care that it sells a broken solution as long as it's not obvious and someone has been paid. Everyone else pays for the failures. And you cannot sue an ML system really.

<comment><USER8> Why don't we do online learning? Each failure is a new training sample.

Why dont't we do active learning? Or adversarial networks, or....

Sure it is easy to fail in machine learning. But it doesn't mean that we need to undstand how and what our models learn. That's the only big advantage of machine learning: the model learns, so I don't have to.

</comment></comment></comment></comment></comment></comment><comment><USER9> I'm not sure about machine learning specifically, but I heard somewhere that a data scientist is someone who does statistics, on a Mac, in San Francisco.

<comment><USER10> After spending 80% of their time cleaning up garbage input data.

<comment><USER11> ...and aftet spending 15% plotting graphs and diagrams.

</comment></comment></comment><comment><USER12> Inferential statistics is about explaining an observed outcome in terms of its causing factors. Once we have explained it, then we can make predictions. Machine learning skips the explaining part and goes straight to making predictions, without attempting to understand the underlying process that led to the particular outcome. This would be the main difference, in my opinion.

<comment><USER13> Yes, this is my observation as well. Put another way, statistics is primarily concerned with understanding the mechanisms behind something. Predictive power can and will often be sacrificed if it aids explanatory power or conceptual elegance.

By contrast, machine learning is primarily concerned with making the best prediction possible, even if that means sacrificing an understanding of the underlying mechanisms.

The Venn diagram of methods can have a fair degree of overlap.

Neither disposition is right or wrong, but they tend to have natural places where each makes more sense. If you're trying to predict whether a picture is of a cat or a dog, you probably don't care much about the constituent contribution of factors to one pictures dogness or catness. On the other hand, if you're trying to predict traffic collisions based on characteristics of a roadway, you're probably less concerned with the predicted number of crashes and more concerned with the relative contribution of a handful of independent variables.

<comment><USER14> If you are trying to predict the failure modes of a cat/dog discriminator on arbitrary images, an understanding of catness/dogness is more useful.

</comment></comment></comment><comment><USER15> This may sound a little like trivializing, but don't we have to know what "statistics" are and what "machine learning" is to say anything about the difference(s) between them?

Looking at this through even the lens of multinomial logistic regression, or of econometrics generally, I don't think that "statistics draws population inferences from a sample, while machine learning finds generalizable predictive patterns" even makes sense _as a difference_. Any _prediction_ is an _inference_ about the population of future events, or of contemporary events not present in the _sample_. You plug 1000s of events each described by 100 columns into a logistic regression, and you're hoping to get something predictive out of it. Further, as nice as the idea is that you can tease out "factors" from your 100 columns, you don't have to look at "3.3375905e-5 x (spent five years before age 18 in a smoker's home)" for very long to wonder how much 'explanation' you're getting out of the terms in the exponents of your probability functions.

I still can't resist tweaking ML enthusiasts and data scientists: Statistics is what people who know what they're doing are doing. Machine learning is the rest!

<comment><USER16> Everybody owes it to themselves to read Breiman's Two Cultures[1]:

> There are two cultures in the use of statistical modeling to reach > conclusions from data. One assumes that the data are generated by a given > stochastic data model. The other uses algorithmic models and treats the data > mechanism as unknown. The statistical community has been committed to the > almost exclusive use of data models. This commitment has led to irrelevant > theory, questionable conclusions, and has kept statisticians from working on > a large range of interesting current problems. Algorithmic modeling, both in > theory and practice, has developed rapidly in fields outside statistics. It > can be used both on large complex data sets and as a more accurate and > informative alternative to data modeling on smaller data sets. If our goal > as a field is to use data to solve problems, then we need to move away from > exclusive dependence on data models and adopt a more diverse set of tools.

The difference between inference and prediction can be illustrated with something like a decision tree or a random forest. Statistical inference is "the theory, methods, and practice of forming judgments about the parameters of a population and the reliability of statistical relationships, typically on the basis of random sampling." If you look at something like linear regressions, it makes a lot of assumptions about the data[2]: the distribution of residual errors is normal, there's no multicollinearity, etc.

Random forest don't care. Random forest is a set of steps. You follow the steps, you get an answer. You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm.

Algorithms are powerful! Not everything needs to be an inference problem. We're better off when we have lots of tools. You don't have to choose _between_ the two camps. But the "Everything is Statistics!" and the "Everything is Machine Learning!" points of view rob us of ways of thinking about our tools that helps us understand what those tools are.

1) [https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...](https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726)

2) [https://thestatsgeek.com/2013/08/07/assumptions-for- linear-r...](https://thestatsgeek.com/2013/08/07/assumptions-for-linear- regression/)

<comment><USER17> Nice comment, but there's more nuance than you claim.

> You made no assumptions about your data, about the distribution of it, any > of it. You just followed an algorithm.

You made _implicit assumptions_ that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as a leopard). This means that you are liable to overfit to the data, and generalize poorly, or in ways you are not supposed to.

This is what leads to the perspective that "machine learning" might end up being a powerful tool for laundering bias: [https://idlewords.com/talks/sase_panel.htm](https://idlewords.com/talks/sase_panel.htm)

<comment><USER16> I think what you're saying is orthogonal to what I'm saying.

Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating.

What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population.

OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data.

To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).

<comment><USER17> > _machine learning is a powerful way to get pretty good results on a lot of > problem spaces without having to understand the probability function > involved_

The NFL theorems essentially state that for every dataset on which an algorithm generalizes well, there is another on which it generalizes poorly. So, there are _always_ implicit biases, even for a random forest model (eg: if you try to model Boolean functions with a random forest, those functions which can be approximated effectively with few trees will form a set of very small measure. Specifically, it is my intuition that those which have many terms in the sum of products form might need many trees to approximate). It then becomes a question of whether the bias of your models are compatible with the dataset/domain under consideration.

See this Minsky-Sussman koan: [http://www.catb.org/jargon/html/koans.html#id3141241](http://www.catb.org/jargon/html/koans.html#id3141241)

> _Both statistical and ML models suffer from the problem you 're stating._

And that is precisely why I don't see how ML algorithms are "more powerful" than statistics in any way, as per your claim.

<comment><USER16> > And that is precisely why I don't see how ML algorithms are "more powerful" > than statistics in any way, as per your claim.

Well I'm glad you don't see that, but I'm a bit confused why you think I said it because I didn't say it and don't believe it. I said that ML is "powerful," not "more powerful."

</comment></comment></comment></comment></comment></comment><comment><USER18> That's easy - statisticians take pride in models that are understandable, while machine learning practitioners take pride in models that are not.

<comment><USER19> Understandable? Two words: statistical significance :P I've seen university employees who still don't understand what it is and can't explain it...and just about everyone who uses it gets it wrong...

<comment><USER20> Most scientists who use statistics are not statisticians.

<comment><USER21> 95% of scientists who use statistics are not statisticians, plus or minus 5%, 19 times out of 20.

</comment></comment></comment></comment><comment><USER22> The main difference is not in techniques, but in goals. The primary goal of statistics is to help a human make an informed decision by quantifying uncertainty. That quantification of uncertainty can be used to explain to other humans why the decision was made.

The primary goal of machine learning is to help a machine make better decisions. As long as it gets the "right" answer, the explanations to humans are not as important.

<comment><USER23> I think that's close, but disagree with the uncertainty quantification bit. Often in both fields, you'll have a proof that "errors" converge at some asymptotic rate. Often in both fields, when you apply that tool to a finite data set, you don't know the error magnitude in _this_ instance. I don't mean exact error, in the sense of 'if I knew the error, I could correct for it.' I mean you know big-O notation error, but not the constant factors.

I do like the rest of the point you make, and it seems to match OP. Inference vs otherwise. Helping people learn/act/decide based on data data vs helping machines learn/act/decide based on data.

</comment></comment><comment><USER24> Reminded me of this [1] by Prof. Rob Tibshirani.

[1]: [https://statweb.stanford.edu/~tibs/stat315a/glossary.pdf](https://statweb.stanford.edu/~tibs/stat315a/glossary.pdf)

</comment><comment><USER25> I like Tom Mitchell's definition in his machine learning book:

'A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.'

Merriam Webster's definition of statistics:

'a branch of mathematics dealing with the collection, analysis, interpretation, and presentation of masses of numerical data'

So establish a couple of definitions, and take it from there.

I believe such discussions can easily be taken too far: surely the point of a subject is to group together related topics. Who is to say that the set of topics comprising multiple subjects need to be disjoint?

</comment><comment><USER26> The amount of hair-splitting that goes into discussing this subject is unbelievable. Clearly, they both are fairly closely intertwined - <USER2> given that one potential explanation boils down to the motivation of the user.

So do we need an explicit taxonomy to say that one application is machine learning and another is statistics? Or does the form not matter as much as the function?

Pace the OP, who I'm sure didn't do this with this end in mind, but a lot of the cases I've seen of people trying to bring this up come down to wanting to self-identify in a certain way as opposed to actually talking about the subject at hand.

<comment><USER27> I think it's actually both important and possible to differentiate between stats and ML (and AI for that matter). Statisticians, ML engineers, and data scientists all have related but distinct skill sets, knowledge, methodological experience, and worldviews. The identification can be important because it suggests what sorts of tasks and projects an individual is well-suited for. (Notwithstanding the fact that I believe anyone can acquire skills/knowledge to be competent at any of those jobs and transition between them with a bit of effort).

For example, consider the work that the FDA does in evaluating clinical drug trial data. In this case, you need statisticians on staff that can critically analyze the statistical/methodological rigor, not to mention understand the medical domain and relevant regulations/processes. It doesn't matter if they can only use the GUI versions of SAS and have never heard of TensorFlow, their time is better spent deeply understanding nuances of experimental design, sampling methods, causal analysis, estimation, etc. I would argue that a "good" statistician in this context might very well be "not so good" at data analysis, as long as they can clearly and accurately critique experiments and analyses and identify when things go wrong. Also consider—what would happen if you slotted a ML engineer into a role like this?

ML engineers are best at optimizing performance (not just accuracy) at some defined task, where ideally the cost of the model being wrong in some exotic way is not astronomical (e.g. approving a dangerous drug, convicting an innocent person, initiating a stock crash, corrupting the attention span of the human race). A good use case for machine learning comes from the book _Pattern Recognition_ —sorting fish on a conveyor belt. No one gives an ounce of chum about statistical rigor in this case, and the consequences of being inaccurate are quantifiable and manageable. But building and tuning a ML system to do this at an acceptable performance level requires a ton of work. You'd need to define the sensor array / inputs, collect and label data, engineer features (more relevant before deep learning, in the case of computer vision), train and evaluate an object detection model, avoid overfitting the training data, build an anomaly detector to weed out the stray crab, ensure it works fast enough to be used in production, turbo-browse arXiv to make sure your stuff isn't obsolete (damn, it is). This is a vastly different focus than statisticians, who wouldn't get very far with this sort of performance- optimization problem.

Data scientists are somewhere in between a statistician and ML engineer—like a hybrid class in an RPG. They can cast a few key ML spells and have dabbled in arcane math, but can also slice up data goblins by hand (maybe even with style). With a foundation in a bit of everything, they can glue a team together and flexibly grow in multiple directions. But they can also become a waste of shiny gold if pitted against a specialized task without support from, you know, specialists.

</comment></comment><comment><USER28> I can't speak for all of machine learning, but classic statistics has some probabilistic assumptions and uses those to prove some theorems about the results of how statistical methods manipulate data.

E.g., in regression analysis, the assumptions are (i) there really is a _linear mode_ with the variables to be used; (ii) typically the data will not fit the model exactly and instead there are errors; (iii) the errors are in the sense of probability independent, all have the same distribution which is Gaussian with mean zero. Then estimate the regression coefficients and get, say, a F ratio statistic on the fit and t-tests on the coefficients. For details can see, say, Mood, Graybill, and Boas, or Morrison, or Draper and Smith, or Rao, etc. If you need detailed references, request them here, but these are all classic references going back decades. In that case, maybe some of current machine learning does qualify as statistics.

Here regression was just an example, however, apparently especially close to much of current machine learning. But there is no end of (i) having data, (ii) having some probabilistic assumptions, (iii) manipulating the data, (iv) getting results, and (v) proving some theorems about the probabilistic properties of the results.

Might argue that in machine learning the results of the training data on the model yields some estimates of some probabilistic properties that can be used to make probabilistic statements about the results of the trained model on future real data.

<comment><USER29> There are non-parametric statistical methods too though (such as bootstrap methods) which don't make such assumptions.

<comment><USER28> Have two calculus teachers, A and B, each with 20 students. Look at the final exam numbers. Put all the numbers in a bucket, stir briskly, draw out 20 numbers (test scores) and average, average the other 20, and get the difference in the averages. Do this many times. Get the empirical distribution of the differences in the averages.

Now look at the difference in the actual average for A and B. If this is out in a tail of the empirical distribution, then we reject the null hypothesis that the two teachers are equally good.

For this non-parametric, _distribution-free_ , resampling, two-sample test, to make theorems about it, which should, will likely need at least an independence assumption and likely an i.i.d. (independent, identically distributed) assumption. Else maybe each student of teacher B is an older sibling of a student of teacher A!!!!

In a nutshell, what's going on in statistical hypothesis testing is that we make the null hypothesis, and that gives us enough assumptions, e.g., i.i.d., to calculate the probability of our calculated _test statistic_ , e.g., the difference in the two averages, being way out in a tail. Without some such null hypothesis assumptions, we have no basis on which to _reject_ anything, are not testing anything.

There's chance of getting all twisted out of shape _philosophically( here: E.g., I outlined one statistical hypothesis test for the two teachers A and B. Okay, now consider ALL_ reasonably relevant* hypothesis: Maybe we on the test I outlined, the two teachers look very different, not equal, maybe teacher B better. But in ALL those hypothesis tests, maybe in one of the tests the two teachers look equally good or even teacher A looks better. Now what do we do? That is, there is a suspicion that teacher B looked better ONLY because of the particular test we chose. Maybe there has been some research to clean up this issue.

<comment><USER28> Sorry, was way too busy and typed way too fast and had LOTS of typing errors.

</comment></comment></comment></comment><comment><USER30> Statistics is a set of theories and methods that can be successfully executed on any list of numbers and provably provides mathematically valid predictions under a set of conditions that never occurs in the real world.

This means, more precisely, statistics only works on data series that obey the law of large numbers, and combining/using 2 or more statistical predictions nearly always requires total independence of the predictions and their inputs, which is never the case (for one thing, they always occur on the same planet). Furthermore, the reason people make statistical predictions is to change the outcome, but doing anything to change the outcome always invalidates the statistical method used to collect the data. There are a couple of things statistics _never_ does. Used correctly, it can never predict extreme values. It can never correctly predict values in systems that are too complex, where too many independent variables determine the outome. And "too many" is something like 50 to 500. It can never correctly be used to verify if a deliberate change worked.

Machine learning is much the same, except it never provides mathematically valid predictions.

Despite this, it should probably be mentioned that both do provide _useful_ results, occasionally getting things very, very wrong.

[https://xkcd.com/605/](https://xkcd.com/605/)

</comment><comment><USER31> I think a better question is what is the difference between a non linear regression and machine learning!

</comment><comment><USER32> Statistics is good for modelling things that are simplistic & low dimensional. Machine learning is good for modelling things that are nuanced & high dimensional. statisticians want to understand things but imho overestimate human ability to make sense of a complex world. machine learning people want the machine to understand the things for us and then teach us about it when it's making us breakfast.

</comment><comment><USER19> Whatever it is, I imagine it's similar to the difference between math, stats, ml, ai, logic, econometrics, actuaries, epidemiology, etc.

For the record, I don't consider those intellectually separate fields, but do accept them to be culturally separate, for better or worse...

But then I could never understand why physics, biology or chemistry were considered separate fields either...

Or psychology, economics, philosophy, etc etc etc...

<comment><USER33> They're considered separate fields because they focus on different problems which are amenable to different techniques, leaving their expert practitioners with very different knowledge bases. You're right that it is a cultural distinction, but that doesn't mean it isn't an important or practical one.

</comment></comment><comment><USER34> One actually knows what's going on.

</comment><comment><USER35> My $0.02: statistics is generally not very "sensitive" to underlying data -- it deals with small numbers of dimensions/features and lots of similar examples and classification is about falling within easily understood bounds.

Machine learning, on the other hand, can be extremely "sensitive" to underlying data -- it deals with high numbers of dimensions/features on potentially extremely sparse data sets, and a small change in one of them could result in a radically different classification. The potentially accuracy is far, far higher but so is the risk of overfitting.

Or: statistics looks for ranges, machine learning looks for patterns.

</comment><comment><USER36> I agree that in general statistics is more concerned about inference, while machine learning focuses on prediction. On the other hand, there is such a huge overlap between the fields that it is hard to make a distinction. Also there are statistical fields which focus more on prediction the same way as ML does. For example, in geostatistics prediction is often the only goal, e.g. to predict heavy metal concentration across a domain. People accept that it is impossible to explain every bit of spatial variation and just model it by a gaussian process (the same gaussian process used in ML).

</comment><comment><USER37> Statistics + Computer Science = Machine Learning?

I think to me that's the case. But also, Machine Learning describes a computer science goal, to have a computer that learns certain or ideally all parts of its algorithm on its own, either from example or by experimentation.

It just so happens that some of the ideas from statistics lend themselves to help computer science implement such machines that can learn. One can imagine techniques for Machine Learning being discovered in the future which leverage ideas from other fields apart from statistics.

</comment><comment><USER38> One difference is that in Machine Learning you must think of data structures and algorithms. i.e. the practical ways to compute a model. How to represent and transform data while building a model. I think this is given less emphasis in statistics. Standard models are often used and theory is built around these different models. For example aspects such as power calculations for a regression model.

</comment><comment><USER39> Can someone correct me, if I'm wrong, but statistics doesn't allow Turing- complete models, right? Machine learning certainly allows that (for example an RNN).

<comment><USER40> This is kind of a fun question because it places the notion of "computational" computability (classical CS) alongside that of classical computability (finding a model by, say, minimizing an error which in turn boils down to calculus or linear system solution).

Long story short, because of the power of real numbers — with each one encoding an infinite sequence of bits — it's not clear that even "simple" computations important to elementary statistical models, like exponentiating, have low computational complexity. Computing, say, "e" can be done by a Turing machine, but it's nontrivial.

Another observation along the same lines is that some simple statistical models turn out to really complex — like "find a such that the model

      y = sin (a x)

fits sample data (x1, y1),..., where abs(y) < 1". This simple model class is well known to have infinite VC dimension, because a can be arbitrarily large.

Another way to say it may be that Turing completeness is not a very sharp tool to separate model classes.

I found this helpful: [http://www.cs.cmu.edu/~lblum/PAPERS/TuringMeetsNewton.pdf](http://www.cs.cmu.edu/~lblum/PAPERS/TuringMeetsNewton.pdf)

</comment></comment><comment><USER41> One will give you a grant (if you're lucky), the other will give you a multi- million dollar funding.

</comment><comment><USER42> the difference is branding

</comment><comment><USER43> Everything's a function, right?

</comment><comment><USER44> From Michael Jordan's reddit AMA

I personally don't make the distinction between statistics and machine learning that your question seems predicated on.

Also I rarely find it useful to distinguish between theory and practice; their interplay is already profound and will only increase as the systems and problems we consider grow more complex.

Think of the engineering problem of building a bridge. There's a whole food chain of ideas from physics through civil engineering that allow one to design bridges, build them, give guarantees that they won't fall down under certain conditions, tune them to specific settings, etc, etc. I suspect that there are few people involved in this chain who don't make use of "theoretical concepts" and "engineering know-how". It took decades (centuries really) for all of this to develop.

Similarly, Maxwell's equations provide the theory behind electrical engineering, but ideas like impedance matching came into focus as engineers started to learn how to build pipelines and circuits. Those ideas are both theoretical and practical.

We have a similar challenge---how do we take core inferential ideas and turn them into engineering systems that can work under whatever requirements that one has in mind (time, accuracy, cost, etc), that reflect assumptions that are appropriate for the domain, that are clear on what inferences and what decisions are to be made (does one want causes, predictions, variable selection, model selection, ranking, A/B tests, etc, etc), can allow interactions with humans (input of expert knowledge, visualization, personalization, privacy, ethical issues, etc, etc), that scale, that are easy to use and are robust. Indeed, with all due respect to bridge builders (and rocket builders, etc), but I think that we have a domain here that is more complex than any ever confronted in human society.

I don't know what to call the overall field that I have in mind here (it's fine to use "data science" as a placeholder), but the main point is that most people who I know who were trained in statistics or in machine learning implicitly understood themselves as working in this overall field; they don't say "I'm not interested in principles having to do with randomization in data collection, or with how to merge data, or with uncertainty in my predictions, or with evaluating models, or with visualization". Yes, they work on subsets of the overall problem, but they're certainly aware of the overall problem. Different collections of people (your "communities") often tend to have different application domains in mind and that makes some of the details of their current work look superficially different, but there's no actual underlying intellectual distinction, and many of the seeming distinctions are historical accidents.

I also must take issue with your phrase "methods more squarely in the realm of machine learning". I have no idea what this means, or could possibly mean. Throughout the eighties and nineties, it was striking how many times people working within the "ML community" realized that their ideas had had a lengthy pre-history in statistics. Decision trees, nearest neighbor, logistic regression, kernels, PCA, canonical correlation, graphical models, K means and discriminant analysis come to mind, and also many general methodological principles (e.g., method of moments, which is having a mini-renaissance, Bayesian inference methods of all kinds, M estimation, bootstrap, cross- validation, EM, ROC, and of course stochastic gradient descent, whose pre- history goes back to the 50s and beyond), and many many theoretical tools (large deviations, concentrations, empirical processes, Bernstein-von Mises, U statistics, etc). Of course, the "statistics community" was also not ever that well defined, and while ideas such as Kalman filters, HMMs and factor analysis originated outside of the "statistics community" narrowly defined, there were absorbed within statistics because they're clearly about inference. Similarly, layered neural networks can and should be viewed as nonparametric function estimators, objects to be analyzed statistically.

In general, "statistics" refers in part to an analysis style---a statistician is happy to analyze the performance of any system, e.g., a logic-based system, if it takes in data that can be considered random and outputs decisions that can be considered uncertain. A "statistical method" doesn't have to have any probabilities in it per se. (Consider computing the median).

When Leo Breiman developed random forests, was he being a statistician or a machine learner? When my colleagues and I developed latent Dirichlet allocation, were we being statisticians or machine learners? Are the SVM and boosting machine learning while logistic regression is statistics, even though they're solving essentially the same optimization problems up to slightly different shapes in a loss function? Why does anyone think that these are meaningful distinctions?

I don't think that the "ML community" has developed many new inferential principles---or many new optimization principles---but I do think that the community has been exceedingly creative at taking existing ideas across many fields, and mixing and matching them to solve problems in emerging problem domains, and I think that the community has excelled at making creative use of new computing architectures. I would view all of this as the proto emergence of an engineering counterpart to the more purely theoretical investigations that have classically taken place within statistics and optimization.

But one shouldn't definitely not equate statistics or optimization with theory and machine learning with applications. The "statistics community" has also been very applied, it's just that for historical reasons their collaborations have tended to focus on science, medicine and policy rather than engineering. The emergence of the "ML community" has (inter alia) helped to enlargen the scope of "applied statistical inference". It has begun to break down some barriers between engineering thinking (e.g., computer systems thinking) and inferential thinking. And of course it has engendered new theoretical questions.

I could go on (and on), but I'll stop there for now...

</comment><comment><USER45> The difference is in how the output is evaluated.

An ML model is evaluated empirically. You compare to real world results and get an accuracy measure. An ML models tells you what something _should_ be. And if it's a good model, you will get a pretty high frequency of that model telling you what the thing is.

Statistics does something entirely different. It tells you what something _could_ be.

If you flip a coin, an ML model will tell you if it's heads or tails. Statistics will tell you how often it will be heads or tails.

Another way to think about it is the difference between probability and likelihood.

Probability is measured by your theoretical priors and hypotheses. Likelihood is measured by the results of actual trials.

The probability of a fair coin landing on heads is .5

But the likelihood of that happening isn't actually .5 because pure frequentist probabilities depend on some fundamentally problematic things. Like a performative infinite number of trials.

The actual line between ML and statistics is really blurry because all useful statistical models are at least a little Bayesian. Priors get updated with each trial. This is essentially machine learning.

Outside of mostly bad/soft sciences (sociology, psychology, neuroscience, nutrition, and climatology are all pretty godawful about abusing classical statistics) pure frequentist statistics don't get used much because they are really only useful for getting papers published and generating squawking headlines.

Most useful statistical methods _are_ machine learning methods. Specifically, they are applied Bayesian methods with weak, randomized priors. Which is exactly what ML is.

I sound like I hate statistics. I don't really. ML models can do a bunch of wacky things. There isn't a coherent theory behind an ML model. You could point a very good classifier at your wife, and it might tell you [(bird,.1), (apple,.3), (woman,.9)]

That's not a realistic interpretation of what _could_ be. It just happened to get that correct.

A really excellent ML model in 2016 could've given the following result for president [(trump,.8), (obama,.6), (rock,.5)]. And after the fact when we can compare it to what happened, it would seem accurate.

But that doesn't tell us the range of possibilities in our future. The reality in 2016 was that we weren't going to elect a fucking rock as president. There was no chance of that. There is zero chance that if I point my camera at my girlfriend, she might actually be a potato. Yeah, the ML model might be right because it has guessed right, and that's often all we care about.

But if I need to know what my chances are of my girlfriend becoming my wife or the mother of my children. That's where we need statistics. An ML model can't have those kinds of priors baked in unless you force it. And if you do that you're just paying someone to do some really expensive Bayesian regression.

<comment><USER6> No, ML model is not evaluated empirically at all. Measures like accuracy and precision are not empirical. Neither is generalization as evaluated by using some model data set. These measures are _statistical predictions_ that may or may not be correct.

Failure rate in the wild is empirical. Accuracy in the wild, the same.

Since the designer does not have access to real data, they are actually not working empirically at all and as such you get common overfitting and methods that work well on model datasets but not on real data.

<comment><USER45> You know that data scientists have access to real data, right? It's not all just kids doing tricks on kaggle? That we have the ability to actually measure shit?

Do you think that all of this is a total joke? I mean, you're allowed to think that. But, umm, yeah. Real people do real things with real data. Believe it or not.

</comment></comment></comment><comment><USER46> statistics - descriptive, ML - <USER3>

</comment><comment><USER47> Machine Learning = Heuristics + Statistics

</comment><comment><USER48> In short:

Machine Learning - making predictions

Statistics - distilling huge amount of data into a few indicators

Machine Learning uses algorithms, optimization/operations research, differential calculus, probability and statistics as it sees fit.

<comment><USER49> Statistics makes predictions too. Indicators are usually predictions - a mean is called an _expected_ value for a reason.

I find the spectacle of machine-learning practitioners and statisticians furiously using different language to agree with each other quite amusing.</comment></comment>


Ask HN: Replacement backend for RRD - <AUTHOR> I am currently re-engineering an stats tracking, analysis, and alerting application which is based upon rrdtool.  There are currently roughly 20k data points being monitored every minute and pushed to rrdcached. I'm seeing a substantial performance impact as the quantity of data points grow, even though I am running it on a fairly big box, as the initial implementation of our app targeted roughly 1000 data points.
Ultimately I'd like to not stress about scaling the application to 100k data points.  I've thought of using Redis, PostgreSQL and Hadoop/HBase, all which are tackling different domain problems. One of my concerns is if I remove the lossy trending of rrd, the size of the data footprint is substantial at 20k data points (20k * 1440 * data lifetime * per-row-overhead).
Do you have any suggestions on a data backend, preferably something lossy like rrd? <comment><USER2> At a previous job we were using Ganglia (<http://ganglia.sourceforge.net/>) to collect both system metrics and custom metrics from our servers. At my current job, we still use Ganglia for redundant storage of our metrics, but then we feed the data into Graphite (<http://graphite.wikidot.com/>) for graphing. Graphite uses Whisper for it's storage mechanism. It's very similar to RRD. The author wrote a page about why he decided to write his own instead of leveraging RRD (<http://graphite.wikidot.com/whisper>). We've got over 60k metrics in our system; happy to talk shop if you want more info.

</comment><comment><USER3> I've worked a lot with Redis over the past 6 months (my own projects + analytics for yellowpages.ca) and not so much with rrd. Would you be interested in an email exchange?</comment>


TDLib – Build Your Own Telegram Client - <AUTHOR> https://telegram.org/blog/tdlib <comment><USER2> Unless they release the server code, no, you can't build your own "Telegram" yet. The original article's title was misleading, thankfully the HN submission title was amended to correct it. Still useful of course for anyone creating third-party clients, but an entirely different matter.

<comment><USER3> Aye aye! Both Telegram and Signal scare me.

We're building an MIT/ZLIB/Apache2 Open Source End-to-End Encrypted alternative that is a fully P2P decentralized app (dApp), see: [https://hackernoon.com/so-you-want-to-build-a-p2p-twitter- wi...](https://hackernoon.com/so-you-want-to-build-a-p2p-twitter- with-e2e-encryption-f90505b2ff8) !

Currently it supports signed/verified messages, haven't added cyphered/private messages yet to the API, but it is available in the SEA (Security, Encryption, Authorization) utility library underneath - but that should be coming soon, and of course love any contributions!

Also, the "server" has no special application logic on it, it is just another peer in the network that can help with WebRTC signaling or being a websocket relay. It cannot middleman the data, because every message is signed via the "client" (technically there are no server/clients, everybody is just a peer) using latest native Web Crypto API!

Here is a demo of an example social network app that is syncing character-by- character with cryptographically verifiable sources across a test network of real world devices: [https://youtu.be/C3akdQJs55E](https://youtu.be/C3akdQJs55E)

Again, all truly Open Source (ie, MIT/ZLIB/Apache2)!

<comment><USER4> Why does Signal scare you? All of their code is open source[1] and has had multiple public audits[2].

[1]: [https://github.com/signalapp](https://github.com/signalapp)

[2]: [https://eprint.iacr.org/2016/1013.pdf](https://eprint.iacr.org/2016/1013.pdf)

<comment><USER5> It requires you to provide a phone number (read disclose your identity) to use it. And its homepage has the word "privacy" repeated several times. It looks like a poor joke.

<comment><USER4> You can easily use a VoIP number to set up Signal if that's really an issue for you.

The alternative (using Telegram's homegrown and dubious cryptography and lack of actually encrypted group messages) is significantly worse. Here's a nice summary of the issues: [https://security.stackexchange.com/a/49802/43733](https://security.stackexchange.com/a/49802/43733)

<comment><USER6> VoIP is easily traceable too. Pay with a credit card for the number? Use Google Voice? Bam, they've got you.

</comment></comment></comment></comment></comment></comment><comment><USER7> There may be political motive to this.

Why is Telegram X issued by "Telegram Messenger LLP" while the original Telegram is issued by "Telegram LLC"?

Telegram LLC, which issues the original Telegram app on the App Store has been undergoing litigation with regards to its ownership. It is based out of Russia and a company called UCP with Krelim ties who 'bought' VKontakte has been suing Durov for full Telegram ownership pursuant to VKontakte 'purchase'.

Telegram Messenger LLP however is based in London and seemingly wholly owned by Durov crew. In 2014, Telegram Messenger LLP launched Telegram HD, a separate app from Telegram-LLC-issued-Telegram. Now it is launching Telegram X , another separate app (which was until very recently Challegram, an open source Telegram client, winner of Telegram contest, and purchased by Durov) .

Draw your own conclusions.

EDITED for clarity and details

[https://themoscowtimes.com/articles/the-telegram-lawsuits- ex...](https://themoscowtimes.com/articles/the-telegram-lawsuits-explained- pavel-durov-58989)

[http://www.ewdn.com/2014/07/29/pavel-durov-has-cloned- telegr...](http://www.ewdn.com/2014/07/29/pavel-durov-has-cloned-telegram/)

[https://rusletter.com/articles/ucp_requires_to_recognize_tel...](https://rusletter.com/articles/ucp_requires_to_recognize_telegram_as_the_property_of_vkontakte)

[http://www.frandroid.com/android/applications/securite- appli...](http://www.frandroid.com/android/applications/securite- applications/484643_telegram-fait-lacquisition-de-challegram-et-le-renomme-en- telegram-x)

<comment><USER8> > Challegram, an open source Telegram client

Is this true? I can't find the source anywhere. Not even a dated one, if they had re-licensed it.

</comment></comment><comment><USER9> I like how they do very good engineering (just straight C++, Swift, with little to no external dependencies) with a very small team of apparently very talented people. Most of their products are typically owned by one or two people. It's also pretty smart of them to make all of the clients available as open source and keep the server implementations private. This way they don't have to license QT for example.

<comment><USER10> Behold, 12283 lines of very good engineering:

[https://github.com/DrKLO/Telegram/blob/e9e40cb13ea942b148b25...](https://github.com/DrKLO/Telegram/blob/e9e40cb13ea942b148b259c083fb3364a0cd90ea/TMessagesProj/src/main/java/org/telegram/ui/ChatActivity.java)

<comment><USER11> As far as I can see, that's lots of code for generating a UI. Interesting that they do that instead of using the usual xml files for specifying the UI on android; the result is that there's lots of code, a number of statements for each single UI element, and all the listeners and such are declared inline.

The compiled code will be fine, probably. It won't be much different from that would happen if they used the standard method of xml files and separate classes for that stuff, and might even be slightly faster.

But it surely is less maintainable, and the code definitely doesn't look generated to me. Maybe if you count copy-paste as code generation.

I hope they don't use Atom as an editor, because it surely won't be able to cope with such a file. [/s; but might even be true. not tested.]

<comment><USER12> Less maintainable? That's a nightmare. As an Android developer who has struggled with couple of 4 digit lines long activity classes I literally shuddered.

This is as if someone used a tool to translate UI code from some other platform to Android (completely into Java) and forgot to refactor.

</comment></comment></comment></comment><comment><USER13> Telegram has a poor security record, with homegrown encryption algorithms and no end to end encryption by default.

<comment><USER14> You're just repeating something that you read on HN. Entirely false.

<comment><USER15> Are they allowing you to have accounts that don't depend on your phone number yet? I think last I checked you could only login with your phone number, which I consider extremely insecure, especially against a nation state agent (you'd think the Telegram team would care more about that, considering where they come from - and in fact, I believe there were some reports of Russian government hacking people like this with the help of national carriers, too).

<comment><USER16> > (you'd think the Telegram team would care more about that, considering where > they come from)

That assumes the Telegram team aren't part of the nation state apparatus, surely.

<comment><USER14> Do you have any reason to assume they are? Of course, other than the fact that the founder is _RUSSIAN_ _gasp_

<comment><USER16> I don't assume they are. But equally I don't automatically assume they aren't, either.

<comment><USER14> You're just evading my point. Implicitely or explicitely you must have a _working_ hypothesis. When you make decisions, you must assume one or the other.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> The iOS version of Telegram and Telegram X are currently not available in the Appstore.

It should be resolved soon: [https://twitter.com/durov/status/958990254396059648](https://twitter.com/durov/status/958990254396059648)

<comment><USER18> There should be a notification "your app is about to be removed, please contact us to solve the issue." (I'm assuming there wasn't because of other horror stories I've read.) If my business's service was suddenly down for 2-3 days due to some externality I couldn't immediately correct, it would be a death knell.

</comment></comment><comment><USER19> Telegram is such an awesome software, I feel it's such a shame it's not more popular.

</comment><comment><USER20> > P.S. Today, we're also introducing Telegram X for Android, a new official > app.

What is wrong with the current app? Or was that not official? I find it better than the Facebook Messenger (light or not) and it's definitely much better than the Hangouts app from Google.

<comment><USER21> They have had two apps (Telegram vs Telegram Desktop) on macOS too for a while. IIRC one is Electron-ish-based and the other is native Cocoa. The first one got features slightly faster but lacked Secret Chats IIRC.

<comment><USER20> I was actually positively surprised to see the Telegram application is qt based. Not sure which one is it of the two you mention, it's just called "Telegram", and is same on both Windows and OSX.

Explains why it doesn't take a minute to start and doesn't need a gigabyte of memory.

<comment><USER22> Also, I was very surprised to see that it was really QT based in the sense that there was no glue to an attached Chromium or something, just pure QT in C++.

The effort that went into that thing is amazing. It's fast on my 5 years old Pentium.

<comment><USER23> Which one should I be using? MacOS specific one feels polished whereas the Telegram for Desktop one doesn't have this strange issue where media sometimes never sends.

<comment><USER24> I prefer the macOS native client.

</comment></comment></comment></comment></comment></comment><comment><USER25> Title is misleading, I think. You can only build your own client.</comment>


Michigan's Ex-Gov. Snyder Knew About Flint's Toxic Water – and Lied About It - <AUTHOR> https://www.vice.com/en_us/article/z3bdp9/michigans-ex-gov-rick-snyder-knew-about-flints-toxic-waterand-lied-about-it <comment><USER2> Of course he was, the only way this was possible in the first place was by falsifying water test results. You don't just not notice that your city water supply is testing multiple points below the required pH level. And the only people who gain anything at all by falsifying water test results is the state government who forcefully took control of the Flint government via Emergency Manager in order to cut the budget. If they failed to cut the budget they would be called out because both the state and local populous voted against emergency manager takeover, so they forced a way to cut the budget, thinking they wouldn't ever get caught. But they were too stupid to realize managing the water's pH level is what prevents the entire water system from corroding away.

Where it not for the leaded joints spiking lead levels in people, they probably would have gotten away without anybody noticing. Nobody would have noticed anything was off until years later when the water system is leaking like a sieve with nobody to blame for it.

<comment><USER3> The Gov. is an ex-COO from Gateway. I guess this is what they mean by making government run like a business.

EDIT: Sorry, I had the wrong company down as Dell and CEO before. Don't post from memory folks.

<comment><USER4> It's worse than that: he was the ex-COO from Gateway (not Dell). They were essentially a Dell wannabe that was even more poorly run.

<comment><USER5> Gateway 2000 -1998 name change-> Gateway -2007 acquisition-> Acer

eMachines -2004 acquisition-> Gateway

Wikipedia is wrong: not Gateway in 2000, but 1997 saw several sub-$1k computers, and possibly the first was the Compaq Presario powered by the Cyrix MediaGX. I recall taking some community college computer course in sophomore(10)/junior(11) year of high school about 1994 and the lecturer predicting the sub-$1k computer.

[https://www.cnet.com/news/the-secret-history-of-the- sub-1000...](https://www.cnet.com/news/the-secret-history-of-the- sub-1000-computer)

</comment></comment></comment></comment><comment><USER6> And he won't suffer any severe consequences because of it...

<comment><USER7> We wish. I just had a similar political discussion with a close friend. Why is there no accountability in politics/government? How could we go about changing that?

<comment><USER8> You can't. The political machines of these areas need to have significant political issues (think bankruptcy and stewardship to a larger organization Detroit being taken over by the state for example) to cause a change.

Places like Illinois are financially insolvent and are cutting back on all sorts of things.</comment></comment></comment>


OpenPDF – A free Java library for creating and editing PDF files - <AUTHOR> https://github.com/LibrePDF/OpenPDF/ <comment><USER2> I was excited to hear about a PDF library not based on iText but alas, this is just another iText fork, in fact the full fork chain seems to be:

        LibrePDF/OpenPDF
        forked from rtfarte/OpenPDF
        forked from kulatamicuda/iText-4.2.0
        forked from daviddurand/iText-4.2.0
        forked from ymasory/iText-4.2.0

<comment><USER3> I had that same frustration this week. Trying to generate geotiffs from arrays in Python. I found four libraries, all of which just wrap GDAL, all of which have the same quirky issues. Alas. :)

<comment><USER4> If you have a 1:1 mapping between array cells and pixels (no geographic transformations), just use any image format as intermediate. But I am interested what quirky issues you encountered. I found rasterio very nice to work with using numpy arrays.

<comment><USER3> Super obscure issues. When I try to use GDAL to build a geotiff from a bumpy array I get an underlying c++ crash of some sort. I'm not an expert. But it happens only when I try to do it within QGIS. It's something about the compatibility of numpy and gdals underlying c++ APIs.

I wish I was smarter and could debug further. Issue doesn't happen standalone but happens in a qt5 plugin for GDAL. I was hoping to find any other implementation of array to tiff in order to avoid this crash. Alas.

<comment><USER5> tried this outside of QGIS? pip install rasterio, etc.

We heavily use GDAL/rasterio/numpy at work with no real problems.

<comment><USER3> Everything works outside QGIS. There's just something about the environment. I have 7 lines of code that crashes my QGIS every time.

<comment><USER4> Please share those for investigation!

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> If you're after an open source PDF library in Java, check out Apache PDFBox. It's actively maintained and has a ton of features.

[http://pdfbox.apache.org](http://pdfbox.apache.org)

<comment><USER7> Would recommend!

Used PDFBox a couple of projects ago. It was great! Very nice API, works like a charm.

<comment><USER8> I encountered PDFBox for the first time last week and although it looks promising I was discouraged by the lack of documentation. Was there something you found particularly useful, or should I just start browsing the source? Thanks!

<comment><USER7> Yeah, current docs might not be the best, I think they are still working on it.

API was pretty intuitive, plus some googling and reading stackoverflow. And yes, browsing the source. I don't remember encountering any major problems with it.

</comment></comment></comment></comment><comment><USER9> Why is this being upvoted now? People have been forking iText for years, ever since the license was changed to AGPL. This repo doesn't even look very active and the last release was last year.

</comment><comment><USER10> People just can't stand the idea of writing a Forth parser, I guess lol

[https://www.prepressure.com/postscript/basics/history](https://www.prepressure.com/postscript/basics/history)

<comment><USER11> FORTH is a wonderful language, I've written a couple of toy implementations. I just wish I could use it for something "real".

<comment><USER10> Same here! [https://github.com/pmarreck/elixir- snippets/blob/master/rpn_...](https://github.com/pmarreck/elixir- snippets/blob/master/rpn_forth_thing.exs) I found it fun, and it was actually super easy to implement a stack-based language interpreter in an immutable pattern-matching language like Erlang/Elixir.

It's incomplete because I haven't yet figured out how to define the defining character ":"...

<comment><USER12> Back in the C64 days there was a Abacus Forth but it cost was pretty high if I remember right $225???? I got a copy from the software store I work with as a teenager and got one at cost. It was great after coming from basic > assembly (Talk about your learning curves).

</comment></comment></comment></comment><comment><USER13> Is the LGPL part of the license actually LGPL 2+, 2.1+ or 3.0+?

At least some of the file headers say 2+, the readme says 3.0+, but this note was added after the fork, with commit [https://github.com/daviddurand/iText-4.2.0/commit/312abf7b12...](https://github.com/daviddurand/iText-4.2.0/commit/312abf7b1214149406db763829d8dad377061152#diff-04c6e90faac2675aa89e2176d2eec7d8) so it may be in error?

If it was in error, does this affect any merged code pieces/PRs submitted/merged after the readme was changed, or do the license headers per- file take precedence?

<comment><USER14> I have some experience with iText, so possibly I can answer. iText never released a version 4.2.0 for Java. The original forker misunderstood iText versioning and used the number of the C# version. At the time, iText versions for Java were 2.x, and C# were 4.x The code in the forked version is actually the last release of the Java version of iText 2.x, which was the last Java release under the LGPL/Mozilla license.

Thankfully, iText later coordinated release numbers for both platforms to 5.x numbering when they changed the license to AGPL.

So, you should be safe using this version of iText under LGPL or Mozilla, but it's a fairly old release of the library. If you're just using iText for your hacking projects, you're probably better off going with a more recent version.

</comment></comment><comment><USER15> I hope somebody makes an Eclipse plugin out of this, and integrates it into the TeXlipse LaTeX IDE. It is a sad state of affairs that this aging project (last release was in 2011) is still the most versatile TeX editing environment. For example, it is the only one that I am aware of, beyond command-line editors, that that lets you open up an arbitrary number of views of the same file, which is often needed in large TeX projects. Its current PDF viewer (Pdf4Eclipse) is broken on Hi-DPI displays.

</comment><comment><USER16> LibreOffice Draw does a nice job at editing PDF files as well

</comment><comment><USER17> Where is the documentation?

</comment><comment><USER18> Can this convert a html/css document to PDF? or is chrome headless(ignoring overhead) the way to go these days?

<comment><USER19> Prince (commercial) is outstanding, and supports a wide array of issues specific to printed media.

<comment><USER20> $495 for desktop license and $3800 for server. I would love to have something this for the times I could use it, but realistically that's only once or twice per year.

I feel the same way about Adobe's Creative suite subscription. I use it's unique capabilities about once every 6 months. I wish there was a one week license.

</comment></comment></comment><comment><USER21> Does it support Unicode in PDF forms? Feature that poppler still lacks (in 2018!).

</comment><comment><USER22> How well does it handle merging PDFs? I've not found an efficient way for merging many small PDFs in pdfbox/itext without cranking up memory settings.

</comment><comment><USER23> That so bad about AGPL, that require a fork?

<comment><USER24> Imagine this scenario. You want to write your own code ProductX and link it with some GPL code. This brings ProductX also under the GPL license.

You can put ProductX on a server and never actually distribute ProductX to anyone. By not distributing ProductX, you don't have to distribute the ProductX source code to anyone -- although technically it IS under the GPL. Since you're not distributing the binary, you don't have to distribute the source. Yet the public can interact with your server and make use of ProductX's services.

The AGPL prevents that. If you write your code ProductX and link it with AGPL code, then ProductX comes under the AGPL license. But now, merely letting the public interact with ProductX on a server requires you to distribute the AGPL source code to ProductX. Now anyone else has the ProductX source code and can compete with you.

The AGPL is also a way for the author of an AGPL library to make money. A library such as iText. If you want to use iText with a proprietary ProductX, then you need to buy a separate commercial license for iText.

If the developer's intent in ProductX is to keep the source code private, then the developer cannot link ProductX with any AGPL licensed code such as iText -- unless the developer is willing to pay for a commercial license to iText.

A developer should never have code that they didn't write themselves, such as a library, unless that library is under a license that the developer can always remain in full compliance with. That includes proprietary and commercial libraries as well as open source libraries. While AGPL is "open source" if you cannot comply with the terms, then don't use it.

Finally, avoid any code you find on the internet that has "no license". If you use such code, the copyright owner of that code could sue you for copyright infringement. "But wait!" they say. "I would never sue anybody, I just want as many people as possible to use my code, so I don't put it under any license." I say: If you're not going to sue me, then put that promise in writing. it's called a license. If you're such a good guy and aren't going to sue me, then put it in writing like all other open source licensed code.

<comment><USER25> Thanks for the explanation! Does anyone actually release code due to this license (which they wouldn't have released otherwise)? I'd imagine generally if a commercial developer saw a license like that they would just choose not to use the AGPL code. I know I won't use any code or libraries which place these sorts of obligations on me as a developer, unless it's in a product that I was planning to release the source of anyway.

<comment><USER26> If a commercial developer saw a license like that... they could pay for a commercial licence. Presumably a commercial developer thinks software is worth paying for.

</comment></comment></comment></comment><comment><USER27> Why the fork? ... oh LICENSE change

</comment><comment><USER28> Took a glance at this one not so long ago - quite a nice tool to auto-generate PDF docs yet not that good at editing. For such a purpose I still use this PDFfiller [https://w9.pdffiller.com](https://w9.pdffiller.com) but it's a paid one and written in JS. Nevertheless it's well-crafted enough to do all the works on visuals and overall layout</comment>


Chart of YC companies' hosting decisions - <AUTHOR> http://jpf.github.com/domain-profiler/ycombinator.html

<comment><USER2> It's surprising how many cells of that table say "GoDaddy."

<comment><USER3> s/surprising/depressing/

GoDaddy's "used car lot" salesmanship makes me loathe the company. I'm in process of transferring all my domains elsewhere.

<comment><USER4> Likewise. Have a fairly budget-price but not-so-shady registrar in mind?

[edit] WTF guys, why am I being modded up? I asked a question without providing additional insight - this doesn't deserve +points.

<comment><USER5> I got tired of GoDaddy a while ago and moved all of my domains to NameCheap. They've been great, although private registrations cost a couple dollars after the first year.

Edit: <http://www.namecheap.com>

<comment><USER6> I second namecheap. I've used them for several years and have around 30 domains registered there (including Divvyshot). It's worth paying a little more for a more intuitive interface. I've never felt afraid of being tricked into buying something I didn't want.

Another suggestion: Use everydns.net for DNS. It's donation-based (<http://www.everydns.com/donate.php>), powerful, reliable, and run by David Ulevitch of opendns.com.

<comment><USER7> I will never ever trust _anything_ even remotely associated with OpenDNS or it's progenitors.

They're not Open in any way, and their service is not DNS either. They're just an 'underdog' Verisign -- same bullshit, different self-image.

</comment></comment></comment></comment></comment></comment><comment><USER8> I'm surprised that Linode isn't on there at all, although it's good to see BitPusher get mentioned for their work on Wufoo (good bunch of guys over there).

</comment><comment><USER9> Email should include outgoing. Tipjoy uses <http://authsmtp.com>

Google hosted email for outbound kinda stinks

<comment><USER10> I'm just about to switch to Google for mail hosting, and your comment worries me a little.

What kind of trouble have people had with Google's outgoing mail servers?

<comment><USER11> I think they might just be talking about using it for mass-outbound messages from an app, not for actual human-to human messages. (Never had a problem with Gmail for my personal/work e-mail, anyway.)

<comment><USER9> You're right. We still use it for inbound, and human based outbound. We do things like send monthly reminders to pay for Tipjoy, meaning lots of emails sent regularly.

<comment><USER12> Is it possible to have multiple email providers on a single domain? How do you do that?

</comment></comment></comment></comment></comment><comment><USER13> HN has had several slicehost vs. linode threads. Comments seems they were an even match (different but even). So why are there no linode users in this list?

I'm asking as a new startup eyeing both as potential host.

<comment><USER14> also, bumptechnologies.com is using linode as DNS. Doesn't linode only offer DNS to domains hosted there?

<comment><USER15> Correct, both our website and our matching infrastructure is hosted at Linode.

</comment></comment></comment><comment><USER16> This is pretty much the exact breakdown of information I have had in the back of my mind to find for quite a while. It's very helpful — perhaps the most useful thing I've seen posted on HN since I started reading. (At least, most useful for me!) Of course, I wouldn't want to jump off a bridge just because everyone else was doing it, but researching all the available providers for all these hosting requirements is a potentially endless job. It's nice to see the decisions other companies have made, to crowdsource it a bit and narrow down the options.

<comment><AUTHOR> Thanks Cody! I wrote the code that generated that report for the same reason: I wanted to be able to research what decisions other domain owners have made. I'm happy to hear that you found this information useful too!

Did you see the code I put on GitHub? <http://github.com/jpf/domain-profiler>

<comment><USER17> Great work. I immediately stored it to help my future hosting decisions. Small feature suggestions: include percentages in the pie charts to make them more readable. (Especially for colour blind)

</comment></comment></comment><comment><USER18> Could somebody please explain why you would use a DNS provider that is not your web host?

I've always just used the host's DNS servers for my domains, but I'm curious as to the decision making of others... are there real concerns that would lead you to separate these services?

<comment><USER19> Many places you lease a server from do not typically provide DNS servers. Also, you get the advantage of always-speedy, geographically diverse name lookup.

When I leased my own server, I had a painful experiment with hosting my own DNS on the same box. After I gave up on that, I admit I snooped on the provider YC used and have never had a problem.

</comment></comment><comment><USER20> Interesting, but in the first two there are slices labeled both "other/self hosted" and "self hosted."

<comment><AUTHOR> "self hosted" means that the IP or Whois lookup matched the second level domain - a high degree of confidence that the service is self hosted.

"other/self hosted" is basically a catch all. Originally just labeled "other", I added "self hosted" later since that is often (but not always) the case.

</comment></comment><comment><USER21> does anyone have experience with <http://www.fdcservers.net/> ? I was looking around for unmetered 100MBps and what I am looking at looks really good and cheap. I have a dedicated server at local ISP and one at theplanet.com - I am not really satisfied with theplanet.com one and prices for local ISP (though superb ping, of course) are a riot.

</comment><comment><USER22> Wow, nothing on Google App Engine or Azure?

<comment><USER17> Well, Google App Engine isn't exactly suitable for innovative applications since you're locked to what their environment offers you (e.g. no background processes!?). Azure I don't know, is that generally available yet?

<comment><USER23> App Engine does offer you cron jobs now, at least.

</comment></comment></comment><comment><USER24> I wonder why Google App Engine doesn't show up even once. I thought it was a pretty viable and flexible option.

</comment><comment><USER25> Cool! Looks like most folks using GoDaddy as registrar switch to some other service (typically web host?) for the DNS. How come?

Are any of the companies with Softlayer using their Cloudlayer offering? If yes, how do you find it?

</comment><comment><USER26> Does choice of hosting provider figure in selection of startups? Will one startup get preference over another otherwise equal startup if the former is (not) using a particular hosting provider/mail/registrar?

</comment><comment><USER27> Pie charts considered harmful:

<http://en.wikipedia.org/wiki/Pie_chart#Discussion_on_use>

<comment><AUTHOR> The code is on github. Send me a patch ;-)

</comment></comment><comment><USER28> It's not surprising how many cells say Google.

</comment><comment><USER29> I'm curious. Does hosting with Amazon imply EC2? Or S3 static files? Or some other form of hosting I am unaware of?

<comment><AUTHOR> It (in almost every case) implies hosting on EC2. I haven't added detection of image host to my script yet.

</comment></comment><comment><USER30> Nice! I'd also be interested in the techology decisions (languages, frameworks..).

</comment><comment><USER31> Umm, what about multiple CN SSL certificates?

<comment><AUTHOR> I didn't think to check for those. Great idea, thanks!

<comment><USER32> I think multi-CN is pretty rare. It's Subject Alt Names to look out for!

<https://owa.central1.com/> for example.

</comment></comment></comment><comment><USER33> I noticed a bunch of people are using enom for email. I assume this is because of their hosted exchange. Does anyone have any experience with enom's standard hosted exchange and using it using mobile devices such as BB/iphone

</comment><comment><USER34> oh noes, you made a pie chart - the most useless chart type of all!

<comment><AUTHOR> <http://laughingsquid.com/accurate-pie-chart/>

<comment><USER34> thank you! morons who down-voted my comment don't seem to be able to understand...

<comment><USER35> Making it as a joke didn't work on this forum because people generally downvote short jokes they don't get. The rule about humor here that seems to work here is to make sure it's actually funny.</comment></comment></comment></comment>


Original Apple CEO talks about most creative use of Apple II - <AUTHOR> http://www.businessinsider.com/apples-first-ceo-michael-scott-2011-5 ...I guess the best example was that we had a contest very early on at Apple where we awarded an extra Apple for the unusual use of the machine.
The winner, and what we never realized then, and what we don't think is still true now, is you don't realize how many different ways it can impact you.
The first use was the person that swore by it, a new father who had a new son that the colic, and took the original Apple II, did his own software, hooked up a washing machine motor, the tilt mechanism from a pinball machine, a microphone to the crib. So when the kid was getting restless, the machine would come on and rock the bed.
He actually had statistics to show how much more sleep he and his wife were getting because the kid would partly wake up, then go back to sleep. And that won first prize for a different use of the Apple, and I think that's still true today. People are still finding uses. You put it out in one area, and then it gets out into a lot of others... <comment><AUTHOR> ...we had a contest very early on at Apple where we awarded an extra Apple for the unusual use of the machine.

The winner, and what we never realized then, and what we don't think is still true now, is you don't realize how many different ways it can impact you.

The first use was the person that swore by it, a new father who had a new son that the colic, and took the original Apple II, did his own software, hooked up a washing machine motor, the tilt mechanism from a pinball machine, a microphone to the crib. So when the kid was getting restless, the machine would come on and rock the bed.

He actually had statistics to show how much more sleep he and his wife were getting because the kid would partly wake up, then go back to sleep. And that won first prize for a different use of the Apple, and I think that's still true today. People are still finding uses. You put it out in one area, and then it gets out into a lot of others.</comment>


Juan Enriquez: Beyond the crisis, mindboggling science and the arrival of Homo evolutis - <AUTHOR> http://ted.com/index.php/talks/juan_enriquez_shares_mindboggling_new_science.html <comment><AUTHOR> "What I'm going to try and do in the next 17 and a half minutes is talk first about the flames. Then where we are in the economy. I'm going to take three trends that have taken place at Ted in the next 25 years and try and bring them together and give you a sense of what the ultimate reboot looks like. Those three trends are the ability to engineer cells, the ability to engineer tissues, and robots."</comment>


Alas, HyperCard (1998) - <AUTHOR> http://db.tidbits.com/article/5155

<comment><USER2> I have to credit HyperCard/HyperTalk with giving me my start in programming. Although I started programming in BASIC on a Commodore setup at a pretty young age, it was HyperCard along about the seventh grade that got me thinking about more complex concepts.

When I was a high school freshman about 17 years ago (yuk), I wrote a complicated self-modifying program in HyperCard. I wanted to write a program that could understand human sentences and reply accordingly, and figured that self-modifying code would be the best way to go about that. Of course it never quite succeeded, but the program was stable and could run for a very long time, and gradually improve. Debugging it was a real adventure.

There was a way to add 'external resources' to HyperCard stacks, and then call them from HyperTalk. I can't remember what they were called now, but there were 'functions' and 'procedures' I think. With those, you could get color working, or do really almost anything on a Mac, all from HyperCard.

They were compiled bits of code, so naturally my interest shifted towards making those instead. I wanted to try writing my own, instead of browsing eWorld and the nascent AOL for stuff written by that Peter what's-his-name.

Somewhere I was told that they were written in Pascal, so I got my hands on a copy of THINK Pascal, and started playing with that. It came with a really wonderful hyperlinked library of the MacOS toolbox. From there, to C, to assembly, and on and on.

I still have a real soft spot in my heart for HyperCard. If there were a modern equivalent, it would be perfect for teaching programming to beginners.

EDIT: I have an idea. If anybody happens to have an old pile of HyperTalk laying around, please email me. Address is in my profile.

<comment><USER3> I also credit HyperCard for really getting me started. Yeah, I learned a little BASIC, but I could write real programs in HyperCard. It was amazing and breathtaking. HyperCard was to the Mac as Applesoft BASIC was to the Apple II.

Since most stacks were interpreted, not compiled, you could just look at the source. HyperTalk is why I learned how to read code. I still remember when I finally realized what HyperCard meant when it said 'too much recursion'. It was like a flashbulb going off in my brain.

The modern equivalent, in terms of 'wow, how do I do that?' is obviously Javascript, but HyperTalk was just easier to read and get started with.

A minor happy note: iOS autocorrects the capitalization in HyperTalk and HyperCard. That just made me very happy.

</comment></comment><comment><USER4> Interesting that <http://apple.com/hypercard> gets redirected to wikipedia article (<http://en.wikipedia.org/wiki/HyperCard>).

</comment><comment><USER5> Doesn't Apple realize they'd hit another big Home Run if they brought HyperCard back for the iPad, iPhone, & iTouch. It would a huge hit. Well, at least with me :-)

<comment><USER6> They already rejected RunRev(<http://www.runrev.com>) for the iPhone and iPad. The modern day descendant of Hypercard. They have no interest in bringing back Hypercard.

RunRev's CEO thoughts: [http://www.runrev.com/company/runrev-blog/revmobile- and-appl...](http://www.runrev.com/company/runrev-blog/revmobile-and-apples- iphone-sdk-agreement)

</comment></comment><comment><USER7> I first learned programming using Hypercard. This was back in the mid 90s. It was huge fun. HyperTalk was close to English and was the easiest programming language I've ever encountered. From HyperTalk I branched out to other script languages - AppleScript, PHP, Perl. I did all kinds of programming for 10 years, before I began learning "real" languages like Java.

In general, script languages make it easier for people to get started with programming, but HyperCard was an environment that simplified a great deal. It was even easier to learn that your average script language. It made assumptions for you, to help you get started. Yet it was amazing how much you could do with it.

I think HyperCard got flattened by the Web. I do not think it is coming back. Nowadays if people want an easy-to-learn environment that allows them to build a wide range of software, there is the web.

If there is a market niche to be filled, it might be a package that tames the Web somewhat. The web is a little too wide open, and the beginner is confronted with a lot of choices before they have the knowledge to make those choices. HyperCard reduced the number of decisions that you had to make up front. Yes, there are packages like Dreamweaver and Flash that help with web and multimedia production, but they do not hold your hand like HyperCard did. Both of them are for professionals, they are scary if you are first learning. A package that makes a lot of assumptions for you, to get you started, might find a niche, even now. And even more so for something like the iPhone, where it is a little easier to make starting assumptions.

For me, the story of HyperCard is one of missed opportunities. In 1995 I went to the big tech/photo expo at the Javitts Center in New York City. At the time, interactive multi-media CD ROMs were suppose to be the next big thing (there is a scene in Jurassic Park, which came out in 1994, where the young girl expresses excitement over "an interactive CD ROM". After the web took off, it was hard to remember why CD ROMs were suppose to be a big deal). Macromedia was there, promoting Director. I looked at Director, but I was sure that this niche was going to end up being owned by HyperCard. I felt that anything so easy to learn was going to get a huge following, and then a lot of those people would evolve in serious professionals who would use the tool they were familiar with to keep building stuff (which is what made PHP successful, just a few years later).

Macromedia adjusted quickly to the Web. They took Director and stripped it down and came out with a simple product called Flash. Which eventually grew up to take over the space in exactly the way that I thought HyperCard was going to do.

Apple had something important in their hands in 1995. But they made bad decisions. Really, it is remarkable how many bad decisions they made in the mid 90s.

</comment><comment><USER8> This article is a great insight about what was hot 12 years ago - a little time capsule, if you will. The web (or browser to be precise) is mentioned only in passing; the author could to not have seen that it would be _the_ platform which most of the software innovations (and a significant number of social ones) in the following decade would rely on.

<comment><USER9> A time capsule into the mind of a 'legacy system' programmer. He's bemoaning Apple finally dropping the last vaporous attempts to keep Hypercard relevant, but the market had moved on years earlier.

What was actually hot 12 years ago: Java applets, Macromedia Flash, early dynamic HTML browsers, web scripting languages (ASP/Perl/etc), dot.com stocks. People did understand where the future lay.

</comment></comment><comment><USER10> For a long time I've had a wonderful HyperCard application (not written by me), and I've dreamed for years how I would just love to be able to disassemble it and re-write it in a modern language, like Python. Unfortunately, I have no idea where to find something to even run it with, much less look into the internals. Any ideas?

<comment><USER11> Others have mentioned RunRev (<http://www.runrev.com>); there's also TileStack (<http://tilestack.com>). I'm not sure if RunRev will do anything for you, but TileStack seems to have limited facility to run existing HyperCard code; see [http://tilestack.com/stacks/?search=%28tags%3Ahypercard%29&#...</a>.

</comment></comment><comment><USER12> Just curious -- is there some context to this twelve-year-old story?

I miss Hypercard too, but I had been building stuff on the web for four years by the time its demise became official.

<comment><USER13> I don't know what's up, but in the last week I've seen several references to Hypercard out of the blue. A modern Hypercard for the iPad or iPhone would rock.

</comment></comment><comment><USER14> I don't know if many of you have looked closely at Apple's Keynote, but it's a stones throw from being an update to hypercard. It just needs to add support for scripting and you're done.</comment>