The Forgotten Solution: Superdeterminism - <AUTHOR> https://backreaction.blogspot.com/2019/07/the-forgotten-solution-superdeterminism.html <comment><USER2> "The Facts" basically say that among the statements, "Your experiment design isn't predestined by the universe to make it accidentally seem like quantum mechanics is true," "the state of the universe today is all you need to know to predict the state of the universe tomorrow," and "an experiment only has one outcome," there is at least one lie. If the first one is a lie that's superdeterminism, the second one the Copenhagen interpretation and the last one Many Worlds. What bothers me about getting philosophical is, philosophers will attempt to choose one or more based on intellectual aesthetic criteria that we developed from the womb onwards in the macroscopic world, while in reality the only legitimate answer is "we don't know." I think that is broadly speaking a problem that hampers the effectiveness of philosophy, there is not enough willingness to say "the present information does not permit a conclusion."

<comment><USER3> This comment started out good but then stumbled in to some odd critique of philosophy as being incapable of dealing with unknowable things when indeed that seems to be perfectly within the purview of philosophy.

Humes Problem of Induction, for instance, is exactly an example of philosophical practice grappling with these unanswerables.

<comment><USER2> So, Hume didn't stop ethical philosophers from trying to derive morals, he only stopped some of them. Wittgenstein and Borges didn't stop philosophers from trying to "beat" language games using only language: they only stopped some philosophers. I'm not saying that there aren't visionary heroes who realize that some discussions aren't going to go anywhere for fundamental reasons; instead, I'm highlighting the fact that even when they do, "all of philosophy" almost never reaches a consensus about quitting the debate. In math, when they deduced that the Axiom of Choice was always going to be an axiom, everybody quit looking for ways to confirm or refute it. I think it's a weakness of philosophy that similar things can't happen.

<comment><USER3> Hume's problem of induction is not about deriving moral principles (I believe you got it confused with the "is-ought" dichotomy.) Most philosophers have largely given up on giving a rational deductive basis for why we should believe in induction, so if anything that seems to be a perfect example of what you're describing.

I don't think the nature of quantum reality is anywhere near as settled. For decades, we thought that it was impossible to test local hidden-variable theories. Thank god some people were still working on the problem!

</comment></comment></comment></comment><comment><USER4> I don't think the problem with superdeterminism is the lack of free will, but with the way it doesn't really give you anything mentally to work with. It posits some early state from which everything could be deterministically extrapolated... except that state is both very complicated and completely hidden. It takes all of the probabilities and shoves them in a black box and says, "The answers exist, and they're in there. But you can't actually look in the box for the answers. You have to go do the experiment and wait for the speed of light to propagate the answer to you."

Like all interpretations, it's mathematically equivalent to any other. It's just a question of what helps you think about the problem, and I don't think many people find it very edifying. You can replace the box with a random number generator, which is at least small enough to fit in your pocket. The superdeterminism box appears to have been crammed full of untold centillions of answers... none of which are accessible beforehand.

If there were reason to think that the superdeterminism box were somehow smaller -- if it all really came down to just one random bit, say, that had been magnified by chaotic interactions to appear like more -- that would attract some attention. And I suppose it would be conceptually testable, by running Laplace's demon in reverse, except that that's not possible either from inside the universe.

So it doesn't really come as a surprise that superdeterminism falls behind MWI or Copenhagen or even pilot wave, because each of those hands you something that you can use to mentally organize the world. Superdeterminism just seems to hand you a catchprase, "As it was foretold in the Long Ago -- but which I just found out about".

<comment><USER5> Whats wrong with a pseudo-random number generator? You start the universe with 1 million random bits and then just iterate your function on them. How would we detect repetition at the 2^1 mil level? Maybe the universe would repeat itself after a while, but how would we know?

Superdeterminism also plays nicely with the simulation hypothesis. You seed the virtual machine with some randomness and the physical laws and then you run the simulation.

<comment><USER4> I don't believe you'd need even a million random bits. It's conceivable that only a few random bits are actually required, and then let iteration take care of the rest.

There's nothing wrong with that. I just don't think people find it very useful as an organizing principle, so it doesn't attract a lot of attention.

</comment></comment></comment><comment><USER6> The thing about superdeterminism is that it's only interesting if you want to argue philosophy. If you're dealing with hidden variables (or even measurement errors) the only practical tool in your box for handling them is probability distributions.

So either way, you've got a probability distribution. And at this point people just apply Occam's Razor and get on with their lives. You can theorize an infinite number of systems that work exactly like the real world. The question is whether they're useful.

<comment><USER7> Could an untestable underlying theory inspire new models and implications that are testable?

<comment><USER8> In a general case, yes. This happened recently with gravitational waves. The construction of multiple gravitational wave detectors across the planet allowed us to test a previously untestable hypothesis about the number of dimensions that gravity can act in.

[https://www.youtube.com/watch?v=3HYw6vPR9qU](https://www.youtube.com/watch?v=3HYw6vPR9qU)

<comment><USER9> > allowed us to test a previously untestable hypothesis

So it is testable now? Is everyone here using the word "untestable" in the same way, i.e. "untestable today with the current state of the art, but it might be tomorrow" vs. "untestable ever, as a principle, even with perfect tech"

<comment><USER10> Untestable is a layman's word, everyone here should be saying "falsifiable".

Many Worlds isn't falsifiable and hence untestable; but untestable can also mean not possible to test with current tech. Falsifiable is more accurate here.

<comment><USER11> "Falsifiable" is a fantastic term but the philosophy of science did not stop with Popper, and it's not the only term we have available to discuss competing theories.

<comment><USER10> What other terms would you use? "Falsifiable" is more accurate that "untestable"?

<comment><USER11> "More accurate" is a good way to phrase it. "More accurate" also describes GR when you compare it to Newtonian mechanics. Strictly speaking, Newtonian mechanics is _false_ and has been _falsified_ if you subscribe to the Popperian view. However, if Newtonion mechanics is false, is it useless? No.

As it turns out, there are a few problems with the epistemology of falsifiability. The main ones I can think of are:

1\. Duhem-Quine problem - there is a large number of auxiliary hypotheses under test in any experiment, in addition to the primary hypotheses. These very numerous and we need a framework for deciding how to apply the results of our experiment to the many hypotheses.

2\. Statistical claims may be unfalsifiable. Consider a theory that claims that a coin flip has a 50% probability of being heads... how does one falsify this? One can't strictly falsify it, but you can show that the evidence is unlikely given the hypothesis. So we need some framework that connects statistical and probabilistic evidence to our knowledge of the world.

Or in summary, the problems with falsifiability are that falsifying a theory doesn't give us the information that we want, and it's impossible to falsify many theories. To abuse analogies, falsifiability is kind of like trying to cross a river with your car, when there's no bridge and the car won't start.

One approach other than Popperian falsifiability is a Bayesian system of belief and likelihood. This only one direction that the philosophy of science is exploring, but it is probably the one most familiar to HN readers.

<comment><USER10> Amazing, I appreciate the depth you went in and will be reading more on these topics. Thank you.

Any resource you could recommend for Bayesian system of belief? I understand Bayesian probability/math but I haven't explored it as a philosophy.

<comment><USER11> I don't have resources for Bayesian systems of belief. I do like _Error and the Growth of Experimental Knowledge_ by Deborah Mayo, but the book is a bit intimidating. I admit I haven't finished reading it, either.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> In Superdeterminism each time a particle has to collapse, instead of rolling a dice it looks into a secret table of hidden variables that was calculated at the beginning of the universe. The table was calculated carefully so the apparent random choices follow all the laws of quantum mechanics, and the results are equivalent to what you would expect if any of the other interpretations where correct.

To calculate this secret table you must simulate all the interactions and path in the universe until it ends, because you must know which particles will be entangled, which result will have the "random" generator in the experiments, ...

So the universe is only a movie that follows the random choices made at the beginning of the universe. But the choices are not arbitrary, they have the correct values so when the events really happen they follow the laws of physics. For example, the random choices at the beginning of the universe make it look that you can't transmit information faster than light.

Physics study the laws of the real universe, but we can redefine Physics as the study of the laws that study the random number generator. Both real- Physics and initial-rng-Physics follow special relativity. Bot agree about QM. Both agree about the Bell inequality.

So with Superdeterminism we solve the problem of QM in the real word, because everything we is already determined. Now the problem is how the RNG at the beginning of the universe work to simulate QM and all the other effects. Let's call the study of the RNG Physics. Now the problem is as hard as before Superdeterminism.

<comment><USER13> What you describe is not superdeterminism, but a replay of a non local theory. The important part happens in the first run when you calculate the table.

What superdeterminism says, is that there exists local and deterministic evaluation rule that will compute consecutive states of the universe, but simply because of the way the rule works experimenters far away end up always choosing the experiments that yield correct results.

Superdeterminism is unpopular because the existence of such evaluation rule seems very unlikely.

<comment><USER12> From the article:

> _Where do these correlations ultimately come from? Well, they come from > where everything ultimately comes from, that is from the initial state of > the universe. And that's where most people walk off: They think that you > need to precisely choose the initial conditions of the universe to arrange > quanta in Anton Zeilinger's brain just so that he'll end up turning a knob > left rather than right. Besides sounding entirely nuts, it's also a useless > idea, because how the hell would you ever calculate anything with it? And if > it's unfalsifiable but useless, then indeed it isn't science. So, frowning > at superdeterminism is not entirely unjustified._

<comment><USER13> Yes, despite the title, the articles has better arguments against superdeterminism than for it.

</comment></comment></comment></comment><comment> IX-103 I'm not fond of superdeterminism since it's not that useful for making predictions. Any purely deterministic model has implications for free will, so that doesn't seem to be a legitimate criticism.

Actually I would like to know more about provable violations if Bell's theorem as I am somewhat attached to local determinism and haven't seen an experiment that I would consider convincing. I mean the theories behind the experiments are sound, but I'm not sure they're actually measuring what they think they are measuring due to limitations in the experiment setup -- in order to prove a violation of locality your system cannot be in a cyclostationary equilibrium.

In such an equilibrium the system state effectively becomes a standing wave so you risk measuring an effect that was actually a result of a previous cycle and mistakenly interpret it as being a result of the current cycle -- implying a violation in locality because the "cause" was outside of the light cone of the effect. Note that this is analogous to confusing the group and phase velocities of a radio wave ([https://www.quora.com/What-is-the-difference- between-phase-v...](https://www.quora.com/What-is-the-difference-between- phase-velocity-and-group-velocity)).

<comment><USER14> _> In such an equilibrium the system state effectively becomes a standing wave so you risk measuring an effect that was actually a result of a previous cycle and mistakenly interpret it as being a result of the current cycle_

I don't know where you're getting this from, but it doesn't describe quantum systems on which Bell inequality violations have been experimentally confirmed (such as photon pairs from parametric down conversion).

The only "loophole" that has not been completely closed at this point is that we don't have 100% efficient detectors, but we have detectors that are well over 90% efficient so the claim that somehow all the stuff that will "fix" the Bell inequality violations is hiding in the small percentage of photons not being detected isn't very compelling.

<comment> IX-103 In short the problem I'm addressing is that the interpretation of the experiment assumes that the system is memoryless, so that the only thing being measured is the interaction with the particles being measured.

In the experiments generating the photon pairs from parametric downconversion, for example, does the entire system start up, send 1 photon which gets split into the entangled photon pairs which then go to the detectors -- with no other photons generated?

If there is a warm-up period for the equipment or other photons are emitted or absorbed then there is the potential for memory effects that could interfere with the measurements.

For instance if we treat light as a wave then the cosine correlation with angle we see in the basic "two entangled photons with polarizing lenses experiment" is exactly what we would expect to see. The difficulty is simply resolving this with the particle nature of photons. If the experimental system has memory then it could easily have the phase of the effective wave or some other function of the history of photons encoded in the state of the system.

There are probably some ways to compensate for these memory effects and demonstrate their (non)existence, but I am not a physicist.

<comment><USER14> _> the problem I'm addressing is that the interpretation of the experiment assumes that the system is memoryless_

That's easy to verify by testing the various components--parametric down conversion, prisms, beam splitters, etc.--and showing that if you shine repeated photons on them from the same source, prepared in the same state, they all come out in the same state, or more generally give the same results. All of the optical components involved in these experiments have been tested in this way: if they had failed such tests, they wouldn't be used in experiments because we wouldn't be able to be confident in their behavior.

 _> n the experiments generating the photon pairs from parametric downconversion, for example, does the entire system start up, send 1 photon which gets split into the entangled photon pairs which then go to the detectors -- with no other photons generated?_

For current photon sources, it's impossible to control exactly when they emit a photon. The sources are so inefficient (in terms of converting input energy into photons that are useful for the experiment) that they end up emitting photons slowly enough that only one at a time is inside the apparatus. However, a typical experiment does not use just one photon. It has to take data from many photons because the results are statistical, so you need enough runs to do statistics.

 _> If the experimental system has memory then it could easily have the phase of the effective wave or some other function of the history of photons encoded in the state of the system._

We know how to design systems that do this: they're called "detectors" and "computers that store data". But such systems have to be carefully designed to do those jobs. Optical components like prisms and beam splitters are not designed to do that: they're designed to do exactly the opposite, to act the same way on every photon that comes into them in the same input state. As I noted above, those components have been extensively tested to make sure they do in fact do that; if they didn't, they wouldn't be used in experiments.

<comment> IX-103 _> if you shine repeated photons on them from the same source, prepared in the same state, they all come out in the same state, or more generally give the same results._

Those kinds of measurements would violate the uncertainty principle. You can't know the complete state going in to the system or the complete state going out. You can run some tests and justify other assumptions based accepted theories. We generally have a good idea what happens when lots of photons pass through these components. We have some ideas of what happens to single photons (in a statistical sense), but the fundamental question we are investigating is whether there even _is_ a local deterministic description of what happens to single photons passing through the component.

 _> The sources are so inefficient (in terms of converting input energy into photons that are useful for the experiment)... a typical experiment does not use just one photon. It has to take data from many photons_

I was aware of that and it's part of my criticism. If the emitter were to only emit useable photons when it's "in the right state", what stops the "right state" for emitting photons to become correlated with the polarizers?

There are a bunch of "unusable" photons bouncing around interacting with everything and transporting global state. Then there are the "usable" photons that get reflected, absorbed and re-emitted by components of the test bed. Any time they interact with anything they modify the state of whatever they touch. What happens to the photons that reflect off of the polarizers and travel back into the emitter?

If a photon bounces off of a mirror it had to have 1. transfered momentum to whatever it hit, and 2. induced a sufficiently strong opposing electromagnetic field to cause the photon to be reflected or re-emitted. While these are tiny effects they are roughly the same order as the effects that caused the photon to be reflected in the first place, and they all require a change in state of the mirror so that momentum is conserved and Maxwell's laws are not violated (my guess would be that this could cause shifts in electron orbital or proton spin orientation, but that's a bit beyond me).

<comment><USER14> _> Those kinds of measurements would violate the uncertainty principle._

No, they don't. The uncertainty principle places limits on measurements of non-commuting observables on the same system. We are not talking about that here. See below.

 _> You can't know the complete state going in to the system_

Sure you can: just prepare the system in a known state. For example, pass your photon through a vertically oriented polarizing filter: if it comes through, it must be vertically polarized, so you have complete knowledge of its polarization state. (You might have to try multiple photons to get one that passes through: that's why photon sources in these experiments are often inefficient.)

 _> or the complete state going out_

Sure you can: you measure it. For example, you pass the vertically polarized photon that just came through your vertical polarization filter through a beam splitter, and you have detectors at each output of the beam splitter. Exactly one detector will fire for each photon.

 _> If the emitter were to only emit useable photons when it's "in the right state", what stops the "right state" for emitting photons to become correlated with the polarizers?_

 _> There are a bunch of "unusable" photons bouncing around interacting with everything and transporting global state._

It looks like you don't have a good understanding of how the "emitter" works. What you are calling the "emitter" is really a filter, like the vertical polarizer described above: it throws away the photons coming from a source (like a laser) that don't meet a particular requirement (like vertical polarization). The thrown away photons are either absorbed (as in the case of the polarizer) or they just pass through the apparatus altogether and fly away (as in the case of parametric down conversion, for example: only a small percentage of the laser photons will be down converted, the rest just fly away and are gone).

In no case are the photons not used kept "bouncing around". They're gone. And the photons in the "right" state are just the ones that make it through the filter and are therefore in a known state when they come out, because that's how the filter works: the filter is uncorrelated with what's inside the experiment because, again, that's how the filter works (and it is tested to make sure it works that way).

 _> What happens to the photons that reflect off of the polarizers and travel back into the emitter?_

There aren't any. See above.

 _> If a photon bounces off of a mirror it had to have 1. transfered momentum to whatever it hit, and 2. induced a sufficiently strong opposing electromagnetic field to cause the photon to be reflected or re-emitted._

1\. Yes, but in these experiments the mirror is fixed to the Earth, so the momentum is transferred to the Earth, which means it's effectively gone. The entire Earth is not going to have a "memory" that can become correlated with the rest of the experiment.

2\. No. You are thinking of it classically, but we are not talking about a classical process.

</comment></comment></comment></comment></comment></comment><comment><USER15> There is something very Taoistic about superdeterminism.

And there is something very Taoistic about homotopy type theory too.

Also, I feel that both superdeterminism and homotopy type theory have traces of the holographic principle in them in a somewhat conceptual or abstract way.

Perhaps there exists a nice correspondence between superdeterminism and homotopy type theory that can be used to extend (in a purely functional and categorical way) the simulation hypothesis into a full-fledged theory (and perhaps with its own nice little axiomatic system) to make sense of reality.

</comment><comment><USER16> The problem is that superdeterminism contains a principle of explosion:

[http://en.wikipedia.org/wiki/Principle_of_explosion](http://en.wikipedia.org/wiki/Principle_of_explosion)

If superdeterminism explains quantum mechanics, why not cosmic inflation? Why not matter asymmetry? Why not abiogenesis? Why not Brexit? Superdeterminism, by construction, can explain _everything_ — and all there's left to do is pray to God.

<comment><USER15> I will have to disagree with that.

Firstly I don't think anyone has actually formalised superdeterminism in a way that the principle of explosion can be logically introduced to formally undermine superdeterminism. What you are doing here is akin to stretching the conceptual relevance of Godel's incompleteness theorems and trying to use it to prove or disprove the existence of God.

Basically I don't see how it makes sense to say that superdeterminism contains a principle of explosion. Perhaps my interpretation of superdeterminism is very different from yours. Or maybe I simply don't see the picture as you do. If that is the case please enlighten me.

Secondly I think you are missing the point of superdeterminism here.

There is something very computational (and perhaps Taoistic) about superdeterminism. Apparently under this framework the whole notion of "explaining things" is nullified and becomes meaningless. It occurs to me that our everyday notion of "explaining things" exists at a lower abstraction level and thus loses relevance in the face of superdeterminism. I believe if you really want to undermine superdeterminism as a theory (or as a philosophy), the more relevant question here to ask is: is there anything useful/meaningful about reality (or the universe) that can be inferred assuming superdeterminism? And then of course if you are a scientist you would then ask: are they experimentally verifiable?

<comment><USER17> > I believe if you really want to undermine superdeterminism as a theory (or > as a philosophy), the more relevant question here to ask is: is there > anything useful/meaningful about reality (or the universe) that can be > inferred assuming superdeterminism? And then of course if you are a > scientist you would then ask: are they experimentally verifiable?

To even ask the question you must deny your own premise. If indeed superdeterminism is true, then any experimental verification is nullified by definition: the results of any and all experimentation is itself superdetermined regardless of any scientific framework.

<comment><USER15> The last part was supposed to be taken with a grain of irony in the spirit of Anton Zeilinger:

"We always implicitly assume the freedom of the experimentalist... This fundamental assumption is essential to doing science. If this were not true, then, I suggest, it would make no sense at all to ask nature questions in an experiment, since then nature could determine what our questions are, and that could guide our questions such that we arrive at a false picture of nature."

I guess that is the problem most people have with superdeterminism. Intellectuals in this day of age are too scientifically trained to have any romanticisation for reality under frameworks like synchronicity (even though it was popularised by Pauli before it lost mainstream appeal after actual fundings went into unfruitful statistical research in the 80s) or (in the newtonian time) alchemy and the love for God. This is why superdeterminism is so unpopular.

I just really like superdeterminism because I think it is cute and I believe in the Tao.

<comment><USER18> What does superdeterminism have to do with the Tao?

<comment><USER15> There was something formless and perfect

before the universe was born.

It is serene. Empty.

Solitary. Unchanging.

Infinite. Eternally present.

It is the mother of the universe.

For lack of a better name,

I call it the Tao. It flows through all things,

inside and outside, and returns

to the origin of all things. The Tao is great.

The universe is great.

Earth is great.

Man is great.

These are the four great powers. Man follows the earth.

Earth follows the universe.

The universe follows the Tao.

The Tao follows only itself.

<comment><USER18> That's beautiful.

</comment></comment></comment></comment></comment></comment></comment><comment><USER19> Can someone explain more clearly how being in a deterministic universe resolves the "problem" of Bell inequalities? It seems like even if the universe were deterministic it would not cause the classic polarizing-filters Bell inequality to seem "reasonable". In fact it makes it seem less reasonable to me!

<comment><USER13> The determinism doesn't solve the problem, it makes the non-locality more visible because some people think that state of two particles far away influencing one another is somehow worse than the wave function in whole universe changing it's value at once.

The superdeterminism "solves" the problem by claiming that there is no problem to begin with, and the results look non-local only because the experimenters always pick experiments that look non-local.

How can a local deterministic theory create such complex behavior as thinking people, and at the same time constrain it in a way that time taken to play mario level is correlated with a photon experiment a year later, is left for the reader as an exercise.

</comment></comment><comment><USER15> There was something formless and perfect before the universe was born.

It is serene. Empty.

Solitary. Unchanging.

Infinite. Eternally present.

It is the mother of the universe.

For lack of a better name,

I call it the Tao. It flows through all things,

inside and outside, and returns

to the origin of all things. The Tao is great.

The universe is great.

Earth is great.

Man is great.

These are the four great powers. Man follows the earth.

Earth follows the universe.

The universe follows the Tao.

The Tao follows only itself.

</comment><comment><USER20> If Superdeterminism means that the initial state of the universe is such that the universe _appears_ to follow quantum mechanics... why? Why would every single decision have its resolution set in a manner that appears to follow QM?

</comment><comment><USER17> Superdeterminism is a self-defeating philosophy. In essence, it cedes everything to random chance, and makes all scientific inquiry meaningless. There is no longer any "why" or "how." There is merely, "That's just the way it is." Any apparent order or structure which might be observed is exactly that: merely apparent. Therefore any attempt to understand the universe is vain.

It is little better than the presumption that planets move because a prime mover moves them. It is, in essence, to give as the final answer: "Planets move as they do because they cannot do anything else."

<comment><USER21> > _In essence, it cedes everything to random chance, and makes all scientific > inquiry meaningless. There is no longer any "why" or "how." There is merely, > "That's just the way it is."_

Which doesn't make it wrong...

<comment><USER22> And indeed, is based on the same math as the more mainstream interpretations.</comment></comment></comment>


UK bank stock prices collapse by 30% - <AUTHOR> http://www.bbc.com/news/business-36611512 <comment><USER2> Some investors are going to make utter bank on the panic.</comment>


Ask HN: What does your production machine learning pipeline look like? - <AUTHOR> Doing some design for an upcoming project and taking a survey.
I'll go first. Model training happened nightly on a Spark cluster. This output a PMML-based SVM model. The model was instantiated on a cluster of compute servers running Openscoring. A thin Node web service wrapper used the Openscoring cluster to serve realtime client prediction requests. Dataset size in the hundred millions of examples with hundreds of features. Handled thousands of requests per second, no problem.
Separating the training technology from the execution technology was nice but the PMML format is limiting in the kinds of models you can use that both you trainer and executor will support. What are people doing who use same tech for both? For something like Tensorflow, I assume you must have to save the model as binary from the train step and then send it off to the prediction cluster to be instantiated again for execution? <comment><USER4> (ML engineer from Dropbox here)

For deep learning oriented projects, we train on EC2 GPU instances, generally p2.xlarge instances these days to get the Nvidia K80 GPUs. We can spin up many of these in parallel if we are doing model architecture searches or hyperparameter exploration.

We have an in-house data turking setup where we can efficiently roll new UIs to get ground-truth data for given problems, generally in the thousands to tens of thousands of real data. We also use data augmentation where possible, synthetically generating millions of example data points, combining it with the real turked data for fine tuning. Note that we never look at or train with real user data, unless explicitly donated, so data efficiency is important to us.

We've standardized on TensorFlow these days, doing inference on CPUs currently on Dropbox's compute infrastructure. We have a jail setup based off of LXC and Provost that allows us to safely execute these trained models in a contained way, responding to user requests as they come in. We use standard distributed systems plumbing for RPCs, queues, etc. to respond to user requests and route them to trained models. We version our trained models, and have an in-house experiments framework that we use to deploy new models and test them against subsets of user traffic to see how they are doing as we ramp up a new model.

Most of our day-to-day work is in Python, with occasional use of C++; other parts of Dropbox sometimes use Go and Rust, though we haven't had need for that on the ML team. Note that Dropbox is one of the largest users of Python in the world (Guido van Rossum actually works here).

BTW, the Machine Learning team at Dropbox is hiring. Come join us! Details: [https://www.dropbox.com/jobs/listing/533100](https://www.dropbox.com/jobs/listing/533100)

<comment><USER5> > we never look at or _train_ with real user data, unless explicitly donated

What is the rationale behind this? Why not train?

<comment><USER6> Couldn't user data make it into the model, and be reverse-engineered from it, if you used it to train the model?

I mean, machine learning models aren't designed as one-way cryptographic hash functions, so you can't be sure that some of the training data can be inferred through the model's behavior.

<comment><USER7> Yes! there is an interesting paper on this subject: [https://arxiv.org/abs/1609.02943](https://arxiv.org/abs/1609.02943)

<comment><USER8> Counterpoint: [https://arxiv.org/abs/1701.04082](https://arxiv.org/abs/1701.04082)

Source: [https://news.ycombinator.com/item?id=13824047](https://news.ycombinator.com/item?id=13824047)

</comment></comment></comment></comment></comment><comment><USER9> Fraud detection at GoCardless (YC 11). We use the same tech for training and production classification: logistic regression (sklearn).

\</comment><comment>-

 __ _Training /retraining_ __

\- train on an ad-hoc basis, every few months right now moving to more frequently and regularly as we streamline the process

\- training done locally, in memory (we're "medium data" so no need for distributed process), using a version-controlled ipython notebook

\- we extract model and preprocessing parameters from the model and preprocessors that were fit in the retraining process, dump to a json config file in the production classification repo

\</comment><comment>-

 __ _Production classification_ __

\- we classify activity in our system on a nightly cron*

\- as part of the cron we instantiate the model using config dumped from the retraining process. This means the config is fully readable in the git history (amazing for debugging by wider eng. team if something goes wrong)

\- classifications and P(fraud) gets sent to the GoCardless core payments service which then decides whether to escalate cases for manual review

\</comment><comment>-

* We're a payments company, processing Direct Debit bank-to-bank transfers. Inter-bank Direct Debit payments happen slowly (typically >2 days) so we don't need a live service for classifications.

Quite simple as production ML services go, but it's currently only 2 people working on this (we're hiring!).

<comment><AUTHOR> When using sklearn, I've seen a lot of folks just pickle the model and use that as the interchange format. I like the human-readable interchange format you are using better. I assume you just rolled your own. Why not something like PMML?

<comment><USER9> Yep, we made our own. I haven't heard of PMML before - quite cool! What we've made is a bit more readable for what we're using it for though, IMO. Looks like this:

        {
            "intercept": 1.0,

            "features": {
                "feature_1": {
                    "coefficient": 1.0,
                    "range": [0.1, 10.0],
                    "mean_feature_score": 1.0,
                    "imputation_value": 1.0
                },
                {
                    ....
                }
            }
        }

<comment><USER10> Is this open source? We were looking for something like this.

<comment><USER9> Sadly not. I'd be totally up for open sourcing if there's clear demand. If you can find it, send me an email at angus@{company_I_work_at}.com

Note that it's very tied down to our use case right now: only compatible with Logistic Regression, and currently it assumes fixed hyperparameters (will change this in future though), assumes a production pipeline of min-max scaling, imputation, then classification.

</comment></comment></comment></comment></comment><comment><USER11> My case is probably not quite what you're looking for, but I'll describe it anyway. I develop models for automated EEG analysis. The models are all in a standalone program that neurologists use. Their computers may or may not be connected to the internet, so all computation has to be local.

I have a set of training records (usually on the order of ~100). I do three folds, and train models on 2/3 of the data, validate on the last 1/3\. I train using Tensorflow, usually experimenting with a few hundred different architectures. Then I combine the best models we have trained on the three folds into an ensemble. The ensemble can be too big to be practical, so sometimes the ensemble is distilled to a smaller model by training to the ensemble outputs. (The models have to run in real-time, and there are a lot, so it should take any particular model no more than ~10ms to process 1 second of data.) The final model is then tested on a completely independent dataset to determine our performance.

As these models are training I develop different experiments to try to probe the behavior of the models to make sure that they're working as expected. This usually motivates new architectures or features to experiment with.

Once the model has been trained, I export all the model weights to a bunch of big static C++ arrays. I've written my own feed-forward NN layers (fully connected, convolutional, and LSTM) to use these arrays of weights in our C++ code. (Tensorflow Serving didn't seem to make much sense for our use case and it was just easier to write the basic NN layers.)

When our resident neurologist feels like it's working well enough, we apply to the FDA for approval, and, who knows, maybe in a few years it'll be allowed in the product!

<comment><USER12> I assume these models, and your end-product goal, is to make some kind of diagnosis, or flag anomalies? This is very interesting, and any more info on what your use-case or end-product would be great.

<comment><USER11> The end goal is to make routine EEG interpretation easier for neurologists. The usual way to look at an EEG is to page through the record 10 seconds at a time, and look for different (sometime subtle) features to make a diagnosis. For a 24 hour record, that's 8600 pages the neurologist has to look through, the vast majority of which contain nothing interesting.

One of the main features of our software is epileptogenic spike detection. Epileptogenic spikes are little blips in the record and are indicative that the patient has epilepsy. Our models are now good enough at finding these spikes as human neurologists. There may be only a handful of these spikes in the record, so finding them automatically can be a huge time savings for the neurologist.

Our software also finds and eliminates various EEG artifacts. Blinking, for example, causes a huge artifact in the records. (It turns out that your eye is a dipole, and rolls upward every time you blink, which produces a huge signal.) The next major goal is seizure detection. It turns out there are a lot of things that can look a lot like a seizure in an EEG record. Even disconnecting the electrodes can look like a seizure.

The holy grail would be to give the program an EEG record and have it spit out the neurologist's report. We're not there yet, but we're working towards it. (By the way, we're hiring another ML engineer and software engineer. Email email@example.com if you're interested!)

<comment><USER13> That sounds like a very rewarding application that will be a great net benefit for humanity.

I'm looking at using machine learning on time series data in a similar way, replacing EEG with 3D motion data. However, ML is not my forte and I am just cutting my teeth. I would really appreciate if you be able to share some of the 'buzzwords' or ML topics that I might look into as I explore this work?

<comment><USER11> I have found recurrent NNs and their variants (LSTMs and GRUs) to be very useful for the time series I've been working with. It can also be useful to use 1-d convolutional NNs as well. Depending on your data, sometimes working in frequency space can be easier (i.e., train on the FFT of your data). I'm still a neophyte in this area (my background is in astronomy), so I'm learning, too!

<comment><USER13> Many thanks.

</comment></comment></comment></comment></comment></comment><comment><USER14> We wrote a blog post detailing a lot of how we do our streaming ML system at Distil ([https://resources.distilnetworks.com/all-blog- posts/building...](https://resources.distilnetworks.com/all-blog- posts/building-a-streaming-log-processing-and-machine-learning-system-at- distil)) We classify web traffic as humans or bots in realtime.

Scoring: Essentially raw logs go to kafka. Java processes then read raw logs and aggregate the per-user state in real-time. State is stored in Redis. When a user's state is updated it is sent into another kafka topic that is the "features" topic. Features are simultaneously saved to HDFS for future training data and then consumed by a Storm cluster for doing prediction. Storm is running pickled scikit-learn models in bolts that read in the features and output scores. Scores are sent into a "score" kafka topic. Downstream systems reading the scores can read from this kafka topic.

Time from receiving a log to producing a score is ~x seconds or so.

Training:

Training data is stored into HDFS from the real-time system, so our training and production data is identical. We have IPython notebooks that pull in the training data, build a model and sckikit-learn feature transformation pipeline. This is pickled and saved to HDFS for versioning. The storm cluster when it starts a topology loads the appropriate model from HDFS for classifying data.

<comment><USER10> I never thought that ipython notebooks could be used to build production models.

Can you talk about this? How big a machine do you need to run notebooks effectively ? Why a notebook...Why not a Python script,etc

<comment><USER14> > How big a machine do you need to run notebooks effectively ?

\- not too big. I've run tests with performance vs. sampling of data and for the models we use we can sample a significant amount (i.e. use a fraction of the full training data). \- If we did need a bigger machine, we're using ensemble models that parallel train nicely.

> Why a notebook...Why not a Python script

I was originally going to use a Python script, but I found it useful to have the notebook output inline performance charts and metrics. It's easier to contain them in the notebook than output a bunch of image artifacts that have to be added to VC. This way I can pull open the notebook and scroll through to check all my visual metrics.

I'm not opposed to ditching the notebook for training entirely, but for now it works just fine.

<comment><USER10> so you build a notebook and play around with it... and then run this notebook in an automated way ? so u can open the notebook anytime and work with it ?

I really love this quick iterative way of working (atleast in the early days). Could you talk about your production setup of training ? I'm just concerned about performance, etc - is it OK to train manually each time (by opening the notebook, etc)

<comment><USER14> So far our models remain fairly stable in the ~weeks timeframe. If we needed to train daily or similar I would invest time in something other than a notebook. But, right now, it's easier to have the training steps documented in the notebook that does the actual training than to build a separate system and document it.

Not claiming it's a good way of doing this, but just how it is right now.

</comment></comment></comment></comment></comment><comment><USER15> Conversational ML Pipeline (Myra Labs).

We have a few interesting challenges. Our system is set up to create models for a variety of customers, and thus needs to be able to build, update and serve a couple of thousand models at any given time. Since we need GPUs, this has required some custom work rather than being able to piggy back on existing frameworks. We leverage Tensorflow for most things.

\- We have a distributed job processor cluster running on GPU nodes. All models are trained on this. We wrote a custom framework in Python for this. It can train any kind of model (keras - tf/theano, sklearn, et al).

\- We store our models on S3. Versioning of models becomes important, and we looked around to see how others do it before we rolled our own.

\- We have a serving system that is a heavily modified/forked version of Tensorflow Serving, written in C++. Again on GPU machines. Why C++? Because we can leverage binary model formats and reduce memory usage/make things more efficient. The reason we forked TF Serving almost a year ago was that we had a large number of features we needed that TF's rapid versioning cycle/breaking API updates just didn't let us do. This system is able to load and unload models dynamically from the S3 source, as well as distribute them across nodes and balance queries into them. It can load all TF graphs, but also supports running other kinds of models through callouts to other "engines" (eg, if we want to use sklearn or CNTK in the future).

<comment><USER10> _> Versioning of models becomes important, and we looked around to see how others do it before we rolled our own._

could you talk about model versioning ? not necessarily your own - <USER2> interesting and relevant examples you found elsewhere.

<comment><USER16> Not sure if this was one of the examples the parent looked at in their search, but Pachyderm is a tool that's built specifically for training and versioning ML models (and any other data pipeline). You can express your analysis operations as Docker images so it should support all of the ML tools listed in this thread.

Full disclosure: I'm one of the creators of the project.

</comment></comment></comment><comment><USER17> I work at Spotify on ML. I spent quite sometime working on our production vector model pipelines.

We have three components:

1\. Training data: Scalding pipelines (could be any large scale production framework) to preprocess training data and reduce it to a few hundred gigs.

2\. Run latent vector models (Matrix factorization, Embedding models etc) to generate item vectors. This stage is designed to be non specific to framework. We use typed avro objects to communicate with the previous stage.

3\. Production deployment of models. Essentially glue code that ensures that models are versioned, atomic (user, music vectors are all from the same batch) and deployed into music recommendation and realtime frameworks.

Read here[1] more.

[1] [https://labs.spotify.com/2016/08/07/commodity-music-ml- servi...](https://labs.spotify.com/2016/08/07/commodity-music-ml-services/)

</comment><comment><USER18> I find a lot of these frameworks to be overkill.

A rating event comes into our web server (python), then sent to SQS, then the event is pulled by our custom artificial neural network (deep learning) script, --written in Go. A model is trained and serialized. Next, the serialized model is uploaded to Postgres where it is fetched by the web service (also written in Go) to serve predictions.

We update our models within 15 seconds of a user rating. Every month with millions of ratings, we re-train millions of models and serve billions of predictions.

<comment><USER19> Interesting. Do you mind going into more detail?

I'm thinking of implementing a similar system – though not using Go.

<comment><USER18> What sort of details are you looking for?

<comment><USER19> Mostly persistence and scaling.

I assume you're using a jsonb column, and going horizontal?

<comment><USER18> Honestly, we haven't needed to scale horizontally. The Postgres column is a jsonb, although probably not necessary since we don't need to query the serialized model. If our reads ever start to slow down, the plan is to just add a follower to read from.

<comment><USER19> Woops. I was referring to the prediction tier.

</comment></comment></comment></comment></comment></comment><comment><USER20> At Custora (YC W11), analyzing consumer behavior and presenting the insights through a web-based interface.

We have an in-house system for modeling DAGs of statistics, perhaps like Airflow (I've only read about Airflow, never used it, so can't comment more there). Computed values at different nodes in the DAG can be cached by their arguments. So for example, a fitted model would be a very logical node to cache, as you'll have many other potential statistical requests that would depend on the model.

Refitting the model would entail either clearing that node; if you changed the inputs to the fit, a separate stat would be cached and other dependent stats would know the difference and point to the right stat depending on the arguments you passed.

A lot of our models are Bayesian in nature, so generating predictions is typically a two-part process: training parameters, which is slow, can happen infrequently and which need not critically include all of the latest data, and applying the posterior update, which is faster and which needs to be redone on every data update. (We import data in batch.) Retraining is somewhat ad-hoc right now, although we've got an active project on the docket to systematize this and produce streamlined before-and-after comparisons.

Computation work is dispatched to EC2 instances by a Redis-based job manager (qless, developed by Moz, formerly SEOmoz). We do the stats work in R, in- memory. For things like order and transactional data this is feasible even for relatively large retailers, but we're gradually looking at involving Spark more so that we have the capacity for larger analyses. (We already do use Spark for some non-ML tasks, like customer data import.)

</comment><comment><USER21> Thanks everyone for the useful answers.

I'll go too: We have 20M customers with a history 1-1000 activities per customer, updated once daily. We do nightly runs of a collaborative filtering algo to produce per-customer suggested activities, and push this to a sql database. Business users can load recommendations for each customer from this db...

</comment><comment><USER22> We use scikit-learn to train the models every few weeks when we get more labeled data. Once a model is trained we use joblib to save the entire pipeline (normalization, feature processing etc). In production we have a thin Rest wrapper that loads the model pipeline to memory and serves prediction requests. We scale the number of these servers based on the load.

<comment><USER23> Same here.

Works well as long as you're disciplined w/ your code and have regression tests, continuous integration.

The hard part is ETLing data for training in a consistent way.

</comment></comment><comment><USER24> (ML engineer from Twizoo here)

We use a multitude of models - linear and non-linear, supervised and unsupervised - to surface the best user-generated content from social for our clients.

Our ML stack is written almost entirely in Python - because of the readily available excellent libraries that give you many of the tools you'll need in your ML toolbox (matplotlib, numpy, pandas, scikit-image, scikit-learn, scipy to mention just a few). We have some more sophisticated image processing in C++. We find ourselves attacking quite a number of machine learning problems so to make our workflow more efficient we have built a layer of functionality above these libraries to manage the flow of data and speed up model prototyping and assessment. Processing input data and model fitting is usually done locally, with any hyper-parameter grid search or other computationally intensive tasks being run on a remote, cloud hosted cluster spun up on-demand.

Our tech is built around lambda architecture principles - we have a live path that processes new social content to the extent required to make content immediately available to relevant clients, and a batch path that processes complete datasets daily. Our ML models get used on both paths.

With a neat set of Python libraries for data manipulation and model definitions and our own model prototyping and assessment rig the biggest challenge is always building big, quality datasets. This continues to be our biggest learning and we are always keen to explore how others tackle this. We rely on a combination of in-house turking (which we consider a vital model development stage), external turking services (for volume) and have additionally looked at some knowledge transfer techniques.

See [https://www.periscope.tv/w/1RDGlYXeDpOJL](https://www.periscope.tv/w/1RDGlYXeDpOJL) for a quick talk where I expanded on why and how Twizoo uses ML.

</comment><comment><USER25> Deepdetect for both dev and prod, with the minimum code in front of it. This setup is definitely not able to accommodate all modern ML needs, but the fast and secure model update from dev to prod is the easiest for us. Disclaimer: DD author so the bias is very high, apologies for this, maybe my comment will remain useful to some.

</comment><comment><USER26> Data Engineer at Thread (YC '12) here.

We're a relatively small team so it's nothing terribly sophisticated - <USER3> jobs prep log data for a nightly model training, done with sklearn - the model is just a python pickle that get loaded into our ranking system, Thimble. All done on EC2.

We also have a real-time pipeline (Bazalgette) that pulls events off a Kinesis stream and turns them into features, saved to Redis.

We're hiring a software engineer (in London) at the moment so if this sounds like the kind of tech you'd be interested in working alongside, drop me an email (in my profile) or look at [https://thread.com/jobs](https://thread.com/jobs)

</comment><comment><USER27> (product person at Appuri in Seattle)

We work with customer data, so lots of time-series. We use SQL for exploration and feature engineering. Model-building happens inside a Docker container that runs on ECS. Scores and predictions are inserted back into the data warehouse (Redshift). It's a simple, practical system that works well without needing a ton of infrastructure.

</comment><comment><USER28> These replies are all very valuable, so appreciate everyone's comments. I have been involved in implementing a risk model for a financial services company, and the system is written completely in R and split as follows:

live model: R script listening to redis list, data is json, scores customer applications as they come in; if needed can trivially scale more workers.

batch update: also R script, generates model files that live system uses and runs nightly on cron, past models are backed up so rollbacks are possible.

The major pain points have been ensuring the live and batch systems match exactly, as we have to rewrite the code to do the same thing on both of them. Unfortunately, since we're doing quite a bit of processing to engineer features from multiple data sources this couldn't easily be expressed as a pipeline in scikit learn.

Redis has been invaluable in managing the queues and glueing the different systems together.

</comment><comment><USER29> Data Engineer at Shopify here.

We use ML in both streaming and batch use cases to do things like fraud detection, app/theme recommendations and to give merchants insights into their customers and business.

Our ML architecture is very similar to the OP, we use PySpark to train models and fill some of the missing gaps using scikit-learn. The batch jobs consume and produce data to HDFS and our streaming pipeline read off a Kafka Stream.

We've been experimenting with exporting models to PMML and using the JPMML Evaluator to serve them. However in some use cases we've had to rewrite models in ruby so that they can be used in synchronous paths with our rails app, these are usually very simple things like logistic regression.

Our ML pipeline is in its infancy, theres still a lot of work to be done, if this is the sort of thing you'd like to build shoot me a message at

firstname.lastname@example.com

<comment><USER14> Is Shopify open to remote work?

<comment><USER29> Some of our positions are open to remote work. But unfortunately there are no remote positions in the ML and Data teams. We do have offices in several Canadian cities and do work on getting a visa for candidates.

heres a link to our positions [https://jobs.lever.co/shopify?lever- via=XBuWsYM_Q2](https://jobs.lever.co/shopify?lever-via=XBuWsYM_Q2)

</comment></comment></comment><comment><USER30> When I joined Airbnb, we did something similar with PMML on the risk team. The issue I had with PMML is that sometimes the scores would differ in weird ways due to the differences in the model implementation. Also, it limited what models we could use.

I moved to our ML Infra team and built a service to allow inference on the pickled sklearn model or serialized TensorFlow model. This also worked with our ml framework: Aerosolve. (If you're using TensorFlow only then you can use TensorFlow serving.) One of the primary goals I had was to make everything config driven so data scientists could launch models without engineering help. This has worked pretty well. We've put about 20 models into production on this service in the last year.

For batch scoring, the solution is similar. We use Airflow for scheduling and can just do inference on the model in the airflow worker. It's easy enough when it's python or spark because Airflow plays nicely with those.

Some complications occur in:

1) Dependency management. We let users package up their own virtualenvs and use them in both the real time scoring and batch scoring. But it's a bit clunky in certain cases and we're moving to docker.

2) Heterogenous resource requirements. If you need a lot of memory or gpu access then it can be a bit tricky to do that with Airflow. We're working on integrating it with docker and kubernetes to solve this at the moment. Typically we use yarn for problems like this but our ML use cases require some things that are just hard to do on our hadoop cluster. I'm not a big fan of yarn for ML. No one has needed GPUs for real time prediction yet so I'll cross that bridge when we get to it.

3) Online / offline consistency. Sometimes we'd end up re-implementing transform logic in Java so we could run it in production. This led to lots of sadness when things didn't quite match up. Now, we try to make it easy to deploy the whole sklearn pipeline including custom python transformations. That way we have some confidence we're running the exact same code at training and inference time. Similarly, we built a framework to compute offline aggregations in hive and upload the results to a KV store for use in online scoring. We have the ability to augment this with streaming data to reduce the data delay on the aggregation. This introduced some lambda architecture requirements that were kind of interesting. We actually ran into quite a few scaling issues here so we've gone through a couple iterations as new projects come in. I'm currently working on the third generation of this framework.

These are some of the issues we tackled early on with the ML Infra team. We're now focusing on improving the quality of our offline training tools. Particularly we're trying to make:

1) A seamless process for going from a notebook to production that integrates all these tools.

2) A good notebook environment that includes simple provisioning of resources for large training jobs.

3) Sample notebooks for common ML tasks that encode best practices and make it easier for people to get their initial model quickly.

4) A system for making it easier to do complex training data backfills.

Outside the ML Infra team we have a few ML focused product teams that own their own infrastructure and optimize it for very specific needs they have.

</comment><comment><USER31> Can anyone comment on the relative computational requirements of training vs using the classifier? antognini described a system that trained in tensorflow and then is used by a simple C++ program (not on a GPU?).

Do you use the same resources/hardware for both training and executing?

<comment><USER4> At Dropbox we use different machines for training and inference, as their computational needs are so different. We train on Amazon GPU machines, while we deploy on our own Dropbox compute infrastructure on CPU machines.

</comment></comment><comment><USER32> Training/retraining: Done arbitrarily, mostly locally, sometimes distribute. Done with either TensorFlow or Torch. We have a custom backend for it. Often times linked with Keras.

Inference: For our custom platform, we have our own framework (in progress). For other products where our hardware is unavailable, we will use either MXNet mobile or a custom framework on mobile frameworks. For deployments where we have the luxury of a cloud link, we will either use TensorFlow serving (with a custom backend once it's done) or Flask linked to TF/Caffe/Keras (also with a custom backend once it's done).

</comment><comment><USER33> I'm interested to see if [https://computes.io](https://computes.io) could be used for ML.

</comment><comment><USER34> For you feedback I'm offering karma or bug bounties (if you can identify me on github)!

Forestry High Resolution Inventory - Predicting Forest Attributes at Landscape Scale \</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>----

This a really promising business line. Our pipeline is not very automated and it has no application level data management - we're a ways away from that (e.g. the code isn't even under version control).

For our current business and data volumes, the system works.

We get target attributes (forest characteristics, e.g. species composition) by doing field surveys or other methods (e.g. expert photo interpretation).

We acquire Lidar, Color Infrared data, and Climatic models to develop landscape-scale features. For the lidar derived features we use LAStools. We use Safe software FME for generating some features from the color infrared data (e.g. vegetation indices). We use regional climate models suitable for the area of interest for the climatic indices.

The reference data and the target attributes are spatially joined. We end up with a lot of features and use the subselect library's "improve" function in R in an iterative fashion to reduce the number of features; leave-one-out LDA is used to assess the performance of the subset of features. If the target variable is not categorical, we have to classify it in order to run the LDA. The procedure produces a lot of candidate feature sets; our process to select a particular feature-set is human driven. We do not have a formalized or explicit rule. The subset of feature chosen are used in KNN routine. Some components are in python and state is shared using the filesystem.

We do a lot of tuning on a project by project basis.

At the end of the prediction, there are several transformations to derive other forest characteristics and munge the data into regional standards.

The whole modeling process is done on a somewhat high powered desktop. There is one directory that holds all the code and configs; scripts are invoked which pull settings from a corresponding config. Some of the code is stateful (i.e. stores configuration) and the configs are global (their location is not abstracted) so in order to run this process concurrently, it has to be deployed in separate machines.

Municipal Sewer Backup \</comment><comment></comment><comment></comment><comment>----

We ported some of the components described above to predict sewer backup risk. Key components were broken into R and python libraries and dockerized. The libraries are here pylearn - [https://github.com/tesera/pylearn](https://github.com/tesera/pylearn) and rlearn - [https://github.com/tesera/rlearn](https://github.com/tesera/rlearn).

A python library (learn-cli, which we haven't open sourced) uses r2py to coordinate/share state between the two libraries. The process for training still requires a user to select a model from all of the candidates that the variable selection routine produces. This selection is made a prediction time; all the candidate models are stored in one file and an id is specified for which one to use. learn-cli is dockerized and we have it deployed on ECS. It scales pretty well.

This solved many of the challenges in the forestry pipeline, but we haven't been able to bring everything from the forestry model into deployment this way due to a gap in data science and development. I've been looking into Azure Machine Learning as a possible solution for this. I have benchmarked some builtin models there and gotten identical performance as with our highly customized process.

\</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>-----

Would love to hear your advice for formalizing & automating, or alternatives to, our process for the forestry model pipeline.

Also, if you have a highly automated machine learning pipeline - what are your data scientist responsibilities? It's not clear to me how our data scientist jobs would evolve if they didn't have to manually run several scripts for fit and select a model and generating the predictions.

<comment><USER35> I'd love to hear more about what you're doing. I'm looking a bit into this space and could share some of what I've seen in the market - my email is in my profile.

<comment><USER8> FYI: HN email field isn't public; you have to put it in the about field too.</comment></comment></comment>


Cheating our children: Suspicious school test scores across the nation - <AUTHOR> http://www.ajc.com/news/cheating-our-children-suspicious-1397022.html <comment><USER3> The problem with education is that these tests treat students as a commodity. "Standardized" tests tend not to be even remotely useful in diagnosing the positive social capability of a human being.

It is a travesty that American teachers and administrators are put in situations where they feel that they have to falsify results because their students are not reaching them. The learning capabilities and specialties of students are as diverse as the rest of human nature, and the best way to create proper incentives for students and teachers is to build a pervasive culture that rewards learning and meaningful contribution to society. Perhaps American education needs to edit its message: aligning extracurricular American cultural values to highlight the positive effects of learning on success will likely increase test scores more than the next set of standardized whatevers that get thrown at them by an overwhelmed, out-of-touch committee.

</comment><comment><USER4> Ensuring that scores are reported correctly won't even start to fix our educational system. We don't need to cut down on cheating; rather, we need to realize that more data is not better data, that teachers know a lot more about teaching than we do, that the teachers were doing a hell of a lot better before we started measuring them, and that we should just let them do their jobs.

<comment><USER5> _...teachers know a lot more about teaching than we do, that the teachers were doing a hell of a lot better before we started measuring them..._

How do you know this? Even in principle, how _could_ you know this?

To me, it sounds like "I was such a good government contractor before they started auditing me and making sure I actually delivered what was promised."

<comment><USER4> There are ways to evaluate teachers other than running tests. In particular, I was in about ninth grade when NCLB hit, and its effects started rolling in in 11th grade. I was sufficiently aware of the situation to observe. It was a lot more like "The government contractors building my bridge were meeting all their deadlines until their managers started demanding documentation in triplicate for every nail they hammered in."

<comment><USER5> Without measurement, how can you know objectives were being met?

<comment><USER4> Again, more measurement is not better measurement. I spent fifty hours a week in a classroom for nearly twelve years. Quite a bit of that was watching my fellow classmates struggle to understand things that I'd learned from the textbook at the beginning of the year. You get pretty good at it.

Also, around 10th grade, every single teacher in the entire school rewrote their syllabus. From there on, the ones with appropriate degrees in math and science - you know, the ones that actually knew what they were doing - complained bitterly about how hard it was to teach both the stuff that mattered and the stuff that was on the test.

<comment><USER5> _Again, more measurement is not better measurement._

Again, how do you know objectives were being met? Please clearly state your evaluation procedure.

 _...complained bitterly about how hard it was to teach both the stuff that mattered and the stuff that was on the test._

I've made similar complaints - Calc syllabi suck. Too many stupid algebra tricks, too little conceptual reasoning. This is a problem with the syllabus, not the test.

All the test does is reveals when you stop doing your job and teach whatever you feel like, rather than what you are being paid to teach.

<comment><USER6> Actually, I'd suggest the tests only reveal when the teacher stops teaching to the test. Also, it's really not fair to hold the teacher's job ransom and say that if they don't teach the way the test demands, they should lose their job. How about giving them the freedom to teach based on what they feel is good for their students?

<comment><USER7> > _How about giving them the freedom to teach based on what they feel is good > for their students?_

The controversy about teaching children to read using either 'whole words' or 'phonics' (and sub-forms of phonics) is a nice example of some teachers not working to an evidence base and causing harm to their students.

<comment><USER4> Even if they aren't teaching from an evidence base and from proper research, they're better at defining requirements than the legislators simply because they're the ones in the classroom and observing their students' progress. I assume you're a programmer, so I'll reframe the question: would you rather design your APIs yourself or have the CEO of your large corporation do it for you?

<comment><USER7> You know that doctors say that this treatment works, because they see it works.

And then someone does a double-blind controlled trial and it turns out that treatment is useless.

(<http://www.nejm.org/doi/full/10.1056/NEJMoa013259>)

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> OK, so they analyzed 69,000 schools by comparing test scores to a model they created predicting what the scores should be, and 200 out of the 69,000, or 0.3%, didn't match the model. (Note: main article says 200 districts, but the associated article showing the data says 200 schools <http://www.ajc.com/news/cheating-our-children-1393866.html>)

It's amazing that 99.7% of schools matched their model. Must be a very good model.

It's ridiculous that they are using this to say there is evidence of cheating though since, even though their model is 99.7% accurate, it is not 100% accurate, and that's to be expected.

The methodololgy also relies on questionable assumptions. They look at 3 year slices to find cases where test scores jumped up one year and then back down the next. They consider these evidence of systematic, district wide cheating. That is unproven. If it was cheating, why would the scores go back down. Is there a correlation then between years of upswing and schools being on the list of underperforming schools about to be forcibly shut down by the feds? That might be evidence of cheating if so. But if that was so it would surely be claimed in the study. Also, district wide cheating means a mass conspiracy in which hundreds of teachers managed to, under direction of a coordinated central authority, successfully cheat for one year but not the next, and to keep the secret, and did so at hundreds of schools. That's quite a conspiracy.

I am sure cheating takes place from time to time, but it is far more likely to happen at the individual teacher level than as something happening district wide and thus coming from a central source because the larger a conspiracy gets, the less likely it is that everyone can keep the secret. Checking for schools that have a single year with higher scores and finding a small number that do is poor evidence of a cheating conspiracy.

<comment><AUTHOR> I appreciate your thoughtful comments, droithomme, on many of the HN threads on education reform. You are correct that the statistical model used by the newspaper report needs more validation. On the other hand, as the story relates, the model is a generalization of a model developed during reporting by the same newspaper about known cheating incidents that could be independently verified by other means, including witness reports of overt conduct that breached test security or altered student-submitted answers after the test booklets were submitted to teachers or school principals. The earlier reporting by the same newspaper revealed that sometimes agreements ("conspiracies") to cheat on tests were rather widespread in entire schools or even entire school districts, and stayed secret for a time (at least) because all the school officials involved were united in having an interest in presenting fake test scores.

I think that the scores bouncing up but then back down is, again, an empirical observation from the known cases of cheating in the area the newspaper reported on most closely a few years ago. As the submitted article here suggests, it is certainly notionally possible that a group of students might spurt forward suddenly in academic performance if they enjoy the instruction of a particularly good teacher. But, if genuine learning had taken place, it is notionally much harder to find the same group of students fall back the next year to a level lower than the level they reached the previous school year. It is much more likely that students might progress in grades from a teacher who is eager to look good, even if it means cheating, to a teacher who is not in on the cheating plot, or who at least is less effective at cheating in a manner that massively boosts scores.

I think some of the other claims that you suggest would be made by the newspaper, if the statistical model were definitely backed up by other evidence of cheating, could be made if journalists in the various localities involved look for other sources of information. Not in all cases, as you correctly point out, but in many cases, additional reporting leg work could probably find direct evidence of violation of test security before the test resulting in exact "teaching to the test" item by item, or alteration of student item responses after students submit their test booklets to school officials for scoring. That is currently an open question, but based on known previous examples in various parts of the United States, there is enough smoke here to prompt journalists to look for sources of fire.

<comment><USER8> Interesting, I didn't catch from the article that this three-year bump model had been tested on multiple incidents and found to correlate. It mentions in the third paragraph their methodology used "a pattern that, in Atlanta, indicated cheating in multiple schools". It seems Atlanta was the only place they are claiming there was a correlation, but even of this single data point they offer no specific evidence beyond the red dot in their graph. I did find it interesting that the biggest recent case where there was systematic corruption within a system was in Atlanta, and the newspaper here trying to show that cheating is common throughout the rest of the US and not just something special in Atlanta is Atlanta's leading newspaper.

Recent analyses of data on 18,000 teachers in New Yock City has shown that year to year testing results are not as consistent as one would assume.

[http://news.firedoglake.com/2012/03/07/data-analysis-of- valu...](http://news.firedoglake.com/2012/03/07/data-analysis-of-value-added- teacher-model-shows-no-correlation/)

> Gary Rubinstein has been analyzing the data at his site, and he's found some > incredible results. First, he found that there's almost no correlation among > teachers year-over-year in the data. A teacher is likely to be judged as > effective one year and ineffective the next. The average change in > performance was a relatively large 25 points, and it did not fit with > commonly accepted beliefs that teachers improve performance over time, > particularly between the first and second year.

[http://garyrubinstein.teachforus.org/2012/02/26/analyzing- re...](http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc- value-added-data-part-1/)

> Looking over the data, I found that 50% of the teachers had a 21 point > 'swing' one way or the other. There were even teachers who had gone up or > down as much as 80 points. The average change was 25 points. I also noticed > that 49% of the teachers got lower value-added in 2010 than they did in > 2009, contrary to my experience that most teachers improve from year to > year.

> With a correlation coefficient of .35, the scatter plot shows that teachers > are not consistent from year to year ... nor do a good number of them go up

These sorts of results where the data is available to the public for analysis makes me skeptical that testing results are actually consistent from year to year and that inconsistencies are evidence of cheating.

I contrast these findings to the AJC article's claim that "Experts say student learning doesn't typically jump backwards." The article does not state which experts they refer to, what their qualifications are, or what exactly they have said. That is suspicious to me, especially since the article has many quotes where they are no so shy about identifying their sources. Shouldn't it be even more critical to identify specific sources when using an appeal to authority argument citing "experts"? Especially when in cases where we have actual data, such as the study I cite above, scores did in fact jump backwards 49% of the time. This difference between empirical data and vague assertions by unnamed experts is suspicious. I must wonder who are these experts.

Mind you, I think there is cheating. I am highly skeptical that the AJC's method has been shown to be a valid way to identify where it is happening.

</comment></comment></comment><comment><USER9> Its not as simple as that. From everything I hear, the teachers and school administrators are being squeezed between a rock and a hard place.

I would guess that there is a subculture of school administrators who feel that No Child Left Behind etc. are hurting curriculum and unfair to teachers and schools.

They have to decide between being honest about their scores or firing a bunch of teachers and/or possibly having their schools shut down, and/or having class sizes they can't cope with due to the need to hold back huge numbers of students.

I'm not a teacher but I have heard a few teachers talk about this. I think its sort like what happens quite often between a manager and his employees: the manager makes some proclamations that are impractical and often unrealistic, and some significant portion of the employees decide they just have to smile and nod, but basically they are ignoring it as invalid. So in this case they feel they are forced to falsify the results.

I think that probably there are some schools in well-off neighborhoods where kids are advantaged and the administrators don't run into these issues that they consider moral dilemmas. The administrators and teachers in those schools may say that the other teachers deserved to be fired based on the poor performance of their students, but those privileged teachers aren't aware of the disadvantages that the other students are working against and therefore don't realize that students' failure to meet objectives often isn't their teacher's fault.

Personally I think that national measurements and guidelines are very important, but fine-grained command centralized control over an entire system has been proven time and again to be repressive and result in failure.

This goes for all sorts of systems, not just education. It is good to try to come up with standards, do holistic analysis, have common data formats and an egalitarian shared perspective and goals. But a system needs to be highly localized and non-homogeneous to be efficient, robust, and have the freedom to evolve and adapt to local conditions.

<comment><USER5> From everything I hear, the traders and managers at failing hedge funds are being squeezed between a rock and a hard place.

They have to decide between being lying about their losses or fire a bunch of people and possibly have their hedge funds shut down.

I'm not a trader, but I have heard a few traders talk about this. I think it's sort of like what happens quite often between an investor and his fund: he makes some proclamations that are impractical and the market doesn't cooperate, and some significant portion of the employees just smile and nod but ignore it and keep collecting their base/bonus. So in this case, they feel they are forced to commit fraud and hide their losses.

I think that probably there are some desks in well-run firms where profits are high and traders don't run into these issues that they consider moral dilemmas. The managers and traders in those firms may say the other traders deserved to be fired based on their losses, but those privileged traders aren't aware of the disadvantages that the other traders are working against and don't realize the market simply moved against them.

Just thought I'd apply the same logic to another profession, albeit a profession for which most people hold few positive feelings.

<comment><USER10> You left out the part parallel to "I would guess that there is a subculture of school administrators who feel that No Child Left Behind etc. are hurting curriculum and unfair to teachers and schools."

The standards schools are being measured against (and rewarded or punished for their performance relative to) are far less objective than "made or lost money". You have not applied the same logic to a different profession, because it misses out on the fundamental point of the comment you are parodying - <USER2> the standard is, in many cases, judging the schools wrongly.

The dilemma is not whether or not to lie about failing. The dilemma is whether or not to lie about a shitty metric that _says_ you are failing when you can see clearly that you are not.

I'm not going to make a claim one way or another about what I think of that argument. I'm just pointing out that the argument you are parodying is not the one you appear to think it is.

<comment><USER5> The standard trading desks are held to is also often less objective than "made or lost money". It's "made or lost money relative to some arbitrary risk- adjusted benchmark". If the risk guys disagrees with your model about how dangerous your strategy is, they could easily mark down your +400bps alpha to +100bps.

And yes, there is a "subculture of traders who feel that the policies enforced by Risk are hurting profits and unfair to traders." Actually it's not so much a subculture as the dominant culture.

<comment><USER10> Fair enough. That wasn't at all clear from your earlier comment, though. In light of this information I'm inclined to agree that there is some similarity. I think there's still a pretty colossal difference, too, though - the risk guys generally work for the same firm, don't they?

To reflect back across the metaphor, in the education world the "risk guys" are a single group of people at a national entity most people already don't trust to manage money, and they are unilaterally declaring the standards for _everyone's_ risk.

</comment></comment></comment></comment></comment><comment><USER11> When you tie incentives and benefits to test score performance, something's gonna give. You can either institute auditing of schools to guard against cheating, or have less incentives riding on school test scores.

If you think about it, the schools with poor test scores are the schools that need more help, not less. The schools performing well are doing great with what they have, they don't need more. Only if a school consistently performs poorly with more help should some punishment take place.</comment>


The Rust split: stable vs. nightly - <AUTHOR> http://xion.io/post/programming/rust-nightly-vs-stable.html <comment><USER2> This is incorrect and seems inflammatory. Please don't turn this into Python 2 vs 3 debate, because this is nothing of the sort.

Nightly isn't stable because we don't want features to land prematurely. Rust is a great language precisely because it it well thought out. Don't rush things into stable and leave us with legacy cruft for the rest of time. That's dangerous and hurts everyone.

Anyone developing their library as nightly-only should expect to see little adoption. I only develop against stable and I'm quite productive with the tooling. Iron and Hyper are excellent web frameworks, and rustc_serialize gets the job done.

I can't wait to see Steve or another core contributor weigh in on this piece.

<comment><USER3> The biggest difference to the situation with python 2 vs 3 is that code written in stable rust also runs in unstable rust.

A related thing: So far I have only written code using stable rust. I see some people using nightly, but I've never seen someone talking about using beta. Are there numbers about beta downloads? Does beta still fulfill its task if barely anyone is using it?

<comment><USER4> There's some discussion of allowing beta for nightly Firefox builds. Since the Rust release cycle is slightly shorter than the Firefox one, this means Firefox release will always ship on stable Rust. Meanwhile, nightly can get bug fixes or features slightly ahead of the normal schedule.

</comment></comment></comment><comment><USER5> I maintain a dozen or two Rust libraries and programs, and close to half of them are real-world production code.

Exactly _one_ of them requires nightly Rust, and that's because it uses a feature (Macros 1.1) scheduled to show up in stable in just under 6 weeks.

There are two reasons not to use nightly:

1\. Stuff breaks. I remember the bad old days before Rust 1.0 and its iron- clad stability guarantees. I spent my time fixing a endless stream of GitHub issues reports across a half dozen projects. Modern nightly isn't so bad, but I'm still guiltily sitting on a PR about nightly breakage (as usual). But something that works in stable Rust just keeps working.

2\. Stuff _goes away_. Nightly only features do get removed, and then you're stuck only building with an ancient nightly build.

</comment><comment><USER6> There's so much incorrect information in this post that I almost know where to begin to refute it. An RFC to stabilize extensions was accepted literally last week. The vast majority of code that is only on nightly is because of unstable deriving, which will be stable in less than six weeks. The community is emphatically not embracing nightly, exactly the opposite. A Python scenario is impossible because stable is a strict subset of nightly, and even using unstable features on nightly requires explicit opt-in. And so on.

</comment><comment><USER7> This seems like mostly hyperbole. Rust 1.15, which is a bit over a month or so away, is slated to have macros 1.1 which covers much of the need for compiler plugins. As I understand it macros 1.1 was in part prioritised by the Rust team so that users wouldn't have to use nightlies.

</comment><comment><USER8> If no one develops serious bleeding edge bullshit against rust nightly, the features that are only in rust nightly aren't gonna get any real-world experience that can inform whether they should go into rust stable, so it sounds like the dichotomy is working as designed to me.

</comment><comment><USER9> I do agree that there is a problem with the nightly/stable split in Rust, but I think a big part of this problem will be solved by macros 1.1: [https://github.com/rust- lang/rfcs/blob/master/text/1681-macr...](https://github.com/rust- lang/rfcs/blob/master/text/1681-macros-1.1.md)

</comment><comment><USER10> The comments on /r/rust are worth reading

[https://www.reddit.com/r/rust/comments/5k4sep/a_tale_of_two_...](https://www.reddit.com/r/rust/comments/5k4sep/a_tale_of_two_rusts/)

</comment><comment><USER11> One real issue with the nightly/stable split is that it hasn't been entirely clear when things will actually be making it into stable. I understand that software development is full of uncertainties and sometimes you just don't know how long an implementation will take until you actually build it. But right now, to someone not involved in developing Rust itself, it's hard to get an idea of what the nightly-only features are, where they are in the stabilization process, and when they might be expected to appear in stable. For a concrete example, I only found out that custom derive will be out in 1.15 because I happened to read Hacker News. And there doesn't seem to be any index of RFCs or where they are in the process at all, or at least not anywhere that I could easily find. In the core team's defense, they're eyeballs deep in this stuff all day every day, so they have a pretty good idea about what is where and how far along things are. But us users who only glance at what's in each new release and only upgrade every 6 months would really like to know whether to bother to bother installing nightly for the shiny new feature or just wait for the next release that will have it and be actually stable.

<comment><USER12> Following this week in rust helps me keep track of what's happening. I also occasionally skim the the internals discourse to see what kind of topics are being discussed.

</comment></comment><comment><USER13> There's another distinction that is not made here!

I recommend the nightly toolchain for development, but to target the stable Rust platform. This is possible because Rust is that good and unbuggy. Use nightly to get the best development tools, stable for the best platform (that doesn't break builds randomly).

</comment><comment><USER14> > at least one feature that won't get stabilized ever: > #![feature(plugin)]

> This is how they are utilized by Diesel, for example, a popular ORM and SQL > query interface; or Serde, a serialization framework.

Diesel and Serde are going to be fully available on Rust 1.15, which is < 6 weeks away. This is actually an example of the complete opposite: the Rust project is working hard to get important projects to work on stable Rust so that people _don 't_ need nightlies.

</comment><comment><USER15> Dependence on the nightly channel has been one of the biggest risks in Rust. The problem is not solved yet, but it will be. The Rust stable / nightly split is disimilar to the Python 2 / 3 transition in important ways. The notion that 'nightly Rust is the only Rust' is premature, and Rustaceans should be cautious about encouraging that notion.

It's undeniable that there is a significant amount of Rust software still reliant on unstable features, but I have not seen any numbers that would put it in perspective. Having consistent measurements of nightly vs stable adoption would help a lot, inform any conversation, and efforts to move off nightly.

So I think it's premature to say that 'nightly Rust is the only Rust', The article suggests that people are using nightly as the de-facto recommended toolchain, without evidence (though I'm sure somebody has recommended nightly be default). I'd encourage people not to recommend nightly and for Rust library developers to aim for their crates to be compatible with stable. I personally don't have the same sense as the author that Rust application or library developers are using nightly more than they have in the past. Rather I know that developers of key nightly-only libraries are actively involved in ecosystem migration off of nightly.

One of the biggest difference between the 2 / 3 split is that stable is generally compatible with nightly, and is intended to catch up with nightly, whereas Python 2 is not such a strict subset of Python 3 and its development is done. That is, nightly is intended to be a staging ground for stable; Python 3 does not have that same relationship to Python 2.

(Edit: I stated this awkwardly. In this comparison nightly=Python 2, stable=Python 3. The point stands that the relationships between the two branches are different between the two languages. Nightly development feeds into Stable, Python 2 doesn't feed into Python 3, it is not under active development.)

Because of this relationship there's still reason to have confidence this dependency on nightly will shrink over time. But not disappear. We would always expect some people to depend on nightly. That is its intended purpose - as a breeding ground for future features.

Some of the biggest features keeping people off stable Rust are coming, and their biggest users are actively migrating. In the near future there will be fewer reasons to use nightly, not more. And beyond that the reasons to stick to nightly will evolve as Rust evolves. The next demographic of users who will be stuck on nightly will be embedded developers, and that contingent will increase over time. But even as that group increases, the group that is freed up to use stable Rust will grow also. And slowly the stable ecosystem will coalesce for all classes of users and all types of code.

Though I don't believe it's happening to the extent the author does, I'd rather not see Rustaceans who care about Rust's future to adopt a nighly-first mindset. More things will get more stable, faster, by working together. But regardless of people's dispositions to the stable / nightly split, more of the ecosystem is going to work on stable over time - that is how the Rust feature pipeline works - and eventually stable Rust will be the only Rust.

I still expect it to all work out ok.

</comment><comment><USER16> i often encounter xion in the rust irc and he is always a great help. cheers.

<comment><USER7> It seems odd that someone connected with the community would make a statement like "Stable Rust is dead. Nightly Rust is the only Rust." given much of his argument deflates in a month or so. Perhaps important developments are not being disseminated as well as I'd thought.

<comment><USER6> AFAICT the author appears to be hastily overreacting to the announcement of Rocket yesterday, which is a web framework that uses unstable features.</comment></comment></comment>


Children need to be bored, so I'm smashing the Wii - <AUTHOR> http://www.telegraph.co.uk/comment/columnists/nigelfarndale/6890370/Children-need-to-be-bored-so-Im-smashing-the-Wii.html

<comment><USER2> (Commenting didn't seem to want to work on the site, so I'll comment here.)

Don't blame video games for discouraging reading; blame yourselves for getting the wrong video games. I grew up on Final Fantasy, Dragon Warrior, Civilization, etc. These involved massive amounts of reading, all of which was fun because it let me discover worlds that I was able to interact with.

<comment><USER3> You can't force them to play those games; they want to play the games their friends are playing... basically guitar hero</comment></comment>


Using Google Earth to save Haiti - <AUTHOR> http://dailycaller.com/2010/02/18/haiti-helps-pentagon-dissipate-the-fog-of-war/

<comment><AUTHOR> The main product referenced about 2/3 of the way down (UDOP) was built by my company (Thermopylae Sciences & Technologies) in the 2 weeks after the Haiti earthquake struck. It lets users upload, create, and edit content within their browser, as well as giving them access to a number of common tools, all in one place. It's currently the best visualization tool out there, and it's being used by a whole bunch of different agencies to coordinate relief efforts and spread information.

I'd be happy to answer any questions about it -- I know the article is a little light on technical details.</comment>