Curious Link Between the Fly-By Anomaly and the "Impossible" EmDrive Thruster - <AUTHOR> https://www.technologyreview.com/s/601299/the-curious-link-between-the-fly-by-anomaly-and-the-impossible-emdrive-thruster/ <comment><USER3> I think what I most enjoy about the possibility of the EmDrive is that it's similar to how things were often discovered during the "golden age" of science.

Often, engineers or inventors would create something and then scientists would have to explain why it happened. In the last few generations, physicists and mathematicians would come up with theories and engineers would have to build equipment to test those theories.

The EmDrive is one of the rare modern situations where someone has engineered a device that shouldn't work according to what we know and the scientists are having to come up with the explanation.

Personally, solving a mystery is more exciting than purely intellectual theories and the EmDrive has created a very interesting mystery.

<comment><USER4> > _I think what I most enjoy about the possibility of the EmDrive is that it > 's similar to how things were often discovered during the "golden age" of > science._

There were also a lot of false discoveries. Someone made a machine that does something, and after a few iterations it was discovered that it was only a bad measurement (or a scam). It happens also with theoretical ideas, the most famous are aether and flogisto. They were good ideas, they explained a lot of experiments, but after a time it became obvious that they were wrong.

> _The EmDrive is one of the rare modern situations where someone has > engineered a device that shouldn 't work according to what we know and the > scientists are having to come up with the explanation._

The obvious explanation is that they have an error in the measurement. Probably something get very hot and the thermal effect produce a tiny force, some experiments show that it need some time to build up and it continue a little after the current is off, like a thermal effect. Another possible explanation is that the high currents produce a magnetic field that produce a tiny force. They are putting a lot of electricity and energy in a small device, and they only measure a tiny force, it's very difficult to rule out experimental errors. A similar case are the faster than light neutrinos, that were the product of a bad measurement.

<comment><USER5> Measurement error is a fine, and even necessary hypothesis in attempting to explain the observed effect. Multiple, independent replications tend to cast doubt on the ultimate viability of that hypothesis, however. How likely is it that all of these people at all of these labs are making the same measurement error?

And then, on top of that, when the observed effect comports with a theory that also predicts another, well-established observed effect?

It's been said that the most important phrase you will ever hear a scientist utter isn't, "I've found it!", but rather, "Well, _that 's_ strange."

This is very much in the, "Well, that's strange" class of things. Either way, we're going to learn something about how the universe works, or about our ability to measure it, or both. This should be celebrated, not dismissed as mere "measurement error."

<comment><USER6> People noticed an effect every time the Dean Drive was turned on... but it is just a stiction engine and obviously won't work in space. It could be this drive is actually generating magnetic forces or moving air around and thus "creating thrust" by pushing off its surroundings, which won't work in a vacuum.

<comment><USER7> ... which was already tested for. vacuum or not seems to make no difference on the effect.

<comment><USER8> It could also push off the Earth magnetic field.

<comment><USER7> Then it should depend on the orientation (and altitude, geographical location). According to [1] this was tested for.

[1] [http://emdrive.com/faq.html](http://emdrive.com/faq.html)

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> > Since then, something interesting has happened. Various teams around the > world have begun to build their own versions of the EmDrive and put them > through their paces. And to everyone's surprise, they've begun to reproduce > Shawyer's results. The EmDrive, it seems, really does produce thrust.

That's a misleading statement. I'm passingly familiar with a few of the experiments they're referring to, and none of them both produced significant results and were performed by groups which seemed un-suspect. I'm not aware of any peer reviewed paper on this stuff, and I don't personally know any non- laypeople who believe there is anything actually remarkable happening here.

[edit] fixed some grammar

<comment><USER10> The article mentions NASA tested it. Is that untrue or less true than it sounds? Genuinely curious.

<comment><USER11> Here is a good summary of the state of EmDrive testing (as of a year ago): [https://www.reddit.com/comments/34cq1b/](https://www.reddit.com/comments/34cq1b/)

TL;DR - a small, experimental division within NASA called Eagleworks tested the device. They are a very small group with very limited funding tasked with exploring unconventional theories around advanced propulsion. Their results were not published in a peer-reviewed journal, and there is considerable disagreement as to the validity of their experiment. They are continuing to refine their experiments, and the last update provided is that they intend to publish a peer-reviewed paper describing an experiment that successfully breached 100uN of thrust, which was the target needed for JPL and others to attempt to replicate their results.

<comment><USER12> Thanks for the link - that comment is great.

I gotta quote one line that is just cool, though:

      4. A test at 50 W of power during which an
      interferometer (a modified Michelson device)
      was used to measure the stretching and compressing
      of spacetime within the device, which produced
      initial results that were consistent with an
      Alcubierre drive fluctuation.

Which is just really cool. The followup sums up my fascination with the whole thing niceley:

      Test #4 was performed, essentially, on a whim
      by the research team as they were bouncing ideas
      off each other, and was entirely unexpected. They
      are extremely hesitant to draw any conclusions based
      on test #4, although they certainly found it interesting.

That right there is science in progress: a hint of something interesting, a test leading to another test leading to another, each interesting in their own right. It's still not clear if something spectacular is going on, but I think it's undeniably interesting. Careful reexamination can yeild all manner of useful insights!

(Off topic, but along the lines of closer examination, see [https://news.ycombinator.com/item?id=9020065](https://news.ycombinator.com/item?id=9020065) for how some careful (and extensive!) experimentation helped illuminate sodium's reaction with water. Not just how much work was needed to tease out the details!)

<comment><USER13> > The most exciting phrase to hear in science, the one that heralds new > discoveries, is not 'Eureka!' but 'That's funny...' \- Isaac Asimov.

This could be complete bunkum but it's fun watching anyway as an interested bystander.

</comment></comment></comment></comment></comment><comment><USER14> I don't think there's anything real here. McCulloch's papers on arXiv seem very confused. Eg., this paragraph:

"In this scheme there is a minimum allowed acceleration which depends on a Hubble scale Θ, so, if Θ has increased in cosmic time, there should be a positive correlation between the anomalous centripetal acceleration seen in equivalent galaxies, and their distance from us, since the more distant ones are seen further back in time when, if the universe has indeed been expanding, Θ was smaller. The mass to light ratio (M/L) does seem to increase as we look further away. The M/L ratio of the Sun is 1 by definition, for nearby stars it is 2, for galaxies' it is 50, for galaxy pairs it is 100 and for clusters it is 300. As an aside: equation (11) could be used to model inflation, since when Θ was small in the early universe the minimum acceleration is predicted to be larger." ([http://arxiv.org/pdf/astro- ph/0612599v1.pdf](http://arxiv.org/pdf/astro-ph/0612599v1.pdf))

If an effect was stronger in the early universe, you'd expect to see a big correlation between the effect size in a galaxy, and that galaxy's redshift z. It wouldn't make any sense to say that "galaxies" have a ratio of 50, since there are galaxies at every redshift; many are nearby and have redshifts of almost zero, while the Ultra Deep Field galaxies have very large redshifts of up to ~10. If the number is really the same for "galaxies" in general, that means there's no distance dependence, but McCulloch doesn't seem to realize this. He seems to imply that nearby stars have a higher mass/luminosity ratio because of their distance compared to the Sun (?!), but the time-delay effect for anything in the Milky Way is negligible (< 0.0005% of the universe's age). In reality, nearby areas of space will have higher ratios than the Sun just because they contain many objects which, unlike the Sun, don't emit much light (red/brown/white dwarfs, gas and dust, etc.). Likewise, he seems to imply that "galaxy clusters" are farther away than "galaxies", but most galaxies are part of clusters, and we can observe both galaxies and galaxy clusters at both small and large redshifts.

<comment><USER15> Isn't he saying the 'standard' M/L ratio for galaxies is 50... and that by looking at a lot of galaxies at different distances you see a trend that the more distant ones are more >50 ?

</comment></comment><comment><USER16> The hard part for physicists to accept is that the theory requires the speed of light to change.

It would be interesting to see how the theory behind the Unruh radiation works with "The quantum vacuum as the origin of the speed of light" ([http://arxiv.org/abs/1302.6165#](http://arxiv.org/abs/1302.6165#))

Or also MOND ([https://en.wikipedia.org/wiki/Modified_Newtonian_dynamics](https://en.wikipedia.org/wiki/Modified_Newtonian_dynamics)) Which also has predictions based on very low accelerations.

It seems like the theories could be related.

(says an ex Nuclear physicist who's now doing computers).

<comment><USER17> > The hard part for physicists to accept is that the theory requires the speed > of light to change.

I'm a complete layman here, so I don't understand how the link you shared says that the speed of light is changing; it seems more like a change in the way we _define_ the speed of light. Is that right?

If the actual speed of light is to change, how is that possible? We've experimentally verified it up one side and down the other, so this seems like a huuuuuge discovery.

<comment><USER18> So there's c, which is essentially a conversion factor between space and time, and also happens to be the speed at which massless particles move. If photons happen to not be massless, the speed of light might change, though 'c' wouldn't have to.

</comment></comment></comment><comment><USER19> The sentiment I hear in this thread is like those that dismiss cold fusion. Sure, we can all make statements like "obviously, someone screwed up" but it is another thing entirely to have to the patience to simply cite the experiments that disprove the proposed effect. I don't know what to make of cold fusion, but I also know for a fact that neither do physicists, and instead of studying it, they're just simply saying because there is no theory it must not work. Same with this stuff.

The one YouTube guy discovered the beaded-chain lifting effect, and _then_ it had to be studied to find out what was going on. Obviously that was an easily reproduced experiment.

So with this thing, we must find conclusively the unmeasured heat or ions or whatever and show a repeatable method for such mistakes. That is my opinion about science, of course I probably lost most scientists with my first sentence.

<comment><USER20> Extraordinary claims require extraordinary proof, and the history of physics is littered with intentional fraudsters and sloppy experimentalists. Disproving every perpetual motion machine via experiment would take a lot of effort from scientists who don't want to do it. If you think those experiments should be done, then by all means do them.

<comment><USER19> Fair enough, but the problem with perpetual motion stuff is that no energy can be extracted or the time until the device stops is very long, not that they don't do what people claim that they do, which is work more or less perpetually.

We need to find that kind of explanation for this supposed propulsion.

<comment><USER21> These devices work "more or less perpetually" the same way 1000.0f is "more or less infinite".

<comment><USER19> I upvoted you because you're right, but maybe you and I should take a few months off and time some of the better ones! Worthless machines but I've heard of some lasting a great deal of time.

</comment></comment></comment></comment></comment><comment><AUTHOR> I'm curious to what degree Unruh radiation (and the resulting consequences this idea relies on) 'established physics'?

By extension, I think this is the most interesting article I've seen to date on the EmDrive: it seems to have a basis in a fairly non-controversial result of GR, which in turn is something which nicely explains an otherwise bizarre physical phenomenon. And, to top it off, there are a number of falsifiable predictions which are within our ability to test. I'm interested in whether any of my assumptions are wrong.

<comment><USER22> > I'm curious to what degree Unruh radiation (and the resulting consequences > this idea relies on) 'established physics'?

Most theorists think Hawking–Unruh radiation has to exist, I guess because the calculation is relatively straightforward. You just need special relativity and some basic field theory.

Hawking–Unruh radiation has never been observed in a gravitational system, and doing so would be very difficult because it requires a truly enormous acceleration in order to produce a horizon with a temperature that is an appreciable fraction of 1 K.

There are certain regimes in fluid mechanics where the mathematics governing the fluid looks like the mathematics governing relativity. Some people have produced phenomena in those systems that look like Hawking–Unruh radiation. Some people further claim that this demonstrates the existence of Hawking–Unruh radiation for gravitational systems, but I find that reasoning to be specious.

</comment></comment><comment><USER23> Extraordinary claims require extraordinary yada yada.

A working reactionless drive isn't just extraordinary. It's utterly mind- boggling. The least interesting thing is that it's a free-energy device.

It requires breaking spatial symmetry. If it works, it's not some edge-case theoretical law that's being broken. It's the geometry of space.

[https://en.wikipedia.org/wiki/Noether%27s_theorem](https://en.wikipedia.org/wiki/Noether%27s_theorem)

<comment><USER24> Even if this works, _reactionless_ drive would be my last hypothesis after exhausting all possible things it might be interacting with via some as-yet- unknown means: dark matter, nearby gravitational bodies, etc.

The infinite energy device claim is easy enough to test. Since a working infinite energy device would be perverse, I'd predict that an EmDrive rigged up so as to produce infinite energy would fail to do so. Exactly _how_ it fails to do so might tell us what's going on. Do the energy requirements rise with momentum (as they should in a sane universe), or do you pass a point at which the effect ceases, or does something truly wacky occur like space-time distortion in such a way as to cancel the effect?

My reading of the McCullough hypothesis is that it's pushing against all matter in the universe at once, or some such thing, but that could be way off. I'm not a physicist.

</comment></comment><comment><USER25> This is probably incredibly naive... but couldn't this all be put to bed by someone putting such a drive on a cubesat, lifting it into orbit, and seeing if it works under practical conditions?

Maybe Hawking and Milner should be considering this for Starshot?

<comment><USER26> Sure, the tricky part is finding funding

<comment><USER27> How much funding?

A Falcon 9 can lift 10,000 kg to LOE (or more). Let's say we could get a simple Em-drive and basic satellite and solar panel in for under 100kg. (We can do better, but let's start there). That's 1/100 of the load, so (in theory) we could maybe pay 1/100 of the $60m cost for a Falcon 9 lift, or $600,000.

Half a million bucks plus some change, let's say.

Edit: Less if we can get on board a previously-enjoyed rocket!

<comment><USER17> You also have to add in the payload integration fees, testing (very, very expensive), the actual payload development, insurance, and also be in a launch configuration such that you can piggyback behind a payload taking up most of the F9 capacity.

I'd be astonished if the cost of this was less than $2-3M.

</comment></comment></comment></comment><comment><USER28> refuted by reddit

[https://www.reddit.com/r/EmDrive/comments/4fm2ac/the_curious...](https://www.reddit.com/r/EmDrive/comments/4fm2ac/the_curious_link_between_the_flyby_anomaly_and/)

> ... (McCulloch) proposes a constant term that modifies the acceleration > corresponding to the inertial mass. He says torsion balance experiments > can't detect it because torsion balance experiments measure differences in > acceleration. But he's wrong because since it's a constant term he > "predicts", it should manifest in the Eotvos parameter. Torsion balance > experiments have gone well beyond the limit to detect this. But it's > irrelevant because he completely misunderstands all the theory he bases this > on.

</comment><comment><USER29> The article leaves out all the numbers and so skips over one little problem. The momentum some people think they might have observed (if it's not experimental error) is compatible with what you'd get by using a microwave antenna as a thruster. Just ordinary radiation pressure, with the puzzle of how the radiation could be escaping the cavity.

But the minimal measurement results I've seen are not compatible with the radiation pressure multiplied by some large factor for the Q of the cavity, which seems to be the claim from some. That really would violate our understanding of conservation of momentum, rather than violating our assumptions about where the momentum goes in this experiment. And that seems to be ruled out experimentally so far.

</comment><comment><USER30> > At very small accelerations, the wavelengths become so large they can no > longer fit in the observable universe. When this happens, inertia can take > only certain whole-wavelength values and so jumps from one value to the > next.

So the EmDrive glitches the universe size? This is hilarious.

<comment><USER31> This was also my first reaction. I don't see how energy could be quantized with the inverse of the diameter of the observable universe.

Could someone explain the current thinking around how energy quanta relate to the size of the universe, or rather why wavelengths larger than the universe are impossible?

If the latter were true, I'd expect that an energy quantum corresponding to 0.99 * universe would be equally impossible as 1.01 * universe, and that only integer multiples of the corresponding frequency would be allowed (i.e. a harmonic series across the universe).

<comment><USER32> The corollary is that you'd be able to measure the size of the universe by observing quantisation effects.

This seems unlikely, because - unless I've missed something obvious - you'd have to be able to set up a wave with a phase velocity > c.

</comment></comment></comment><comment><USER33> Photons must behave as if they have mass. From TFA:

McCulloch's theory could help to change that, although it is hardly a mainstream idea. It makes two challenging assumptions. The first is that photons have inertial mass.

When I was taking college physics, there was a question on the exam about radiation pressure. I missed that day, so had no idea how to solve it. "a 5mW laser is reflected off a mirror (perpendicular) what is the force exerted on the mirror"? Later looking it up in the book there was a page on this and a derivation using electromagnetic theory. In the exam however, I decided to convert 1 second of laser energy to mass, bounce it off the mirror at speed=c, compute the force and change in momentum (over change in time which was 1s). I got the right answer of course.

The logic is simple. If we can convert back and forth between matter and energy, any experimental setup must obey conservation of momentum and it's CG must not move. So a laser inside a closed spaceship would actually be tranfering mass (as energy) from one end to the other. The net effect must be the same as if that mass was moved any other way.

I derived a general expression for radiation pressure after the exam and it's identical to the EM one from the book. Photons behave - and must behave - <USER2> if they have mass with a velocity of c. By the same reasoning, gravity must bend light rays, though I have not compared this prediction to that of relativity.

</comment><comment><USER34> I can't help but look at things like this as hacks in/of the simulation we are probably living in.

<comment><USER35> Interestingly, that would imply we are possibly living in an imperfect simulation, one that hypothetically does not replicate the physics of a real non-simulated universe.

If that is true, why? I can think of four possible reasons:

1) Is it because the simulation was designed in such a way that it leaves its inhabitants clues that reveal the true nature of the universe? This lets them figure out the truth once their society and knowledge become sufficiently advanced.

2) That it's impossible to simulate a real universe? This would imply any sufficiently advanced society will be able to detect that they do in fact live within a simulation.

3) That the creators made a mistake and it's not a fundamental limitation. It's simply an error made by whatever civilisation created this simulation, they messed something up.

4) That it's possible to create a perfect simulation, but for some reason they decided to make a simulation that used less resources and this brings errors or the need of hacks to get it working. Now we might be exploring these hacks.

</comment></comment><comment><USER36> This is fascinating stuff. It is also good to see more and more respectable names looking at this. Good things will come from that regardless of the overall outcome of the emdrive.

</comment><comment><USER24> "Yes, they're still using warp drives to heat food..." \- alien observers

</comment><comment><USER37> So is there any microwave radiation escaping the cone? If more of it escapes from one side than the other, won't the momentum of the escaping photons (radiation) be the reaction of the thrust produced? Why does it need special physics? I am probably missing something here, but what is it?

</comment><comment><USER38> One of the main issues I see with the EmDrive in general is time invariance. And especially with the theory Mike McCulloch put forward. If photons cause this process for a positive change in momentum, the same process should happen in reverse with a negative change in momentum.

</comment><comment><USER39> People have been doing "research" into reactionless propulsion for a long time, such as this science fiction writer:

[http://rexresearch.com/dean/stine.htm](http://rexresearch.com/dean/stine.htm)

</comment><comment><USER40> > inertia must quantized at small accelerations.

as everything does :) I think it is quantized at all levels, it just becomes noticeable at low levels as usually.

<comment><USER41> Everything seems to be limited by the precision our simulation is running on :)

</comment></comment><comment><USER42> Humans have learned a lot of things. However they still have to learn about the universe because there are non-naturally-observable physical phenomena and we have no evidence that we discovered them all.

Edit: These so called "laws" are laws in our minds and things like EmDrive show us that our minds can expand forming new "laws".

</comment><comment><USER43> For anybody who's interested, there's a NASA forum devoted to the EmDrive:

[http://forum.nasaspaceflight.com/index.php?topic=39772.1440](http://forum.nasaspaceflight.com/index.php?topic=39772.1440)

</comment><comment><USER44> Inertia is quantized? Do we actually live inside a computer, because that's really weird. Physicists, help explain this outside of the universe uses a lookup table to approximate inertia.

</comment><comment><USER45> Supposing this theory is true, how big an emdrive does it predict is needed to lift a reasonable payload, say 100kg?

<comment><USER46> The EmDrive would never be powerful enough to be a lifting engine; but what it would be is an incredibly useful engine once your mass is in orbit.

</comment></comment><comment><USER47> wonder if this inertial/acceleration effect is what is responsible for "dark matter" in the universe, i.e. galaxy-scale gravitational anomalies that don't agree with existing newtonian and GR models.

</comment><comment><USER48> Would be very interesting to see an actual implementation.

</comment><comment><USER49> Meta: Wow that drop down menu at the top is really annoying

</comment><comment><USER50> Is there a thing that allows for Science Annonymous?

</comment><comment><USER51> How does this relate to the spinning black/white thing?

<comment><USER52> The Black White thing is a Crookes Radiometer[1] and it rotates because of differential heating of the black and white sides of the plates interacting with air molecules to create a tiny air flow from the light side to the dark side of the plate.

The device is under _partial_ vacuum. Under total vacuum, no rotation will occur, if the air pressure is too high, drag forces outpace the tiny thrust.

[https://en.wikipedia.org/wiki/Crookes_radiometer](https://en.wikipedia.org/wiki/Crookes_radiometer)

</comment></comment><comment><USER53> Man, this is the perfect news story for 4:20.</comment>


Pre-order: Play for Scala by Peter Hilton, et al.  - johnx123-up http://www.manning.com/hilton/

<comment><AUTHOR> Was searching for a book like this just the other day.

</comment><comment><USER2> Anyone with any experience using Play?

<comment><USER3> I was at 3 Day Startup San Antonio this weekend and used Play to prototype a platform for e-learning. Deployed it to Heroku and I was happy with it.

It wasn't my first time using Play but I'm still getting a handle on it. I'm about to place a pre-order for this book as I think Play has a lot of promise but is still missing a few important features. It's also severely lacking in documentation.

I also used Play to build the front-end for my thesis project. My thesis was about taking a set of related Wikipedia documents, running them through Stanford's NLP library and using that to construct a semantic graph. The front-end for traversing that graph and integrating with Google Maps was all built in Play. By the time I was done it could show and justify a connection between the string "Britain" and the sentence "The Allies met at the Tehran and Yalta conferences to [...]" and place those connections on a map.

<comment><USER4> > I also used Play to build the front-end for my thesis project …. By the time > I was done it could show and justify a connection between the string > "Britain" and the sentence "The Allies met at the Tehran and Yalta > conferences to [...]" and place those connections on a map.

This sounds amazing. Is there a demo online?

<comment><USER3> Sorry, not yet. My algorithm had far too low of a signal/noise ratio for me to be confident in showing it off.

While the connections it found made sense it was effectively no more than string matching in most cases. My hypothesis was that using something called a "dependency grammar" I could quickly extract relations between nouns in a sentence. My running every sentence in a document through that I could construct a map/reduce algorithm to build a map of entity relations in a document and further reduce that across a set of documents.

I used the Wikipedia article on World War II and every article linked to from there as my input set.

Though I'm done with the "thesis" portion of it, I'm working on ways to rewrite it. I have a few ideas involving machine learning that I think would significantly improve my results.

I wrote it all in Scala though Play was only used to display results from the database.

</comment></comment></comment></comment><comment><USER5> no shipping to Uruguay :S</comment>


Recommended Self-Study Path for Statistics - <AUTHOR> I'm a middle manager at a large tech company that recently took responsibility for a few engineering teams that do some stats heavy work.  Each producing forecasts where we talk about accuracy of predictions, some that use machine learning & the like.
I have a fairly shallow understanding.  I took a basic stats class in undergrad 20 years ago.  Over the years, I've seen various analysis so I'm not totally lost when people talk about p-values (but also have a lot of gaps where some of the details are lost on me).
I'd like to strengthen my understanding so I can better understand/appreciate/represent what the teams are doing.  Any recommendations on a course of study?
FWIW, I've considered just buying a text book or hiring a tutor.  I looked at classes at the local community colleges, but both the syllabus suggests a fairly slow pace (and most things I feel comfortable with) and when I took a previous class (iOS development) it was so dang slow in pacing. <comment><USER3> Hello, I studied economics and statistics in college and grad school, and worked as a teaching assistant for undergraduate statistics courses. Here is a short, annotated bibliography of my favorite statistics books.

1\. Ayres, Ian (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart

[Good introductory summary of the main concepts in statistics with many real- world examples]

2\. Bernstein, Peter (1996) Against the Gods: Remarkable Story of Risk

[Intellectual history of statistics, accessible to beginning students.]

3\. Healey, Joseph (2005) Statistics: A Tool for Social Research, 7E

[This is the text book that was used in the undergraduate statistics courses while I was working as a teaching assistant at UC Santa Cruz.]

4\. Kahneman, Daniel (2011) Thinking, Fast and Slow

[Kahneman combines cognitive psychology with statistical concepts; highly recommended]

5\. Silver, Nate (2012) Signal and the Noise: Why So Many Predictions Fail, but Some Don't

[Silver's book offers an excellent summary of major concepts in statistics and how they are applied to real-world problems]

6\. Taleb, Nassim Nicholas (2005) Fooled by Randomness, 2E

_________ (2010) Black Swan: Impact of the Highly Improbable, 2E

[Important critique of statistics and how it is mis-used and mis-applied, particularly in econometrics]

Hope this helps. Shoot me an email if you have any questions. Good luck. email@example.com

<comment><USER4> These books are more for casual reading, except for Book 3. If I may, do you have more recommendations similar to Book 3? In addition to my personal interest, my read of the OP request was he looking for more technical details. For myself, what books are good for a second or third course in stats? I have a finance background, so I'm familiar with the intro stuff in Book 3.

As a recommendation to the OP, "Collective Intelligence" by Toby Segaran is amazing.

<comment><USER3> The only advanced textbook on statistics that I would recommend is: Kennedy, Peter (2008) Guide to Econometrics, 6E

</comment></comment></comment><comment><USER5> I encounter this a lot at work. People needing more advanced stats skills for a new role and not having much training in it or if they did it was years ago. (I work in finance which has become increasingly quant and stats heavy - faster than training in it has).

My advice: Figure out exactly what type of stats work your teams are doing. Make a list of those topics. Random example: are those Kolmogorov–Smirnov or Mann–Whitney tests? Then hire a tutor who knows that stuff - maybe a grad student somewhere, can be remote over skype even.

If you are not 100% sure what you are looking at at work and what to put on this list of topics...hire a tutor and show them stuff from work (if the work is proprietary/confidential, recreate it with dummy data or just give rough examples) and ask what topics would be needed to nail one's understanding of this work.

Statistics is a huge subject and if you buy a textbook you may spend a ton of time on stuff that's just not relevant when you could be going a bit deeper into a sub-topic that is very relevant to your work. Also a lot of what looks like statistics is actually found under applied math books/courses not statistics.

Lastly, in case this needs be said, after you get the basics on a stats topic, the most important question to ask a stats tutor is "where do people usually fuck up when doing this?"

 _Stats in practice is often more about not making errors than it is about accuracy. Find out where people often fuck it up, especially as a manager and 2x as they are not statisticians either it sounds like._

<comment><USER6> "Stats in practice is often more about not making errors than it is about accuracy."

What's the nuance? (Serious question)

<comment><USER5> TL;DR What to focus on? Some statistics advice: Don't be a liar. Don't be a biased idiot. Don't fuck up. The software should handle the rest.

Nuance is a fair serious question. And this could easily turn into a debate of semantics or philosophy (will add links at bottom tho^).

But what I meant was statistics in practice isn't about proof of some truth but about chance of disproof. An analogy in jurisprudence: there is a difference between "not guilty" and "innocent".

Someone may or may not be "innocent". There's even presumption of innocence. But then in practice, lawyers give evidence to a jury to decide beyond a reasonable doubt if someone is "guilty" or "not guilty".

What's the focus? It sure looks like the work is more focused on "not guilty" vs "innocent".

Furthermore, in statistics there are errors...eg statistical errors, random errors, systematic errors, type 1 errors, non-sampling errors...lots of errors. You can't eliminate them. But you can be aware of them and reduce them where possible.

Now, statistical software deals with errors to the extent statistics techniques exist and the technology can handle the process. Sort of like spellcheck.

But software can't fix everything. Most importantly it can't fix if the person using software is an idiot.

Too many times I have looked like an idiot for sending an email where spellcheck put the wrong word. What to do? I could write a new algo to make spellcheck better or I can just double check the email next time.

^Links to semantics and philosophy stuff: Some fields try to have precise, official definitions for words like "error" and "accuracy".

See ISO 5725 or longer list of examples on wikipedia: [https://en.wikipedia.org/wiki/Accuracy_and_precision](https://en.wikipedia.org/wiki/Accuracy_and_precision)

Of course, philosophy also addresses the nuances. Way more fun to read than ISO technical documentation.

Short list of philosophy of statistics issues on wiki: [https://en.wikipedia.org/wiki/Philosophy_of_statistics](https://en.wikipedia.org/wiki/Philosophy_of_statistics).

Better, longer list, which is worth reading as it includes more interesting and broader philosophy of science issues: [http://plato.stanford.edu/entries/statistics/](http://plato.stanford.edu/entries/statistics/)

If lists of philosophies are overwhelming and you want one random example of it... _What is the probability the sun rises tomorrow?_

Long post. Lastly, a joke: 'A physicist, an engineer, and a statistician go duck hunting. They spot a duck in the distance and the physicist takes the first shot, but just misses left. The engineer shoots next, but just misses right. The statistician yells, "we got it!"'.

[Edit] At this point I might as well add Buffet's 2 rules for investing: "Rule No. 1: Never lose money. Rule No. 2: Never forget rule No.1"

</comment></comment></comment><comment><USER7> Get your own consigliere. Hire a consultant to advise/mentor you individually.

You might find one on Hourly Nerd > [https://hourlynerd.com/your- matches/information-technology-a...](https://hourlynerd.com/your- matches/information-technology-and-services/manage-business/)

<comment><USER8> I agree with the above recommendation.

I have read John Cook's blog for some time and think he might be a good fit as a consultant for bringing you up to speed and reviewing your current teams' work.

[http://www.johndcook.com/blog/](http://www.johndcook.com/blog/)

I suspect it would be worth your time to get a consultant like him to come in for a week, review all the work going on in your org, and maybe lay out a learning plan or crash course in developing a safety net for you.

</comment></comment><comment><USER9> Allen Downey offers many free and excellent textbooks online from his Green Tea Press [0].

You can learn statistics, Bayesian reasoning, and a bunch of other stuff.

Sites like Coursera, edx, and Udacity all have courses for other presentations and applications of statistics at pretty much every degree of difficulty.

[0] [http://greenteapress.com/wp/](http://greenteapress.com/wp/)

</comment><comment><USER10> You can take free online courses to refresh your Statistics knowledge.

Coursera [https://www.coursera.org/browse/data-science/probability- and...](https://www.coursera.org/browse/data-science/probability-and- statistics?languages=en)

Coursera - Making sense of Data [http://academictorrents.com/details/a0cbaf3e03e0893085b6fbdc...](http://academictorrents.com/details/a0cbaf3e03e0893085b6fbdc97cb6220896dddf2)

MIT 6.041 Probabilistic Systems Analysis and Applied Probability [https://www.youtube.com/playlist?list=PLUl4u3cNGP61MdtwGTqZA...](https://www.youtube.com/playlist?list=PLUl4u3cNGP61MdtwGTqZA0MreSaDybji8)

Statistics 110: Probability - <USER2> [https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6...](https://www.youtube.com/playlist?list=PL2SOU6wwxB0uwwH80KTQ6ht66KWxbzTIo)

Udacity also has few courses on Statistics.

</comment><comment><USER11> Old but gold. Kennedys 'A guide to econometrics'. Not so much a textbook as a book that ties theory to practice and explains common pitfalls and intuitions. It's one upmanship in statistical practice.

You shouldn't try to learn stat on par with your teams. Learn to ask the right questions.

If you prefer learning by doing then Elements of statistical learning would give you some modern skills plus add good questions (model testing and prediction are imho more important than base skills, and central to the work of Tibshirani et.al.) to your book.

I think the coaching approach in the other response thread is worthwile as well. If you weren't really into stats before and haven't read up when it wasn't part of your day job, the route of learning the skillset seems a detour. Possible if motivated ofc, but you need advice on managing stat heavy teams. That is a different, though related ballpark.

</comment><comment><USER12> I'm working my way through these videos as way of refreshing my basic knowledge of stats.

[https://www.youtube.com/playlist?list=PL5102DFDC6790F3D0](https://www.youtube.com/playlist?list=PL5102DFDC6790F3D0)

Very well explained.

</comment><comment><USER13> I'm currently addressing this need where I work and posting my training materials at [http://RForecasting.com](http://RForecasting.com) as they are developed. Also I like [https://www.amazon.com/Data-Matters-Conceptual- Statistics-Ra...](https://www.amazon.com/Data-Matters-Conceptual-Statistics- Random/dp/0470412283), but you can see I'm biased on that.

</comment><comment><USER14> CMU's self-paced and free Open Learning Initiative classes on statistics might be helpful early on:

Empirical Research Methods: [http://oli.cmu.edu/courses/future/empirical- research-methods...](http://oli.cmu.edu/courses/future/empirical-research- methods/)

Probability and Statistics: [http://oli.cmu.edu/courses/free-open/statistics- course-detai...](http://oli.cmu.edu/courses/free-open/statistics-course- details/)

Statistical Reasoning: [http://oli.cmu.edu/courses/free-open/statistical- reasoning-c...](http://oli.cmu.edu/courses/free-open/statistical-reasoning- course-details/)

</comment><comment><USER15> This free online textbook using R is what I used when I last taught statistics in a college setting : [https://www.openintro.org/stat/textbook.php](https://www.openintro.org/stat/textbook.php) .

</comment><comment><USER16> Discovering Statistics Using R by Andy Field is a fantastic and entertaining book.

<comment><USER14> Andy Field also posted some great lectures that pair well with Discovering Statistics here: [https://www.youtube.com/user/ProfAndyField/videos](https://www.youtube.com/user/ProfAndyField/videos)

<comment><AUTHOR> This + the book look like a great recommendation. Thank you both!

</comment></comment></comment><comment><USER17> Why not to ask your developers how they studied this field?

</comment><comment><USER18> I've heard good things about Naked Statistics by Charles Wheelan, but I've never actually read it myself

</comment><comment><USER19> Statistics 110 Harvard University video lectures by Joe Blitzstein is a good place to get you started.</comment>


Ask HN: How does UAE block encrypted calls like Watsapp and Telegram? - <AUTHOR> <comment><USER2> This might be a better question on the StackExchange for Information Security [0], they'll give you better and more in-depth explanations than what Hacker News was designed for.

[0]: [https://security.stackexchange.com/](https://security.stackexchange.com/)

</comment><comment><USER3> Possibly related thread that I commented in last year:

[https://news.ycombinator.com/item?id=12039133](https://news.ycombinator.com/item?id=12039133)</comment>


Experts Say U.S. Recession Ended in June 2009 - <AUTHOR> http://www.dailyfinance.com/story/nber-great-recession-ended-in-june-2009/19640587/

<comment><USER2> When did the second recession start, then?

<comment><USER3> This is a rear-view mirror sort of thing, so it may have already but they don't yet have the data needed or are still making up their minds.

The general theory I've heard is that companies re-built their inventories after they let them get low in the extreme uncertainty following the 2008 fall financial shocks. That appears to have tapered off earlier this year and we could be treading water or going back down right now, but as mentioned above it can't yet be called.</comment></comment>


Is Java the platform of the future? by Mark Little, JBoss Tech Lead - <AUTHOR> http://blog.jelastic.com/2012/04/02/is-java-the-platform-of-the-future/

<comment><USER2> Java isn't 'cool' anymore. No one wants to write Java. They want to use the newest flavour of the month.

While it might be possible that everyone gets tired of these and returns to Java, I don't think it's very probable.

If I had to bet, it would be on JavaScript. It's everywhere. It has begun catching the interest of more than just 'browser scripting' and with a little help from some core enhancements in the coming generations, it could really become the 'platform of the future'.</comment>


The best description of the fed bailout - <AUTHOR> http://angrybear.blogspot.com/2008/09/your-urgent-help-needed.html

<comment><USER2> Have you noticed that one of the main results of this saving of things which are "too big too fail" is a concentration of those big businesses into even bigger business, a greater proportion of which are now too big to fail? The rhetoric we had become used to hearing - that liquidity is subtle but very important (which it is) has changed to language of protection and we see policy responses to this. For example - the bans on non-naked shorting hits the liquidity of the options market (because it's harder to do things like hedging when you want to deal in options) and leads to further stagnation.

</comment><comment><USER3> its not that big, should have pasted it:

Dear American:

I need to ask you to support an urgent secret business relationship with a transfer of funds of great magnitude. I am Ministry of the Treasury of the Republic of America. My country has had crisis that has caused the need for large transfer of funds of 800 billion dollars US. If you would assist me in this transfer, it would be most profitable to you. I am working with Mr. Phil Gram, lobbyist for UBS, who will be my replacement as Ministry of the Treasury in January. As a Senator, you may know him as the leader of the American banking deregulation movement in the 1990s. This transactin is 100% safe. This is a matter of great urgency. We need a blank check. We need the funds as quickly as possible. We cannot directly transfer these funds in the names of our close friends because we are constantly under surveillance. My family lawyer advised me that I should look for a reliable and trustworthy person who will act as a next of kin so the funds can be transferred. Please reply with all of your bank account, IRA and college fund account numbers and those of your children and grandchildren to email@example.com so that we may transfer your commission for this transaction. After I receive that information, I will respond with detailed information about safeguards that will be used to protect the funds. Yours Faithfully Minister of Treasury Paulson

<comment><USER4> Yes, but the formatting is important. The screwed up line breaks are not an accident!

</comment></comment><comment><USER5> Here's another good one: <http://www.buymyshitpile.com/>

</comment><comment><USER6> Wow, an article about the bailout that's actually relevant to hackers ;-)

Good surprise.

</comment><comment><USER7> It would have been more convincing if everything is in CAPS.

</comment><comment><USER8> i am still laughing. that is the funniest thing i have read in a while

</comment><comment><USER9> Errr, Americans seem to need somebody to blame for this crisis. But before we lynch Bernake and Paulson, we should all take a good look in the mirror.

<comment><USER10> umm have you actually seen the terms of the bailout that Paulson is trying to sell?

the bailout "shall be limited to $700,000,000,000 outstanding at any one time." - the key words are 'at any one time' meaning this is a blank check for Paulson that he can just keep on drawing

It gets even better: "Decisions by the Secretary pursuant to the authority of this Act are non-reviewable and committed to agency discretion, and may not be reviewed by any court of law or any administrative agency"

so Paulson gets a blank check and we can't see what he's doing with our money - awesome! sign me up

to top it off with icing: "The Secretary is authorized to take such actions as the Secretary deems necessary to carry out the authorities in this Act, including, without limitation... designating financial institutions as financial agents of the Government, and they shall perform all such reasonable duties related to this Act as financial agents of the Government as may be required of them;"

so totalitarian socialism replaces free market enterprise? o yeah what happens to the dollar?

<comment><USER9> Who would you rather be in charge of the bailout? The President? Congress? ... Ill put my trust in Paulson, he and Bernake seem to be the only ones who have a clue as to what is going on and how to fix it.

<comment><USER11> Definitely, Paulson knows more about this than Congress. But Paulson has the perspective that comes with being a former Goldman Sachs CEO, so he might (knowingly or unknowingly) be biased towards helping out the banks too much.

<comment><USER12> Not only that, but the lack of "checks and balances" in the whole deal is shocking.

</comment></comment></comment></comment></comment><comment><USER13> This is a generational phenomenon. The housing-market racket of the late '90s and early 2000s was a case of a large, dominant generation (Baby Boomers; b. 1943-58ish) enriching itself at the expense of a small, recessive generation (Gen X; 1958-80). It occurred during a period when the Baby Boomers were selling houses (that they inherited from parents, or that they were selling in order to downscale to smaller, more maintainable dwellings), when young families (Xers) were buying them, and when Millennials were (thankfully, from my perspective) still renting. All of these bailouts, in my opinion, are the same thing. Bush-style "conservatism" is clearly _not_ future-oriented, and not very conservative either. Cutting taxes while starting a war? Come on. It's a bunch of obnoxious Baby Boomers with no foresight plundering the future.

</comment><comment><USER14> How does this contribute to your understanding of anything? Flagged.</comment>


Ask HN: Ideas/problems that are hi-tech in nature and hard to implement. - <AUTHOR>

From last few years I have been doing many interesting projects that aimed to solve some real world problems but they were mostly not too technical in nature(mostly web application involving db or simple mobile apps).
Now, I am craving to write some code that solves some technologically difficult problem(but still worth solving).
Do you have any such ideas? Do you have any such technological problems with no solution available at this moment? <comment><USER2> Why not scour gitHub for "technologically difficult" projects to contribute towards? That way you can pick ones suited to your expertise/field-of- knowledge.

<comment><AUTHOR> That's also an option. I will go through that. Btw, any recommendation for such projects at gitHub?</comment></comment>