New theory may explain the notorious cold fusion experiment from two decades ago - <AUTHOR> http://discovermagazine.com/2012/nov/27-big-idea-bring-back-the-cold-fusion-dream/ <comment><USER4> I'd like to humbly request that popular science articles geared toward the general public refrain from using the word "theory" in the non-scientific sense. This article even includes the phrase "only a theory"!

I know there are more syllables, but can we start using the word "hypothesis" when it is correct to do so?

It's hard enough combatting anti-science ideologies and repeatedly having to explain that the word "theory" doesn't mean what you think it means. Please, journalists, stop making it even more difficult by using the non-scientific connotation of "theory" when you mean "hypothesis".

<comment><USER5> I know you say that because of this other utterly stupid cultural battle, but the use of "theory" in that article is largely correct. It would certainly not be correct to call their theory an hypothesis.

Theories can be wimpy or really strong, can be good or bad. They are the connective tissue you use to try and explain certain phenomena. From these explanations arise predictions that help you formulate hypothesis. You then try to falsify those hypothesis and if you fail your theory is strengthened (and if you succeed your theory is wrong and at the very least has to be modified or thrown out altogether).

A hypothesis is not a theory in waiting. You make a similar mistake as Creationists do when they call Evolution "just a theory".

<comment><USER6> Did people ever talk about the "hypothesis of relativity?" It seems like it was a "theory" before they managed to confirm it.

<comment><USER7> It is worth noting that the theory of Special Relativity is confirmed by the Michelson–Morley experiment, which happened 25 years _before_ Einstein came up with his explanation. In the interim period, Lorentz had managed to come up with all of the contractions in Einstein's theory based on experimental data, but had no explanation of why it was true.

Einstein's insight was to come up with a single coherent theory that explained everything which had been observed, all together.

Therefore there was no period before they managed to confirm it.

The general theory was produced in 1916, when people were preoccupied by WW I. Its first confirmation was the precession of Mercury's orbit, which had been known about since the 1800s. Most people only heard about it after the second confirmation in 1919 when Eddington measured gravitational lensing during an eclipse. The third confirmation was the measurement of gravitational redshift.

Therefore it too was considered confirmed by the time most heard about it.

I say "considered" because several decades later physicists realized that all three confirmations are explained by a wide variety of plausible theories that do not possess the nonlinear characteristics of general relativity. (Removing nonlinearity would be desirable because it would make it easier to reconcile gravity and quantum mechanics.) This drove the development of better confirmations.

Note that theories are usually developed to explain existing observation, so the presence of confirmation before explanatory theory is actually quite common.

<comment><USER8> Could you please expand on that, or give a reference. What are the newer theories that do not possess the nonlinear characteristics ?

<comment><USER7> I may well be misremrembering a comment in a class that I took over 20 years ago.

Wikipedia lists a number of alternatives to general relativity. But I can't find any that look linear. There may not have been a linear theory, just a set of heuristics that any linear theory would be likely to support. In particular when I took general relativity, my professor made the following points.

Let's model a concrete photon coming out of a gravitational potential well. We can calculate exactly how much mass it should lose. Apply the particle wave duality to that energy loss we get GR's gravitational redshift to first order. Based on the wave interpretation, you can predict, to first order, the amount that time slows in a gravitational well.

To model Mercury, assume that the Sun creates a potential well. At all points in its orbit, assume that Mercury moves as Special Relativity says that a body with fixed angular momentum and the appropriate kinetic energy for that radius should move. (I forget whether you need to toss in the aforementioned time dilation.) To first order, you recover the GR correction to the precession of Mercury.

I forget the argument for the bending of light (maybe just throw gravitational time dilation into the mix?), but there is a first-order heuristic that can give the correct figure there as well.

The point is that if a series of heuristic arguments pull out the correct prediction for the original 3 tests, then _any_ theory that tries to reasonably combine QM, special relativity, and gravity, is likely to give similar predictions to first order.

That professor claimed that the first test which he did not have a heuristic explanation for involved the actual time that light took to get from one point to another. Looking on Wikipedia, I think he's talking about <http://en.wikipedia.org/wiki/Shapiro_delay>.

</comment></comment></comment></comment></comment></comment><comment><USER9> A recent pre-print paper suggests that a few mistakes in Widom-Larsen's calculations lead to overestimating the amount of energy that can be produced:

      A correct calculation gives a neutron production rate from (1) about
      300 times smaller than what estimated in [1–3], for the value of the
      mass renormalization factor β ≈ 20 considered there. In turn, it is
      questionable that values of β can be realized, in particular for
      bound electrons, so large as to give rise to useful nuclear
      transmutation rates. A more detailed analysis of the attainable
      values of β is needed to obtain more definite conclusions on this
      interesting phenomenon, should it exist at all.

<http://arxiv.org/abs/1209.6501> quote is from the "Conclusions" section

<comment><USER10> Our response to Ciuchi et al.'s preprint is as follows:

"Erroneous wave functions of Ciuchi et al. for collective modes in neutron production on metallic hydride cathodes" A. Widom, Y. Srivastava, and L Larsen (v1 Oct. 17, 2012) <http://arxiv.org/pdf/1210.5212v1.pdf>

Abstract: "There is a recent comment [1] concerning the theory of collective many body effects on the neutron production rates in a chemical battery cathode. Ciuchi et al. employ an inverse beta decay expression that contains a two body amplitude. Only one electron and one proton may exist in the Ciuchi et al. model initial state wave function. A flaw in their reasoning is that one cannot in reality describe collective many body correlations with only a two particle wave function. One needs very many particles to describe collective effects. In the model wave functions of Ciuchi et al. there are no metallic hydrides, there are no cathodes and there are no chemical batteries. Employing a wave function with only one electron and one proton is inadequate for describing collective metallic hydride surface quantum plasma physics in cathodes accurately."

Conclusions: "No significant argument has been provided against our nuclear physics results. The experimental evidence of neutron production and nuclear transmutations in properly designed plasma discharge electrolytic cells [5] agrees with our theoretical analysis and belies the theoretical arguments given in [1] against a hefty production of neutrons in hydride cells."

Lewis Larsen President and CEO Lattice Energy LLC

</comment></comment><comment><USER11> I found this article to be overly simplistic in its explanations, here's the relevant paper if that helps (although it's still fairly impenetrable, but it does have more details): [http://www.springerlink.com/content/77127077754p1788/fulltex...](http://www.springerlink.com/content/77127077754p1788/fulltext.pdf)

Note that ultimately it still comes down to low-energy fusion, in this case it's the fusion of two neutrons and Li-6 into Beryllium-8 which then decays into He-4 nuclei. This is somewhat similar to the CNO cycle whereby you keep adding protons to C-12 and the subsequent products until ultimately you end up with C-12 again plus a He-4 nuclei, and a lot of fusion energy along the way.

Fortunately there are several clear-cut experiments that can be conducted to verify each element of this theory, so we should see some progress on this (one way or the other) in the coming years.

<comment><USER12> It's a stretch of terminology to call neutron capture "fusion". This would make all (non-spontaneous) fission "fusion" too. Neutron capture is trivial to accomplish; there's no Coulomb barrier, as neutrons are uncharged. What's nontrivial, and an extraordinary-claim-demanding-extraordinary-evidence, is the production of free neutrons.

They don't have extraordinary evidence, or any evidence. Apparently I'm supposed be enthused about this future experiment, the outcome of which _could_ be extraordinary, although there's no particular prior expectation for this.

</comment></comment><comment><USER13> From the submitted article: "Zawodny has designed a stamp-size array of metal tiles to test the theory. According to Larsen's paper, the properties of some of the tiles should make it easier for electrons and protons to merge and form neutrons. If Zawodny observes evidence of neutron production, then he plans to do a follow-up experiment to see if those neutrons are fueling radioactive decay. Even if he gets the expected results, though, it would take several years and many corroborating experiments before LENRs could be considered confirmed."

. . . .

"So far, Larsen still has only a theory and some circumstantial evidence."

In other words, the operative word in the subtitle of the submitted article (used as the submission headline here) is "may." There doesn't seem to be any strong evidence yet that the process described in the article actually produces net energy output.

P.S. I heartily approve of (and have upvoted) the first comment this comment received, pointing out that there isn't even evidence that the speculated process described in the article even exists.

<comment><USER12> _There doesn't seem to any strong evidence yet that the process described in the article actually produces net energy output._

Massive overstatement. They don't have evidence the process _exists_ in any form.

<comment><USER14> I'm sorry, but that's just not true. There have been 1000+ replications / reports of excess energy from this very same "anomalous energy effect" - <USER2> hard part is the how and why.

See the very same Joseph Zawodny speak of "It has the demonstrated ability to produce excess amounts of energy.." in a much more candid way: <http://www.youtube.com/watch?v=JBlKc0TaqPs>

For a good compilation of LENR related evidence, see this presentation from ICCF-17 in September: <http://www.slideshare.net/tylervan/lenr>

<comment><USER15> > There have been 1000+ replications / reports of excess energy [...]

I followed the links, but the links I couldn't find an article with enough technical details about the experiments. I think that you know more about what is published in the subject, so can you please give me:

* One link to a paper where the excess of energy is clearly measure experimentally, that is published in a peer reviewed indexed journal with a high impact index.

* One link to an article with the technical details to reproduce the effect in the laboratory and measure the excess of energy. (Something like "Experimental LENR for dummies" would be nice.)

<comment><USER14> how about a whole library of them? <http://lenr-canr.org/>

<comment><USER15> Two is better! I have enough physical background to understand these papers but I'm not an expert in the subject. And I have to do my real work, so I have enough time to read two or three papers, but I don't have time to read 1000 of them.

I saw the webpage only for a few minutes, and most of the links look like special edition books and conference proceedings. I couldn't find the information I was looking for.

Calorimetric measures are extremely difficult to do correctly, so I would like to see:

 _) One paper in a well known journal, to be sure that an independent expert in the subject has read it thoughtfully.

_ ) One clear description of the experimental setup, so I could try to convince some friends to reproduce it.

</comment></comment></comment></comment></comment></comment><comment><USER7> If anyone qualified wants to dig in, [http://www.slideshare.net/lewisglarsen/lattice-energy-llc- in...](http://www.slideshare.net/lewisglarsen/lattice-energy-llc-index-to- documents-re-widomlarsen-theory-of-lenrssept-4-2012) provides a series of references to their theory.

</comment><comment><USER16> Surely Lithium decay would produce radiation at a very specific frequency that could be detected?

And wouldn't putting a sample of the water (post-experiment) through a mass- spectrometer immediately reveal the presence of Beryllium?

<comment><USER17> When I spent my entire Christmas vacation absorbing LENR a couple of years ago, it was just that... slight traces of reaction byproducts, and a few traces of pits in neutron detection gel, so, yes? The problem at the time was why so little of such things if it is fusion, but now we have this theory.

</comment></comment><comment><USER18> I was a little surprised when then said that the electron combines with the proton to form a neutron, but apparently it is called "electron capture" (see <http://en.wikipedia.org/wiki/Electron_capture>). In case anyone else was wondering...

<comment><USER12> Somewhat importantly, no one else claims isolated protons (hydrogen nuclei, which are stable) undergo EC. It's an effect of unstable, radioactive nuclei with too many protons and excess energy. Not free protons. In fact, naively that violates mass conservation: a proton+electron combined weigh less than a neutron. With EC, there is excess mass in the unstable nucleus. Here they are claiming something entirely different: the excess energy comes in from other electrons, in a coherent quantum mechanical effect. (An "effective particle", a "heavy electron", takes the place of a physical electron). It's not the same.

</comment></comment><comment><USER19> One thing that I like about Widom-Larsen theory is that it attempts to reason about possible quantum effects that could cause different sorts of reactions then fusion. While I was a tike back when Pons and Fleischmann were doing their experiments, I always thought the fusion explanation was bunk.

Extremely low energy neutrons created through quantum interactions makes a lot more sense then something majickally defeating the coulomb barrier with regularity (sure there are tunneling effects, but those don't happen with regularity.) While this area of physics is full of some quacks (Rossi for instance), I am glad this is starting to get more attention over the last few years. There is so much that happens as a result of quantum physics that is just counter intuitive, and really so much is left to understand about how the realm of the small effects our macro level reality. I really wish Feynman would have been alive when the first cold fusion craze was announced, perhaps he could have had clues to offer that made people think twice about possible quantum effects instead of the obvious craziness of fusion.

</comment><comment><USER20> Can someone much smarter with physics than myself explain what the hubbub was a couple of years back when 60 Minutes and a few other sources were starting to rehabilitate Fleischmann?

<http://www.cbsnews.com/8301-18560_162-4952167.html> <http://www.cbsnews.com/video/watch/?id=4955212n>

Is this the same phenomenon?

Edit: Additional question - If the palladium based device described in 60 Minutes consistently outputs more energy than input (they just don't know how long each reaction will take, exactly how much energy will be output, or quite frankly how it works), wouldn't it be a trivial matter to daisy chain hundreds of these devices and get a relatively more consistent average energy output?

</comment><comment> at-fates-hands I remember my Physics teacher discounting this straight away and wouldn't let us talk about it at all. He simply replied, "They will be shown as frauds and quietly retire, never to be heard from again."

It stuck with me and I was glad to see someone revisiting the idea again.

</comment><comment><USER21> If anyone is interested, I've found Eugene Mallove's "Fire from Ice" to be a fantastic read in documenting cold fusion and its troubled past.

[http://www.amazon.com/Fire-Ice-Searching-Behind- Fusion/dp/18...](http://www.amazon.com/Fire-Ice-Searching-Behind- Fusion/dp/1892925028/ref=sr_1_1?s=books&ie=UTF8&qid=1351024905&sr=1-1&keywords=fire+from+ice)

</comment><comment><USER22> Oh, hey... that's my uncle.

</comment><comment><USER14> I've been following the Rossi Ecat spectacle since stumbling upon it in 2011, and I think LENR (cold fusion) will be the biggest story on the planet in a few months time. I'm a 'believer'.

The Believers movie just came out a week ago: <http://www.thebelieversmovie.com/>

Also Rossi is featured in this month's Popular Science: [http://pesn.com/2012/10/16/9602208_Andrea-Rossis_Black- Box--...](http://pesn.com/2012/10/16/9602208_Andrea-Rossis_Black-Box-- by_Popular-Science/)

<comment><USER23> I don't understand how anyone can believe Rossi.

He has a mysterious device. He refuses to submit it to tests where anyone but him gets to measure the sytem's energy inputs and outputs. It hasn't been properly evaluated by third parties. He makes wild claims about both the product and how well it's selling. AND he has a history of this if you read about his previous ventures.

It has every hallmark of a con. At this point I'm genuinely interested - <USER3> can you believe it?

</comment></comment><comment><USER24> doesn't anybody know that there is already a field tested product coming to market?

<http://ecat.com/>

<comment><USER25> I'm following the ecat and Rossi story about the possible LENR device that he has (supposedly) built. Everything regarding this matter is surrounded by mystery and a lot of people objecting the tests.

I hope to see some serious development from other parties like NASA. Could be really cool.</comment></comment>


Show HN: Is this useful to those of you hiring programmers? - <AUTHOR>

I created http://devtest.net as a weekend project, and I'm pretty proud of it from a technical and UI standpoint.
The problem is, I've built this by myself in a bit of a vacuum, to "scratch my own itch", and I don't know if this is useful to anyone other than me. I'm also not so hot on the business side and don't really know what I'm doing.
Basically, I'm looking for some advice, now that I've actually launched, what now? One thing that this has shown me is that I'm capable of putting together a pretty decent web application on my own in my spare time. Should I stick with this idea, or should I look for someone more business minded with a better idea to implement and partner with them? <comment><USER2> That's pretty cool actually. I'm going to send the link to my boss, I could see him being interested. We have our own internal coding test we usually give applicants, but he might find some appeal in having something a little more formalized and centralized.

It might also be cool if you could let users upload their own test somehow, along with a spec that would let you automatically "grade" or evaluate the submission in some fashion (maybe using something like Fitnesse).

Edit: upon further review I see that you do support custom tests. Cool, I'll definitely dig in and give this a more thorough look later. First impression is very favorable though.

</comment><comment><USER3> I think this is a great idea. I've come up with this problem in the past not being a technical person myself. Especially when vetting out new hires early on. In the past I've worked in organizations that have their own internal testing but those are often baselined against the person who originally wrote them. The great thing with your service is that you could also, once you've gained traction, normalize the data and create "the" benchmark to score testers. You can even splice by geographic region, etc. You could go further once you establish your service as the "industry" standard allow programmers themselves self-test and post (show off) their scores on resumes, linkedin, github, etc. in effect quantifying their qualifications. Could even explore a partnership with CodeAcademy downstream. What you need to do now is get some validation, a way to say your tests are sufficient and/or above average. Then you need to go to market and drive awareness, lots of outlets to seed this. I'd be happy to bounce some ideas with you. Would you mind checking out my app too? www.whisper.me thanks

<comment><AUTHOR> I like it, we're using basecamp at the moment and the new version is just not quite doing it for us. I'll be investigating for my day job.

</comment></comment><comment><USER4> Nice work! I'm in the process of hiring right now, and I was just thinking this morning that I needed a solution like this to give a code test to a remote candidate.

<comment><AUTHOR> Awesome, if you sign up I'll give you some free credits and help with your test if you like!

<comment><USER5> I wouldn't mind testing it too!

<comment><AUTHOR> No problem, sign up and I'll add some credits to your account, it'd be really good to have some feedback.

</comment></comment></comment></comment><comment><USER6> Great idea, and definitely something I've been looking for!

As you get more users and traction, you can even implement some forms of anti- plagiarism (the test should still be allowed to pass, but simply flagged for the reviewer of the results).

I've just signed up and I wouldn't mind getting a few credits to test out your app as well :).

<comment><AUTHOR> Awesome, credits added!

</comment></comment><comment><USER7> LinkedIn advertising might be good for you. Submit the site to any group devoted to engineering HR. Create a list of 100 target businesses and start cold calling them. Start an affiliate program. *choose three of those things and you should be on your way to a solid revenue stream.

<comment><USER7> By the way, I love the site, don't know how well it works or not, but assuming it provides a great value to companies hiring, I'd love to be a business partner in the venture.

</comment></comment><comment><USER8> I talked to a lot of people about almost this exact idea in late 2011. Though I didn't build it because I got involved in something else, everyone I talked to was very enthusiastic about the idea (e.g. ready to hand over a credit card).

I think you should pursue this idea further!

</comment><comment><USER9> I think you people killed this guy's server :P

And after reading the positive comments I really want to see what this app looks like and what it does!

<comment><AUTHOR> Weird, none of my monitoring is showing any downtime...

Is it still not working for you?

</comment></comment><comment><USER9> Wow, it's an amazing app. Congratulations.

<comment><AUTHOR> Thanks for looking!

</comment></comment><comment><USER10> Just Awesome!

<comment><AUTHOR> Thanks!</comment></comment>


Ask HN: How are you using App.net - <AUTHOR>

I have my App.net account, but I hardly use it. I was wondering how you guys are using App.net?  Who are some good people to follow? Do you post to App.net and Twitter or only App.net now? <comment><USER2> I mostly browse the Global timeline. It doesn't move too fast yet, maybe 10-20 messages per minute. I jump into interesting conversations, which results in finding interesting people to follow.

You should post the same question on app.net if you haven't already, someone will answer.

<comment><AUTHOR> I've always been more of a lurker. But, I like the idea of jumping in on interesting conversations. Thanks!</comment></comment>


Ask HN: Java/Groovy/Scala/Clojure/Ruby for your startup? - <AUTHOR>

What JVM Language are you using for your startup? if any. Java/Groovy/Scala/Clojure/Ruby?<p>For what layers? And why? <comment><USER2> Not actually using any, but if scale problems come, we have the following order: Jruby, Clojure, Scala. The reasons behind the order:

* Jruby, because we really like the level of metaprogramming of ruby, the readability, the ammount of libraries, the effort Charles Nutter put into making it a very good port of ruby, and the library support that feels like ruby (dsl-ish, easy to use, easy to change, with tests as documentation, etc), even if you are not using Rails.

* Clojure, because it is dynamic, and enforces immutable state (most of the time), it has great metaprogramming support as well (which is very important for readbility), and you can use type annotations to remove all reflection code, and use direct invoking (as fast as java).

* Scala, because it is much more convenient to use than Java (and has other advantages, <http://bit.ly/4DIKy4>), while still being compiled to very efficient java bytecode. Its static typing has issues (<http://bit.ly/asQ7FC>), as its lack of metaprogramming. Implicit conversions, on the other hand, can deal with it for most part.

*Java/Groovy: Scala should be the lowest level we would like to handle. Groovy feels a bit like Ruby, it has more metaprogramming facilities (like categories, and AST manipulations are builtin, unlike Ruby's Parse Tree, which doesn't work on 1.9). However, Jruby is the same language we use everywhere else, and Groovy is not faster than clojure. So it is a "master of none" type of language. Not to mention that its stack trace is riddled with java garbage, and it feels a lot like Java, which is really a put off.

This is mosty for backend, as with are probably sticking with Sinatra on front end for a looong while. Twitter uses Rails on front end even nowadays, so it seems like we won't have too much problem with it as well.

<comment><USER3> I wouldn't necessarily put Clojure between JRuby and Scala (I hope I am not misinterpreting). I think it should be more along side of JRuby depending on the type of problem you are trying to tackle, it supports a totally different paradigm of programming.

<comment><USER4> The problem is that there really isnt any definition of languages within the java landscape right now. Meaning theres a bunch of languages but noen really stand out for any specifc uses with the expection of people that know them well and are using them well. So its very confusing to know which to use.

For example Grails and groovy were supposed to be big on the java framework but due to seveeral reasons they seem to be really more niche. PArtly because it seems the people behind both of them just arent dealing wiht the problems such as bugs quickly enough.

Scala is an interesting language but again there doesnt seem to be a heavy push for it. Expect for peopel that are using it and know what its strengths are.

All of which leads to alot of confusion about which technology to use.

Then theres the issue of frameworks. Theres a million of them but none of them really stand out. ANd personally i dont think it helps that oracle is no running things.

On the other hand ruby is known by many people as being pretty powerful and used by alot of startups. Same thing with php. How many startups do you know that use java? I know foursqaure uses liftweb but thats about it.

<comment><USER2> Just because you are not aware of what is being used, does not mean that it is not used. Scala startups: KaChing (follow <http://twitter.com/eishay> for more info, he talks a lot about the Lean startup practises on Kaching as well) and Etherpad (wave-like product, bought by google last year). Eishay also gave a great hint for startups looking for scalable technologies: <http://twitter.com/eishay/statuses/20478841811>

Flightcaster, which is funded by YCombinator, uses clojure and Ruby on heroku (more on their infoq profile: <http://www.infoq.com/articles/flightcaster- clojure-rails>).

Most startups are not actually vocal about what they use. Most of them actually are not heard, so they could be as vocal as they wanted and still noone would hear them.

</comment></comment></comment></comment><comment><USER5> I know that the University of Oslo started on a rather big Groovy project, but after a few months ported everything over to Ruby (not sure if it runs on JRuby though). As far as I remember, they felt that Groovy was a terrible "clone" of Ruby: Long and Java-ish backtraces, small community and much broken code.

<http://news.ycombinator.com/user?id=thomasfl> worked on the project, so maybe he'll chime in with more details.

<comment><USER6> I'm not at a startup (anymore... we got bought), but we're using Groovy for unit/functional testing.

It's a nice compromise when you want something that cuts through boilerplate with metaprogramming/duck typing but can keep tight interop with Java where you need it... and you need a quick learning curve for Java devs.

That said, stack traces are _horrible_ , debugging in Eclipse is difficult to impossible depending on how "dynamic" you get, there's no interactive debugger like pdb.set_trace() (that I know of) and the flexibility of the syntax can get out of hand quick. Great for limited use, like making mini-languages and glue for testing. Wouldn't want to support a large application written in it.

<comment><USER7> Groovy also has a faster static version, Groovy++ at <http://code.google.com/p/groovypptest> which can intermingle with the more scripty Groovy code using annotations.

</comment></comment></comment><comment><USER8> You should use what you are comfortable with and most productive. If Visual Basic make your product work, go with it.

</comment><comment><USER9> Avoid to use Java directly, if you need any Java libraries choose Clojure to wrap it, that strategy works well for me. Scala is another alternative here. It's a bit more complicated to work with Java libraries but great for writing code.

For a general programing task my choice is Python, stable, mature, well documented with an extensive library support. iPython is a fantastic shell for it, that's my interface to Python libraries and programs, good for developing, running code, debugging or for a quick peep to the documentation. Highly recommended.

<comment><USER10> I know that dissing Java is cool these days, but some arguments would've been neat.

<comment><USER9> From my experience Java programming is too slow and inflexible for a startup while there are good and important libraries, that's why I suggest clojure to use them. That's based on first hand experience using various Java projects for a large traffic site, including Cassandra, Solr, hadoop, restlet, etc. We started with java for coding all of these but replaced most of the code later and not for fun. :-)

<comment><USER10> That's just too general a comment, in my opinion. Saying "java is too slow for a startup" is like saying "Windows is too primitive for a startup". What if you're making Windows-specific desktop software? Or are we only allowed to talk about ruby-on-rails twitter mashups here anymore?

</comment></comment></comment></comment><comment><USER11> If you do decide to go with Java directly consider including Guava (Google's open source common library).

<http://code.google.com/p/guava-libraries/>

We really like the support for higher-order functions, like Iterables.transform (aka map) and Functions.compose amongst others. There are loads of useful tools including MapMaker, which is great for building caches. The downside is that Function<X,Y> is an interface and implementations are often inner classes, which can be syntactically quite messy.

We've recently open-sourced our common library, which extends Guava with a few additional features we've found useful:

[http://github.com/mbst/common/tree/master/src/main/java/com/...](http://github.com/mbst/common/tree/master/src/main/java/com/metabroadcast/common/)

</comment><comment><USER12> python. python beats ruby in community imo. if you're just doing a webapp then rails wins, but with scipy conferences and researchers from astronomy to biology putting out python packages, if you're doing something outside the normal webapp you'll probably find a more useful community with python.

<comment><USER13> This opinion sounds very one sided. Do you have any experience within the ruby community, personally?

<comment><USER12> It is very one sided as almost everything I've seen about Ruby is Rails focused, especially outside of japan. Please fill me in where you think the Ruby community has more to offer in a scientific environment.

<comment><USER13> This is a completely different statement than your original one. I think there's plenty of support for non-webapp Ruby, and that its very competitive with Python outside of web application arena (look at Chef or Puppet for a couple examples). But Python is _clearly_ the dominant language in a scientific environment, at least between it and Ruby. But like I said, that wasn't your initial statement.

</comment></comment></comment></comment><comment><USER14> I'm really curious for the purpose of this question. Are you trying to find the most popular language of choice among HN hackers that have their own startup? Shouldn't you at least mention "and what does your startup do"? Or at this point, "startup" is automatically assumed to be a "webapp"?

My startup specializes in bringing other people's ideas into life. Most of our clients are also startups. Depending on their needs, we use Ruby on Rails, PHP, Java, Objective C/Cocoa, or C++. For my other startup, which is a webapp, I use Ruby on Rails.

</comment><comment><USER15> Ruby, because it's relatively easy to find Ruby developers who want to work in a startup environment.</comment>


As oil and gas exports surge, West Texas becomes the world's "extraction colony" - <AUTHOR> https://www.texastribune.org/2018/10/11/west-texas-becomes-worlds-extraction-colony-oil-gas-exports-surge/ <comment><USER2> Texan here. I'm anti fossil fuels, pro solar and wind. I've stopped trying to have any conversations with other Texans about this topic. Texans are very pro fossil fuels, on balance. I'm saying this as an Austinite.

It's interesting to see this in the Texas Tribune.

From time to time I have reason to drive through or fly over (I know, hypocrite) West Texas, and passing through the oil fields one can literally watch the world burn. It's unreal.

Edit: Just a note on the people in the energy industry. They're good people on balance. I might have come across as incidentally disparaging people in the energy industry. I have many friends and family of friends that have spent their entire career in the industry. It's like Silicon Valley and tech's problems. NYC and the financial crisis. Economic and financial incentives supported by cultural norms. In Silicon Valley the confidence that technology will improve our future means sometimes it is developed without pause for impact. In Texas a strong Christian ethos encourages a certain view of "stewardship" of the earth that does not mesh well with the constantly evolving culture of environmentalism. But as people they're good people.

<comment><USER3> Houstonite here that works in the energy industry. Im not sure you are talking to the right people. Im not anti fossil fuels, but Im not really "pro" fossil fuels either. If you look at the oil and gas business, there really just isn't any viable alternatives available on a large scale at this moment. I would echo some other comments in that Texas has a lot of solar and wind farms. My father in law has a ranch out in west Texas and every surrounding property has wind or solar generation on it. I am encouraged by this, and while I don't really like looking out at the landscape to see a bunch of windmills, I think it is important to try and reduce our fossil fuel use. The fact of the matter though is that until we give up plastic, and figure out how to store electricity, we need nat gas or nuclear for electricity. Ive read conflicting data about the worlds lithium supply, but it may or may not be able to replace crude for transportation. Electric planes still seem a decade off or more. If crude went away tomorrow, I would still work in the energy industry so I don't care if that were to happen. I can say that I have been encouraged by some of the internal projects some "big oil" companies are taking on. I wish it could be a case of just stop using oil. Life as we know it would cease if we did that.

<comment><USER4> Former Houston old-energy chemical engineer here (also grew up in west Texas). I switched careers to cleantech, and I see now something I didn't see back when I was in old-energy: being neutral about fossil fuels is just as bad as being pro fossil fuels.

First, that attitude makes you, your family, and your friends more susceptible to propaganda from pro-fossil fuel interests. You're more apt to believe crazy stuff like we'd have to stop using plastic or that nuclear would have to also go away. You're more apt to doubt the science behind climate change. You're more apt to be mislead on the economics or feasibility of renewables.

Second, that attitude makes you think we're not in a rush, so we can afford to wait until things are figured out more. It makes you think that we have to figure out storage for wind and solar and airplanes, even though we don't have to worry about that until we reach much higher levels of penetration than we currently have (and we're already making good progress nonetheless). It makes you think that we'd fall into chaos if we use less natural gas and oil, even though it's going to be a gradual transition and will be replaced with equal or better alternatives.

Unfortunately, because of previous neutral attitudes like yours, we're now in a goddamn race for our lives as the nightmares of climate change are starting to happen (how many Americans died in Puerto Rico? California's wildfires? etc.). Nature doesn't give a fuck. It's too late to be neutral anymore.

So from one Texas energy worker to another, I beg you to stop being neutral. Unlike most other people, you actually have domain expertise to have a substantial impact fighting climate change. Remaining neutral when you work in energy is just making everyone else's lives worse.

<comment><USER5> Your entire reply is extreme in the sense it is all absolutes. Please provide sources that we are in a 'race for your lives'. And please don't make assumptions about other people or their lives without more information. It gives you the appearance of a paranoid alarmist.

<comment><USER4> > Your entire reply is extreme in the sense it is all absolutes. Please > provide sources that we are in a 'race for your lives'.

Graph: [https://www.ipcc.ch/site/assets/uploads/sites/2/2018/10/SPM3...](https://www.ipcc.ch/site/assets/uploads/sites/2/2018/10/SPM3a.png)

Report: [https://www.ipcc.ch/sr15/](https://www.ipcc.ch/sr15/)

News: [https://www.nytimes.com/2018/10/07/climate/ipcc-climate- repo...](https://www.nytimes.com/2018/10/07/climate/ipcc-climate- report-2040.html)

This is the carbon emission reduction curve we have to hit if we want to hit our Paris Agreement targets. That's a pretty steep drop off if you ask me, and certainly falls into the category of a "race for your lives".

Do you disagree? Please provide sources that we aren't in such a rush to curtail fossil emissions (because there's mountains of science saying we are).

> It gives you the appearance of a paranoid alarmist.

How can I be paranoid if the science backs me up? However, I do think being an alarmist is appropriate for the situation. If that graph isn't a mash-the-big- red-button moment, I don't know what would be.

<comment><USER6> I believe that there is a gap in your argument. We have to hustle to hit our Paris Agreement targets, but missing them isn't exactly equivalent to dying - at least, I don't think you've shown it to be. That's why it's not "a race for our lives".

But, you may say, a lot of people will die from climate change. (And a lot _definitely_ die from pollution.) A lot die from malaria, too, but we don't have a "race for our lives" on that one. Cynically, most people don't consider a bunch of _other_ people dying as a "race for our lives".

(We did more or less have a "race for our lives" on AIDS, though. It seems that we start to care when it's hitting closer to home, and we think that it could be _us_. That's the problem with your argument. You haven't shown people enough evidence that they think it will be _them_ dying, so they don't think it's "a race for our lives".)

<comment><USER4> Hmmm, for non-scientific discussions, I tend to like the XKCD scale for global temperature averages.

[https://xkcd.com/1379/](https://xkcd.com/1379/)

Going above Paris gets into the "there be dragons" realm of human survival. The Pentagon is predicting mass migrations, wars, famine, and economic implosion[1]. I feel like that certainly constitutes "race for our lives" level of rhetoric.

Usually, when someone balks at strong rhetoric on climate change they are (1) unaware of the scale and severity of the problem or (2) don't see those affected as within their "our" definition (e.g. it will happen to _other_ people, not _my_ people). The former is slowly being solved through increased education and awareness, the later will likely always be present due to human tribalism and racism.

As far as the malaria theory, it's a very different strategy when you are actively causing the problem you are trying to solve. It's as if we are repeatedly shooting ourselves in the foot, then some of us say we should stop, then you say why are you so focused on the foot-shooting vs cancer killing us. One is an immediately addressable problem caused by ourselves, one is a natural threat that is not under our direct and immediate control (thus takes much longer to understand and solve).

[1]: [https://www.nytimes.com/2014/10/14/us/pentagon-says- global-w...](https://www.nytimes.com/2014/10/14/us/pentagon-says-global- warming-presents-immediate-security-threat.html)

<comment><USER7> IIRC, Eocene period had atmospheric CO2 up to 2000 ppm, the known max. I figure we'll meet that target (ocean acidification, thawing tundra, desertification). So life will somehow survive. Though I wouldn't place bets that we humans will squeeze thru that keyhole.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> This is a complete house of cards, for multiple reasons. It's yet another boom that will result in a lot of hardship for people.

First, all the fracking in Texas is fueled by cheap interest rates and private equity. The industry as a whole is losing money -- to the tune of $600 billion since 2007. And when investors (I'm looking at you pension funds) finally realize that there is no path to making these companies profitable, especially when the decline rate on their wells is 70% in the first year, they will stop pouring money in. A lot of companies will vanish when this happens.

Second, the media loves to play up the notion of "energy independence" and headlines of the United States producing more than Russia/Saudi Arabia, but neglects to understand the differentials in crude quality. The US (especially Texas) is producing light oil, which increasingly is not wanted in the industry. The refineries in the US have consumed all they can and are increasingly in need of heavier oil to mix with, hence why you see the US exporting our oil and importing oil from Saudi Arabia and elsewhere. When exports have been maxed out and foreign buyers slow their purchases these companies will have nowhere to send their oil and will need to slow drilling.

<comment><USER9> The worst part is that private companies make the profit while tax payers clean up the mess. Reverse socialism at its worst.

<comment><USER10> How will tax payers clean up the mess ? What do you mean by it ?

<comment><USER11> Many of the extraction companies will eventually go bankrupt and liquidate, leaving toxic waste sites behind. Then the government ( funded by taxpayers) will have to step in and clean up the mess.

</comment></comment></comment></comment><comment><USER12> "Residents with no say in these decisions are stuck with the consequences" \- I'm pretty sure that these residents get to vote for state assembly people who obviously support the policy, to vote for city council people who support the policy, for US Congress, House and Senate, and for the presidency. Saying that you "don't have a say" when a democratic decision goes against you is pretty misleading - perhaps more accurate to say "some residents, who disagree with the democratically taken decisions at local, state and federal levels..."

<comment><USER13> Perhaps there are no candidates who represent the residents views.

Then they wouldn't have a say.

<comment><USER12> Sure. But they have the right to run for office - so if _none_ of the 100's - 1000's of people who _do_ represent their views chose to run, isn't that a choice too?

Wouldn't a simpler explanation be that the more popular choice won i.e. overall, the people of the city decided that the benefits outweighed the harm? And as more evidence of harm is accumulated, I'm sure their views on regulation will change. But I think it's unhealthy to imply that when "our side" loses, then the results are not legitimate, but when "our side" wins, only then is it "democracy in action", "will of the people" etc

<comment><USER14> Of course that's exactly what happened. The anti views are the minority. Their complaint is actually that they're losing out to a more popular position in a Democracy. The oil industry jobs pay extraordinarily well compared to (non- existent) alternatives that were there before the oil boom. There was no economy in West Texas before the fracking output soared. There's nothing out there other than Midland and Odessa, and their populations hadn't changed much in 30 years. The choice is extraction economy or no economy. And if by some unexpected effort + resources the state decided to invest vast tax revenue to build up a non-extraction economy in the area, it will take decades anyway.

<comment><USER15> As usual an unpopular yet sound argument from adventured. The real losers are people downvoting because of their shallow mental models.

West Texas oil is making US more secure and access to cheap energy will not only make things better in Texas, but Mexico is going build on this supply of NG.

West Texas is extraction colony, for America. The rest of the world is buying because it is sweet as hell and produced in most stable region on the planet.

</comment></comment></comment></comment></comment><comment><USER16> There seems to be a fair amount of cognitive dissonance here.

The central complaint almost seems to be that it's getting exported, rather than getting used domestically.

And as a non American I'm struggling to understand the lack of environmental opposition. Is the article biased, or does environmentalism not exist in Texas?

Finally speaking as someone from the UK. This is scary. Theres been an ongoing battle over fracking, with the first well drilled this year. Hopefully we have better standards in place.

<comment><USER17> Plenty of environmentalists all over the place. plenty of people not happy about it. None you will hear about because who would you hear it from? The 9 o'Clock news?(Brought to you by Ford and Chevron.) Even "grass roots" social media platforms; You subscribe to any local Texas groups on twitter/insta/whatever?

Texas is a single state with 4 times the land mass of the entire UK. Do the people Glasgow stop what they are doing, take vacation time and go to protest a new Tesco building over a historical site in London? Likewise people in Houston have little reason and large costs to fight something happening 500 miles away. Nor do they have anything to say about or vote to cast in (almost) any elections that would have (directly) altered policy in the area, which is again, 500 miles(800km) away.

<comment><USER16> The article doesn't contain the voices I would expect. I'm wondering whether that is an article bias, or just that those voices aren't present.

I don't know the ideology of the newspaper, I don't know the local politics. Which is why I asked the question.

No offense was intended.

</comment></comment></comment><comment> village-idiot Curious what that'll do to West Texas' non-extraction economy.

Once a whole nation state depends on extraction, they usually lose other forms of economic activity. This is usually called "Dutch Disease" for historical reasons. But what happens when a small region within a country becomes entirely dependent on resource extraction?

<comment><USER18> There never was a non-extraction economy in West Texas. The place is empty. Some cattle. Fire ants. And the occasional SS retiree living in a mobile home in the middle of nowhere (such as the lady highlighted in the article).

Weep over the rainforests. Weep over the wetlands. Heck, you can even weep over the drained swamps. But, spare your tears for West Texas... there is NOTHING you could do to that terrain or ecosystem that wouldn't be an improvement over the wasteland that it now is.

<comment><USER19> How about weeping over natural gas flaring equivalent to daily exhaust emitted by about 2.7 million cars?

<comment><USER20> It's a temporary thing.

Once the number of LNG liquefaction plants in North America more than doubles, NG will be worth collecting for export to markets that pay more than domestic buyers.

[https://en.wikipedia.org/wiki/List_of_LNG_terminals#North_Am...](https://en.wikipedia.org/wiki/List_of_LNG_terminals#North_America)

The oil market is much more globalized than NG.

</comment></comment></comment></comment><comment><USER21> I don't own a car. I also think climate change is similar to fighting obesity: "you must change your behavior today or something will kill you in x years" Usually people have a wake up call(cardiac event, can't run up stairs without being winded, etc), before they get really serious about weight loss. What would be a similar wake up call for the planet? People don't attribute floods and stuff to climate change, so therefore it's not a wake up call.

<comment><USER22> I'd wager that most people's wake up call would be weather that's more extreme than climate change will likely produce. I imagine climate change to be worrying when plants don't grow around me and that probably will never happen, some just plants will recede into colder ranges and plants from warmer ranges will come in.

<comment><USER23> If you live in the right place that is. There are countries worrying about completely disappearing (due to raising sea levels) right now.

</comment></comment></comment><comment><USER24> Environment aside, what's the expected long term impact of lifting oil export restrictions?

<comment><USER25> The restrictions were intended to make sure America was energy independent. Since that goal was accomplished, there isn't really a reason to keep them.

Increasing American exports hurts countries that depend on energy exports. America can manage both with and without exports, but other countries can not.

The next step in my mind is increasing non-oil American energy and exporting even more oil, this would further destabilize those oil-dependent countries (most of whom are not friends of the US).

<comment><USER14> Venezuela is one of the first test cases for those consequences. Their system put them down, the US no longer needing/wanting their sour crude will make it far more difficult to get back up. US oil imports from Venezuela were near one million barrels per day as recently as 2012. It's down about 40% from there, and the US now buys around 60% of their total oil output. It has also been hitting Cuba indirectly (as a Venezuela dependent) for some time and now appears to be entering another stage there.[1] By the time Venezuela gets itself back together as a functioning country and could think about rebuilding its output, the world is unlikely to have much demand for its oil. The IEA is expecting their oil production to drop below one million barrels per day soon (from 3.5 million barrels per day in ~1998). Their estimated extractable reserves was always a bit farcical, however even if you chopped that down by 2/3, roughly 100 billion barrels, that still ends up being several trillion dollars in national wealth that will mostly never be realized.

[1] [https://www.reuters.com/article/us-cuba-economy/reality- bite...](https://www.reuters.com/article/us-cuba-economy/reality-bites-cuba- plans-more-austerity-as-finances-worsen-idUSKCN1OR0VN)

<comment><USER26> You cannot put the collapse of Venezuela on US policies... The US didn't put Chavez / Maduro in power, and they didn't ask them to oppress their own people.

</comment></comment></comment></comment><comment><USER27> In _The Dictator 's Handbook_, by Bruce bueno de Mesquita and Alastair Smith, the authors outline why those who live in an extraction economy often do so under what we term dictatorships. Briefly, the supporters that are needed to maintain that extraction economy are a very small subset of the population and, from a government perspective, are not essential. The supporters you end up needing are the portion of the working class needed to run the extraction operation and a police force to keep the rest of the population in check. (Well, and a treasurer and some other bureaucrats, but the application here is to Texas in the U.S., where those services, and an army to defend your territory, are provided by the federal government.) In banana and oil republics, it's actually beneficial for the ruling class if most of the population is weakened and starving, as they are less able to organize.

Contrast this with a classical democracy, where the support base is the general population, and the treasury is made possible by the increased productivity of an educated populace who have access to infrastructure that supports their economic activity. In that instance, it is, broadly speaking, the health of the populace that feeds the health of the state.

I bring this contrast up at this time for two reasons. First, in the present case of an extraction colony existing inside a U.S. state, we can find a coherent theory of the difference between Republican/Conservative policies, such as states rights, authoritarianism, veneration of the mythology of the (by definition unempowered) individual, and left policies (I leave out Democrats, our centrist party): The former policies are tailored to preserve capital concentrations, favor extractive economic activity, curb controls on pollution, and suppress unions, all with the all-too-easy resort to violence (or it's equivalent, the all-too-common occurrence of looking the other way). The latter policies, from the left, are the opposite, intended to protect the (in fact) powerless and to broaden economic benefit, often through redistribution. Republicanism in this country, allied with business, exists in order to set up and protect banana republics at the state level.

Second, we can point to other trends that are worrying in the U.S. For example, infrastructure decay should be seen as a lagging indicator of the economic vitality of a democratic nation; if the benefit of infrastructure accrues indirectly to the ruling class through the productivity of the working classes, then it (infrastructure) will be maintained. After that benefit has gone away (say after manufacturing, technology, and information jobs have moved overseas) the infrastructure can be allowed to decay, as other economic systems arise to extract what's left. Unfortunately, what's left is the remaining savings of the working classes. Think ad revenue for eyeballs, opaque healthcare markets and lack of medical accountability to generate and capitalize on health calamity, cornered apartment markets to extract rent, isolated extraction economies like Dakota shale and the Permian basin, farmland, and so on. No longer relevant to the productivity of a nation, the citizen becomes, at best, a resource; at worst, a burden.

Anyhow, for the reasons above, I find the presence of these extractive economies inside the U.S. worrying, more so if they are increasing. The dynamics of power, namely wealth extraction in order to pay supporters, do not bode well if it really has come to this.

<comment><USER28> The US is just returning to its traditional political economy, where the land is pillaged by elite interests, the rabble is dealt with as it can be, and any "freedom" anyone has is totally coincidental and mainly stems from the failure of elites to even pay attention to them because they are making so much more money subjugating and impoverishing rabble elsewhere.

The mid-20th century was a bizarre anomaly brought on by a number of non- repeatable factors, most notably WW2, the arms race, and Jim Crow, where we accidentally had broad-based economic growth that included a (almost totally white portion of the) working class. For centuries before that, the entire economic purpose of America was to be a literal extraction colony.

The infrastructure needed to run the wealth pumps (for example, the pipelines needed to pump crude from the Permian to the Gulf, or the West Coast ports needed to import Chinese-made dollar store tchotchkes) will not decay, but things like public transit, water treatment, and midcentury highway tunnels will probably get worse, at least outside elite enclaves like New York and San Francisco.</comment></comment>


Quantcast Web Site Rankings | You'll be suprised at some of the sites on this list - <AUTHOR> http://www.quantcast.com/top-sites-1

<comment><USER2> I do not see "news.yc". I am not happy. Maybe that is good sign. If i went where 99% of the population goes to do things, find things, see things, then I must be in real trouble.</comment>


Getting Started with MapD, Part 2: Electricity Dataset - <AUTHOR> http://randyzwitch.com/mapd-pjm-electricity-data/ <comment><USER2> I want to believe that GPUs can accelerate databases but I rarely see head to head comparisons. Is 100ms for this query really that good? With only 4mm rows how would vanilla Postgres hold up, not to mention something like Citus or Memsql (both of which are at the core my startup's big data pipeline).

<comment><USER3> 100ms for this query is good compared to PostgreSQL (~4s when running the query as-is, ~275ms when using a window function) considering that MapD doesn't seem to utilize an index for it.

That being said a 146MB CSV with 5 million rows is not a challenge for vanilla PostgreSQL either. If you're willing to add an index and use window functions (which MapD doesn't seem to support), you can run this query in ~34ms: [https://gist.github.com/felixge/fffef4bc41a5e87e26f62c2a5c8e...](https://gist.github.com/felixge/fffef4bc41a5e87e26f62c2a5c8ed5bd)

I have no experience with MapD, but people often significantly underestimate the performance of relational databases and reach for the big/expensive toys too soon.

<comment><USER4> You are right, as a demonstration there is a trade-off between using a real dataset and making the data large enough to demonstrate the performance gains. You are proving the other part of my argument though, in that you don't need to add an index (though, the comparison between using a window function in Postgres is fair, since that would be the canonical way to write that query)

As far as the "big/expensive toys", I'm using the Community Edition, which is FOSS.

<comment><USER3> I guess you're just trying to show how to use MapD rather than showing when it makes sense to use it, which is cool. For the latter, I found the results over here fairly interesting: [http://tech.marksblogg.com/benchmarks.html](http://tech.marksblogg.com/benchmarks.html)

As far as "big/expensive toys" go, I was referring to total cost of ownership of putting something like MapD in production vs PostgreSQL. The latter can run on any machine (no GPU needed), there are plenty of hosted services, and there is a large pool of people familiar with it. If this is just about running some analysis on a local machine, that argument doesn't apply so much :).

<comment><USER5> Note that MapD can run on CPU as well and is generally quite fast. I haven't benchmarked but on a dataset of this size the numbers might be similar.

I'd also say that while this is a great example of getting up and running with MapD the system really shines when doing interactive queries over datasets from the 100M to 100B record range without requiring indexes.

</comment></comment></comment></comment></comment><comment><USER6> The exact same query takes 60-70ms in SQLite for me. This is not a very good MapD advertisement...

edit: Actually, I get this (uncached) in the sqlite3 CLI (the 60-70ms where via sqlitebrowser):

        Run Time: real 0.018 user 0.018164 sys 0.000135

Here is the database if you want to play around with it. [https://transfer.sh/VkWXe/sqlite.db.gz](https://transfer.sh/VkWXe/sqlite.db.gz)

<comment><USER7> GPU databases have a higher latency, due to the need to copy data to and from the GPU. The data processed has to end up in the host RAM, to be sent to the client - it can't be sent directly from the GPU RAM.

The GPU really only helps when you need a higher throughput - eg. you're analyzing many billions of rows, or are running a relatively compute-intensive query.

<comment><USER6> Yeah, I know. Maybe the post(s) should not talk about performance for those badly fitting examples then.

<comment><USER8> hi - i took a look at the sqlite db. not to nitpick, but you have 4 indexes there?

the overhead of indexes at tiny data sizes like this is minimal, so no surprise that sqlite or pg will compare favorably at laptop scale (FWIW, i ran this on a 4-core macbook pro with no GPU (i.e CPU only) and without indexes, mapd runs the query at ~1.5 sec while sqlite takes ~6).

however, the bigger point is about how this experience scales. take a look at this public demo - [https://www.mapd.com/demos/ships](https://www.mapd.com/demos/ships). it's 11 billion geo data points, and you'll see the same low millisecond response that FridgeSeal is talking about - again, no indexing (to verify, go to dev tools in your browser, turn on SQLLogging(true) in the console and look at the query times reported for the round trip. i'm sure you appreciate that the overhead (both creation and maintenance) of indexing scales with data size, plus sqlite/pg dont (AFAIK) offer anything by way of GPU-rendered charting.

<comment><USER6> Of course I use indexes. It's utterly silly not to do so.

My point was that the submitted post did not make a convincing argument pro MapD when query times are that much slower than a trivial SQLite database.

<comment><USER8> i'm not questioning your use of indexes. i'm questioning the basis of the unqualified assertion that 'query times are slower than a SQLite database' \- with indexes off, and both running on CPUs, the basis of comparison isn't tilted one way or the other, and then it isnt clear at all that mapd is 'much slower'.

</comment></comment></comment></comment></comment></comment><comment><USER6> I meant to post some SQLite and Postgres numbers but the CSV that the code generates (on "wget [http://www.pjm.com/pub/operations/hist-meter- load/{1993..201...](http://www.pjm.com/pub/operations/hist-meter- load/{1993..2018}-hourly-loads.xls")) is garbage:

        2010-01-15 00:00:00,DPL,24,2017.425
        ...
        ,1/3/2017,1,RTO
        ...
        7/5/2017,VMEU,22,97.495

<comment><USER4> The Python code works for Python 3 and pandas v0.22.0. There are some rows that are mangled, but my assumption was this was an artifact of reading Excel files, and I didn't go back and fix every single row in the dataset.

      ACTUAL_DATE,ZONE_NAME,HOUR_ENDING,MW
      1993-01-01 00:00:00,AE,1,855.0
      1993-01-01 00:00:00,BGE,1,2358.0
      1993-01-01 00:00:00,DPL,1,1150.0
      1993-01-01 00:00:00,GPU,1,3772.0
      1993-01-01 00:00:00,JCPL,1,1632.0
      1993-01-01 00:00:00,METED,1,929.0
      1993-01-01 00:00:00,PECO,1,3015.0
      1993-01-01 00:00:00,PENLC,1,1211.0
      1993-01-01 00:00:00,PEPCO,1,2137.0

<comment><USER6> I tried both Python 2.7.14 and Python 3.6.4. pandas==0.22.0 and xlrd==1.1.0. [https://transfer.sh/2T4fw/hourly_loads.csv.gz](https://transfer.sh/2T4fw/hourly_loads.csv.gz) is what I get.

It looks like dates are stored not as dates but strings in the more recent spreadsheets.

edit: dateutil's parser.parse(x) & strftime("%Y-%m-%d") helps</comment></comment></comment>


Why we started CoreOS - <AUTHOR> https://coreos.com/blog/why-we-started-coreos <comment><USER3> If Equifax were running a vulnerable version of Apache Struts on CoreOS the the hack would have still happened.

That the CoreOS CEO has chosen to use the Equifax hack as a marketing opportunity is equally distasteful and disingenuous.

Also invoking the founding fathers, the Bill of Rights and the great Dr Martin Luther King in the as part of that marketing pitch is the height of bombast and absurdity.

This blog post could have been written by Gavin Belson.

<comment><USER4> One of the features of their stack is live single-click or even auto-update of you technology stack while your application keeps running. While that doesn't necessarily mean that you will keep your stack up-to-date, it makes patching security issues considerably easier and less painful, which means it's far less likely that you will be running a known-vulnerable version. So it's not entirely BS, even though _why_ CoreOS helps security isn't spelled out here. I happen to have watched a talk where Alex Polvi did spell this out just last night, which is the only reason I know this. But yes, the rhetoric is a bit heavy-handed.

<comment><USER5> This is built-in functionality of Kubernetes (& OpenShift) called Rolling Update [0]/Rolling Deployments [1] and Image Triggers [2].

[0] [https://kubernetes.io/docs/tasks/run-application/rolling- upd...](https://kubernetes.io/docs/tasks/run-application/rolling-update- replication-controller/)

[1] [https://docs.openshift.com/container- platform/3.6/dev_guide/...](https://docs.openshift.com/container- platform/3.6/dev_guide/deployments/deployment_strategies.html#rolling- strategy)

[2] [https://docs.openshift.com/container- platform/3.6/dev_guide/...](https://docs.openshift.com/container- platform/3.6/dev_guide/builds/triggering_builds.html#image-change-triggers)

</comment></comment></comment><comment><USER6> "Install kubernetes in 15 minutes or less -> bare metal" leads to this:

[https://coreos.com/tectonic/docs/latest/install/bare- metal/r...](https://coreos.com/tectonic/docs/latest/install/bare- metal/requirements.html)

There are over 15 non-trivial subtasks there (counting "foo and bar" under one bullet as two subtasks). Worse, this is just prerequisites for before the actual installation!

I really want something like what CoreOS claims to be to exist, but always feel like the victim of some inside joke when I actually try to install it.

Does anyone know of any alternatives that can be setup in a matter of hours and not days?

<comment><USER7> docker swarm. it really is as simple as installing the docker daemon on a cluster of machines and issuing a command to link them together.

<comment><USER8> I have to agree. Swarm is probably the easiest thing to get going besides the Docker engine itself. Honestly I felt like I might have been missing some step or component when I set up a swarm, because it was just really easy. Then I could start swarm services by making some slight modifications to my pre- existing Docker Compose files. It is very very easy.

<comment><USER9> And also runs very stable. It's not perfect, but I've certainly run into fewer bugs than when I was trying to get Kubernetes set up.

</comment></comment></comment></comment><comment><USER10> I disagree with this article. Coreos wouldn't have helped at all in this particular case. First of all, struts is not a system dependency. So it wouldn't be auto updated.

Well okay, how about Claire, the vulnerability scanner? It scans container layers, aka system dependencies. They don't scan your maven dependencies.(maybe I'm wrong here). So your entire security relies on someone not updating dependencies for a project often. Which was already the case. What did we gain? In this particular case, nothing

Of course kubernetes makes rolling out updates a lot easier. And thus a team might patch often. But that is not the OS.

Of course for system level security vulns, coreos is great and on the right track. But saying they would have caught Equifax is a big stretch. Equifax was not a big hack with multiple zero days. It was a team who didn't update their application.

</comment><comment><USER11> While admirable, I don't really see how CoreOS automatically makes the internet more secure as they claim. I'm assuming this blog is targeted at other tech folks so definitely expected a little more detail than a small blurb.

I say this as fan of CoreOS and really appreciate their OpenSource work.

<comment><AUTHOR> In a few words. With CoreOS Tectonic we are reducing the toil teams endure to update both application infrastructure and applications. We believe that security begins with a simple regular processes for getting updates to software out the door and that our tools and products remove toil from these processes.

Two sides of the problem:

Infra: CoreOS Tectonic[1] provides one-click updates of the entire app infra stack from the VM/bare-metal Linux instances[2] through the Tectonic control plane including Kubernetes, identity services, monitoring tooling, etc[3]. We call this automated operations.

App: By leveraging Kubernetes APIs application teams can roll. Further container scanning tools enable app teams to scan in-use containers for CVEs.

[1] [https://coreos.com/tectonic](https://coreos.com/tectonic)

[2] [https://coreos.com/blog/introducing-container-linux- update-o...](https://coreos.com/blog/introducing-container-linux-update- operator)

[3] [https://coreos.com/blog/announcing- tectonic-1.7.1](https://coreos.com/blog/announcing-tectonic-1.7.1)

<comment><USER12> A few years ago I worked at a big company that desperately needed something like this. Without it we more than did our part to make the internet less secure.

Without containers, software releases and infrastructure upgrades were highly interdependent. The result was that the software never released and the infrastructure never got upgraded.

Being able to upgrade individual services, independently of the infrastructure, is a bigger enabler than you would think in a large company. When you enable this, teams are suddenly able to release more often. Features ship faster and the lifetime of application vulnerabilities shortens.

Meanwhile, if Tectonic works as advertised, your infrastructure can auto- update but continue to provide a stable API to the services it supports. Again, the lifetime of vulnerabilities shortens, potentially by a lot.

</comment></comment></comment><comment><USER13> I like CoreOS, and I know this is some post that is meant for marketing purposes. But it's a shame they're propagating a myth that good security is the result of good tools.

It's a common fallacy to think that these security problems have a technical solution first. They don't. CoreOS makes tools that helps security aware companies/organizations implement security, but being sufficiently security aware is the _first step_. Buying and using CoreOS products and support does not help you anything if you don't know how to use it well, or if your organization doesn't allow the engineers to use them effectively. It's become abundantly clear that mismanagement is the root cause of the Equifax hack, and that the vulnerable Struts server is just a symptom of it. A fool with a tool is still a fool.

It's attractive to think CoreOS's products, or some other vendor's product, would have avoided this. But given the mismanagement it seems unlikely, and at best it would have just plugged a hole until another one popped up at some later time. Who knows that they already plugged some earlier worse holes with some other security products. Making the tools to make the internet more secure is the _easy part_. The hard part is getting everyone to use them in the right way. If tools and only buying things were the answer, the internet would have been a much more secure place already.

So keep doing you CoreOS, keep making those tools. But please don't oversell yourself.

</comment><comment><USER14> > I believe in freedom. Founding fathers, Martin Luther King, Jr. – that kind > of freedom. It's the same kind of freedom that motivates my passion for free > software.

Oh boy, this CEO is laying it on pretty thick here... I'm a pretty steadfast advocate for free software--and I think open software does make a better society--but even then, I think it's a bit strange to use a comparison to MLK for something that seems an order of magnitude less important.

It's giving me serious flashbacks to Silicon Valley.

[https://www.youtube.com/watch?v=J-GVd_HLlps](https://www.youtube.com/watch?v=J-GVd_HLlps)

</comment><comment><USER15> This looks like some epic PR talk for marketing purposes. Maybe they want to sell it to Equifax like companies?

<comment><USER16> And likely not letting a crisis go to waste.

</comment></comment><comment><USER17> An auto-updating OS doesn't really help you with web application vulnerabilities. Even something like Clair would be unlikely to help - <USER2> likely knew about the numerous struts vulnerabilities they faced and simply chose not to care (and prioritize other more exciting business prospects - like selling identity theft protection)

</comment><comment><USER18> I like their product, but evoking the founding fathers of the US and MLK in talking about a more secure OS seems a bit much.

<comment><USER19> Especially since it took another 80 years for the country that the 'freedom- loving' Founding Fathers created to actually get rid of slavery. Buggered if I know why some people get so religiose about the Founding Fathers when they missed such simple things like "hey, freedom for all means no slaves. And women get the vote as well."

<comment><USER20> They enacted tremendous social change for their time, but they weren't dictators. Even if they wanted to abolish slavery or enable women to vote, it would have never happened because those were very unpopular views at the time. Only a (benevolent?) dictatorial regime can act against the popular will of the people to enact changes faster than the people can accept them.

Societal change has a hull speed, and attempting to exceed that hull speed will result in massive push-back and ultimately failure even if the change in question is viewed as 'obvious' in historical hindsight.

We can see the exact same thing happening today with the legalization of gay marriage or drugs. Both will be viewed as obvious a generation from now, but today they face tremendous pushback. That's not the fault of governance, that's just governance imposing changes that approach the societal hull speed on those issues.

<comment><USER21> Also, it took a war to get rid of slavery.

I can understand why a new nation battling off a major world power for independence didn't have a civil war at the same time.

Sometimes, you have to pick your battles.

</comment></comment></comment></comment><comment><USER22> Usually blog posts like this come at pivotal times in a company's lifetime.

There doesn't seem to be anything yet. Interesting.

<comment><USER23> My guess is it's easier to get attention for selling cloud security after something like the Equifax breach happens.

</comment></comment><comment><USER24> For everyone criticizing this because there's not much actual substance, I'm fairly certain this piece wasn't aimed at you - it was aimed at people that get taken by this kind of rhetoric over actual technology. If it comes down to the usual players in enterprise software or CoreOS / Tectonic, I'd be perfectly happy to see that stack as an option in the F500. I'm sure there's a lot of reservations we all have about the tone and content, but as long as the fundamental goal isn't just selling I can let it slide.

</comment><comment><USER25> As far as I know, if a malicious user compromised the CoreOS update central and pushed a new rolling update, then everyone using CoreOS and (presumably automatically) received the rolling update will also be hacked, the end. This is an even bigger evil. It's no difference to a botnet . I don't know if CoreOS has some kind of update signing/verification or what but based on my assertion I wouldn't suggest using it.

</comment><comment><USER26> "At CoreOS, our aim is to arm these companies with the tools to build their cloud services – and run our digital lives – correctly. We're also dedicated to making it so easy to run these highly complex systems that they take care of themselves. What if they'll never miss an update again. They'll have all the security features turned on by default and new versions of applications will ship quickly and safely." What a load of absolute dreck this article is.

</comment><comment><USER27> Off-topic minor note to site owner: You have two shortcut icon references in your head, and one of them references the relative ico/favicon.png which 404's. Chrome falls back but many others do not and you get no favicon.

</comment><comment><USER28> Why do people lie about founding stories. Most of them boil down to two things:

1> We like making money 2> We enjoy working in the domain the company operates in.

Everything else is a usually made up crap for branding, PR and to feed tech journalists.

</comment><comment><USER29> It was my understanding that CoreOS was started as a minimal Linux distro tooled for hosting and orchestrating Docker containers. IMHO, Docker still has a long way to go to impress the security community.</comment>


Why we picked Clojure - dale-cooper https://medium.com/@metabase/why-we-picked-clojure-448bf759dc83 <comment><AUTHOR> Clojure would be a nice environment to work with if it wasn't running on JVM. It renders Clojure a resource hog, and the startup time is awful. A big no-no for most of my use cases.

> Python's mysql and postgres database drivers required compilation. This made > both developer machine setup as well as deployment more complicated than it > really needed to be.

Not quite. _You_ made setup more complicated than it needed to be. Ever heard of binary packages?

<comment><USER2> "Clojure would be a nice environment to work with if it wasn't running on JVM. It renders Clojure a resource hog, and the startup time is awful. A big no-no for most of my use cases."

Then don't use it, right? There are plenty of use cases for which Clojure makes sense, and where running on the JVM is a perk.

<comment><AUTHOR> > Then don't use it, right?

I don't. Though I would like to from time to time.

<comment><USER3> ClojureScript is these days selfhosting. If you want to write and run small scripts you could give Planck a try (ClojureScript REPL). The startup time is much smaller.

[http://blog.fikesfarm.com/posts/2015-08-22-planck- works.html](http://blog.fikesfarm.com/posts/2015-08-22-planck-works.html)</comment></comment></comment></comment>


Serious MacKeeper vulnerability found - <AUTHOR> http://www.thesafemac.com/serious-mackeeper-vulnerability-found/

<comment><USER2> MacKeeper is just malware, from my humble opinion...</comment>


'Armchair woodchoppers' make DIY timber guide surprise bestseller - <AUTHOR> http://www.theguardian.com/books/2015/nov/16/armchair-woodchoppers-guide-surprise-bestseller-norwegian-wood <comment><USER2> This one is barely even a submarine. That would imply the advertising / PR is disguised in some way. It's more like a cruise ship with an advert on the side telling you to buy this book as a Christmas stocking filler. The only real surprise is there's not a link at the bottom of the page to the "Guardian Bookshop". [http://www.paulgraham.com/submarine.html](http://www.paulgraham.com/submarine.html)

</comment><comment><USER3> This sounds like a non-parody version of the Swedish 1961 classic " _En liten bok om att bränna löv, ris, kvistar och annat avfall i ett hörn av trädgården_ " ([https://www.goodreads.com/book/show/9646686-en-liten-bok- om-...](https://www.goodreads.com/book/show/9646686-en-liten-bok-om-att-br- nna-l-v-ris-kvistar-och-annat-avfall-i-ett-h-rn))

<comment><USER4> "A Little Book About Burning Leaves, Shrubbery, Twigs and Other Waste in a Corner of the Garden"</comment></comment>


Mary and Josh release software - <AUTHOR> http://docs.pythonpackages.com/en/latest/faq.html#q-what-does-pythonpackages-com-do-for-me-that-i-can-t-do-myself

<comment><USER2> Sheer insanity. Mary is a moron and her app users are prime targets for hackers.

Just because the test coverage passes, doesn't mean the code doesn't do something nefarious behind the scenes.

If Mary doesn't review the code before release, she deserves the rapture her customers will bestow upon her when she finds out Josh was paid by a competitor to decimate her business.

It's not that I believe people are inherently evil, I just believe that you need to review every patch to make sure it does what it's supposed to do and _no more_.

<comment><AUTHOR> Was it not clear from the story that she was able to review the code? It's GitHub… is this any better?

        Mary is naturally skeptical at first. The "old" Mary would never have
        considered
        performing a release without testing it on her laptop first! However, since
        her
        code has 100% test coverage and Josh is a reliable community member and she
        can
        ensure that the tests have passed and she is able to review the code sanity
        and
        security online via GitHub, she happily agrees to perform the release.</comment></comment>