Show HN: DataJoy, an online Python and R editor for scientists - <AUTHOR> https://www.getdatajoy.com/

<comment><AUTHOR> DataJoy is our second product (we're also the team behind ShareLaTeX (www.sharelatex.com)). We realised that a lot of the technical problems we had solved to make ShareLaTeX possible could also apply to more general environments, like Python and R which are also very common in academia. We're hoping to make these tools easier for scientists to use, both when getting started and when collaborating on their data analysis, data clean up, statistics, etc. Let us know what you think!

<comment><USER2> ShareLaTeX is a fantastic product, thank you for making it. It's very much a Google Doc for LaTeX. I used it very extensively when I was in university. Given how many auxiliary files are generated/needed when compiling LaTeX, keeping those files in the cloud was very painless.

I think an interesting avenue to explore with that product would be more templates/easy-to-start documents/quick tutorials.

LaTeX is the standard in some parts of academia (math, CS, stats) and not others (my brother, a biochemistry student at a public research university with 30k+ students, say nobody has even heard of it there) - I think the amount of market that is still using Word for academic papers (often because they do not know of LaTeX or think they don't have time to learn it) is very significant. ShareLaTeX is definitely a step in making LaTeX more accessible for that segment but there's definitely a long road ahead too.

</comment></comment><comment><USER3> Hey, I just signed up and I'm a bit unclear on how the "reproducible" part works. If I download my files they don't contain e.g. a requirements.txt?

<comment><AUTHOR> This is a very good point, that we haven't worked out a good solution to yet (it's a notoriously hard problem). A requirements.txt file would certainly help, but then that's the sort of thing you'd probably need to maintain yourself? Fully reproducible environments that can be run offline is a stretch goal of ours for the future though.

The 'reproducible' part that we mentioned was meant to refer to the fact that it's easily reproducible on DataJoy. I.e. if you share you work with a collaborator, you're not going to have any headaches around getting the scripts working on their system, installing the right packages again, etc.

</comment></comment><comment><USER4> Are there any future plans to incorporate ipython notebooks?

If you could run ipython notebooks as a service, I would think you would get a significant number of users.

</comment><comment><USER5> Nice idea. How does this compare to someline like Cloud 9?

Minor interface note: I think the Run Line / Run Script buttons look a little out of place and too big.

<comment><AUTHOR> Thanks. DataJoy is aimed at a totally different market from Cloud 9. We want to make it easy for scientists and people doing data analysis, statistics, etc rather that developing software packages. The difference is that this requires a much more interactive environment where the code you're running is constantly evolving and being iterated on (if you've ever generated a plot, you'll know how many times it takes to get perfect). So we're aiming to work well for that use case.

<comment><USER2> You are definitely correct when it comes to non-software developers needing tools to do data analysis that allow for a lot more iteration and ad hoc exploration.

How do you see this competing with / differentiating itself from RStudio? The python support?

In the statistics department I was recently at, R was the lingua franca and Rstudio was how we wrote it. I have not found a better product yet for ad hoc exploration than Rstudio but will certainly give this a go.</comment></comment></comment>


When will machines outthink us? - <AUTHOR> http://www.msnbc.msn.com/id/20676037/

<comment><USER2> Strong AI (the creation of a general intelligence capable of learning new things in the way humans can, as opposed to weak AI which includes programming machines to do specific tasks that are usually considered to require intelligence, such as playing chess or flying a helicopter) is about 20 years away...and it has been about 20 years away ever since the 1950's.

The traditional approaches to build a strong AI all involved forms of explicit knowledge representation: where facts about the world would be stored in memory and then pieced together to come to new (hopefully intelligent) conclusions and plan actions. Most people quickly realized that this approach doesn't scale at all: most knowledge that humans have about the world is very context-dependent and fuzzy. The new approaches around machine learning and probabilistic modeling are more promising but almost all of them involve building mathematical models around a specific problem rather than building a general learning machine.

Most leading AI researchers in academia aren't even trying to build a general- purpose intelligence anymore, instead focusing on solving domain-specific problems, and the field of AI has been much more successful over the last fifteen years or so as a result.

<comment><USER3> So do you think we are closer to getting to AI than we were in the past or closer? Certainly, we have better understanding of cognition today. Modern understanding of the mind like we have now didn't really exists in decades previous along with better imaging technologies. Is there some reason why we can't build a general learning machine?</comment></comment>


The Elements of HTML - <AUTHOR> http://w3c.github.io/elements-of-html/

<comment><USER2> Please use [http://meiert.com/en/indices/html- elements/](http://meiert.com/en/indices/html-elements/) instead, as it's based on the WHATWG HTML Living Standard rather than W3C's versioned fork.

<comment><USER3> Two things about that:

1\. What are the differences? I've no idea what the significance of the source is.

2\. The one you linked is hella unreadable. OP's table is, due to the color differences, extremely easy to skim, while the one you linked has only very similar black&white x and checkmark symbols, which require closer inspection to differentiate.

<comment><USER4> The "main" development site for HTML5 is the WHATWG. That's where all the browser implementors convened when they lost faith in the W3C.

The WHATWG doesn't bother with versions, there will be no HTML6, for example. They just continually update the HTML5 spec. It's a living spec.

The W3C kind of tracks the WHATWG progress and tags certain points with version numbers (that's where the "5.1" comes from).

Although they do some development on their own, and sometimes both sides disagree and the specs diverge. At some point WHATWG removed an element that W3C retained or the other way around.

I have no idea if they still diverge. I just discard the W3C when it comes to HTML5 and follow WHATWG.

Well, practically speaking, I follow what the Mozilla Development Network documents. :-)

<comment><USER5> _> Well, practically speaking, I follow what the Mozilla Development Network documents. :-)_

Yeah, I don't quite know when such a table is very useful anyway. What I check is: (A) [http://www.whatwg.org/specs/web-apps/current- work/multipage/...](http://www.whatwg.org/specs/web-apps/current- work/multipage/section-index.html#elements-1) (B) [http://caniuse.com/](http://caniuse.com/)

<comment><USER6> I love [http://caniuse.com](http://caniuse.com) but I wish they had more detail. For instance, on the XHR page, I'd like to know which browsers support addEventHandler syntax and which ones require the use of onreadystatechange.

</comment></comment></comment></comment></comment><comment><USER7> No blink tag documentation. I can't use this thing.

<comment><USER8> I believe it is because it is a non-standard tag...

</comment></comment><comment><USER9> The browser-tag relationship often happens to be more nuanced than just "supports" or "not supports". I'd recommend skimming an MDN article in case of doubt. They also include browser compatibility tables for quick reference and link to W3C and WHATWG specs.

</comment><comment><USER10> Another index that's a lot more usable than this one IMHO is:

[http://developers.whatwg.org/section- index.html#index](http://developers.whatwg.org/section-index.html#index)

<comment><USER11> It serves a different purpose.

Your link does not show how HTML evolved/what tags are new, the OP link doesn't show what each element actually _is_.

<comment><USER12> Here's one also by WHATWG, with all the relevant changes:

[http://meiert.com/en/indices/html- elements/](http://meiert.com/en/indices/html-elements/)

</comment></comment></comment><comment><USER13> What is the point of things like hgroup and rbc that are listed as not available in any of the versions on the chart?

<comment><USER14> hgroup was part of the "original" HTML5 spec, but it won't be part of HTML5 going forward as the living spec grows up a bit. I think it's still useful: [http://www.whatwg.org/specs/web-apps/current- work/multipage/...](http://www.whatwg.org/specs/web-apps/current- work/multipage/sections.html#the-hgroup-element)

</comment></comment><comment><USER15> Are frames really deprecated in HTML 5.1? Why does the documentation not tell that?

<comment><USER16> It's on the list of obsolete, non-conforming features [http://www.w3.org/TR/html5/obsolete.html#non-conforming- feat...](http://www.w3.org/TR/html5/obsolete.html#non-conforming-features)

</comment></comment><comment><USER8> Is it me or the <marquee> is missing?

<comment><USER8> Oh, may be because is a on-standard...

</comment></comment><comment><USER17> <center> is not HTML5? why? i love <center> & they standardised all other worst practices of the browser vendors, but this one they missed?

<comment><USER18> Probably because it's presentational, rather than "semantic". You're supposed to use CSS to do this.

<comment><USER19> If only they could retrofit <center> as a semantic tag, like they did with <small>.

<comment><USER20> <center>, though, has no semantic implication. <small> can be taken to mean "here be subsidiary information not worthy of large print (or in which we wish to hide critical information we do not with the reader to pay any attention to)". Centred text (or figures) is, and always has been, a stylistic decision, and there are already containers that are semantically meaningful to which that stylistic decision may be applied.

<comment><USER17> <center>-al (central) information maybe? but yeah, i understand why they don't standardise it, but still i prefer a working <center> tag over

      {
          display: block;
          margin-left: auto;
          margin-right: auto;
      }

<comment><USER21> Agree. And also no valign="middle" in HTML5 <td> elements drives me crazy. I know I'm "supposed" to use CSS, but CSS should be simple. I shouldn't have to fight with silly non-intuitive negative margins and nested div hell just to center something. And then deal with IE's own artistic interpretation of that div hell.

<comment><USER22> You don't need to use negative margins or anything. You just need the flexible box model ([http://philipwalton.github.io/solved-by- flexbox/demos/vertic...](http://philipwalton.github.io/solved-by- flexbox/demos/vertical-centering/)).</comment></comment></comment></comment></comment></comment></comment>


Google Deprecates "Sort by Date" Search Function - <AUTHOR> https://twitter.com/searchliaison/status/1115706765088182272 <comment><AUTHOR> Even if Google's dates are not "high confidence," this function was still useful outside of News search. Very disappointed to see this be retired.

<comment><USER2> Might be connected to YT breaking search sort after the mosque video.</comment></comment>


Bill Gates @ World Economic Forum: A new view for entrepreneurship - <AUTHOR> http://online.wsj.com/public/article_print/SB120113473219511791.html

<comment><AUTHOR> Full Video: <http://www.youtube.com/watch?v=Ql-Mtlx31e8>

Text of Speech: [http://www.microsoft.com/Presspass/exec/billg/speeches/2008/...](http://www.microsoft.com/Presspass/exec/billg/speeches/2008/01-24WEFDavos.mspx)

Why you should watch/read: "The potential of a big financial return for innovation unleashes a broad set of talented people in pursuit of many different discoveries. This system, driven by self-interest, is responsible for the incredible innovations that have improved so many lives."</comment>


AWS Achieves PCI DSS 2.0 Validated Service Provider Status - <AUTHOR> http://aws.typepad.com/aws/2010/12/aws-achieves-pci-dss-20-validated-service-provider-status.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AmazonWebServicesBlog+(Amazon+Web+Services+Blog) <comment><USER3> I ran down the implications of this: [http://blog.meatinthesky.com/pci- compliance-in-the-cloud-thi...](http://blog.meatinthesky.com/pci-compliance- in-the-cloud-this-changes-ever)

<comment><USER4> You seem to assert that almost every startup that accepts credit cards is not PCI compliant, and further, operates on an environment which at times in the past has been known to be moderately insecure (e.g. hypervisor attacks). That's a pretty bold claim. Do you have any evidence to back it up? (e.g. name companies which are knowingly operating without PCI compliance when they're required to have it)

<comment><USER3> I'm saying every startup that hosts their checkout pages in the cloud is - <USER2> definition - in violation. If you know a startup that accepts credit cards and doesn't have the secure server that accepts the credit card input on a rack somewhere - whether in their office, in their apartment, in a colo, or whatever - then that startup is, _by definition_ , not PCI compliant.

There is a simple and easy way to get into compliance without moving your host - use hosted payment pages. PayPal, Recurly, Braintree, and the other top-tier providers all have hosted payment pages.

Every startup I have been employed by or consulted for w/r/t payments either has a physical box or uses hosted payment pages.

<comment><USER4> Right, so do you know of any that are in violation?

I know of a bunch of startups that use a hosted page (e.g. Paypal, Google Checkout, authorize.net SIM, whatever), and a bunch of startups that host their main site in the cloud but have a physical server for accepting CCs, but I don't know of any that accept CCs directly on a VM. I'd be pretty surprised to hear about it.

</comment></comment></comment></comment><comment><USER5> This is great. I can't wait for the other vendors to follow suit.</comment>


BlackBerry outage: The questions still unanswered - <AUTHOR> http://www.computerworlduk.com/news/mobile-wireless/3310435/blackberry-outage-the-unanswered-questions/

<comment><USER2> RIM MD Stephen Bates was just on a London TV channel and gave a fairly dumb PR performance which is odd because he does much better on pre-record <http://www.bbc.co.uk/news/technology-15272310>

I'm not an addict but a lot of my friends (in Africa) are and they are so totally p __ __d (and some are losing actual business) that they will be acting on their iPhone desires very soon.

Anyone here know more about what's really going on?

Anyone with the ear of RIM? A refund would really go down well with my friends.</comment>


Hospital computers across England shut down by ransom cyberattack - <AUTHOR> https://www.rt.com/uk/388115-nhs-hospitals-cyber-attack/ <comment><USER2> It has been targeted to various Spanish companies as well: [http://elpais.com/elpais/2017/05/12/inenglish/1494588595_636...](http://elpais.com/elpais/2017/05/12/inenglish/1494588595_636306.html)

</comment><comment><USER3> Many dupes of this story today</comment>


A Mildly Satanic New Video Game That You Can Only Play Online with Money - <AUTHOR> http://kotaku.com/a-mildly-satanic-new-video-game-that-you-must-play-onli-1638083439/

<comment><USER2> This is a new trend starting in gaming industry. Apple has now started approving skill-based "gambling" games. One such example is Dollar Candy created by Ian Ippolito (who also created PlanetSourceCode.com and rentacoder.com). These games are addictive, you play against real human and there is money on the line in each game.

Gambling had been banned for a reason for long time in many cultures - it's all too common for poorest people to blow up their earnings in the hope of a jackpot. They do so not because they are less smart but because they are too desperate and don't have any other opportunity to making it through. Now game creators have found a loop hole in the law aka "skill-based" games. I can see lot of people barely making ends burning their kids school money in these stuff while standing in grocery lines.

</comment><comment><AUTHOR> What I thought was compelling is that a well-known indie game developer had the really smart idea to target the online-poker skilled-game competition niche that's opened by short-sighted US laws

</comment><comment><USER3> I set my expectations low when I saw the empty hype the beginning of the article was building up, but this game actually sounds quite strategically interesting.

I definitely see why this is the kind of game that isn't the same without money, much like poker and backgammon. I hope there's some kind of micropayment platform that it could eventually support to avoid the credit- card fees and the check-mailing fees.

(No, not bitcoin. Most people can't use bitcoin.)

</comment><comment><USER4> The success of this game is partly predicated on player trust, i.e. how do I know that it's not being rigged against me by the operator? It is a game of skill, but gives the player imperfect information (unlike chess), leaving the possibility of various forms of cheating and fraud.

I am interested to see how this could be addressed.

</comment><comment><USER5> Jason Rohrer's games are fantastic - I'd recommend his game Passage, which only lasts for five minutes.

[http://hcsoftware.sourceforge.net/passage/](http://hcsoftware.sourceforge.net/passage/)</comment>


Polynomial time factoring algorithm using Bayesian arithmetic - <AUTHOR> http://arxiv.org/abs/1212.4969

<comment><USER2> When I opened this page, I was going to be the third comment, then I got caught up trying to explain why their linked paper [1] (proving P=NP) is non- sense. Unfourtuantly, my habit of assuming that my not understanding something just means that I need to re-read it or play with that segment of the proof myself costed me more time than I thought (even after I wrote in my partially completed reply that the introduction alone was sufficient to dismiss the paper as nonsense). Anyway, here is my overkill analysis:

I would be highly skeptical of their claim. First of all, if they did prove P=NP (and long enough ago to write another paper), then I cannot imagine why this is the first I am hearing about it, as solving this problem would likely get a prominent spot in mainstream news, and definitively its own article on HN.

Beyond that, the paper proving P=NP, even to my amateur eye, looks like garbage. From the P=NP paper: "What is the powerful ingredient which allows a dramatic speed-up of quantum computation over classical computation ? We propose that this ingredient is an implicit use of the Bayesian probability theory. Furthermore, we argue that both classical and quantum computation are special cases of probability reasoning. On these grounds, introducing Bayesian probability theory in classical computation as well, we reduce a typical NP problem, namely 3-SAT, to a linear programming problem. According to algorithmic complexity theory, this proves that P=NP."

"Any logical algorithm can be formulated as a linear programming problem. Specially, a basic question of logical satisfiability with n variables, namely 3-SAT, is equivalent to a linear programming problem with O(n^3) unknowns and even in general with O(n) unknowns. According to algorithmic complexity theory, this proves that P=NP."

"Again, the main reasons of the supposedly quantum efficiency are basically unknown, but the common wisdom is that entanglement should be the key ingredient. Indeed, in the quantum community, it is 'widely believed that classical systems cannot simulate highly entangled quantum systems' [13]. By contrast, we have nevertheless argued [14] that the concept of entanglement is actually a quite classical attribute of contextual systems. Furthermore, any classical computer can be regarded as a highly entangled system as far as the entanglement is measured between the binary digits of the processed data during the computation. In addition, we shall argue that the crucial ingredient is not in the least a Hilbert space with its full quantum machinery but only a flexible randomization, or to be exact, the implicit use of Bayes probabilities. This is like a grin without a cat in Alice and Bob adventures in Quantum Land."

They devote two pages to history and what should be common knowledge to anyone interested in this paper.

The actual proof is mathametical gibberish, however there are surprisingly large contiguous sections of the proof that, by themselves, work.

[1] <http://arxiv.org/pdf/1205.6658v2.pdf>

</comment><comment><USER3> OK, I've really only skimmed it, but here's an impression ...

He spends a huge amount of time setting up the very very specific equations, and then waves a magic wand saying "This is a Linear Programming Problem which can be solved in polynomial time."

It looks to me like an integer linear programming problem, and that's known to be NP-Complete. There doesn't seem to be anywhere that he takes a non-integer solution and converts it in polynomial time to an integer solution (or provably fails), nor is there anywhere that shows how to find an integer solution in among all the possibly exponentially many solutions it might produce.

<comment><AUTHOR> As far as I understand from the two papers, and I haven't finished reading them, nor do I expect to understand it all, but this appears to be very much an LP problem, not ILP, from what I've read so far.

<comment><USER3> I don't have time now, and I'm unlikely to have time in the next six months, but it looks awfully like he's setting up a standard LP problem, but I don't see how he's extracting integer solutions from it.

I expect there to be errors, and I expect them to be subtle, and I expect that if they're pointed out he'll push them around to some other place where they're hard to find. Having said that, I haven't read it thoroughly (and won't have the time to) and I'm disinclined to. Not that that's any great loss - I'm not an expert.

</comment></comment></comment><comment><USER4> You might as well start with his proof of P=NP

<http://arxiv.org/pdf/1205.6658.pdf>

</comment><comment><USER5> The abstract states, "When applied to NP-complete algorithms, this leads to the fundamental conclusion that P = NP".

Isn't this a much more important conclusion?

<comment><USER6> Agreed. Lots of papers like this are submitted, but none have been right so far. Is there some reason to trust this one in particular? I'm not sufficiently familiar with this material to evaluate it.

<comment><AUTHOR> This happened a few years ago when someone posted some code on GitHub that they claimed demonstrated P=NP, however it turned out not to.

This paper on the other hand is a more theoretical approach and appears to be more concrete than previous attempts that I'm aware of.

While it may or may not be correct, (experience hinting at the latter) it is interesting news, and an interesting paper nonetheless.

<comment><USER3> Actually, there appears to be nothing new, structurally or ideas-wise, and I don't think it is very interesting. The concreteness is all about creating a system of equations in terms of the bits in a multiplication. That's really trivial. Large and messy, but nothing new.

There's no concrete indication that I've found in this paper about how to solve those equations.

</comment></comment></comment></comment><comment><USER7> Red flags: a single author, use of nonstandard terms like "algorithmic complexity theory" (the standard term is simply "complexity theory"), appeals to quantum computing theory (which is only tangentially related to the topic of the paper), author only has two papers, no coauthors, and begins with awkwardly irrelevant statements like, "Arithmetic is a part of abstract mathematics."</comment>


Respondent Robotics: Simulating Responses to Likert-Scale Survey Items - <AUTHOR> http://journals.sagepub.com/doi/full/10.1177/2158244018764803 <comment><AUTHOR> Submitter here. It seems some types of surveys in psychology may be more affected by the words chosen than previously accounted for.</comment>


US Navy X-47B Robot Fighter Jet Completes First Phase Of Testing - <AUTHOR> http://singularityhub.com/2012/06/29/us-navy-x-47b-robot-fighter-jet-completes-first-phase-of-testing/ <comment><USER2> In _The Terminator_ , Skynet is supposed to have achieved sentience and turned against its masters. But if you look at something like this from the point of view of someone in a part of the world where the political structure is not aligned with US hegemony, the difference between an advanced AI which has "turned" and a robotic army that is ultimately still controlled by humans is probably academic. I suppose that in the latter scenario there is still in theory some humanity to appeal to, but I somehow doubt that's what's running through your head when you're being chased by a faceless killing machine.

Then again, a closed-cockpit jet fighter isn't exactly a cuddly sight regardless.

<comment><USER3> Humanity?

[http://www.iraqwarlogs.com/2010/10/23/us- commits-%E2%80%9Cwa...](http://www.iraqwarlogs.com/2010/10/23/us- commits-%E2%80%9Cwar-crime%E2%80%9D-as-apache-helicopter-cleared-to-gun- surrendering-insurgents/)

No, no humanity.

[http://www.channel4.com/news/iraqs-secret-war-logs- surrender...](http://www.channel4.com/news/iraqs-secret-war-logs-surrender- and-die)

Watch the second video. US Military considers that it is not possible to surrender to aircrafts. So, if someone tries to surrender, they are still valid targets, and then eliminated.

So, basically, it's the same as fighting with a totally autonomous machine. There's no humanity in that kind of fight.

<comment><USER2> Well, that's the point, right? People are worried about hostile AI, and how scary it would be to be confronted by an enemy possessing overwhelming resources and no emotional common ground. But much of the world already has that experience.

</comment></comment></comment><comment><USER4> These sorts of weapons move the balance of power to where the only reasonable defenses against the US are nuclear and bioweapons, and therefore that is where you should start investing your research efforts in if you are not an ally.

<comment><USER5> How does this type of aircraft shift the balance of power? It doesn't appear to do anything "better" than a manned aircraft aside from not having a pilot to lose onboard.

<comment><USER6> 1\. With in-air refueling, a greatly increased time in the air vs manned aircraft.

2\. Ability to outmaneuver any manned aircraft by exceeding the human g limit.

3\. Greatly lowered cost in both manufacture and pilot training (or lack of).

4\. When fully realized, AI that is superior to human pilots -- increasing overall combat effectiveness and reducing error.

<comment><USER7> 5\. Reconnaissance

6\. Stealth

7\. Autonomy

8\. Cost of pilot death (human loss)

9\. Reduced training time

<comment><USER8> 0\. No risk of losing a human, which makes it easier to decide to attack or go to war, and so you look more bad ass in the balance of power.

The increased willingness and ability to attack is a bad thing.

</comment></comment></comment></comment></comment><comment><USER9> Fighter? Hardly:

>The mission of the Navy Unmanned Combat Air System (UCAS) Aircraft Carrier Demonstration (UCAS-D) is to mature technologies for a carrier suitable, low observable relevant, unmanned air system in support of a potential follow-on acquisition milestone for an unmanned air system capable of providing persistent, penetrating surveillance, and penetrating strike capability in high threat areas.

Primarily a recon craft with strike capability.

</comment><comment><USER10> So how many G's can one of these pull? I imagine it's much higher than any human fighter since the upper limit is what can safely hold the craft together.

<comment><USER11> I wouldn't imagine it needs to pull very much at all. There are no more dogfights anymore, it's just electronic countermeasures vs electronic detection, jamming and spoofing.

<comment><USER8> Dodging SAMs could involve a lot of Gs.

<comment><USER10> That's more along the lines I was thinking. Don't fighters still do evasive maneuvers?

Of course, the SR-71 Blackbirds had one of the most effective solutions. Fly fast and unpredictable vectors. Anything on an interception vector would be miles off by the time it reached it's predicted location.

</comment></comment></comment></comment><comment><USER12> I wonder how they develop the code for these systems. You'd hope it might be something approaching the standards, rigours, and processes of the people who write the Space Shuttle code, but I have the sad feeling it probably isn't.

<comment><USER13> I am pretty sure it is pretty strickt. As far as I understand code for airplanes in general is much more encapsulated to do one thing well without any resuability of the code as such. It's in other words strictly causal.

Fighter jets might have more of an abstract layer for all the AI stuff so this might leave more room for an actual OS to reprogram and optimize on top off.

<comment><USER13> It might be perfectly possible that I am completely off on this. But my understanding from reading about it is that software is written very differently in airplanes than in most other fields.

Down vote is fine if I am wrong. But would appreciate to learn how I am wrong then.

</comment></comment></comment><comment><USER14> Eliezer's unfriendly AI does not need to be smarter necessarily, just better armed.

</comment><comment><USER15> Impressive, but seems like a really, really bad idea.

</comment><comment><USER16> Super high tech, advanced-robot fighter jet... not able to retract landing gear? (from video). Did that bother anyone else?

<comment><AUTHOR> Generally, the first test flights won't have an aircraft raise its gear.

[http://www.upi.com/Business_News/Security- Industry/2011/02/0...](http://www.upi.com/Business_News/Security- Industry/2011/02/07/Navy-flight-tests-new-UAV-for- carriers/UPI-66291297084580/)

</comment></comment><comment><USER17> Shouldn't that title read "Cyberdyne Systems X-47B"?

</comment><comment><USER18> Most disappointing title ever.

It's just an unmanned jet.

was expecting something like robotech/macross</comment>


CDC employee Timothy Cunningham went missing more than a week ago - <AUTHOR> https://www.nbcnews.com/news/us-news/cdc-employee-timothy-cunningham-went-missing-more-week-ago-n850721 <comment><USER2> Sounds like a really great fellow. I hope they find him and can help him.</comment>


The Christian Withdrawal Experiment - <AUTHOR> https://www.theatlantic.com/magazine/archive/2020/01/retreat-christian-soldiers/603043/ <comment><USER2> "Latin is the only foreign language offered, and teachers favor blackboards over computers. A classical education, the school believes, is the foundation of students' future."

Long before I took an interest in programming and computer languages, I studied Latin and Spanish in high school and had worked out a grammar for a language I called "Simplica" which would be roughly based on Latin but purely phonetic and where all verbs would be regular. I was into language design long before I ever started applying it to computers.</comment>


Music is our universal language. Join the conversation - <AUTHOR> http://blog.elementsme.com/join-the-conversation/

<comment><USER2> Check out the landing page: it's awesome.</comment>


One Giant Leap for SQL: MySQL 8.0 Released - <AUTHOR> https://modern-sql.com/blog/2018-04/mysql-8.0 <comment><USER2> PostgreSQL 9.5 manual: "Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table."

MySQL 8.0 manual: "Queries that skip locked rows return an inconsistent view of the data. SKIP LOCKED is therefore not suitable for general transactional work. However, it may be used to avoid lock contention when multiple sessions access the same queue-like table."

Flattering (I wrote those words). Of course we borrow the SKIP LOCKED syntax from yet another database (though the idea is much older, under different names).

</comment><comment> Digital-Citizen The article says MySQL "the most popular free SQL database" but gives no reference for this claim. It seems to me that every system I know of has at least one install of SQLite somewhere (be it as something that was installed with the OS or in some commonly installed application). SQLite seems to agree with my assessment in [https://sqlite.org/mostdeployed.html](https://sqlite.org/mostdeployed.html) so I wonder where did the quoted statement above come from, and what's the basis for it?

<comment><USER3> Full quote.

> MySQL is very popular. According to db-engines.com, it's the second most > popular SQL database overall. More importantly: it is, by a huge margin, the > most popular free SQL database.

The second sentence is linked: [https://db-engines.com/en/ranking](https://db- engines.com/en/ranking)

The third sentence refers to the same source.

<comment> Digital-Citizen Thanks for the link, but I find this highly unlikely to be what the article or db-engines.com claims. The data comes from vague sources that don't jibe with number of installations. The word "popular" is used in such a perverse way it renders the db-engines.com ranking and that claim in the article impossible to believe.

That ranking site claims "The DB-Engines Ranking ranks database management systems according to their popularity." but then includes concepts such as "Number of mentions of the system on websites", "General interest in the system", and "Relevance in social networks" as part of the ranking (see [https://db-engines.com/en/ranking_definition](https://db- engines.com/en/ranking_definition) for details). In fact, the one thing one would typically consider to be "popularity" \-- installed base size -- is not considered at all ("The DB-Engines Ranking does not measure the number of installations of the systems, or their use within IT systems."). In fact the paragraph from which I drew that quote highlights how ridiculous this site is: to understand that paragraph's use of the word "popular" you have to use the common definition, the definition they won't use in computing their so-called "popularity" ranking. This tells me that db-engine.com's ranking is not to be taken seriously.

As SQLite's page I pointed to claims, virtually every modern operating system and many popular programs come with at least one copy of SQLite. But then again, understanding the previous sentence requires using the word "popular" in a way that most people use that word most of the time, not the perverse way db-engines.com used the word. Put differently, Oracle, db-engines.com #1 pick, doesn't run on iOS, MacOS, GNU/Linux, and some variants of Windows. That means there are a considerable number of systems that host at least one copy of SQLite and will never host Oracle.

I find it very difficult to believe that there are more running or installed instances of Oracle on the world's computers than there are SQLite. The original article is plainly wrong. MySQL is nowhere nearly as "popular" as is claimed and the backing to believe otherwise hinges on a highly dubious source.

</comment></comment></comment><comment><USER4> This looks like a great place to tell you about the burning white hot rage I have for Debian after my difficulties with MySQL this past weekend.

I was working on script that builds a LEMP stack with Vagrant on a Debian box. And when the script was upgraded to use Debian Stretch, everything we did to customize MySQL configuration broke, and I spent many hours of anguish not knowing why.

Until I stumbled onto the fact that Debian decided that in version 9, they'd start installing MariaDB when you typed "apt-get install mysql-server". I cannot fully explain the anger I have for this dark pattern: When I say "install x", you should either install x or say you don't have it, not install a competitor you think is close enough.

<comment><USER5> The way MySQL was being handled as a hybrid of open-source and closed-source for tests was causing a lot of trouble for organizations like Debian that wanted to ship truly open-source software. MariaDB was, at the time, a pretty close fork with the right license model so it was a good drop-in substitute.

These situations are rare, but it's not without precedent.

<comment><USER4> To be clear: I have no issue with Debian shipping MariaDB. I have an issue with Debian pretending MariaDB is MySQL.

If I run "apt-get install mysql-server" and get something other than MySQL, we now have a trust issue: The Debian repository cannot be trusted.

<comment><USER6> MySQL is not available in Debian repository since MariaDB has replaced it.

mysql-server package installs mariadb so that they can handle upgrades gracefully in a way that won't leave existing users in cold.

They would choose to do it the way you said but then the people who are upgrading their systems would be annoyed instead of you. Tough choice but either way it's a minor annoyance to be honest.

Addition:

Also it's not as if it isn't _very_ clear what you're installing:

      $ apt install mysql-server
      ....
      The following NEW packages will be installed:
        default-mysql-server galera-3 libdbd-mysql-perl libhtml-template-perl libterm-readkey-perl mariadb-client-10.1
        mariadb-client-core-10.1 mariadb-common mariadb-server-10.1 mariadb-server-core-10.1 mysql-server
      ....

      $ man mysql
      MYSQL(1)                                                               MariaDB Database System                                                              MYSQL(1)

      NAME
             mysql - the MariaDB command-line tool

      ....

It is not aim of distribution maintainers to hide facts or make you use counterfeit programs, they just want to help users with transition.

<comment><USER7> > _Also it 's not as if it isn't very clear what you're installing_

I certainly disagree that this qualifies as "very clear". The user asked to have mysql installed and the very first item on the list of stuff-to-be- installed is called "default-mysql-server". That certainly sounds like positive confirmation that you are indeed getting what you asked for.

Now, if you keep reading, you will see "mariadb-server-10.1" and stuff on that list, and if you also know two pieces of information (that mariadb is a fork of mysql, and that someone at Debian contemplated the idea of substituting one for the other), then you can see that you _may_ have concluded the wrong thing and you can backtrack from there. Though really, at this point you still have conflicting information and it isn't conclusive.

So, I wouldn't say "very clear" is accurate. More like "possible to decipher if you have all the relevant context and are paying close attention".

As UIs go, this definitely violates the principle of least astonishment.

<comment><USER6> Okay, these are all fair points regarding package installation

(though I don't buy this was an enough excuse for GP in this specific case because it becomes clear at the moment you run `mysql' command)

</comment></comment></comment></comment></comment></comment><comment><USER8> I'm having a hard time to pick between mysql8 and postgresql10.

MySQL is under Oracle is one of the major concern, plus it's said to be good for less-write-more-read workload, which is fine.

PostgreSQL is more on the open source side, however bugs like this really concerns me: [https://lwn.net/Articles/752063/](https://lwn.net/Articles/752063/) \-- basically, PostgreSQL does not do its own DIO instead it depends on file system and it can lead to data corruption, and there is no quick fix...

tried to install mysql8.0 on ubuntu16.04 failed so far, need a vm to test it out sometime.

<comment><USER9> From MySQL's documentation:

[https://dev.mysql.com/doc/refman/8.0/en/innodb- parameters.ht...](https://dev.mysql.com/doc/refman/8.0/en/innodb- parameters.html)

> Caution: Many operating systems and some disk hardware fool the flush-to- > disk operation. They may tell mysqld that the flush has taken place, even > though it has not. In this case, the durability of transactions is not > guaranteed even with the setting 1, and in the worst case, a power outage > can corrupt InnoDB data. Using a battery-backed disk cache in the SCSI disk > controller or in the disk itself speeds up file flushes, and makes the > operation safer. You can also try to disable the caching of disk writes in > hardware caches.

<comment><USER10> Pardon my ignorance, but do you know if this would affect PG as well, since it's an OS/hardware sourced issue?

<comment><USER11> It's same issue grandparent mentioned.

</comment></comment></comment></comment><comment><USER12> Oh. Apparently I don't know SQL nearly as well as I thought.

<comment><USER5> You could write an entire book on SQL and have the same thought.

<comment><USER13> The more you know SQL, the more you don't know SQL.

<comment><USER14> As your circle of knowledge increases, so does your circumference of ignorance.

</comment></comment></comment></comment><comment><USER15> SQL seems to be evolving by tacking on features willy-nilly, with no clear holistic philosophy, and worse, the standards which emerged aren't followed by all major implementations and even standards which are followed differ starkly in terms of performance across implementations. The product is that we all write implementation-specific SQL, which largely defeats the purpose of a standard language altogether.

I think if someone came along and fixed these major issues and gave it a modern syntax without significantly degrading performance, it would be sufficiently different that the XKCD "competing standards" meme would not apply.

I would love to hear from someone with database specialization.

<comment><USER16> Browsing the featured added by SQL standards on Wikipedia:

SQL:2003 added XML functions

SQL:2006 more XML!

SQL:2008 XQuery!

SQL:2016 JSON!

My guess it's not so much "willy-nilly" but more a sign of "what was popular."

<comment><USER17> Calling it now: in 2020 they will add YAML support.

<comment><USER18> WASM for SQL

<comment><USER19> Definitely blockchain.

</comment></comment></comment></comment></comment><comment><USER20> Do not be mislead, MySQL is the PHP of the database world.

[https://slideshare.net/anandology/ten-reasons-to-prefer- post...](https://slideshare.net/anandology/ten-reasons-to-prefer-postgresql- to-mysql)

</comment><comment><USER21> _The reason I think a single database release has a dramatic effect on the entire SQL ecosystem is simple: MySQL is the weakest link in the chain._

That's a heckuva deep cut.

I wonder why/how MySQL got so popular, when in my head the go-to is Postgres (perhaps because I worked on Django/Rails stuff early on). Is it because of Oracle and their marketing/sales work?

<comment><USER22> > I wonder why/how MySQL got so popular, when in my head the go-to is Postgres > (perhaps because I worked on Django/Rails stuff early on). Is it because of > Oracle and their marketing/sales work?

MySQL's popularity long predates the Oracle acquisition. MySQL got popular because it was usually available (alongside PHP) on shared hosting and had very fast read performances (using the default MyISAM storage, at the cost of write performances or actually caring about your data). What with the web being front-loaded on read, that made it a pretty good fit.

Meanwhile historically postgres was byzantine to set up and _slow_. It cared much more for your data and had plenty of features (custom types! procedural languages! transactional DDL!), but it didn't really work OOTB (still doesn't, IIRC the base configuration still assumes rotating rust and 128MB RAM or some shit) and wasn't available on most hosting platforms, especially not the really cheap shared hosts.

Also while Django defaulted to postgres, I'm pretty sure Rails used to kinda- sorta default to mysql (I'm talking 10 years back, many plugins/extensions didn't work properly on pg).

<comment><USER21> Makes sense, thanks for the thorough answer! The popularity of the LAMP stack predates me a bit, so I didn't know as much about the history.

</comment></comment></comment><comment><USER23> Celso's quote applies equally well to SQLite, which is SQL compliant until it's not. Glad to see MySQL improvements, but wish SQLite was also getting similar upgrades. It was a bit depressing to see the solid column of Xs by SQLite for all these great new features the author was relishing.

<comment><USER3> > It was a bit depressing to see the solid column of Xs by SQLite

Except WITH [RECURSIVE] which is in SQLite since early 2014. Damn, I forgot to use that trivia in my MySQL bashing... ;)

Other than that, I met Richard Hipp in person some years back. He is a VERY pragmatic guy. I wrote an article about it:

[https://use-the-index-luke.com/blog/2014-05/what-i-learned-a...](https://use- the-index-luke.com/blog/2014-05/what-i-learned-about-sqlite-at-a-postgresql- conference)

</comment></comment><comment><AUTHOR> I'm happy to see that MySQL is progressing, but I think that enumerating its adherence to the latest SQL standards overlooks some of its other serious flaws from the perspective of operations and reliability. Below are two of the worst operational situations I've encountered with MySQL. I've run several other databases in high-volume production environments and never encountered those kinds of problems. That MySQL's developers would have made these choices that result in those kinds of problems is truly mind-boggling to me.

* If you have pre-5.6.4 date/time columns in your table and perform any kind of ALTER TABLE statement on that table (even one that doesn't involve the date/time columns), it will TAKE A TABLE LOCK AND REWRITE YOUR ENTIRE TABLE to upgrade to the new date/time types. In other words, your carefully-crafted online DDL statement will become fully offline and blocking for the entirety of the operation. To add insult to injury, the full table upgrade was UNAVOIDABLE until 5.6.24 when an option (still defaulted to off!) was added to decline the automatic upgrade of date/time columns. If you couldn't upgrade to 5.6.24, you had two choices with any table with pre-5.6.4 types: make no DDL changes of any kind to it or accept downtime while the full table rewrite was performed. To be as fair as possible, this is documented in the MySQL docs, but it is mind-blowing to me that any database team would release this behavior into production. In other words, in what world is the upgrade of date/time types to add a bit more fractional precision so important that all online DDL operations on that table will be silently, automatically, and unavoidably converted to offline operations in order to perform the upgrade? To me, this is indicative of the same mindset that released MySQL for so many years with the unsafe and silent downgrading of data as the default mode of operation.

* Dropping a table takes a global lock that prevents the execution of ANY QUERY until the underlying files for the table are removed from the filesystem. Under many circumstances, this would go unnoticed, but I experienced a 7-minute production outage when I dropped a 700GB table that was no longer used. Apparently, this delay is due to the time it takes the underlying Linux filesystems to delete the large table file. This was an RDS instance, so I had no visibility into the filesystem used and it was probably exacerbated by the EBS backing for the RDS instance, but still, what database takes out a GLOBAL LOCK TO DROP A TABLE? After the incident, I googled for and found this description of the problem ([https://www.percona.com/blog/2009/06/16/slow-drop-table/](https://www.percona.com/blog/2009/06/16/slow-drop-table/)) which isn't well-documented. It's almost as if you have to anticipate every possible way in which MySQL could screw you and then google for it if you want to avoid production downtime.

</comment><comment><USER24> Anyone know when this will make it to AUR on Arch? I've been waiting for CTEs for a long time!

</comment><comment><USER25> I have one in my backpack I am parking now.

</comment><comment><USER26> Are the feature comparison tables in this article supposed to reflect the state of SQL databases as of several years ago?

I've been using Postgres very consistently for the past half decade (as a dev, not a DBA) and it's had a `GROUP BY` and `WHERE` clause for at least as long.

I'm not totally sure what " _base table_ `UNIQUE`" is supposed to mean in this context, but if it's just the bare `UNIQUE` constraint, Postgres has also had that for a long, long while.

<comment><USER27> Yeah, later on in the article it has an "X" for PostgreSQL for "JSON_OBJECTAGG" and "JSON_ARRAYAGG" and I'm 100% sure postgres has the former (I used it like 10 minutes ago), and like 99% sure it has the latter.

<comment><USER3> [https://www.postgresql.org/search/?q=json_objectagg](https://www.postgresql.org/search/?q=json_objectagg)

But there is a patch pending for this. Not sure if this gonna be included in 11.

<comment><USER27> It's json_object_agg in PostgreSQL

[https://www.postgresql.org/docs/current/static/functions- agg...](https://www.postgresql.org/docs/current/static/functions- aggregate.html)

<comment><USER3> Those gory details I've mentioned in the article.

This is how standard SQL JSON_OBJECTAGG works: [https://modern- sql.com/blog/2017-06/whats-new-in-sql-2016#js...](https://modern- sql.com/blog/2017-06/whats-new-in-sql-2016#json_objectagg)

It can actually do much more:

<JSON object aggregate constructor> ::= JSON_OBJECTAGG <left paren> <JSON name and value> [ <JSON constructor null clause> ] [ <JSON key uniqueness constraint> ] [ <JSON output clause> ] <right paren>

This free technical report of ISO has some examples: [http://standards.iso.org/ittf/PubliclyAvailableStandards/c06...](http://standards.iso.org/ittf/PubliclyAvailableStandards/c067367_ISO_IEC_TR_19075-6_2017.zip)

<comment><USER27> I haven't done any exact test, but it looks identical to how the json_object_agg function in postgres works, but with a different syntax.

Is it really that different to not even warrant a note in the table?

Clearly you know better than I do, but the big red X surprised me as the fantastic JSON support is one of the reasons why I tend to reach for Postgres so often over other DB engines.

<comment><USER3> A big red X doesn't mean the database doesn't have other, similar but proprietary functionality. It just means: it doesn't support this function in a standard confirming way yet.

There is nothing against using non-standard functionality in that case. In this particular case. PostgreSQL got JSON support (~2012?) long before it was added to the standard (2016). Obviously, their JSON functions differ from the standard (as they didn't lobby their functions into the standard).

However, slowly but surely database will add the standard functionality, which makes life easier.

Having that said, the standard JSON_OBJECTAGG is more powerful as it gives you control over null handling (NULL ON NULL -vs- ABSENT ON NULL) and allows checking for duplicate keys ([(WITH|WITHOUT) UNIQUE KEYS]).

<comment><USER22> Maybe add some little footnotes to the X? It took me a few minutes of thinking "Of course postgres has json object aggregation here it's working perfectly and it's documented" before I realised that sure it existed but it's not the standard version, at all, which is probably why it's marked as a no, because it's not Troel's page and Markus really really cares about the standard.</comment></comment></comment></comment></comment></comment></comment></comment>


Amazon's cloud business is competing with its own customers - <AUTHOR> https://www.cnbc.com/2018/11/30/aws-is-competing-with-its-customers.html <comment><USER4> The trend I've noticed is that AWS makes a lot of very half-baked services that appear to compete with third-parties, but provide about 10% of the features and usability. Maybe they're hoping that's enough for the bulk of users (and maybe it is), but from what I've seen, if you want to do anything serious, you still have to use a third-party.

Take CloudWatch for example. It's technically "competing" with Splunk, Datadog, and other logging/metrics services. But the usability of CloudWatch is beyond poor - it's slow, the UI is terrible, and has tons of limitations. It provides just barely enough usability that hobbyist users can use it. But most serious businesses will revert to Splunk or Sumo Logic or Datadog.

Another issue I have with AWS is that they simply provide too many ways to do the same thing. I want to run a container. Should I use ElasticBeanstalk? ECS? Fargate? EC2 with Docker? I've been using AWS for years, so I get the subtle differences, but some users just want to click a button and be done with it, which is where third-parties can excel.

<comment><USER5> This is not unique, it is the "integrated" vs "best of breed" offering. It's like the oldest business school IT case. Amazon isn't trying to sell the best anything, they're selling one spot where you can get everything you need even if some of it is crappy.

<comment><USER6> "Overwhelming mediocrity"

<comment><USER7> Software industry's evolution in a nutshell. New tools are this to old tools. SaaS are this to desktop software.

</comment></comment></comment></comment><comment><USER8> That's why I think MongoDB's new license which prevents AWS etc from hosting as a service without open sourcing makes sense ([https://news.ycombinator.com/item?id=18229452](https://news.ycombinator.com/item?id=18229452)). Amazon barely contributes anything back to the community and is known for destroying OSS businesses.

<comment><USER9> I wonder with Amazon if we're seeing a failure mode of Apache and BSD licenses, and if the future OSS might trend back towards more aggressively being copy left... or some place in between.

<comment><USER10> I'm actually not sure how this is a failure mode of those licenses.

I thought the point of permissive licenses over copyleft licenses is that you're explicitly granting companies the right to make money off your code without giving anything back.

<comment><USER11> It's the difference between believing that what matters is user rights and sharing code, and believing that what matters is the world containing more cool stuff.

Tim O'Reilly has a story (which I'll probably butcher) where after publishing the BSD manuals as books he was accused by someone at a conference of stealing the author's work. One of the authors stood up and said something like "Actually what Tim is doing is exactly what we want - spreading the material as widely as possible and making it as useful as possible. We knew exactly what we were doing choosing that license".

It's the same spirit in which MacOS being based on BSD is fine, even if not a single line of code is shared back. The question isn't "What did Apple share back to BSD programmers recently?", it's "Is the world a better place with MacOS in it as something people can choose to use?".

The same applies to Amazon. It simply wouldn't have been possible to build something like Amazon purely on proprietary code, running only proprietary OSes and services. Even if they had, it would have been a lot less interesting. I think it's pretty clear that the world is a better place with services like AWS available. As long as they follow the licenses of the software they use, there isn't a problem, but if some licensors disagree they can always change their license.

</comment></comment></comment></comment><comment><USER12> I am personally much happier to see AWS implement managed versions of OSS like Kubernetes and Kafka, than to only see them offer their proprietary versions (Kinesis & ECS).

The fact that all of these services are based on OSS software is what will help keep Amazon in check as they continue to grow and dominate, because at the end of the day if AWS starts to turn into Oracle and get abusive with their pricing/licensing then the customer has a lot more leverage to move to another cloud or self host.

I am not holding my breath for some kind of government driven intervention, nor have I heard an idea of how that would work that makes sense. I think OSS is what is saving us here, and we should celebrate when a project becomes popular enough for AWS to offer it.

Is it fair to the developers that they don't get compensated, nope, but isn't part of the point of OSS to make sure that great solutions get reused and widely deployed?

I think companies building great software can continue to make money servicing it even with AWS in the picture, but I think if you want to make unicorn returns you are going to have to move further up the stack.

Honestly, at some point I hope we are no longer even taking about low level components like Kubernetes and Kafka and that we look back and laugh at the fact that we used to have to license and get support for these things individually. I do also hope that we manage to find a sustainable (if not equitable) model to support the development of these tools.

All the clouds are built on top of the Linux kernel and nobody really expects the kernel devs to get a cut of all those profits, but they seem to have a sustainable model for development, I hope we can continue to figure out good ways to do this for more and more OSS tools.

<comment><USER13> The story I heard about Kinesis (and it could be a misunderstanding) is that Amazon originally wanted to offer managed Kafka but they couldn't get it working with their infrastructure and started from scratch.

MSK makes sense now as Kafka has matured significantly, while Amazon has had experience in porting OSS onto their infrastructure stack (Aurora, managed ActiveMQ, EKS).

<comment><USER14> Kinesis is based on DynamoDB, so not from scratch...

<comment><USER15> Is there any public information available about this?

</comment></comment></comment></comment><comment><USER16> There is nothing that prevents AWS from using it's own systems as business intelligence for creating their own competitive products.

Think about it like "store brand" software. If people seem to be installing docker on 40% of AWS instances, then evolving Amazon ECS to include an Amazon branded Docker clone, effectively replacing Docker, is a no-brainer.

Seems like this is the way they are going. No need to acquire companies if you can just build infrastructure or services natively that work more easily.

Terrifying times out there for starting a software company.

<comment><USER17> It should fix itself eventually, either companies will choose to use Google or Microsoft rather than AWS or the government will smack AWS with monopoly laws. The main concern there is that Bezos has become so entrenched in DC with the washington post, DOD cloud contract, and new headquarters that it's hard to tell if we can rely on the government to do anything

<comment><USER16> Honestly I doubt it.

1\. If Amazon is providing a service that is as good or easier to implement than others, people don't care enough about competition or open markets to decide based on that.

2\. From the govt perspective, I'm on the front lines of the DoD cloud discussions and I can tell you that even though we don't want a single provider, and hopefully I can push hard to ensure we have multiple providers, Amazon has the mindshare in the govt and growing.

</comment></comment></comment><comment><USER18> Disclosure: I work on Google Cloud.

For what it's worth, I often recommend customers look to our partners (e.g., Elastic with their managed offering). The people who build the product will know it best. Moreover, they would be way better suited to running say a multi-cloud variant, or staying with you as you move between cloud providers.

There are still barriers though for third-parties on a cloud platform, some of which we as providers can slowly chip away at:

\- Integration/Feel: You want to be in the console and feel like a natural part of the platform. This is double-edged though, because holding all partners to such a high bar (say a Google internal API review... ha!) is unrealistic.

\- Support: Who do customers call? Their overall cloud provider or their specific service provider? If the customer is having trouble with their Elastic cluster, is it because we're having a GCE outage or did the customer write a query-of-death? For the latter, our front-line support won't be much help. [Edit: And you want to be able to still have a very fast "Hey! My stuff is down!" path. I think technological things could make automatic handoff here much better, but I believe it'll be "not great" for a while.]

\- Economics: most service providers are going to be smaller than our largest customers. Those customers have usually signed up for multi-year commitments with high volumes resulting in a discount. If your service pricing model is "on-demand GCE pricing plus our service fee", you suddenly look over 2x as expensive to their TCO folks (just from missing Committed Use, or RIs with AWS). Unbundling the resources from the management fee would make this easier, but I've not seen many service providers attempt it.

We owe you much better docs (start here [1]), but we're trying to get there on at least some of these. All of the "human" factors though are still going to be a challenge, no matter how much software gets written.

[1] [https://cloud.google.com/service- infrastructure/docs/tenancy...](https://cloud.google.com/service- infrastructure/docs/tenancy-units-tutorial)

</comment><comment><USER9> Amazon will probably go broad, not deep. And that can be (and is) leveraged by the right companies.

The trick will be differentiating within that space. Take hosted search - <USER2> Elasticsearch for example doesn't necessarily prevent companies like Bonsai, Elastic, or Searchstax from standing out for the right use cases. Most importantly these companies can provide a lot of high end support and combine with other industry focused products and services (like Elastic XPack) that Amazon doesn't provide for the niches these companies work within.

We probably need to assume Amazon will host X thing and work to find the niches deeper in X that need to be served.

<comment><USER19> The challenge is to convince customers to pay premium for services/products from i.e. Bonsai/Elastic/etc than to use a general purpose, darn cheap, Cloud- based offering.

Amazon scales. Niche does not.

</comment></comment><comment><USER20> I would imagine this should be a primary concern for a lot of companies operating in Google, Microsofts, and Amazon's cloud.

You are dealing with companies that have the power and money to become your main competitor overnight, and you don't know what kind of business intelligence they are gaining from you being on their hardware, and if there is espionage going on with your software and service. How honest are they being?

Judging by the size, scope, behaviors, and attitudes of the companies, I think it's turning into a risk to host on their platforms if you work in a space they may be interested into getting into.

<comment><USER21> You don't need any espionage to implement and release the new Amazon Kafka product. Kafka is an open source package after all.

Profiting from any marginal gain from such espionage would be deterred by the huge reputational risk for AWS, were they caught.

<comment><USER22> You do need espionage to know how many of your customers are using Kafka on their EC2 instances to make that product decision though.

</comment></comment></comment><comment><USER23> Yeah, it's getting tricky as an AWS user. Some of the stuff we're building will now be directly competing with Amazon as well. And they keep moving more and more in the direction of competing with stuff people are building on their platform.

That said, the name of the game is "Co-opetition"[1]. Everybody is simultaneously competing and cooperating.

Going beyond that though, it does lead us to wonder at what point it would make sense for us to move off of AWS and build everything on bare-metal, using our own servers in a co-lo center, etc. Of course that takes a lot more up- front $$$, and we wouldn't have the same economies of scale as Amazon, so...

Yeah, it's complicated.

[1]: [https://en.wikipedia.org/wiki/Coopetition](https://en.wikipedia.org/wiki/Coopetition)

<comment><USER24> are there competitors to AWS that want to stay in just of cloud hosting?

<comment><USER25> Yeah, traditional hosting companies such OVH...there was rackspace as well but it's gone now.

<comment><USER26> ovh and hertzner are also vastly undercutting amazon in the EU.

Amazon is crazy expensive. especially if you consider the far lower labour costs in most of the EU.

</comment></comment></comment></comment><comment><USER27> Its been this way since the beginning. In the first wave of cloud services there were companies that built UIs to manage aws. Managed databases. PaaS offerings, queues, load balancers, etc, etc.

If it didnt make so much sense to keep building these tools yourself you would almost be able to suggest these other companies built expensive MVPs for AWS.

</comment><comment><USER28> I dont fully agree, as others noted by here, there are plenty of products that compete with others and co-exist, for example terraform.

There are some things though that I bet their customers are asking badly, take for example the Transit Gateway, one might thing that it is trying to put out the vendors that users use for their Transit VPC, for example Cisco, Fortinet, Palo alto to name a few, however, the Transit VPC is ugly, tough to maintain and cant scale, yes, the vendors make big money from licences for Transit VPC, but the need of a native service was there, the vendors will still sell licences, just not for Transit VPC, they will be able to focus on other parts that AWS dont prodive a lot.

</comment><comment><USER29> Aka the "Apple app store" and "Trader Joes" model of competition: Build a platform for other people to innovate on top of, then steal their best ideas after you've validated that they work well on your own platform.

<comment><USER30> trying to see how this is the Trader Joe's model.

i thought what Trader Joe's did was find food makers that sell products at other retailers (e.g. Naked Juice, sold for example at at Whole Foods) and then negotiate a deal _with that same food maker_ to supply _that same product_ to TJ's under the store label and at a lower price.

if I'm not mistaken, TJ's also develops its own product concepts (e.g. Thai Chili Cashew Nuts or their Winter blend coffee) in conjunction with some outside food makers and sells those in its TJ's stores too.

but, I'd be interested in finding some examples of where TJ's basically ripped off one of their own supplier's ideas and had it made by someone else for their store label. interesting stuff!

<comment><USER29> Hmm I was under the impression that they were ripping off those companies, but you could certainly be right. I just saw the pattern of (for example) Snap Pea crisps under a non-TJ brand get popular at TJ's, then TJ's labeled version comes out and the other one disappears.

<comment><USER30> You could be right too. I'm not sure.

Trader Joe's is very secretive about who supplies its products. When I google for something like "Who makes Trader Joe's $product" I find various articles that _seem like_ they've actually uncovered suppliers in certain cases, and I think those suppliers are pretty willing to sell a version of their existing products at TJs for lower prices. But I certainly don't know the whole story.

One other thing I notice about TJs is that it does not carry a "full" product line like a normal grocery store. TJ's often seems to cherry pick or negotiate relatively good deals from outside food suppliers where they can. And if they can't do that, in many cases, they just don't offer that product at all. Still, they offer a wide enough product line to keep the customers flowing in.

[http://www.time.com/money/4894722/who-makes-trader-joes- prod...](http://www.time.com/money/4894722/who-makes-trader-joes-products/)

[https://www.eater.com/2017/8/9/16099028/trader-joes- products](https://www.eater.com/2017/8/9/16099028/trader-joes-products)

</comment></comment></comment></comment><comment><USER31> This is absolutely correct, they have done for a long time. They look at what their customers build, see what grows a lot, and then build (often a shitty version) themselves. They don't even pretend to do otherwise, and everybody in the space knows it.

I was chatting to the CEO of a cloud product who went to reinvent and was relieved to find out that Amazon had not launched a competing product. But we both know it's a "when, not if" situation.

Part of the reason my new startup is on GCP. I don't want to AWS to have any of our internal growth numbers.

</comment><comment><USER32> On the hand, AWS's largest customer is Netflix. Amazon's Prime Video is a direct competitor.

Reposting an earlier link.

[http://scripting.com/stories/2007/04/04/coexistingWithPlatfo...](http://scripting.com/stories/2007/04/04/coexistingWithPlatformVend.html)

 _Sometimes developers choose a niche that's either directly in the path of the vendor, or even worse, on the roadmap of the vendor. In those cases, they don't really deserve our sympathy."_

</comment><comment><USER33> I remember reading, probably about 10 years ago, something by I think Tim Bray where he argued (I will have to paraphrase as I cannot find it anymore) that developing applications that would be beneficial to all users of a platform on that platform was a waste of time because it was to the advantage of the platform itself to replicate the functionality once it saw how well it was doing in the market.

Therefore if you do not want to compete with the same platform on which you've based your solution you should make solutions that are niche specific - <USER3> believe the example was a Apple and a dentistry service, because no matter how much revenue your dentistry service made Apple would not go and copy it because it had nothing to do with their business.

</comment><comment><USER14> AWS is dominant and doing pretty well in the IaaS space, but lack of enough good quality middlewares and enterprise software comparing with its competitors, Azure and Oracle.

Maybe this claim is not fair for DynamoDB, Kinesis, Lambda and etc. But these are just managed service. Customers could not get a binary to play around on their own server. So I could not call them software or middleware.

So their strategy is simply making the most popular software running on IaaS as a managed service, like managed Hadoop, Spark, K8S and now Kafka...

From a customer's perspective, it is pretty good since managed service just works without hassles to manage servers.

</comment><comment><USER34> While a lot of these offerings are subpar compared to their competitors. I think one big advantage AWS has it that I don't have to talk to a sales team to get something working. Like, go to Confluent's site. The business plan is request a quote; I am not a fan of this model at all, why not keep things self serve and pricing open. This makes your bootstrap time significantly longer.

Also, with AWS's offerings you get better management with security groups etc.

</comment><comment><USER35> This surprises anyone? Amazon retail has been competing with it's own customers (mostly in the form of Amazon Basics) for years now...

</comment><comment><USER36> Amazon is also using their consumer goods sales data of their merchants to determine what products they should themselves profitably create and sell. For instance Amazon began directly selling Amazon branded DisplayPort to HDMI adapters after they determined they could profitably undercut the merchants on their platform. Unfortunately I can't dig up the link.

</comment><comment><USER37> Coopretition is nothing new - look at every utility, telecom, cableco, and others - its a normal way of existing when markets and technologies are heavily interlinked.

</comment><comment><USER38> This just in: Ditch digging business, which got into the shovel business, is also digging graves potentially competing with its shovel customers...

</comment><comment><USER39> Have you tried MyAirBridge? I find it to be the best provider for online data storage and sata sharing. Super reliable and safe

</comment><comment><USER40> It's not just AWS that competes with its customers, it's Amazon in general.

</comment><comment><USER41> A few thoughts. 1) This is nothing surprising. People who build businesses based largely around filling in gaps in AWS's managed services hopefully expect and plan on AWS to attempt to fill that gap if that niche truly is profitable.

2) AWS is going to build what its customers ask for. I work for a relatively big consumer of AWS services, and we have a great relationship with our customer support team, who connect us to the engineering teams building the tools we use, and without fail down to the team, they are always very interested in how they can better serve our needs and build products we want to use. And if they hear "Kinesis is fine for some things, but we really want to also do Kafka" from enough people, they will build a Kafka managed service, just like they built ElasticSearch service, a Redis service, an RDS. And why shouldn't they?

3) For customers, AWS's cost model is far better than most managed cloud services. I don't know anything about Confluent's managed Kafka offering, but having everything metered in tiny increments is a huge reduction in the barrier to entry. If we want to build on a managed Kafka platform, but we want to start slow and grow as we need, it's so much better to be able to just spin up a tiny version to try things out, tear it down, take a break for other priorities, then come back with more confidence and more experience and iterate into building something real. Whereas too many cloud services require a huge lock-in agreement up front before you can get any support, or often any service.

And it's not like the lock-in is necessary. These companies could be integrating their services to allow potential customers to leverage their AWS infrastructure. There's no reason these third parties can't use IAM credentials for authentication, and publish their services to customers over PrivateLink, a tool AWS built for explicitly this sort of purpose--to enable cloud providers to provide tight integrations with their shared customer base.

4) In terms of product quality, I don't think Confluent et al have anything to be worried about. AWS "competes" with Dropbox, technically, but is WorkDocs really serious competition? AWS's managed services have their place, but you can't use RDS to replace the flexibility and scale you can achieve by running PostgreSQL yourself. Their managed ElasticSearch and Redis products are okay, but ultimately very rigid, and nonsensical to use beyond mid-range scale. Their managed k8s service is something of a joke at this point (though I'm sure it will get better). If Confluent is truly offering a strong managed Kafka product, they will be just fine.

5) In fact, I suspect Confluent will get a lot more customers as a result of this announcement, as management at hundreds of mid-sized companies will now be convinced thanks to the AWS marketing that this Kafka thing their engineers have been begging for for years must be really key now, but they'll figure out how the AWS version doesn't quite live up and will end up being really expensive to scale to their company's needs. Then they'll look a little deeper and find Confluent. So I hope their marketing and sales teams are ready for the influx. It's coming.

<comment><USER14> Confluent would be fine for sure... Usually, the open source projects would have close-sourced "enterprise feature" to make money.

</comment></comment><comment><USER42> This is an ongoing trend. Same thing is happening right now with Twilio with their Flex offering competing with longtime customers such as Talkdesk.

</comment><comment><USER43> AWS's managed FTP service too

<comment><USER44> In all honestly though, "SFTP-as-as-service" doesn't sound like a very sexy or sustainable business model.

<comment><USER43> Sure, not "sexy" enough for the HN crowd but it sucks to be these guys [https://hostedftp.com](https://hostedftp.com) since [https://aws.amazon.com/sftp/](https://aws.amazon.com/sftp/) was announced

</comment></comment></comment><comment><USER45> This is why public cloud companies should be more regulated like telephony companies are, ie like utilities. They give you broadband but not necessarily try to sell you 15 kinds of database engines, or trying to build the same things other businesses try to build using their lines.

<comment><USER32> It kills me that a website full of well paid people in IT who all benefit from the free market always cry for regulation.

I guess you don't realize how many people in middle America would love to "tax the rich" and if you're making six figures "they" consider you rich.

<comment><USER46> As an aside, if you make six figures, you are rich.

<comment><USER47> Agreed. Plus many six figure salaries are attached to equal or greater amounts of equity, 401K match and typically low cost health insurance. So while it's not jet-set wealth it is more than enough to have a very comfortable life with few things out of reach for you.</comment></comment></comment></comment>


Why French Kids Don't Have ADHD - <AUTHOR> https://www.psychologytoday.com/us/blog/suffer-the-children/201203/why-french-kids-dont-have-adhd <comment><USER2> There are a few things that are incorrect in this article:

* Indeed French kids could be diagnosed as having ADHD, for example in CRA [0] (in French). I know it first hand.

* Spanking is forbidden in France by law and most parents do not spank.

* French kid education does not have the high qualities that are alleged in this article. In fact the current government is making great steps toward the right direction [1] (in French).

* It is a kind of discourse where "the lawn is greener in this foreign mysterious country".

[0] [https://fr.wikipedia.org/wiki/Centre_Ressources_Autisme](https://fr.wikipedia.org/wiki/Centre_Ressources_Autisme)

[1] [http://www.cafepedagogique.net/lexpresso/Pages/2018/01/08012...](http://www.cafepedagogique.net/lexpresso/Pages/2018/01/08012018Article636509921667912467.aspx)

<comment><USER3> >* Spanking is forbidden in France by law and most parents do not spank.

To be fair, while I believe most French parents do not spank their children, nor did it in recent years, the Law is rather new:

[https://www.huffingtonpost.com/entry/france-bans-spanking- ki...](https://www.huffingtonpost.com/entry/france-bans-spanking-kids-with- new-law_us_5873ccc1e4b02b5f858a39ff)

and is not a crime, as said in the above article is more a symbolic act to make clear that it shouldn't be done than anything else.

But, let's set aside France and french kids, more widely mosr European countries have a percentage of diagnosed ADHD that is a fraction of those in the US, and even if the numbers of diagnoses tend to increase, it is IMHO largely because of the US influence (not only in psichology and psichiatry, also in "way of life").

Some reference:

[http://menzonius.nl/index.php/article-review-the- impending-g...](http://menzonius.nl/index.php/article-review-the-impending- globalization-of-adhd-notes-on-the-expansion-and-growth-of-a-medicalized- disorder/)

<comment><USER2> > while I believe most French parents do not spank their children, nor did it > in recent years, the Law is rather new:

I think the French law on this topic is more or less aligned on international standards which are quite recent at WHO or in EU.

But one of the first regulations in France about spanking in school was passed in 1795 (even if I saw some violent teachers in the 60")

<comment><USER3> >I think the French law on this topic is more or less aligned on international standards which are quite recent at WHO or in EU.

Maybe it is _now_ , actually France received a complaint by the Council of Europe in 2015 for NOT having such a Law (together with a few other EU countries) or not having it "clear enough":

[https://www.coe.int/bs/web/portal/full- news/-/asset_publishe...](https://www.coe.int/bs/web/portal/full- news/-/asset_publisher/Dgh51iCGvfbg/content/corporal-punishment-of-children)

[http://www.endcorporalpunishment.org/assets/pdfs/press- relea...](http://www.endcorporalpunishment.org/assets/pdfs/press- releases/France-collective-complaint-decision-statement-EN.pdf)

[https://web.archive.org/web/20150305031324/http://www.coe.in...](https://web.archive.org/web/20150305031324/http://www.coe.int/t/dghl/monitoring/socialcharter/Complaints/CC92Merits_en.pdf)

Of course one thing is the Law, and another thing is practice, as said I believe that in most EU countries, even those that do not have a "proper" Law the actual use of corporal punishments (within the family) is largely a thing of the past.

About corporal punishment in schools or anyway outside the family/paretns (which is another thing), I believe it is forbidden everywhere, and as you mentioned last occurrences were probably in the '60's, see the map here:

[https://en.wikipedia.org/wiki/Corporal_punishment#Legal_stat...](https://en.wikipedia.org/wiki/Corporal_punishment#Legal_status)

[https://en.wikipedia.org/wiki/Corporal_punishment#/media/Fil...](https://en.wikipedia.org/wiki/Corporal_punishment#/media/File:Corporal_punishment_in_Europe.svg)

</comment></comment></comment></comment><comment><USER4> Such bullshit. My father believed this nonsense that "ADHD is a conspiracy (theory) to control children, therefore it doesn't exist and medication is unnecessary." No, they don't understand how the other kids will label the ADHD kid as weird, exclude them and they won't be able to complete tasks they start.. because it's emotional and/or executive function disregulation. Watch the great videos on YT to learn more about the horrible possible side-effects of untreated ADHD: homelessness, joblessness, friendlessness, lovelessness, depression, anxiety, suicide, etc. And ADHD is extremely treatable as well.

\- Lifelong undiagnosed ADHD

</comment><comment><USER5> The article is by a family therapist selling their book/service, not a medical doctor.

Also it's ironic a Bioethics PhD wants to model ADHD after the French system. Their history with psychiatry leaves reason for so few French psychiatrists contributing to the ICD and DSM.

France is better known for their neurological contributions anyways.

</comment><comment><USER6> [https://www.psychologytoday.com/us/blog/pay- attention/201511...](https://www.psychologytoday.com/us/blog/pay- attention/201511/french-kids-do-have-adhd)

<comment><USER2> Indeed and it also well describe my feeling as a parent about France's (sometimes) invisible ADHD.

</comment></comment><comment><USER7> I think the French also take their food very seriously, they believe that their food habits of children are established in the first few years so are much more likely to have mother's that stave off alcholol, cigarettes, paracetamol etc during pregnancy and breastfeeding.

</comment><comment><USER8> Well, it looks to me to be very similar to the obesity epidemic in the US. Both countries are developed. France is not a "foreign mysterious country". Its doctors are actually as good as US ones, it offers a better life expectancy and a better health coverage. And it's true that nearly nobody takes Ritalin in France.

But that does not mean the French situation is perfect. There are several recent articles pointing to ADHD being under-diagnosed in France - with many doctors reluctant to accept it, and/or being very cautious with regards to its dangerous secondary effects.

So it's probably somewhere in-between.

</comment><comment><USER9> I lost interest after this unsubstantiated statement:

These divergent philosophies could account for why French children are generally better-behaved than their American counterparts.

</comment><comment><USER10> Title should indicate publishing year please (2012)</comment>


Detecting Fraud Using Benford's Law - <AUTHOR> http://blog.cluster-text.com/2015/10/20/detecting-fraud-using-benfords-law-mathematical-details/ <comment><USER2> My favorite Radiolab episode talks about Benford's law [http://www.radiolab.org/story/91697-numbers/](http://www.radiolab.org/story/91697-numbers/)</comment>