Interview with Eric Brewer - <AUTHOR> https://medium.com/s-c-a-l-e/google-systems-guru-explains-why-containers-are-the-future-of-computing-87922af2cf95 <comment><USER2> A funny and somewhat off topic story -- back in 2003, before the Google IPO, Google was doing a recruiting event at Berkeley. They brought a few of their folks with them: their founder Larry, one of their female engineers, Marissa, and some others. They did a little talk, and during the Q&A, professor Brewer told Larry that there was an opening in the PhD program and he was welcome to it. Larry politely declined.

Afterwards I asked Larry, "so, do you think you'll ever finish your PhD, either here or at Stanford?". He said, "If this Google thing doesn't work out I might, but I have a feeling it will work out ok."

It amuses me that Professor Brewer is now working for Larry. :)

<comment><USER3> In 2003, Google had over a billion dollars in revenue. I suspect that answer was tongue in cheek.

<comment><USER4> I suspect the question was as well.

</comment></comment></comment><comment><USER5> Great Interview!

I worked as a contractor at Google in 2013 and loved their infrastructure. It was amazing to fire off a Borg job that used hundreds to thousands of servers, and the web based tools for tracking the job, fantastic logging to drill into problems, etc.

And, Borg was two generations ago!

Even though I am very happy doing what I am now, sometimes I literally wake up in the morning thinking about Google's infrastructure. I now use lesser but public services like AppEngine, Heroku, nitrous.io (like Google's web based IDE Cider, a bit) but it is not the same.

BTW, not to be negative, but while Google is a great home for someone like Eric Brewer, it is a shame that many hundreds of future students at UC Berkeley will not have him as a professeur.

<comment><USER6> How hard was it to learn these tools as a contractor? I hear that Google is more aggressively making their tools open-source to help the portability of their engineers' skills. Specifically, friends have told me that onboarding at Google is difficult because every tool is a proprietary one built on top of more proprietary systems. In addition, people who leave Google have trouble interviewing because they rely heavily on tools that are unavailable outside of Google. By beginning to contribute more to open-source, I think that it has the potential to make interviewing at and joining Google a smoother experience.

<comment><USER5> Good question. I found onboarding at Google really fun and interesting. I took several great classes, the best being the end-to-end class that in 8 hours let you write code that used most of Google's infrastructure - that class was so awesome that I would have payed Google for that day at work :-)

Also, they had code labs that are self paced modules for learning specific tech. I didn't spend much time at work doing code labs, but I could access them at home with my corporate laptop and I went through about a dozen of them at home. One other nice thing is even mentioning that you couldn't figure out something from the documentation or code labs would cause someone to jump in to help you.

Also, I don't think that people leaving Google have problems getting other jobs :-) The retention rate at Google is surprising low, given the pleasant atmosphere there. People leave to go elsewhere, start their own companies, etc. I was 63 when I worked there, and although it was probably not the most awesome place I worked, it was really great.

Apply for a job if you are interested, or go the easier route and get a contractor position.

</comment></comment></comment><comment><USER7> One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong- consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement. Now, I see man who invented CAP theorem discuss it while referencing all kinds of NoSQL options to show us the tradeoffs. Yet, there's Google services in production and tech such as FoundationDB doing strong consistency with distributed, high throughput and availability.

[http://www.theregister.co.uk/2012/11/22/foundationdb_fear_of...](http://www.theregister.co.uk/2012/11/22/foundationdb_fear_of_cap_theorem/)

So, why aren't such techs mentioned in these discussions? I liked his explanation of the partitioning problem. Yet, he and NoSQL advocates seem unaware that numerous companies surmounted much of the problem with good design. We might turn CAP theorem into barely an issue if we can get the industry to put the amount of innovation into non-traditional, strong- consistency architectures as they did into weak-consistency architectures. There is hope: Google went from a famous, NoSQL player to inventing an amazing, strong-consistency RDBMS (F1). Let's hope more follow.

[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41344.pdf)

<comment><USER8> > The NoSQL movement didn't notice that NonStop Architecture scaled linearly > to thousands of cores with strong-consistency, five 9's, and SQL support.

Also incredibly expensive. Take this case, Google took off because they were able to scale-out with off-the-shelf hardware, compared to the millions banks were pouring in for scale-up configurations which handled much less load. Scale-up can quickly hit hard limits, before it becomes exponentially expensive to continue on the path. This is true even today.

NoSQL movement, if you wanna call it that, took off because most apps (including Google, including Financial Services, even Health Care) don't need some types of consistency offered by relational databases. Many large apps are significantly de-normalized and have many foreign-key less tables, often filled up by scheduled jobs. That's fine for most apps; NoSQL architectures recognize that and users consider that in design.

For the majority of use-cases out there, NoSQL databases offer enough consistency. For the remaining use-cases, there are tools available in NoSQL databases to make them work, though it requires a bit of work.

<comment><USER9> > For the majority of use-cases out there, NoSQL databases offer enough > consistency.

For applications with a nontrivial data model, ensuring that each logical operation only does a single document update (or that multiple nontransactional updates cause no conflict in maintaining consistency in the presence of other logical operations) is actually really challenging - and it adds a substantial design overhead to every new feature added. I think you're being extremely optimistic in your assertion that NoSQL systems are that widely safely applicable. My experience has been that NoSQL-based systems stay mostly-consistent because they happen to experience very little concurrent activity, not through understanding/design on the users' part.

This is not to make light of the situations where NoSQL systems shine, but the idea that higher levels of consistency are rarely useful does not match my experience at all.

> NoSQL movement, if you wanna call it that, took off because most apps > (including Google, including Financial Services, even Health Care) don't > need some types of consistency offered by relational databases.

I'd say that Mongo (for example) took off because:

\- They really nailed the setup experience for new users (which RDBMSs historically sucked at).

\- The data model is much easier for simple apps.

\- They had some fairly creative techniques for making their system look good - unacknowledged writes, and claims of support for replication which didn't really fully work.

\- Most programmers don't really understand the ins-and-outs of maintaining consistency.

</comment></comment></comment><comment><USER10> I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting.

But it's the less common choice.

I am guessing convenience is more important than the better solution that would ultimately be more just as convenient and more efficient if it gets enough eyeballs?

<comment><USER11> I'd like a real explanation for why containers _and_ unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply run efficient software configured properly on properly utilized hardware. As if regular virtualization alone doesn't give enough headaches in trying to figure out why some subsystem does not perform.

At this rate we'll end up shipping containers as 'apps' to the clients machines with a suitable emulator at some point.

All this luxury comes at (considerable) cost and not everybody seems to be doing the math before deployment which more often than not leads to terrible efficiency.

But we're technology fans, so 'oohh! shiny!'.

<comment><USER12> Do you have experience using containers?

Here's my answer for "why": DRY. Once you've deployed hundreds of servers using the same exact Ubuntu 12.04 LTS kernel base, why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line of code for the kernel and redeploy it every time I add a node?

> _At this rate we 'll end up shipping containers as 'apps' to the clients > machines with a suitable emulator at some point._

That's exactly the point. Care to elaborate on the downside of such a promise?

<comment><USER11> > why not just completely abstract the OS away and focus the attention on > scaling the OS services that matter?

Because it adds a layer that makes no sense unless you have very specific use cases. Though I see the point regarding people efficiency, that one makes good sense (see other comment in this sub-thread)

> Why is that when I decide that I need to scale out, I need to copy every > library of the OS and every line of code for the kernel and redeploy it > every time I add a node?

If you're doing it that way then you are simply doing it wrong. See: chef, configuration management and various deployment services (of which you could argue containers are one off-shoot, but they focus (imo) on the wrong level for all but the largest companies). Containers are like sandboxes with significant overhead for applications that focus on ease of deployment (but that's strange to me because I see that as a one-time cost for most of my own use cases, though I can see how that equation would change if you deploy lots of things configured by lots of different people to a single set of servers, especially if there are conflicting requirements between those deployments).

> Care to elaborate on the downside of such a promise?

That's my personal view of hell, if you don't see any downside there please ignore my vision and continue as if nothing was said.

From open, text based standards to shipping arbitrary binaries in a couple of decades. And I thought GKS was about as bad as it got ;)

<comment><USER12> > _See: chef, configuration management and various deployment services_

A chef script is basically the automation of _" I need to copy every library of the OS and every line of code for the kernel and redeploy it every time I add a node?"_ I'm sorry if you didn't pick up on my implied remark. Two problems are then introduced when automating those actions: (1) it doesn't negate the fact that I need to store and deploy a 700M sized OS layer every time I want to add a node (which takes minutes, not seconds with non- containerized configs) and (2) maintaining config scripts can (not always) be painful (version control, rollbacks, etc)

> _unless you have very specific use cases._

> _Containers are like sandboxes with significant overhead for applications_

Again, do you have experience using containers? You seem awfully dismissive ("you're doing it wrong!") in a way that suggests that you might not entirely understand how they actually work...

<comment><USER13> What I do on our app servers is create a new user for each app (via configuration management).

That gives me both environment separation (A needs Ruby 1.9, B needs Ruby 2.0), resource accounting on a per-app basis, and a repeatable foundation in case I need to re-deploy the server or spin up new instances.

<comment><USER14> That is only true if you use no parts (e.g. shared libraries etc.) of the host environment. That guarantee that you're not inadvertently depending on something on the host system that might change is what containers give you.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> "Heroku took off for doing things with Salesforce" \- This is not my impression at all.

</comment><comment><USER16> Containers, like virtual machines before them, aren't the future of computing. They're how we manage legacy apps.

The future of computing is not this horrible kludge.

<comment><USER17> Disclaimer: I work at Google on Kubernetes

The problem containers are trying to solve is not isolation of environments (though, that's a tremendously good outcome).

The problem is how do you develop microservices that are self-contained, easily composable and inherently portable. It doesn't matter if you're running on Java 1.1 on AIX 3.0 or Node on Ubuntu 26, if I have a ball of computing providing a service in Dubai, and want to move it to Ireland to take advantage of computing space that just opened up, containers make that trivial (and about a million other scenarios).

</comment></comment><comment><USER18> I saw a kubernete talk at a local meetup.

Google have 40 programmers dedicated to that project. It's still very beta btw all programmed in Go.

There's also mesos and I think you can use both in tandem since they're targeting a different thing.

Anyway if anybody is doing or thinking about containers check Kubernete and Mesos out. Also of course docker and rocket. Kubernete officially support docker and will be supporting rocket.

There are also article about how rump kernel are better than containers. Just fyi.

</comment><comment><USER19> I have been using Amazon Web Services and other cloud platforms for over a year now, and I never really felt that VMs were the bottleneck in any way. Can someone explain to me the advantage of containers here?

I know that containers are faster because they don't virtualize the hardware, however it comes at the cost of security.

<comment><USER17> Disclaimer: I work at Google on Kubernetes & containers.

I tried to address it here: [https://news.ycombinator.com/item?id=9570639](https://news.ycombinator.com/item?id=9570639)

But in summary, the basic thing that containerized software offers is inherent portability and simple composition. VMs aren't going anywhere; running a container on top of them just makes everything more powerful.

</comment></comment><comment><USER20> Can someone explain or provide an educated guess about what is the google's strategy with kubernetes here? Surely containers are hot now and it is nice to have a stake in the game but borg has been one of their key competitive advantages. What is the profit in making an open-source alternative?

<comment><USER21> I have been wondering the same thing. I suppose it's partly because it was only going to be a competitive advantage as long as as containers where not commoditized which seems to be the direction its now heading.

</comment></comment><comment><USER22> I love the idea of using containers. Due to linux popularity and google's backing, containers will be next.

But FREEBSD had jails since back in the day. What's the benefit of containers over bsd jails?

<comment><USER23> Linux has copy-on-write block devices that make it possible to efficiently layer container filesystems. FreeBSD has no such thing as far as I know; the best you can do involves hard links (correct me if I'm wrong).

<comment><USER24> [https://en.wikipedia.org/wiki/UnionFS](https://en.wikipedia.org/wiki/UnionFS)

Edit: addendum

Additionally, a hardlink solution (as you point out) really doesn't sound too outrageous to me. A trivial tool to write to manage immutable things like binaries. Or is there some trouble I'm failing to see (possible)?

<comment><USER23> According to the mount_unionfs(8) man page:

> THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN'T WORK) AND > USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM.

<comment><USER24> You're right. My personal reference for unionfs is NetBSD, which doesn't include such a warning [0]; apparently freebsd's implementation needs some love.

[0] [http://netbsd.gw.com/cgi-bin/man-cgi?mount_union++NetBSD- cur...](http://netbsd.gw.com/cgi-bin/man-cgi?mount_union++NetBSD-current)

</comment></comment></comment></comment></comment><comment><USER25> imo, if containers are the future ( very plausible ), then, things like Aws lambda are just as plausible if only just a bit further out.

I think this is the case due to granularity of workloads and what apears to be a continuum in the workload container from metal > vm > containers > lambdas (as first class workloads).

fun stuff

<comment><USER17> Disclaimer: I work at Google on Kubernetes

True, lambda is DEFINITELY an advance, but it's more like the salmon at a buffet that includes steak and chicken. Some applications will need just the ability to run code (lambda), some will need defined environments (containers) and some will need total isolation (VMs/bare metal). You'll see a mix of all of these in every mature environment - some things do not fit. For example, it's super unlikely that you'd be able to run a trading app on Lamdba (needs 10GB/sec of direct network access & memory); similarly it'd be totally unnecessary to run a thumbnail processor on bare metal (though you could, of course).

<comment><USER25> good points but I'm not sure anything about the fundamental lambda architecture prevents it from being applied to high bandwidth or low latency uses. especially if lambda functions are orchest rated to run in the metal ...think: kernel module implements lambda, then add orchestration glue up in user land

</comment></comment></comment><comment><USER26> With all these container talks people forget Solaris zones which were pretty advanced sort of containers in Solaris. However Sun was honest about not mentioning zones as the solution for everything. The most important problem with containerization is that due to dependence of multiple containers on the same kernel of the host or VM OS , any kind of upgrade specially the security patching is virtually impossible to do without taking full downtime.</comment>


Parsing JSON Is a Minefield (2018) - <AUTHOR> http://seriot.ch/parsing_json.php <comment><USER2> I suppose you could saw that parsing _any_ text-based protocol in general "Is a Minefield". They look so simple and "readable", which is why they're appealing initially, but parsing text always involves lots of corner-cases and I've always thought it a huge waste of resources to use text-based protocols for data that's not actually meant for human consumption the vast majority of the time.

Consider something as simple as parsing an integer in a text-based format; there may be whitespace to skip, an optional sign character, and then a loop to accumulate digits and convert them (itself a subtraction, multiply, and add), and there's still the questions of all the invalid cases and what they should do. In contrast, in a binary format, all that's required is to read the data, and the most complex thing which might be required is endianness conversion. Length-prefixed binary formats are almost trivial to parse, on par with reading a field from a struture.

<comment><USER3> Binary formats have their own serious problems.

> Length-prefixed binary formats are almost trivial to parse

They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that.

> the most complex thing which might be required is endianness conversion

That is a gross simplification. When you look at the details of binary representations, things get complex, and you end up with corner cases.

Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0. You can also create two NaN numbers that do not have the same binary representation. You have to choose single or double precision (maybe a benefit, not always). Etc.

Similarly in JSON integers or arrays of integers are nothing special. It is mostly a benefit not to have to specify UInt8Array.

JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology. So far a binary format mutation hasn't beaten JSON, which is telling since binary had the early advantage (well: binary definitely wins in parts of the ecology, just as JSON wins in other parts).

<comment><USER4> > with a binary format you can transmit NaN, Infinity, -infinity, and -0. You > can also create two NaN numbers that do not have the same binary > representation. You have to choose single or double precision (maybe a > benefit, not always). Etc.

Boy you are really stretching to make this sound complicated. It's not. You transmit 4 bytes or 8 bytes. Serialization is a memcpy().

You don't have to think about NaNs and Infinities because they Just Work -- unlike with textual formats where you need to have special representations for them and you have to worry about whether you are possibly losing data by dropping those NaN bits. If you _want_ to drop the NaN bits in a binary format, it's another one-liner to do so.

It's funny that you choose to pick on floating-point numbers here, because converting floating-points to decimal text and back is _insanely_ complicated. One of the best-known implementations of converting FP to text is dtoa(), based on the paper (yes, a whole paper) called "How to Print Floating-Point Numbers Accurately". Here's the code:

[http://www.netlib.org/fp/dtoa.c](http://www.netlib.org/fp/dtoa.c)

Go take a look. I'll wait.

dtoa() is not even the state of the art anymore. Just in the last few years there have been significant advances, e.g. Grisu2, Grisu3, and Dragon4...

Again, in binary formats, all that is replaced by a memcpy() of 4 or 8 bytes.

(A previous rant of mine on this subject: [https://news.ycombinator.com/item?id=17277560](https://news.ycombinator.com/item?id=17277560) )

> > Length-prefixed binary formats are almost trivial to parse

> They definitely are not, as displayed by the fact that binary lengths are > the root cause of a huge number of security flaws. JSON mostly avoids that.

Injection (forgetting to escape embedded text) is the root cause of a huge number of security flaws _for text formats_. Length-prefixed formats do not suffer from this.

What "huge number of security flaws" are you referring to that affect length- delimited values? Buffer overflows? Those aren't caused by values being length-delimited, they are caused by people deserializing variable-length values into fixed-length buffers without a check. That mistake can be made just as easily with a text format as with a binary format. In fact I've seen it much more often with text.

> JSON currently dominates large parts of that ecology.

JSON wins for one simple reason: it's easy for human developers to think about, because they can see what it looks like. This is very comforting. It's wasteful and full of pitfalls ("oops, my int64 silently lost a bunch of bits because JSON numbers are all floating point"), but comforting. Even I find it comforting.

Ironically, writing a full JSON parser from scratch is much more complicated that writing a full Protobuf parser. But developers are more comfortable with the parser being a black box than with the data format itself being a black box. ¯\\_(ツ)_/¯

(Disclosure: I am the author of Protobuf v2 and Cap'n Proto. In addition to binary native formats, both have text-based alternate formats for which I wrote parsers and serializers, and I've also written a few JSON parsers in my time...)

<comment><USER5> I think people undervalue clean-looking (alphabet-only, few special character) things, things that don't require people to use the symbol-parsing part of their brain. Basically easily human-parseable things. I suspect this phenomenon can be observed in the case of relative popularity of JSON, TOML, YAML and Python plus the relative unpopularity of Lisp, Haskell, Rust and XML. And if we look at protobuf in this context it is not easy to parse for humans, which causes people not to want to use it, developers are not

> more comfortable with the parser being a black box

they're more comfortable with the parser being a black box but the format being relatively easy to parse compared to the parser being easy to understand but the format basically unreadable for a human.

<comment><USER6> > I think people undervalue clean-looking (alphabet-only, few special > character) things, things that don't require people to use the symbol- > parsing part of their brain. Basically easily human-parseable things.

The symbol parsing part of the human brain is what parses letters and numbers, as well as other abstract symbols. The division of symbols into letters, numbers, and others is fairly arbitrary. Most people would say "&", but the modern name of that symbol is a smoothing over of the way it was recited when it was considered part of the alphabet and recited with it.

> I suspect this phenomenon can be observed in the case of relative popularity > of JSON, TOML, YAML and Python plus the relative unpopularity of Lisp, > Haskell, Rust, XML.

I suspect not: Lisp and Haskell have _less_ use of non-alphanumeric characters than most more-popular general purpose languages, and not significantly more than Python; also, if this was the phenomenon in play, popularity would be TOML > YAML > JSON but in reality it's closer to the reverse.

<comment><USER5> > The symbol parsing part of the human brain is what parses letters and > numbers, as well as other abstract symbols.

I really don't think that's true when you talk about about someone using the latin alphabet, words in that alphabet compared to some other alphabet (e.g. {}():!) and "words" (or meanings) in those. Just as a crude example parsing "c = a - b", where equals and minus are one symbol each and have been taught for a while, is different from parsing "c := a << b" where ":=" and "<<" basically act as a separate meaning someone has to learn to understand. Similar to the difference of latin alphabet and say simplified Chinese.

> also, if this was the phenomenon in play, popularity would be TOML > YAML > > JSON but in reality it's closer to the reverse.

There could be somewhat of an sigmoid response to the effect, decreased reaction if you go into either extreme compared to deviating from the average.

I'm not a linguist so it is my speculation, so don't take it too seriously :D

</comment></comment></comment></comment></comment></comment><comment><USER7> Because of this article (which I encountered a year ago) I would say Parsing JSON is _no longer_ a minefield.

I had to write my own JSON parser/formatter a year ago (to support Java 1.2 - don't ask) and this article and its supporting github repo ([https://github.com/nst/JSONTestSuite](https://github.com/nst/JSONTestSuite)) was an unexpected gift from the heavens.

<comment><USER8> Wait. How is this no longer a minefield just because there is a test suite that identifies some tricky cases?

Doesn't the test suite's matrix demonstrate that there are tons of cases that aren't handled consistently across these parsers?

<comment><USER7> Good point. I am presuming the test suite is comprehensive. Does it cover 100% of all JSON mines? Probably not. But it surfaced about 30 bugs in my own implementation - things I would have never dreamed of.

So it certainly helped me. And just based on how thorough and insane the test suite is, I think I'm in good hands. Not perfect hands - but definitely a million times better than anything I would have come up with on my own.

The test suite made my parser blow up many times, and for each blow up I got to make a conscious decision in my bugfix: how do I want to handle this?

(I decided to let the 10,000 depth nested {{{{{{{{{{{{{{{{{{"key","value"}}}}}}}}}}}}}}}}}}} guy blow up even though it is legal. Yes, I'm too lazy to implement my own stack.) :-)

</comment></comment></comment><comment><USER9> This might be throwing a lit match into a gasoline refinery, but why not opt for XML in some circumstances?

Between its strong schema and wsdl support for internet standards like soap web services, XML covers a lot of ground that Json encoding doesn't necessarily have without add-ons.

I say this knowing this is an unfashionable opinion and XML has its own weaknesses, but in the spirit of using web standards and LoC approved "archivable formats", IMO there is still a place for XML in many serialization strategies around the computing landscape.

Json is perfect for serializing between client and server operations or in progressive web apps running in JavaScript. It is quite serviceable in other places as well such as microservice REST APIs, but in other areas of the landscape like middleware, database record excerpts, desktop settings, data transfer files, Json is not much better or sometimes even slightly worse than XML.

<comment><USER10> XML cannot be parsed into nested maps/dictionaries/lists/arrays without guidance from a type or a restricted xml structure.

JSON can do that. It also maps pretty seamlessly to types/classes in most languages without annotations, attributes, or other serialization guides.

It also has explicit indicators for lists vs subdocuments vs values for keys, which xml does not. XML tags can repeat, can have subtags, and then there are tag attributes. A JSON document can also be a list, while XML documents must be a tree with a root document.

XML may be acceptable for documents. But seeing as how XHTML was a complete dud, I doubt it is useful even for that.

And we didn't even need to get into the needless complexity of validation, namespaces, and other junk.

<comment><USER11> XML is a perfectly serviceable data exchange format. The parsers and serializers work great when used properly. It's nice to have schema.

But I think people just got sick of XML because it was abused so badly with "web services", SOAP, wsdl and all those horrible technologies from the early naughts. Over-complicated balls of mud that made people miserable.

<comment><USER12> Apple's plist format might be the weirdest abuse of XML as far as I can tell. The SOAP envelopes and shit like that were horrible but plist is plain weird.

Everyone abused XML some way or another. JSON is not that "abusable" I'd say.

<comment><USER13> PLIST is pretty flexible though, the underlying storage can be XML, binary or even JSON now.

<comment><USER14> Implementing a binary plist encoding on you REST endpoints is actually pretty great for iOS devices.

</comment></comment></comment></comment></comment></comment><comment><USER15> Once you're parsed the first minefield, another crop emerges: interpreting the result. Even the range of values seen in the wild for a supposedly simple boolean attribute is just mind-boggling. Setting aside all the noise from jokers trying it on with fuzzing engines, we'll see all of these presented to various APIs:

        true
        false
        null
        0 | 1
        "true" | "false"    (with assorted variation by
        "yes"  | "no"        case and initial character)
        "" | "0" | "1"
        "\u2713"            (hi DHH)
        -1                  (with complements)
        "[object Object]"
        { "value": true }   (and friends)
                            (attribute not present)
        "敵牴"

That last looks like a doozy, but old lags will guess what's going on right away. It's the octets of the 8-bit string "true", misinterpreted as UCS-2 (16-bit wide character) code points and then spat out as UTF8. Google translates it, quite appropriately, as "Enemy".

Oddly though, according to my records, never seen a "NULL".

</comment><comment><USER16> I'm fine with a parser that doesn't get all of the corner cases as long as it fails gracefully.

Really, the only time it would matter is if you are parsing user-provided JSON and said user was trying to exploit your parser somehow.

But 99% of the time, I'm not parsing user-provided JSON, so I don't ever encounter these corner cases and parsing/serialization works great.

<comment><USER7> What about the 2^63 corner-case?

Consider this JSON: {"key": 9223372036854775807}. With most parsers it never fails.

But... some JSON parsers (include JS.eval) parse it to 9223372036854776000 and continue on their merry way.

The problem isn't user-provided JSON here. The problem is user-provided data (or computer-provided data) that's inside the JSON.

rachelbythebay's take ([http://rachelbythebay.com/w/2019/07/21/reliability/](http://rachelbythebay.com/w/2019/07/21/reliability/)):

 _On the other hand, if you only need 53 bits of your 64 bit numbers, and enjoy blowing CPU on ridiculously inefficient marshaling and unmarshaling steps, hey, it 's your funeral._

<comment><USER17> > _some JSON parsers (include JS.eval) parse it to 9223372036854776000 and > continue on their merry way_

This is correct behavior though...? Every number in JSON is implicitly a double-precision float. JSON doesn't distinguish other number types.

If you want that big a string of digits in JSON, put it in a string.

Edit: let me make a more precise statement since several people seem to have a problem with the one above:

Every number that you send to a typical JavaScript JSON parser is implicitly a double-precision float, and _it is correct behavior_ for a JavaScript JSON parser to treat a long string of digits as a double-precision float, even if that results in lost precision.

The JSON specification itself punts on the precise semantic meaning of numbers, leaving it up to producers and consumers of the JSON to coordinate their number interpretation.

<comment><USER18> Every number in JavaScript is, JSON does not specify.

<comment><USER17> "JavaScript Object" is right there in the name.

<comment><USER19> Java is also in the name JavaScript, but we know how much that has to do with it.

<comment><USER20> indeed, we've all read that history. And we all know how much JavasSript has to do with "JavaScript Object Notation", too right? Basically everything?

<comment><USER19> Other than shared syntax, JSON is its own thing.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> Just recently V8 the JS engine rewrote their JSON parsing code to achieve upto 2.7x faster parsing and also making it memory efficient.

Ref link:- [https://v8.dev/blog/v8-release-76](https://v8.dev/blog/v8-release-76)

<comment><USER22> Ah, so that's the source of that Chrome bug that we saw last week. Customers on Chrome for Windows (only that, not Chrome for Linux or macOS) were complaining that the search on our statically-generated documentation site was not working. The search is implemented by a JavaScript file that downloads a JSON containing a search index, and it turns out that this search index had too much nesting for Chrome on Windows's JSON parser. This would reliably produce a stack overflow:

      JSON.parse(Array(3000).join('[')+Array(3000).join(']'))

We were about to report a bug when we noticed that the problem was fixed in Chrome 76, and the users in question were still on Chrome 75.

<comment><USER23> Pretty weird for a JSON parser to be platform dependent.

<comment><USER24> Maybe the stack is shallower on Windows, or the calling convention takes more space per function and is enough to push it over the edge.

<comment><USER25> Recursion based implementation of parsers in languages with limited stack size is a programmer mistake.

<comment><USER18> All languages have limited stack size.

<comment><USER26> Eh... not really if you're referring to call stacks. Rust at one point had growable stacks. That was removed for performance reasons. Haskell with GHC kind of has growable stacks (basically IIRC most function calls occur on the heap) and its stack overflows take a different form. SML I think at one point also had an implementation with a growable call stack.

<comment><USER27> I guess this is a nitpick of limited vs. fixed. Even if you grow the stack, at some point you can't grow anymore.

<comment><USER26> If by limited you mean limited by the amount of memory your machine has then yes it's limited, but I don't think that's what parent was getting at, since in that sense everything about a computer is limited.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER28> Check out the simd JSON project if you're interested in a super fast JSON parser:

[https://github.com/lemire/simdjson](https://github.com/lemire/simdjson)

I've been using to process and maintain giant JSON structures and it's faster than any other parser I've tried. I was able to replace my previous batch job with this as it gives real-time performance.

<comment><USER29> How does it do on the article's test suite?

<comment><USER30> [ Original designer of much of simdjson here ]

We haven't used that particular suite, but almost everything in that suite is something we've thought about. In many cases we do the right thing by not innovating and randomly allowing stuff that isn't in the spec.

I see exactly one thing we didn't think about, as our construction of a parse tree is pretty basic and we don't build an associative structure even when building up an object - thus we would not register an error when confronted with the malformed input listed under "2.4 Objects Duplicated Keys", but happily build a parse tree with duplicated keys (which will be built up strictly as a linear structure, not an associative one).

There seems to be leeway on this point as to what an implementation should do. It certainly doesn't fit our usage model very well to build a associative structure right there on the spot - some of our users wouldn't want that much complexity/overhead.

</comment></comment></comment><comment><USER31> ASN.1

> Abstract Syntax Notation One (ASN.1) is a standard interface description > language for defining data structures that can be serialized and > deserialized in a cross-platform way. It is broadly used in > telecommunications and computer networking, and especially in cryptography.

[https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One](https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One)

Or keep re-inventing the wheel. It's not like the people paying you will notice or care, eh?

<comment><USER32> Asn.1 is incredibly hard to actually implement . There are dozens of cases of security bugs based on bad parsers. Also there are a dozen different encodings of asn.1 data including json (JER). Its age also means that it has a bunch of obsolete datatypes.

Protobuf and friends have most of the power without a lot of the drawbacks.

<comment><USER31> > Asn.1 is incredibly hard to actually implement.

For whom?

> There are dozens of cases of security bugs based on bad parsers.

You're saying this on a thread called "Parsing JSON Is a Minefield", eh?

In any event, this is not unique to ASN.1. I haven't checked but I don't doubt there are similar cases for Protobuf, etc.

> Also there are a dozen different encodings of asn.1 data including json > (JER).

So what? That's the opposite of a problem.

> Its age also means that it has a bunch of obsolete datatypes.

So don't use them.

\- - - -

My point is that if the time and effort that was spent on Protobuf and CapnProto and all the others had somehow been spent instead on perfecting ASN.1 then, uh, that would have been good...

<comment><USER4> > My point is that if the time and effort that was spent on Protobuf and > CapnProto and all the others had somehow been spent instead on perfecting > ASN.1 then, uh, that would have been good...

I wrote proto2 in 20% time at Google and I developed Cap'n Proto entirely on my own time, unpaid. If you think ASN.1 could be perfected with a similar amount of work then why don't you do it?

<comment><USER31> It seems like I may have offended you, I didn't mean to, and I apologize.

I'd love to discuss this but don't want to get in a flame war.

In re: ASN.1, if I ever have to de/serialize some messages again (I'm quasi- retired ATM) I would use ASN1SCC "an ASN.1 compiler that was developed for ESA to cover all data modelling needs of space applications."

> The compiler is targetting safe systems and generate either Spark/Ada or C > code. Runtime library is minimalistic and open-source. The tool handles > custom binary encoding layouts, is fully customizable through a code > templating engine, generates ICDs and automatic test cases."

[https://essr.esa.int/project/asn1scc-asn-1-space- certifiable...](https://essr.esa.int/project/asn1scc-asn-1-space-certifiable- compiler)

</comment></comment></comment></comment></comment><comment><USER33> I miss the days of strongly typed schemas. It's much easier to fail gracefully.

<comment><USER34> Throw your support behind [https://json-schema.org](https://json-schema.org) it's a great effort.

<comment><USER35> Or use something sane like protocol buffers

<comment><USER25> Protobuf crashes with data larger than 2GB (json does not).

This severely limits its usefulness, unless you want to put workarounds of this shortcoming into your application.

Other protobuf-like things like capnproto don't have that restriction, because they don't use 32 bit integers for sizes.

<comment><USER4> You really don't want to put >2GB in a single protobuf (or JSON object). That would imply that in order to extract any one bit of data in that 2GB, you have to parse the entire 2GB. If you have that much data, you want to break it up into smaller chunks and put them in a database or at least a RecordIO.

Cap'n Proto is different, since it's zero-copy and random-access. You can in fact read one bit of data out of a large file in O(1) time by mmap()ing it and using the data structure in-place.

Hence, it makes sense for Cap'n Proto to support much larger messages, but it never made sense for Protobuf to try.

Incidentally the 32-bit limitation on Protobuf is an implementation issue, not fundamental to the format. It's likely some Protobuf implementations do not have this limitation.

(Disclosure: I'm the author of Protobuf v2 and Cap'n Proto.)

</comment></comment></comment></comment></comment><comment><USER36> Parsing is a minefield. General purpose computing systems are minefields. Of all human readable formats I've ever worked with, only S-expressions have proven easier and safer to parse. Json.org even has unambiguous railway diagrams!

<comment><USER37> I wish EDN would catch on. The simplicity of JSON with better number handling, a few more very useful data types like namespaced keywords and sets, arbitrarily complex keys, and an even terser, more readable syntax. [k v k v] beats [k: v, k: v] hands down, and you can use whitespace commas if you want them.

[https://github.com/edn-format/edn](https://github.com/edn-format/edn)

</comment></comment><comment><USER38> JSON sucks. Maybe half our REST bugs are directly related to JSON parsing.

Is that a long or an int? Boolean or the string "true"? Does my library include undefined properties in the JSON? How should I encode and decode this binary blob?

We tried using OpenApi specs on the server and generators to build the clients. In general, the generators are buggy as hell. We eventually gave up as about 1/4 of the endpoints generated directly from our server code didn't work. One look at a spec doc will tell you the complexity is just too high.

We are moving to gRPC. It just works, and takes all the fiddling out of HTTP. It saves us from dev slap fights over stupid cruft like whether an endpoint should be PUT or POST. And saves us a massive amount of time making all those decisions.

<comment><USER39> Off-topic but I'd want to work on a place where half the REST bugs are from JSON parsing.

<comment><USER40> Yeah I don't believe I've _ever_ seen a json parsing problem in 11 years of software development.

</comment></comment></comment><comment><USER41> Plugging my amazing JSON-like format! [https://gitlab.com/rendaw/luxem](https://gitlab.com/rendaw/luxem)

In case anyone doesn't click the link, its much simpler than JSON, leaves interpretation up to the reader, supports polymorphic data, and has some tweaks to make it nicer to edit by hand. I've used this in a bunch of personal projects and maps all the models I've come across perfectly. If you need more power in your format you're better off using Lua than YAML.

I have a Rust Serde implementation 90% complete I could finish up if anyone wants it.

<comment><USER42> Trailing commas are of course very sensible!

This doesn't seem simpler than JSON otherwise, though, e.g. type declarations and optional quotes.

Why is leaving interpretation up to the reader desirable? Shouldn't things always come out the same?

Asterisks are an unusual choice of comment syntax. What if your comment needs to contain an asterisk? Why not "//..." and/or "/* ... */", or "#..."?

</comment></comment><comment><USER43> Compared to XML, Markdown, and other human readable formats, this is...actually not too bad. I was expecting worse.

</comment><comment><USER44> The bigger question is, what is there to be done? What is the road to a more uniform handling of JSON? I've handled some JSON before and it's usually fairly easy untill you catch one of these strange implementation quirks. But I'm not sure that those quirks can be ironed out at this point.

<comment><USER45> Can you help me understand the problem? These things seem like corner cases that you could just Not Do(TM) and then you don't have to worry about it.

What am I missing, when do these gotchas become an actual problem for you as a developer?

I'm not sure I've used any technology that was free of footguns, and JSON appears to have fewer of them than the average programming language or library.

<comment><USER46> I call this kind of answer "The C Answer". "Who cares if this particular combination of code results in undefined behavior? Just don't do it!"

> when do these gotchas become an actual problem for you as a developer?

Whenever you have to deal with JSON produced by "not you", or when you have to deal with JSON that may have been corrupted in some fashion along the way.

<comment><USER45> I'm probably just not used to pure "all behavior is defined" systems, so I appreciate your perspective.

What industry do you work in? I don't see many systems like that in my industry. I work like hell to push things in that direction, but it's a best case of "we went from 5% well defined behavior to 50%" after many years of effort.

</comment></comment></comment></comment><comment><USER47> The product owner perspective on this should be "nothing supports anything unless it is tested".

If you pick up a standard and just assume other products will be able to work with it, you're in for a surprise. I don't care if it's TCP sockets or .ini files; if you didn't test compatibility with the product you expect will interact with yours through the standard, consider it unstable, and don't advertise support for it.

Sometimes you _have_ to support a standard itself, like WPA2, so you implement the standard according to internet engineering best practice: be liberal with what you accept, and conservative with what you transmit (or something to that effect). Then test compatibility with the major products you know will want to use it, and fix the bugs you find.

</comment><comment><USER48> Been of the opinion a while a lot of issues could be resolved if we agreed on a streamable binary format that had good definitions for data types (including integers and dates).

String formats are great an all for viewing in whatever text viewer but so inefficient and then you have the whole escaping string inside of strings and string encoding binary data.

If we all agreed on a binary format then there would be a viewer for it in every debugging tool.

ASN.1, Protobuf, BSON, ION. MSGPACK whatever. I would prefer a binary format that doesn't repeat keys for efficiency where the schema can be sent separately or inlined. But even one that's basically binary JSON with more types would be step up.

</comment><comment><USER49> I'm not a professional coder. And I mostly work with tabular data, in spreadsheets and SQL. I like to get my data as delimited text files. Ideally, delimited with some character that's 100% guaranteed to never occur in the data. In my experience, "|" is often a good option, but you never know. And CSV, even with quotes, can be a nightmare, especially if the data contains addresses. Or names with quoted nicknames.

Anyway, given the choice, I always pick JSON over XML. Because with JSON, I can always identify the data blocks that I need, and parse them out with bash and spreadsheets. Not with XML, however. Just as not with HTML.

</comment><comment><USER50> > For example, Xcode itself will crash when opening a .json file made the > character [ repeated 10000 times, most probably because the JSON syntax > highlighter does not implement a depth limit.

FWIW, this appears to have been fixed recently.

<comment><USER51> Slightly off topic but Xcode crashing has been very common for me, as much as I like macos the tooling leaves me missing Windows development.

</comment></comment><comment><USER52> I certainly would not criticize such a thorough examination for being facile, but I do want to point out that the conclusion "But sometimes, simple specifications just mean hidden complexity" is not supported by the article. Almost all of the end cases are caused by implementors ignoring or extending the simple specification.

</comment><comment><USER53> The crashing test cases look scary from a security perspective, specially in the C-based parsers. Does anyone know if these results are still up to date or if the bugs have already been fixed?

</comment><comment><USER54> Relaxed JSON is pretty good. [http://www.relaxedjson.org/](http://www.relaxedjson.org/)

<comment><USER55> Relax it a bit more and you get Neon: [https://ne-on.org/](https://ne-on.org/)

</comment></comment><comment><USER56> This is all well and good, but which decoders are already in the browser?

XML and JSON last I checked.

</comment><comment><USER57> Again. Parsing is a minefield in general, but parsing JSON is one of the easiest tasks of all serialization formats. It's also the only secure format.

Its various spec bugs (by omission) are not that dramatic, and the various "enhancements" only made it worse, ie more insecure. Still, bad but not a minefield. What worries me most that my JSON module is the defacto perl standard, passes all these tests, was the very first to add all these tests, is the fastest, and still is not included in that list, just some outdated modules which should not be used at all. Checking best practices besides maintaining a spec obviously also is a minefield.

<comment><USER57> I forgot another major JSON minefield problem which is not mentioned nor tested here: stackoverflow.

This is in fact the most important problem to test against, because it might lead to exploitable stack ROP.

JSON is usually parsed recursively, and deeply nested structures are mostly not depth counted. One can trivially construct a nested array or map of 500 to 30000 elements, and at one point the parser either fails or crashes with an overflow. This number is fixed, thus trivially exploitable. The test spec should contain the max. depth for arrays and maps, and if there's a fixed builtin limit, a compile-time limit, implicit limit by crash, or none. non recursive parsers are fine.</comment></comment>


Exploring the Myth of the Chubby Gamer - <AUTHOR> https://www.sciencedirect.com/science/article/abs/pii/S0277953619302941 <comment><AUTHOR> This article is a meta-analysis of various studies on sedentary video gaming and body mass.

It concludes that while "high body mass and obesity are frequently linked to the use of sedentary media, like television (TV) or non-active video games", actual "results do not corroborate the assumption of a strong link between video gaming and body mass as respective associations are small and primarily observed among adults."</comment>


Amazon Announces new Data Warehousing Product - <AUTHOR> http://aws.amazon.com/redshift/ <comment><USER2> This has the potential of really disrupting the enterprise data warehouse sector. All the MPP vendors today (HP Vertica, EMC Greenplum, Teradata) have exhorbitant pricing and ridiculous licensing. With their pricing - 1000 $ per TB per year, I would be really worried if I were Teradata (Not so much if I were IBM).

<comment><USER3> A lot of large enterprises won't be comfortable hosting their data outside of their own data centers. The killer application is making a portable, on premises version of this functionality without the high price.

<comment><USER4> I've heard this argument time and time again in the context of various solutions/technologies. I still today feel that for 95% of the companies that "feel this way", it's simply the result of foolish paranoia among older, upper management. The type of thing that separates a "Fast Company" from stodgy and likely to be disrupted companies.

<comment><USER5> Question: would you trust all your medical data/history and shopping data/history tied up in their cloud?

<comment><USER6> What makes you think it isn't already?

Hospitals & doctors outsource, and as long as the provider is HIPPA compliant (which AWS is[1]), your data is probably out there already.

[1] [http://awsmedia.s3.amazonaws.com/AWS_HIPAA_Whitepaper_Final....](http://awsmedia.s3.amazonaws.com/AWS_HIPAA_Whitepaper_Final.pdf)

</comment></comment></comment></comment></comment><comment><USER7> Have to say that this is pretty amazing. The price is so low that it's a no- brainer to just give it a try. For the same 2TB capability, a Vertica license would run between $20-40K, with high annual subscription fees.

The bigger question for me is why Amazon has been able to figure out the technical details necessary to run this kind of service for this price. It's just ridiculous. Talk about taking the oxygen out of the market...

<comment><USER8> > The bigger question for me is why Amazon has been able to figure out the > technical details necessary to run this kind of service for this price.

I guess they grew the infrastructure for themselves, optimizing it bit by bit over the years. And then noticed that it could be sold too.

<comment><USER9> They seem to have taken their own business requirements for amazon.com and reimplemented them on commodity hardware.

</comment></comment></comment><comment><USER10> Does anyone have insight into how painful it is for non-technical people to _query_ their data warehouses?

I'm building a tool that allows business people and non-technical analysts to query their data warehouses using _natural language_. (Currently, you must ask a technical person to write ad-hoc queries for you, or build you a dashboard. This bogs down your data people.)

Does anyone have insight into the demand for such a product?

[edit: I'd love to chat with anyone with insight into this topic. Reach me at Joseph at metaoptimize dot com]

<comment><USER11> I would suggest reading some books on the topic of Dimensional Modeling [1] such as "The Data Warehouse Toolkit" [2]. The critical thing you need to expose to your users is the ability to ask for things which make sense in their world that are actually really difficult for even an engineer to code. Things like: "Show me average 9am-12pm sales on Mondays, Wednesday and Fridays for 1st quarter, 2012"

      [1] http://en.wikipedia.org/wiki/Dimensional_modeling
      [2] http://www.amazon.com/Data-Warehouse-Toolkit-Complete-Dimensional/dp/0471200247

<comment><USER12> Speaking as someone who does his fair share of dimensional modeling, I would just point out that the example you cite could only involve two tables in a well designed dimensional model (sales fact and time/date dimension, I reckon). The challenge is in getting to that point.

To speak to OPs point about difficulty in querying data warehouses, most business intelligence tools that I'm aware of provide semantic layer[1]-type capabilities, whereby the user interface of the tool is presented in the language of the business domain. Nevertheless, I still agree that this is still difficult work, unfortunately. That it is getting more complicated in some respects, such as through unstructured data, doesn't help either.

[1] <http://en.wikipedia.org/wiki/Semantic_layer>

<comment><USER11> I guess I wasn't clear enough if I came across like my example was complex. It's one easily solved via DM and one that's extremely hard to execute in most non-dimensionally-modeled setups. That's exactly why I'm a huge advocate of DM instead of just throwing a ton of servers, hadoop & MR at everything.

</comment></comment></comment></comment><comment><USER13> I'm curious what technology they are using to power it. According to the website, the technology described seems very similar to what Cloudera recently open sourced (Impala), which sits along side Hadoop allowing ad-hoc MPP style querying on petabytes of data.

<https://github.com/cloudera/impala>

<comment><USER14> I'm guessing it is quite a bit different from that. It is a relational data warehouse. It supports a Postgres protocol and API, which sounds more like what Netezza has built. In fact, I would expect Netezza to be one of the most likely companies to partner with Amazon at this kind of price-point.

<comment><USER15> Other candidates:

* Yahoo's Everest

* Greenplum

* Aster Data

All mentioned here: [http://www.cubrid.org/blog/dev-platform/database- technology-...](http://www.cubrid.org/blog/dev-platform/database-technology- for-large-scale-data/)

The Register wrote that Amazon's solution is a column-oriented database possibly based on Postgres, like Yahoo's:

[http://www.theregister.co.uk/2012/11/28/amazon_aws_redshift_...](http://www.theregister.co.uk/2012/11/28/amazon_aws_redshift_data_warehousing/)

</comment></comment></comment><comment><USER16> Should be interesting if this will be a viable competitor to column oriented sql engines like Vertica or other OLAP solutions like SAP HANA. It would be nice if there was a simple SQL based olap solution that I can spin up for offline reporting that can scale terrabytes of data

</comment><comment><USER17> Now if only Amazon would offer PostgreSQL on normal RDS.

<comment><USER18> Doesn't RDS crap itself whenever there's a core AWS problem?

<comment><USER19> Yes, since it relies on EBS for persistence, which has been one of the flakiest parts of AWS so far.

<comment><USER20> All it takes is an EBS outage to determine this.

In the last one, RDS and ELB had issues (and were flagged as such on their status board) due to needing EBS, but I don't believe DynamoDB was.

</comment></comment></comment></comment><comment><USER7> Update! The entire keynote is now available on youtube: <http://www.youtube.com/watch?v=8FJ5DBLSFe4>

The discussion about Amazon Redshift begins at 52:50 [http://www.youtube.com/watch?feature=player_detailpage&v...](http://www.youtube.com/watch?feature=player_detailpage&v=8FJ5DBLSFe4#t=3175s)

</comment><comment><USER21> What is the use case for something like this versus a regular RDS service?

<comment><USER22> The answer is in the term "data warehousing" -- <http://en.wikipedia.org/wiki/Data_warehouse> \-- which has implication that you're going to be doing data mining on vast amounts of data, often historical data like logs or transaction histories.

Google has systems like this for analyzing its request logs. Think of how many HTTP requests hit Google's front-end servers per second or hour or day. Each one has a few dozen pieces of data associated with it -- URL, client IP, headers, etc. Suppose I want to make a bar chart of how many requests came from France containing a certain header, each day for the last year. The system can do this query quickly if the requests are already bucketed by time interval, organized by column, compressed, and stored so that exactly the information needed can be brought into RAM quickly.

It is a little funny, when you step back, that "storing," "archiving," and "warehousing" are different things and Amazon has services for each. Try explaining the difference between S3, RDS, EBS, Glacier, and Redshift to a layperson.

<comment><USER21> Thanks for the response. Would it make sense to say that this is more likely to be used for metadata (i.e. analytics, logs, etc.) while a normal RDB (or NoSQL DB) would be used for application data (i.e. users, settings, etc.)?

</comment></comment></comment><comment><USER23> It's called Redshift!

wow.. I just finished reading the sci-fi book a few weeks ago - "Redshift Rendezvous" by John E Stith. I wonder if this is where the name comes from? In the book Redshift is the name of the space ship that runs cargo mission through folded space, the obvious problem that since you are traveling within just a few m/s of the speed of light just walking on the ship while underway causes color shift - thus redshift.

I read that Stith has a physic degree and worked as an Engineer for NORAD Cheyenne mountain. That made me really interested in what novel he would come up with. <http://www.neverend.com/short-bio-john-e-stith>

<comment><USER24> Redshift is a real physical phenomena describing the way light wavelengths get "shifted" (stretched, to visualize) towards the red as they are seen coming from something moving away from the observer:

<http://en.wikipedia.org/wiki/Redshift>

<http://en.wikipedia.org/wiki/Hubbles_law>

</comment></comment><comment><USER25> It seems that the price (~$1 / GB / year) in the best case (3 year reserved) is comparable to S3 at its lowest tiers (~$0.1 / GB / month)

<comment><USER26> It's "Price per TB per Year" not GB.

</comment></comment><comment><USER7> Very cool that this will support regular sql queries and queries can be sent using postgresql drivers. Postgresql drivers are super stable and supported everywhere. Driver support is usually overlooked with 'Enterprise' Data Warehousing solutions. I recall that it was really hard to get the Vertica drivers installed and stable under Linux.

I took a few screenshots from the keynote and included one showing the mention of Postgresql and ODBC/JDBC support. Included here if you want to see for yourself: <http://wp.me/p2sRpx-1e>

</comment><comment><USER27> Maybe a naive question, but how does this compare with Google Big Query?

<comment><USER28> <https://developers.google.com/bigquery/docs/pricing#table>

$1474 per TB per year for storage alone ($0.12 * 1024 * 12)

plus

$35.84 per TB queried

Amazon is definitely cheaper.

</comment></comment><comment><USER29> I cannot find information on whether Redshift supports queries in MDX. Lots of DWs today are run on Microsoft SQL Server Analysis Services and its MDX spec is now supported by several DW vendors. MDX support would mean it would be easy to switch the DW engine and leave your visualisation suite (or Excel, what the hell) and make it for an easy switch to the cloud - you'd just pick a different data source in your tool.

</comment><comment><USER30> Looks impressive and very interesting, signed up to review and compare with Teradata/Netezza.

Can we run more complex in-database processes implemented as stored procedures on this platform or is it going to be limited to pure SQL querying/analytics?

And does anyone have an idea how to upload 1 TB of data to this service using Internet connection from your in-house company server? ;)

<comment><USER31> AWS has pretty good support for taking external drives and importing them to S3 which could then be used with this service:

<http://aws.amazon.com/importexport/>

I am assuming that you have 1TB to start, not generating 1TB per day which obviously changes the equation.

</comment></comment><comment><USER32> This looks awesome - we'll definitely be plugging SnowPlow into this.

</comment><comment><USER33> So is this the Amazon clone of Google's Spanner?

<comment><USER34> No. Spanner is a globally distributed database which supports transactions. It is meant for applications which need to make frequent updates to a database, but the storage for the database may be distributed around the world.

Redshift is a different usage model. You upload your data once, then ask questions of it - but you don't update it. Google does have something similar to Redshift: BigQuery (<https://cloud.google.com/products/big-query>).</comment></comment>


Congress wants Twitter to block "propaganda" (political speech it doesn't like) - <AUTHOR> http://telegraph.co.uk/technology/twitter/8972884/Congress-calls-on-Twitter-to-block-Taliban.html

<comment><USER2> Once again, a non-US news site breaks the news that Lieberman wants to censor news for the USA.

What does this say about the USA news sites? Nothing good, it just seems to confirm the "we're already self-censoring" suspicions that a lot of people already have about US news media.

It also says a lot about Lieberman. It says he failed his civics classes in high school, for one thing. He also seems to have a low, low opinion of the American public itself, inf effect saying that the American public can't be trusted to sort out good info from bad.</comment>


How many GNU/Linux users are needed to change a light bulb? - <AUTHOR> http://www.gnu.org/fun/jokes/users-lightbulb.html

<comment><USER2> > 1 to remind everyone that the right name is GNU/Lightbulb.

Haha, opened the page already expecting that. Was not disappointed. I wonder why naming is something that attracts so much energy and strong opinions in open source movements? Are other fields like that as well?

<comment><USER3> There are two major problems in computer science: naming things, cache invalidation, and off-by-one errors. (Twist on the Karlton quote.) I blame the foremost for this issue. Note though that very few of the users needed actually complain about it; the vast majority of the discussion is, predictably, off-topic.</comment></comment>


Message Processing Styles (2016) - <AUTHOR> https://www.tbray.org/ongoing/When/201x/2016/10/23/Message-Processing <comment><USER2> " _The problem here is that when you're building infrastructure you're usually using a statically-typed language, which will have a severe impedance mismatch with JSON's happy-go-lucky attitude: the structure of the data is the structure given by the characters in the JSON text, which is definitely neither known nor deducible at compile time._ "

As an old school network protocol geek, I find laments such as this article inexplicable. Message handling is a variant of good old parsing. The only difference between it and the first couple of weeks of your compilers class is that you want message parsing to be fast (the term of art is 'wire speed') and you don't really care what the error is, if there is one.

Usually, all that means you only look at the parts of the message you need to, and if something is wrong you drop it right there. The more magic in the way, the worse off you are.</comment>


Show HN: FlexDict – Deeply Nested Python Dictionaries with Ease - <AUTHOR> https://github.com/ozturkberkay/FlexDict <comment><USER2> This is cool! Would easily replace the slightly tricky to grok code I came across to make a binary trie last week:

      # in for loop:
          node = trie
          for char in number:
              node = node.setdefault(char, {})

(from here: [https://www.hackerrank.com/challenges/maximum- xor/forum](https://www.hackerrank.com/challenges/maximum-xor/forum))</comment>


Which is the stack you usually use to start your side projects? - <AUTHOR> I've been thinking about starting a project on my own for a long time and I was wondering which stack is usually used.
I've been developing micro service applications for a long time and the best option I can think of is to use the MERN stack.
What is the best option for you? <comment><USER2> Whatever stack you're most comfortable in and will allow you to get up and running sooner than later.

<comment><USER3> This is always the right answer.

<comment><USER4> Is it, though? Side projects can be a fantastic opportunity to try something new. I wouldn't have any of my current go-to stack without such experimentation.

<comment><USER5> Good point - so if the goal of the side project is to learn a new skillset then this is the wrong answer, but if the goal is to build an MVP then it is right - use whatever gets you there quicker.

</comment></comment></comment></comment><comment><USER4> It totally depends on the project, there is so much that can be a side project (even a commercial one). For CLI tools and other utilities, mostly Go. APIs and backends, Scala and Akka HTTP. For storage, usually MySQL, sometimes Kafka. For web frontends either Scala.js, Vue or vanilla JS. For infra, plain Docker or one of my k3s clusters. For IoT/electronics, Arduino and a suitable µC. For parts design and printing usually OpenSCAD and a Prusa Mini. For woodworking, a small Metabo drill, a Dremel and a set of hand tools. For illustrations and the like, an iPad Pro with Pencil and Procreate.

<comment><AUTHOR> Thank you very much for your answer, it has taught me things I didn't know

</comment></comment><comment><USER6> Usually PHP. I grab Laravel, Tailwind CSS, Vue and the needed libraries like VueRouter, Vuex and a few PHP libs and see what happens.

The correct answer is mostly what others have said: whatever you know best. But I would caveat that with this: if this is a project just for fun and learning, pick anything you want and the most important part is have fun. If you are trying to do RAD and get something in a market, grab what you know best and go from there.

</comment><comment><USER7> I generally use Next.js, Bulma, Firebase Auth, and Firestore. For more complex apps, will move Firestore behind a REST API and use react-query for data fetching/caching.

<comment><USER8> > will move Firestore behind a REST API

How do you do that? Do you spin up your own server?

<comment><USER7> With Next.js it's just a matter of dropping a file in the src/pages/api directory. So I'd create API files like create-user.js, update-user.js, etc.

</comment></comment></comment><comment><USER9> Wt _/ C++ for the backend and templating, jQuery+Bootstrap_* for the frontend and PostgresQL for database.

* [https://www.webtoolkit.eu](https://www.webtoolkit.eu) __With Wt there is no requirement for Bootstrap and /or jQuery though.

</comment><comment><USER10> I would love to use assembly x86 on baremetal (i.e., no code that's not written by me executes) as I used to do years ago but I will be needing an infinite time and that would be, career-wise, a complete waste of energy.

</comment><comment><USER11> It certainly depends but for websites bulma and nextjs and strapi for backend

</comment><comment><USER12> For a long time, it used to be PHP + MySQL + JQuery. Now its more like React + Next + Firebase.

Just pick what seems fun and what seems productive to you.

</comment><comment><USER13> i use php + jquery + mysql. I can make pretty much whatever i can think of and there is not much overhead. I have made tonnes of various tools for work using this by just creating a php file and started to code with no bs from stopping me implement the idea i have in my head.

</comment><comment><USER14> Django! I am not good with front end JavaScript though, so it is hard for me to create anything too dynamic.

</comment><comment><USER15> vue.js and svg. i do FUI stuff as side projects

unreal engine for interactive stuff

c++/ swift for code oriented stuff</comment>


Question quality is dropping on Stack Overflow - <AUTHOR> http://meta.stackoverflow.com/questions/252506/question-quality-is-dropping-on-stack-overflow?cb=1

<comment><USER2> SO aims at a simple Q&A format. I wonder why people expect it to yield "great" questions. A lot of actually fruitful topics are closed for provoking discussion.

       "... as they tend to attract opinionated answers and spam.   Instead, describe the problem and what has been done so far to solve it."

Stackoverflow is not an intellectual forum, its a simple question-and-answer database with debate being discouraged, and a very limited space for the kind of questions allowed. This is by design and its great, because finally when asking a question on XML, you are NOT bothered with people starting to discuss whether XML is actually a thing worth considering. But its the same _beneficial_ rules, that in the long run lead to a plattform where people just paste in their error messages and expect other people to dive through those.

<comment><USER3> I don't see how "not a forum for debate" means "a forum for 'plz email teh codez' posts" in your reckoning. There are many fine questions with right answers that are not open-ended navel-gazing.

<comment><USER4> The point is that if debate isn't allowed you aren't likely to have higher quality posters than the average for programmers. And the reality is that there are tons of lazy/stupid programmers out there that just want "plz email teh codez".

The forums that I have seen work best at maintaining abnormally high levels of quality posters are ones where the Q&A is effectively secondary to the debate. The reason is that the fun of debating keeps the high quality posters around and answering shitty/repetitive questions.

<comment><USER3> I think your first conclusion is wrong. From what I have seen, Stack Overflow appears to be fairly well-supplied with quality posters, but they just don't have as many questions as the "plz email teh codez" crowd. The problem isn't that good questions are generally going unanswered, and it isn't even that good and novel questions are in short supply, but that there aren't as many good questions _and_ they're having trouble stemming the tide of bad questions. The disproportionate rise of bad questions is the core problem.

</comment></comment></comment></comment><comment><USER5> I think So is being overrun with "Help Vampires" ( [http://slash7.com/2006/12/22/vampires/](http://slash7.com/2006/12/22/vampires/) )

However, the twist with SO is that people looking to inflate their rep actually like help vampires to a certain extent because their questions are usually quick and easy to answer which increases speed at which they can buff their scores.

Combine these two and you get a community where posting hard, reaching, questions with in-depth answers becomes the exception rather than the norm.

<comment><USER6> The solution to fixing Stack Overflow is ultimately to hide karma points and prevent the distortions it causes. If you take away the external incentives you will hopefully be left with a community who simply help each other.

</comment></comment><comment><USER7> I've been a StackOverflow user since it started, I've been a member for over 3 years, and I can't comment on an answer because I only have 46 rep. I'm sure I hit the site everyday finding value in it, when I started at my last job they gave me a tour of their Android work and joked it was made by StackOverflow. The site is difficult to use, everywhere I click it says I need n more rep to use a piece of functionality.

<comment><USER8> It is intentional that new users are steered towards answers instead of comments. SO is a question & answers site and comments are mostly meant for clarifications, not for discussion. This is something that new users that are used to how forums work are not expecting. The hope is that by the time they have 50 reputation, they understand a bit more how comments should be used on the site.

There are additional reasons, the existing moderator tools would not be sufficient to deal with the kind of abuse that would be possible if anyone could post comments, there is simply no meaningful review of new comments on old posts.

And acquiring 50 reputation is generally not hard, that is 10 upvotes on questions, 5 upvotes on answers or 25 suggested edits that improve other posts. You could also earn 200 reputation on any other Stack Exchange site and be able to comment on all of them.

<comment><USER7> That makes sense, it would be nice if there was a linked site for discussion though. As a lurker my first instinct to dip my toes in the water is to comment on something first.

</comment></comment></comment><comment><USER9> The root cause is the same as this: [https://news.ycombinator.com/item?id=7650799](https://news.ycombinator.com/item?id=7650799) "Why is Stack Overflow so negative of late?"

SO has become the go-to site for everyone who thinks they might want to solve a problem using a computer, and really lame, poorly researched questions get asked by people with few incentives to ask good ones.

<comment><USER10> Seems like instead of rate limiting users ability to ask bad questions, you should rate limit the viewing of bad questions by good answerers. In order to keep up the effort of answering questions, maybe some automation is in order to do best-effort auto answering or collapsing of questions into related ones.

<comment><USER9> I agree with your first sentence for sure. Another way to implement it would be to avoid showing new questions with no upvotes to many users--only show them to a small sample of users and see if in the first ten minutes the question gets voted up or down to gauge its quality. That way we'd limit the "shotgun" effect of sending crappy questions to thousands of people all at once (we sometimes see 6 or 8 downvotes on a single question before it gets closed or deleted, which implies a ton of people saw it despite that it was crap).

<comment><USER10> I like it! Would be sorta like A/B testing the questions...

</comment></comment></comment></comment><comment><USER11> I think Stack Overflow needs to enter a curation phase. There's still lots of gems popping up ( [http://stackoverflow.com/questions/13134825/how-do- functors-...](http://stackoverflow.com/questions/13134825/how-do-functors- work-in-haskell/13137359#13137359) ), but they're so drowned out in the noise. A weekly "best of" for tags I subscribe to - perhaps with bonus points for being featured - would do a bit to boost quality. How would it work though? Have a "curator" role you could gain after a certain point threshold, which allows you to vote for best ofs. . and gain points for consistently accurately suggesting best ofs? If you suggest too much which is never voted for, that counts against you... but if you constantly suggest answers / questions which are featured that counts towards your rep.

Perhaps have a separate "exchange" to achieve this?

</comment><comment><USER12> 100% agree, I even tweeted([https://twitter.com/ppiplewar/status/460199400610856961](https://twitter.com/ppiplewar/status/460199400610856961)) this thing few days ago.

On this days, I only answer questions which are worthful and most of the time it falls under `featured` category.

</comment><comment><USER13> Well, how many "good" questions are there really?

I asked several "good" questions a couple years ago, but it's now been probably 1.5+ years since I've even needed to post a question at all because everything I've needed to know is there (and sometimes in 3 different variations).

<comment><USER14> Yes and if this is true for most of the knowledgeable users most of the time it's not a mystery if fewer good questions are asked. I suppose a large quantity of bad questions could be due to increased exposure.

</comment></comment><comment><USER15> Is it really that the volume of poor questions is increasing, or that the volume of good questions is decreasing? I know I gave up on asking questions there a while ago, and I'd like to believe that the questions I did ask were fairly high-quality.

<comment><USER3> Both, I think. But the curves were somewhat different. The really good questions started trailing off quite some time ago (presumably the low-hanging fruit got picked), but the flood of crap didn't really swell until relatively recently. There are fewer good questions, but there really didn't use to be five or more instances of "How do I make one object talk to another?" every single day. That is a new development.

</comment></comment><comment><USER16> It's really just a geometric growth problem. I've been a member for 5 years and I feel like it hit it's "Eternal September" a couple years ago. And I'm sure the people who have been a member since the beginning feel like it went to hell around about the time I joined! I've often wished they had an option to filter questions by asker's rep score.

</comment><comment><USER17> That's the logical end to "everyone should learn to code". Imagine the same fever happening to painting and we have a paintersoverflow site where people would be asking how to mix primary colors.

</comment><comment><USER18> I wonder if there might not be some currency buy-in model that would work. Instead of your in-site "cred" getting bumped, maybe that cred is worth something tangible. Maybe in order to buy into the site you have deposit $5 worth of cryptocurrency X and to ask a question you have to "pay" a dollar worth of that currency and those dollars are "credited" based on how the answer/upvote pool decides.

I can think of at least 10 problems with this approach, but I think it might be feasible.

</comment><comment><USER19> I guess that's the other side of being mainstream - people, who are not perfect users, are using your product too.

Those not perfect users are not only the ones who post questions but also the ones who answer them just for the points. So closing questions which are just copy pasted error messages is solving only half of the problem.

But the other problem is much more subtle. You can't ban (or whatever) someone, who answers questions... Or can you?

</comment><comment><USER20> I have my gripes with SO but I can't imagine a day without it. The number of views on trivial questions says that finding an answer on SO is easier than the way we document code. If there's supplementary documentation done in a Q&A or FAQ fashion, with the right tweaks to rank up in Google, people would not use SO to ask things like ' how to iterate through a list'.

</comment><comment><USER21> My theory is that the help vampires have a certain geographic distribution, and they come from cultures where this kind of behavior is accepted.

<comment><USER22> give us your insight,dont be shy,where you do you think they come from?

</comment></comment><comment><USER23> Conversely, I've found the answers to be less and less helpful and more rude. Just yesterday I asked a question as I thought my web server was spawning a new port for each request. The response was:

"You are just wrong. There is no port shown in the URLs as the slash after the domain."

He was right, but what is with the need to phrase it that way?

</comment><comment><USER24> SO saves time, but drives away the expertise: If I need a help in a bigger, complex and not so trivial questions such as what framework to choose (e.g. angular vs backbone), chances are that question will be closed as not relevant. That leave very technical specific questions which are easily answered by references book.

</comment><comment><USER25> Perhaps, after all this time, there are fewer questions left to ask.

<comment><USER26> That's my thought.

People who care to do the research probably can find the answer in most cases because it has been asked before.

Leaving the new questions to (generally) be asked by people too lazy to do basic research.

<comment><USER27> Occasionally I come across a problem that I cannot find an answer to. I post a question that meets SO criteria.

Lately I find that people answer less and less. And those that do take the time, often don't follow up if I come up with additional information.

</comment></comment></comment><comment><USER22> I dont think "crappy questions" are a problem so much than the inability to filter the crap. With SO being that popular there will always be loads of crap, the issue is as a user i dont have the tools right now to filter crap(questions being downvoted for instance).SO doesnt want to finish like Quora,does it?</comment>


SyntaxCenter – Quick syntax lookup for 9 languages - <AUTHOR> http://syntaxcenter.com/

<comment><USER2> Hey HN! I'm the creator of this project, would love to get some feedback or suggestions for it. Adding Objective-C to the site soon and then after that, I'll be writing an API!

Cheers, Anthony</comment>


Flash in 2020 (and Beyond) - <AUTHOR> https://community.openfl.org/t/flash-in-2020-and-beyond/12287 <comment> 100-xyz Here is a simple animation editor that works similar to Flash and uses simple English sentences for animation

[https://toonclip.com](https://toonclip.com)</comment>