Arguing against using protobuffers - <AUTHOR> http://reasonablypolymorphic.com/blog/protos-are-wrong/ <comment><USER2> Hi there,

I'm an actual author of Protocol Buffers :)

I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises.

\- D

P.S. I also don't believe the personal attacks to be warranted or productive.

<comment><USER3> I thought Kenton Varda was the author of protobufs?

<comment><USER4> Nope, I'm not the original author -- that would be Jeff Dean and Sanjay Ghemawat (also often credited with inventing things like MapReduce, BigTable, Spanner, ...). I wrote version 2 (a complete rewrite, but largely following the original design) and open sourced it. I stopped working on Protobuf about 8 years ago. Many others who have been on the Protobuf team since can certainly call themselves "authors".

<comment><USER5> > that would be Jeff Dean and Sanjay Ghemawat

or "amateurs", as the post would call them.

</comment></comment></comment></comment><comment><USER6> Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration.

The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types.

I think we're at a juncture where we really desperately need a better, _unified_ way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend.

The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.

<comment><USER7> I'd like to humbly suggest that we use JSON please, in particular:

JSON + JSONSchema[0] +/\- JSON Hyperschema[1] +/\- JSON LD[2]

It's a bit to learn but I _promise_ you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guaranteed to imagine a new (albeit somewhat daunting) future, where APIs and clients can be smarter.

There's no reason I can think of that a sufficiently specified JSON schema cannot be automatically converted into a efficient binary format. I have absolutely zero reason to use Protobuf and it's whole ecosystem of tools if that's the case, I'd much rather go with tools built on a standard like JSON.

GraphQL's functionality is a subset of the above set of tools -- IIRC GraphQL can basically be boiled down to an option (or two) on a resource endpoint -- How you query on the server side does indeed change a bit to support the syntax, but this is basically the same as if you'd implemented with horizontal filtering like in POSTgrest[3], or resource embedding feature[4] (which everyone ends up doing at some point with REST-ish APIs). I honestly don't know who or what is trying so hard to drive GraphQL adoption so hard but I think it's mostly hype, and in terms of general technology, by default it _forces_ clients to know the inner workings of the data model and that's bad for any kind of future in which you _don 't_ want to have to learn X APIs to interact with X systems.

BTW I absolutely love a good well-supported rant that agrees with all my biases (yes, Google as well as every other company has a bunch of amateurs, as well as a small amount of very skilled engineers), and all the problems noted with protobuf _are_ real problems, and this viewpoint is sorely missed in all the talks and conferences where people are touting GRPC as the next sliced bread with Protobufs as the transport (you can change GRPC's transport layer to be JSON for example[5])

[0]: [https://json-schema.org/](https://json-schema.org/)

[1]: [https://datatracker.ietf.org/doc/draft-handrews-json- schema-...](https://datatracker.ietf.org/doc/draft-handrews-json-schema- hyperschema/) (also on json-schema.org)

[2]: [https://json-ld.org/](https://json-ld.org/)

[3]: [https://postgrest.org/en/v5.1/api.html#horizontal- filtering-...](https://postgrest.org/en/v5.1/api.html#horizontal-filtering- rows)

[4]: [https://postgrest.org/en/v5.1/api.html#resource- embedding](https://postgrest.org/en/v5.1/api.html#resource-embedding)

[5]: [https://grpc.io/blog/grpc-with-json](https://grpc.io/blog/grpc-with- json)

<comment><USER8> I'd love to see something with the approximate structure of JSON, but less JavaScript-bound syntax. EDN has caught my eye, but then again I've always been fonder of Lisp-y syntax as a whole.

<comment><USER7> ooh I'd love EDN as well -- but I don't think people are willing to adopt S-expressions :(

</comment></comment></comment></comment><comment><USER9> I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the different databases, proxying, etc. etc.

Your "oneoff" and "repeated" might've been (I dunno really), a decision based on metrics where encoding repeated oneoff would've cost more, or made problems, more testing, or who knows what.

For one, Google was, and I believe still is open to discussing all kinds of matters, and you can get on to a design doc and comment if you like, if you care, if you have something to say... You can definitely influence, and people really did talk (you can even experience this from their public design docs - like from chromium, etc.).

And I love protobufs, so I guess I'm biased...

<comment><USER10> Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!

<comment><USER11> I got bad news for you but almost all programming is just moving bytes from one place to another, and occasionally multiplying, adding, subtracting, or dividing some of them. ;)

<comment><USER9> I wasn't expecting really to work on a next breakthrough algorithm, and such (although folks in my team actually worked on some incredible statistics related stuff - and this is where I understood I know nothing about statistics).

Plenty for everyone, I think. I'm glad I was part of the experience!

As an ex-game tools developer (back then, and back again - current job), it was quite a change for me. If I haven't learned enough, then at least seen things from quite different perspective.

</comment></comment></comment></comment><comment><USER12> > Despite map fields being able to be parameterized, no user-defined types can > be. This means you'll be stuck hand-rolling your own specializations of > common data structures.

What a pain! Well, at least Google won't make that mistake again!

<comment><USER13> My contention with the quoted text is that you probably shouldn't be using elaborate data structures in streams/files. Using DTOs as heap/stack data has bitten me enough times that I'm fairly certain that it's an anti-pattern.

It doesn't matter if you're using a quantum binomial tree in a black hole: save it as a 'stupid' map<k,v> when it hits the network. That way everyone who interacts with your service can decide how they want to represent that structure. You can compose any in-memory data structure you can dream of, you can validate the data with more richness than "missing" or "present", and Protobuf doesn't contaminate your codebase.

Option 3 is the correct choice. Serialization libraries should be amateurish by design.

<comment><USER14> > That way everyone who interacts with your service can decide how they want > to represent that structure.

Protobufs aren't a message format for publicly specced standard wire protocols. They're a serialization layer for polyglot RPC request and response messages. The whole point of them is that you're transporting the same typed data around between different languages, rather than there having to be two conversions (source language -> "standard format"; "standard format" -> dest language).

In this sense, they're a lot like, say, Erlang's External Term Format—driven entirely by the needs of an RPC-oriented distributed-computation wire protocol, doing interchange between nodes that aren't necessarily running the same version of the same software. Except, unlike ETF, Protobufs can be decoded without the backing of a garbage-collected language runtime (e.g. in C++ nodes.)

I'm not saying Protobufs are _the best_ , but you have to understand the problem they're solving—the "rivals" to Protobufs are Cap'n Proto and Thrift, not JSON. JSON can't even be used to do a lot of the stuff Protobufs do (like talk to memory-constrained embedded kernels running on SDN infra.)

> and Protobuf doesn't contaminate your codebase

Just like in HTTP web-app codebases, a codebase _properly_ designed to interact with an [https://en.wikipedia.org/wiki/Enterprise_service_bus](https://en.wikipedia.org/wiki/Enterprise_service_bus) will tend toward Hexagonal architecture—you have your own logic, and then you have an "RPC gateway" that acts as an MVC system, exposing controllers that decode from the RPC format and forward requests into your business logic; and then build responses back into the RPC "view" format.

Once you have that, there's no point in having your application's parsed-out representation of the RPC format be different than the on-the-wire RPC format. The only thing that's touching those RPC messages is your RPC gateway's controller code anyway. So why not touch them with as little up-front design required as possible?

<comment><USER6> In the context of gRPC, it's true that Protobuf is intended to be a wire protocol, but the argument would be more convincing if the code generator toolchain Google fostered created code that integrated better with the languages that people use.

Typically, the data types and interfaces generated by these tools are so poor that you need to build another layer on top that translates between "Protobuf types" and "native types", as you describe in your comment, and shields the application from the nitty-gritty details of gRPC. So investing in gRPC means that when you've generated, say, Go files from your .proto files, you're only halfway done. Protobuf in itself introduces a kind of impedance mismatch, a kind of stupid in-bred cousin whose limited vocabulary has to be translated back and forth into proper language.

So gRPC/Protobuf solves something important at the wire level, but developers really want to productively communicate via APIs, and so what you have is just half the solution.

I wish the Protobuf/gRPC toolchain were organized in such a way that the generated code could actually be used as first-class code. Maybe similar to how parser generators like Lex/YACC or Ragel work, where you provide implementation code that is threaded through the output code.

</comment></comment></comment></comment><comment><USER15> This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like

"Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses."

According to here [https://www.quora.com/How-many-software-engineers-does- Googl...](https://www.quora.com/How-many-software-engineers-does-Google-have) there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be... $3.3 billion a year? And this was 2014, and doesn't include stock compensation

Don't get me wrong - there are definitely problems with protobufs. But this post is citing a lot of stuff to back itself up that doesn't add up.

Also, required vs optional - definitely only optional, there should be nothing required. So I flat out disagree with that one.

<comment><USER16> Google's annual revenue was $110 billion in 2017 [1]. Even if headcount doubled and salary has increased, that's $7 billion a year. That's not peanuts, but at a company level it's not a massive expense.

[1]: [https://www.androidauthority.com/alphabet-q4-2017-earnings-8...](https://www.androidauthority.com/alphabet-q4-2017-earnings-834774/)

<comment><USER15> True. I'm also guessing this number changes a lot with stock grants (that number was just base pay), and that yearly number seems pretty low (NYT has the entry level engineer at 124,000 [https://www.nytimes.com/2017/09/08/technology/google- salarie...](https://www.nytimes.com/2017/09/08/technology/google-salaries- gender-disparity.html)) so glassdoor might not have good data.

I wouldn't be surprised if it was much more than 6%.

<comment><USER16> I also wouldn't be surprised if it was more than 6% - but I very much doubt it could be over 12-15%. That's still not a very high percentage of revenue. It's most definitely a significant cost, but unlikely to be their biggest cost.

</comment></comment></comment></comment><comment><USER17> > _Protobuffers correspond to the data you want to send over the wire, which > is often related but not identical to the actual data the application would > like to work with.(...)_

> _Option 1 is clearly the "right" solution, but its untenable with > protobuffers. The language isn't powerful enough to encode types that can > perform double-duty as both wire and application formats. Which means you'd > need to write a completely separate datatype, evolve it synchronously with > the protobuffer, and explicitly write serialization code between the two._

I'd say, get used to this. This is exactly the same story you have with ORM/ActiveRecord. You either treat it as a different layer and write the translation code that's needed, or mix it with internal logic and pay the price of messy codebase later. The problem isn't protobuff(or ORM)-specific. It's just a consequence of the fact that your internal business model and the data you want to save/send are usually two different things.

<comment><USER18> The worst is when people deny that is the truth.

"Then model is like 80% the same, just use the same class!"

<comment><USER17> Yeah. It's not always obvious at the beginning, though. In my second job, I didn't notice this until it grew to the point we couldn't really implement anything new without spending 90% of the time on the spaghetti the business logic became. My excuse is that we were all new at this kind of work, but the lesson is clear: just don't do that.

</comment></comment></comment><comment><USER19> Put me firmly in the camp of "optional fields are bad." I believe all fields should be required.

I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility.

The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields.

It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.

<comment><USER20> If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non- language specific binary formatters.

<comment><USER14> > every application has to be updated when a field is added, even if they do > not use that field

This is when, in a protocol, you reach for the hammer called "extensibility."

In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMessage.

Then, just make the semantics of your format such that a sum type can have a pragma in its IDL indicating that it isn't required to be decodable for the product-type containing it to be successfully decoded. I.e., the sum type becomes a (DecodeResult = Decoded(sum type) | Opaque(raw wire data)).

Opaque (unrecognized) extension data can't be manipulated by the program, but it can losslessly survive a trip back through the wire encoder. So you can choose to pass it through in your middleware, or not.

Boom: you've reinvented the "chunks" concept of the [https://en.wikipedia.org/wiki/Interchange_File_Format](https://en.wikipedia.org/wiki/Interchange_File_Format), as seen in PNG and ELF.

<comment><USER21> I believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.

<comment><USER14> Nope—there's a very important difference in the two approaches. You have to _choose_ to add the kind of extensibility I was describing to a particular type, in advance, as part of that type's original specification. You have no choice in having optional fields.

With Protobuf optional fields, even if you formally specify a protocol where some particular message is absolutely "final" and will never be extended, anyone can just throw an optional member on there when sending one to you, and your implementation won't reject the struct. This is a big problem if you're trying to specify types that really map one-to-one to fixed-size structs you'll use in the business logic of your non-memory-managed language.

With explicit extension points, you get to dictate where and when extension happens; and, as well, you get to formalize exactly what extensions look like (i.e. you don't have to allow for un-recognized extensions; you can simply have extensions be a finite set of known new extensions, where everything else is a real decode error.)

<comment><USER21> ... however, it's a great feature if the use case is "protocol for client- server network communication," which was the design goal for protobuffers. It follows the "permissive in what you accept, strict in what you emit" design philosophy.

</comment></comment></comment></comment></comment></comment><comment><USER22> Hmmm, I suppose I objectively agree with some of the points the author made, but as someone who works with protocol buffers daily, those issues never actually come to be problematic in practice. In fact, I have nothing but positive things to say about protocol buffers and find them pleasant to work with. Definitely a step up from sending raw JSON down the wire.

Granted, the application I'm working on is fairly boring/vanilla so maybe I don't feel the pain points that come from going off the beaten path.

<comment><USER18> Maybe for the benefit of everyone could you elaborate more on your vanilla application. So we can understand the context of why your problem-free experience differs from the author's or others'.

</comment></comment><comment><USER23> > Give the ability to parameterize product and coproduct types by other types.

This is "worse is better" at work. protobufs are inconsistent and types map to Java. But it's a feature because Java is a language designed for for your average programmer. It avoids words like "coproduct types". Because most people's eyes will glaze over that and their mind will think "this is too complicated, there must be something simpler, like a map or something..."

> "Google is filled with the world's best engineers," and that "anything they > build is, by definition, not built by amateurs.

That's true too though. It's a major factor when picking tools. So protobufs are used because of Google. gRPC is used because of Google. Kubernetes because of Google. Angular because of Google. Go became popular because of Google etc. Of course there are technical advantages to those things and they are great tools, but I think quite often the reason is simply ... because it's Google.

</comment><comment><USER24> It's abundantly clear why this author lasted only a year at Google. Aside from using the non-idiomatic "protobuffers" ...

"nothing wants to inspect only some bits of a message and then forward it on unchanged"

In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic on the wire with a different type, but is much cheaper to parse (example: a deeply nested structure where all of the fields of interest are defined at the top level).

Nobody even within Google will argue that no mistakes have been made with proto. Proto3 is generally acknowledged to have been a big mistake and it doesn't have a lot of traction within Google. Map and OneOf are both basically antagonistic to high performance application code, and people avoid those for that reason. But the author's arguments are pretty weak and his dismissal of Google itself as an existence proof is insufficiently supported.

<comment><USER25> Ah right... when you can't attack the ideas, attack the author. Sandy is a Haskell enthusiast, and it is not a surprise that his criticism of protobuf is inspired heavily by that.

The argument presented is that protobuf does not attempt to even replicate the best practices we already have in terms of data representation. It makes sense why languages like C are constrained in their data representations -- they want to be close to the machine. However, a language meant literally to specify data really ought to be able to handle the basic competencies of its field (like sum and product types and polymorphism) with ease.

There is no excuse in 2018 for a data language without polymorphism and without product and co-product types. Compatibility with legacy or substandard languages is not an excuse for poor design. Compatibility with needed legacy languages should be the thing that's tacked on, not basic functionality.

<comment><USER24> Sure that makes sense if you can write off C++ as a "legacy" language, which is fine if you just can't get your mind wrapped around the scale at which Google operates. Due to the nature of weighted averages, you can't just write off something as being a "Google-only problem". Google and its peers like Amazon and Facebook own a very large fraction of the world's computing resources.

I think the overall mistake you and the author are making is assuming it is desirable for one program examining an encoded message to respect the type system of the program that produced it. That assumption is not obviously correct. The flexibility to just treat a vector of numbers as a vector of numbers, or to skip it, or ignore it, etc are pretty important in practice. That is why protobuf is defined at the level where it is defined. It encodes numbers and strings on the wire. It happens to be a fairly good way to represent search data. It brings no CS academic wankery to the party. It's very practical.

<comment><USER25> C++ is a legacy language when it comes to data representation. Its ideas on what constitutes 'data' are from a different era. That is not a criticism of C++ or its utility. It is just a fact that data representation has changed a lot since the time of C++s development.

Products and coproduct types are not academic wankery.

<comment><USER24> The machine does not think about category theory. The machine thinks about numbers. It can add them together! The way the machine thinks about numbers has not meaningfully changed in 40+ years.

<comment><USER25> Product and coproduct types are not category theory (what is that?)

</comment></comment></comment></comment></comment></comment><comment><USER26> > All you've managed to do is decentralize sanity-checking logic from a well- > defined boundary and push the responsibility of doing it throughout your > entire codebase.

One wonders if the author has practical experience with evolving a complex system composed of multiple independent services without downtime. Centralized sanity checking conflicts with transitioning services stepwise from e.g. foo.a foo.new_a by first deploying producers that create send both foo.a and foo.new_a and then phasing out foo.a after all consumers are updated.

Services which communicate via some general purpose, strongly typed schema language that's still lax enough to evolve backwards-compatibly are far easier to understand, maintain and evolve in my experience than alternatives such as json rest apis, overly tied down static or completely ad hoc protocols.

<comment><USER27> He meant "centralised in the serialisation library". I don't see how that conflicts with versioned evolution, especially because protobufs historically _did_ verify the presence of fields.

The story behind "required considered harmful" at Google is really quite shameful. I was there at the time and couldn't quite believe people were making that argument. Beyond all the logical problems with it, they were basically saying Jeff Dean didn't know what he was doing when he made that decision (except they never actually said it explicitly because the notion itself would have been subjected to ridicule).

I used to like protobufs but I wouldn't use proto3. It seems to have regressed over time instead of getting better. Partly this is because the wire format is not evolvable. Protobufs do not have a "this is protobufs v1" header in them. This also explains the type system issues: Google cannot/will not evolve the wire format so each new version of protobufs is just different tooling and APIs over it.

<comment><USER28> I was there at the time too and couldn't believe people were defending required. I had already run into enough problems with it that I stopped using it earlier, and I worked on a relatively small system with not that many moving parts! I can hardly imagine how much pain required must have caused in something like the indexing pipeline.

The thing I realized was that even if a field is present, you almost always still want to do further checks on it before declaring the whole thing valid and good to go. "required" just tells you that it's an integer, for example, it doesn't tell you if it's in the range [0, 1000). Or that a string is a valid filesystem path. Or that two parallel arrays are the same size. You have to do those things anyway, so the has_... check is no big deal. And making everything optional is really nice when you have to make large changes to your protocol and don't want to have to carry around dummy values forever.

Protocol buffers weren't designed from scratch by Jeff Dean, they evolved from various ad-hoc ways of specifying query parameters (which, ok, were probably mostly written by Jeff), so I don't think it reflects badly on him to say that required was a mistake. Groups were a mistake in retrospect too, but no one thinks that means Jeff is an idiot.

<comment><USER28> Btw, I realize this argument is very similar to the argument for dynamic typing + lots of unit tests instead of static typing: "you have to write the tests anyway, and the tests will check the types".

I'd like to note that even if you're a static typing fan and don't believe that argument for code, you can still believe it for serialization and protocols, because data schemas and protocols evolve in different ways than code does (they're almost duals of each other).

</comment></comment></comment></comment><comment><USER4> Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways.

This article appears to be written by a programming language design theorist who, unfortunately, does not understand (or, perhaps, does not value) practical software engineering. Type theory is a lot of fun to think about, but being simple and elegant from a type theory perspective does not necessarily translate to real value in real systems. Protobuf has undoubtedly, empirically proven its real value in real systems, despite its admittedly large number of warts.

The main thing that the author of this article does not seem to understand -- and, indeed, many PL theorists seem to miss -- is that the main challenge in real-world software engineering is not writing code but _changing_ code once it is written and deployed. In general, type systems can be both helpful and harmful when it comes to changing code -- type systems are invaluable for detecting problems introduced by a change, but an overly-rigid type system can be a hindrance if it means common types of changes are difficult to make.

This is _especially_ true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse.

I don't have time to address all the author's points, so let me choose a few that I think are representative of the misunderstanding.

> _Make all fields in a message required. This makes messages product types._

> _Promote oneof fields to instead be standalone data types. These are > coproduct types._

This seems to miss the point of optional fields. Optional fields are not primarily about nullability but about compatibility. Protobuf's single most important feature is the ability to add new fields over time while maintaining compatibility. This has proven -- in real practice, not in theory -- to be an extremely powerful way to allow protocol evolution. It allows developers to _build new features_ with _minimal work_.

Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time, hence the "required considered harmful" manifesto. In practice, you _want_ to declare all fields optional to give yourself maximum flexibility for change.

The author dismisses this later on:

> _What protobuffers are is permissive. They manage to not shit the bed when > receiving messages from the past or from the future because they make > absolutely no promises about what your data will look like. Everything is > optional! But if you need it anyway, protobuffers will happily cook up and > serve you something that typechecks, regardless of whether or not it 's > meaningful._

In real world practice, the permissiveness of Protocol Buffers has proven to be a powerful way to allow for protocols to change over time.

Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use.

> _oneof fields can 't be repeated._

 _(background: A "oneof" is essentially a tagged union -- a "sum type" for type theorists. A "repeated field" is an array.)_

Two things:

1\. It's that way because the "oneof" pattern long-predates the "oneof" language construct. A "oneof" is actually syntax sugar for a bunch of "optional" fields where exactly one is expected to be filled in. Lots of protocols used this pattern before I added "oneof" to the language, and I wanted those protocols to be able to upgrade to the new construct without breaking compatibility.

You might argue that this is a side-effect of a system evolving over time rather than being designed, and you'd be right. However, there is no such thing as a successful system which was designed perfectly upfront. All successful systems become successful by evolving, and thus you will always see this kind of wart in anything that works well. You should _want_ a system that thinks about its existing users when creating new features, because once you adopt it, you'll be an existing user.

2\. You actually _do not want_ a oneof field to be repeated!

Here's the problem: Say you have your repeated "oneof" representing an array of values where each value can be one of 10 different types. For a concrete example, let's say you're writing a parser and they represent tokens (number, identifier, string, operator, etc.).

Now, at some point later on, you realize there's some additional piece of data you want to attach to every element. In our example, it could be that you now want to record the original source location (line and column number) where the token appeared.

How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each _containing_ a oneof, so that you could add a new field to that message. But because you didn't, you're probably stuck creating a parallel array to store your new field. That sucks.

In every single case where you might want a repeated oneof, you _always_ want to wrap it in a message (product type), and then repeat _that_. That's exactly what you can do with the existing design.

The author's complaints about several other features have similar stories.

> _One possible argument here is that protobuffers will hold onto any > information present in a message that they don 't understand. In principle > this means that it's nondestructive to route a message through an > intermediary that doesn't understand this version of its schema. Surely > that's a win, isn't it?_

> _Granted, on paper it 's a cool feature. But I've never once seen an > application that will actually preserve that property._

OK, well, I've worked on _lots_ of systems -- across three different companies -- where this feature is _essential_.

<comment><USER29> > In every single case where you might want a repeated oneof, you always want > to wrap it in a message (product type), and then repeat that. That's exactly > what you can do with the existing design.

But I could use the same argument for repeated _anything_. Why even allow repeated primitives at all, if your argument is convincing?

<comment><USER4> Repeated primitives were in the first version, before anyone knew better. I'd argue against them in new code.

In Cap'n Proto, I made it possible to upgrade a repeated primitive to a repeated strict whose first field is the primitive type, to avoid this trap.

</comment></comment></comment><comment><USER30> I'm using protobufs for an embedded product, and I think they're awesome. Not the solution to everything - certainly - but definitely a highly useful tool for the right circumstances.

In my case, I have a single .proto file that serves as the input to two software systems: an embedded OS running on ESP8266 hardware, and the web interface served up by that OS to edit OS-specific features/variables/etc.

So, I define in one file, my types that I want shared both with the embedded OS, and with the web interface. On compile, both systems get their relevant generated code, and both systems are therefore able to communicate easily with each other. The user can use the embedded OS to change values coded in the protobuf, and the web interface that the system serves up immediately knows about the change. The web interface, by the way, generates the UI based on the protobuf declarations - meaning that adding a new var to the project is a simple matter of editing the .proto file, and doing a full clean build. Very, very nice: one change, multiple systems updated, and it produces an auto- generated UI for the new vars.

This kind of interoperability is very key - I'm sure there are alternative ways of doing this, but I can't think of anything as smooth and easy to integrate as protobufs have been .. maybe if we'd put a Lua VM in the mix somewhere, this'd be better, but .. protobufs are pretty tight.

</comment><comment><USER15> Mostly lost me at "Make all fields in a message required. This makes messages product types."

Your constraints on protocols can change over time and context, changing something that is required to something that is optional can cause crashes in prod. Unless somehow the 'optional' design has a way to handle that? You also will have to do custom validation anyways, as your storage format will never be able to enforce all constraints you care about.

Also, it would seem like the 'debate' of optional vs required is harmful was resolved, as proto3 makes everything optional.

<comment><USER31> If you would continue reading, there is an explanation of how optional would work almost immediately after this quote.

<comment><USER15> Right - this is the same as having the possibility of both required and optional. I'm saying there should not be any possibility of a 'required' field

<comment><USER31> In context of current protobuf design required was a mistake and optional is clearly better, but author argues about grand type system that is based on different principles, including stronger validation.

Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.

<comment><USER32> From a practical standpoint the problem is that "required" handles a trivial subset of message validation.

I mean, I'm not going to claim that it never happens that your only constraint on a valid value of a message field is "present", but you quite often want to be able to require that one of three fields is set, or a number be between 0 and 1048576, or that a field be equal to an existing user ID, it that a string contain at least one printable non-punctuation, non-space letter.

So no matter your RPC message parsing code, you're going to need a custom bit of code in each of your handlers to say "This isn't a valid message, fuck off". Enforcing "this field should be of this primitive type" saves a lot of time, but it turns out that "this field should exist" doesn't save that much because you still have to write "... and have a sensible value" in your own code.

So you have a language feature which causes problems sometimes and doesn't really help much.

<comment><USER33> > it turns out that "this field should exist" doesn't save that much because > you still have to write "... and have a sensible value" in your own code.

Disagree. Dozens of "x should exist" checks are tedious to write and even more tedious to read, obscuring the more relevant business-specific validation logic. Better to move the low-hanging fruit into the message format.

</comment></comment></comment></comment></comment></comment><comment><USER34> There are enough voices on the net these days that we don't have to put up with this kind of attitude anymore. This sort of stuff was "just the way things were" when we were teenagers visiting forums ran by teenagers. I don't know if the guy is right or wrong, I've just got better things to do than to sift through all the vitriol to get to whatever point he is trying to make.

<comment><USER18> I think a lot of people are still attracted to the "hostile genius" persona, or whatever. Maybe it justifies their own poor behavior.

</comment></comment><comment><USER35> These objections are interesting. I have mixed opinions on protos, I think overall I'm mostly in favor. I'm a bit confused by this set of objections though.

While oneof fields cannot be repeated, oneof fields can be arbitrary protos, so they can _contain_ repeated fields. In other words, you can have a (pseudo- proto)

        oneof {
          RFoo {
            repeated Foo;
          }
          RBar {
            repeated Bar;
          }
        }

so in practice this isn't a restriction. If anything, its a (very minor) api wart.

As for maps, I'd say don't use them. I have, I don't think they're very useful, except for prototyping things. `repeated (string key, string value)` is just as useful for prototyping things, and you should be quick to promote things to fields. Optional fields aren't costly.

I'm also in the minority who thinks that the `has_foo` methods are a smell, and everything should _always_ be set to a default value[0]. If it really matters, one can define a maybe type via a oneof, but that should be the exception, not the norm. Most people disagree with that though, so maybe I'm crazy and you should ignore me.

[0]: But if that's not the case, everything should be optional, nothing required.

<comment><USER36>
        repeated(oneof(Foo, Bar))

is not the same as

        oneof(repeated(Foo), repeated(Bar))

<comment><USER35> Correct, both of those are possible to represent as is though. Each oneof just requires and extra proto to wrap it.

So you have

        repeated OneaofWrapper {
          oneof {
            Foo
            Bar
          }
        }

Or what I did in my previous comment.

</comment></comment></comment><comment><USER37> > By far, the biggest problem with protobuffers is their terrible type-system. > Fans of Java should feel right at home with protobuffers ...

Yeah right, because insulting _only_ Protocol buffer and Google engineering won't be edgy enough for a blog post... :/

</comment><comment><USER38> "Make all fields in a message required. This makes messages product types."

This article was clearly written by amateurs.

</comment><comment><USER39> I've used protocol buffers on a large project, evolving the messages over time. They are a fine choice.

The tone of this article is a little less than objective which I feel discredits it a bit. But hey it made it to the front page of hacker news so...

</comment><comment><USER40> > Make all fields in a message required. This makes messages product types.

Now you lost one of the biggest benefits of using protobufs, the possibility to make things backwards/forwards compatible. When you add a field to the proto, a server using the new version of the proto will fail to deserialize a request from a client using an older version that's missing that field.

The author mentions this later, but doesn't really address it/propose a better solution, as far as I can see.

</comment><comment><USER41> I can not really take someone who calls Jeff Dean an amateur seriously...

</comment><comment><USER42> Once I was faced with a task of serializing object hierarchy to a binary on disk[1], with the requirement of transparent backward compatibility (and desired forward compatibility), and the ability to be accessed from several programming languages in the future.

Protobufs looked promising because of their forward compatibility promise, but it looked like they only work if you write your code from the get-go using them. Rewiring a 15-year-old codebase would be a nightmare, so that was not an option.

Boost::serialization[2] looked promising, because it's non-intrusive -- but even "dumb" forward compatibility (skipping unknown fields) is not supported -- and reading its archives from other languages seems nontrivial.

So I went ahead and rolled out a custom framework with C++ template trickery which was non-intrusive, wrote to SQLite (with a fixed ORM schema), and did the job.

I wondered for a while if I was re-inventing the wheel, but in the end, I am glad I did it:

1\. It wasn't _that_ much work - especially compared to rewiring everything for Protobufs

2\. It made me appreciate the goals and the API of Boost::serialization better - mine ended up being not too dissimilar. It was a great way to learn that - or any other- serialization API - without even looking at it! Now it's "let's see how they did ___" instead of "what does it do and why is this here".

Reading this article made me glad again: partly, for not depending on Protobuf -- and partly, for it having some of the same design flaws that I have (like having difficulty telling a zero-initialized value from a value not present in the data).

But the take-away for me was that sometimes, writing a custom framework _is_ justified. Not all tools are suited for all purposes. Figuring out what one wants, and checking the list against the goals of a project can save a lot of headache.

[1] Binary was a necessity because some of the data is large arrays of full- precision floating point numbers, on the scale of 10-100MB. This was scientific data, no precision loss on saving/loading was acceptable.

[2][https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc...](https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc/index.html)

</comment><comment><USER43> I think it was Churchill said that "protobufs are the worst form of data serialization, except for all those other forms that have been tried from time to time." I really don't understand why you would ever want to serialize data without a description of what can be in the serialized data blob. And yes, I do care about the difference between a single precision float, a double precision float, or an int64. Thank you, typing system.

</comment><comment><USER44> The main point that people are missing is that experienced engineers don't want to work with people who think like the author of this article.

Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages.

No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases.

If you are building a system where your data can be described by protobufs, it may be a good choice. If your data structures don't mash up well and you have to manipulate them heavily, protobufs may be a bad choice.

Instead of pointing out use cases where a different serialization format may be better than protobufs and use cases where protobufs are better, and why, the author is spouting dogma about how protobufs are bad for every use case.

Be wary of working with developers who prefer to argue about why they hate certain technologies instead of providing useful data and ways to solve problems. You don't always have to solve a problem just because you are aware of it but don't go shouting from the rooftops that a technology sucks for every use case under the planet when that's obviously not the case. The author's opinion is more of: I don't like protobufs. Not: protobufs are wrong

<comment><USER14> The author isn't trying to say either of those things, really.

I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a _dogma_ for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad _dogma_ to be stuck with. That is, when they don't work for a use-case, they _really_ don't work for that use-case—and an ESB means having to try to shove a wire format into pretty much _every_ use-case. So "when it's bad, it's _really_ bad" is a bad property for a format aimed specifically _at_ being an enterprise's ESB-interchange-format dogma to have.

(Contrast to, say, JSON-RPC, which I'd place on the opposite end of the scale being described here. JSON-RPC is mediocre at best in terms of data fidelity, performance, etc., but it's _inoffensive_ —when it's bad, it's no worse than when it's good. That makes it a _good_ choice of ESB dogma... even though, in engineering terms, it sucks!)

<comment><USER45> No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot.

It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original.

As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.

<comment><USER14> I mean, all the arguments to support my conclusions, are _their_ arguments. If someone spends 10 pages giving you facts and anecdotes and other types of data, and then uses them to support a "dumb" conclusion, they've still done useful work. They don't realize what exactly it is their data proved, but they _did_ do the work required to prove something. Those facts and anecdotes support a conclusion, whether or not they feed it to you at the end/in the abstract.

And, that being said, I think they're right about the conclusion, too. A shorter way to say "Protocol Buffers are a bad choice of common ESB-bus format, even though being an ESB-bus format is the primary thing they're for and what everyone tries to use them for" is "Protocol Buffers are a wrong design." If something doesn't work when used to do the thing it's _advertised to do_ , then it's broken, even if it _can_ do something _else_.

The only difference between "Protobuffers Are Wrong" and my conclusion is that I'm making the _implicit_ context of their argument _explicit_. Read between the lines of their argument—they _are_ talking about the use of protocol buffers (specifically, gRPC) in an ESB-bus common-format scenario. None of their arguments make sense if they aren't.

</comment></comment></comment></comment><comment><USER46> Many of the restrictions are there because you need to operate as well as you can at the intersection of all target language type systems.

</comment><comment><USER47> Agree with other ppl here, an unnecessarily violent rant, and the point the author makes is kind of lost in the title "Protobuffers Are Wrong". What they really mean is "I think Protobuffers type system could be better".

I recently saw a talk by Rich Hickey about Effective Programs [1]. The talk explains why Hickey favors dynamic types, with EDN [2] and transit [3] proposed as an alternative to more statically typed data exchange formats like protobufs and less structured ones like Json (the talk explains the reasons, nearing the end I think).

1: [https://www.youtube.com/watch?v=2V1FtfBDsLU](https://www.youtube.com/watch?v=2V1FtfBDsLU) 2: [https://github.com/edn-format/edn](https://github.com/edn-format/edn) 3: [https://github.com/cognitect/transit- format](https://github.com/cognitect/transit-format)

</comment><comment><USER48> > Option 1 is clearly the "right" solution, but its untenable with > protobuffers. The language isn't powerful enough to encode types that can > perform double-duty as both wire and application formats. Which means you'd > need to write a completely separate datatype, evolve it synchronously with > the protobuffer, and explicitly write serialization code between the two. > Seeing as most people seem to use protobuffers in order to not write > serialization code, this is obviously never going to happen.

This last is not the case. I've only ever seen it used when people a) already _have_ written serialization code that works with XML/JSON, and b) don't want to write code to serialize to a _binary_ format.

JSON also doesn't let you use objects as object keys, or have any form of polymorphism. It seems like the author wants protobufs to be an RPC system with a modern type system, when in fact it is a _data interchange format_.

<comment><USER49> But the author is only talking about data types. I don't see any reason it shouldn't be designed with composition in mind.

The only restriction I can think of that JSON imposes (in terms of composition) is that keys must be strings. And I don't necessarily think JSON is a great serialization format either.

</comment></comment><comment><USER50> I disagree severely with large swaths of this article. The biggest problem is here:

>Of course, the actual serialization logic is allowed to do something smarter than pushing linked-lists across the network---after all, implementations and semantics don't need to align one-to-one.

I _like_ that I can read a pb dump in hex and have it map quite closely to the protocol definition, rather than being viewed through heaping piles of abstractions. This is only going to make it more confusing and more work to use, with the only gain being to make some functional programmer feel squishy about their beautiful type system.

I largely agree with the "The Lie of Backwards- and Forwards-Compatibility" section. You can feed /dev/urandom into nearly any protobuf definition and it'll happily give you nonsense back, no errors. I understand why this is the case, and I'm not sure it should change, but it's bitten me several times.

</comment><comment><USER51> I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider this then there is only a very very narrow band in which it makes sense to use protobuffers vs JSON (which is a defacto standard) or lesser known solutions that provide superior performance.

<comment><USER52> The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.

<comment><USER24> Proto doesn't really have a schema. You can encode a proto of type X and tell your program to decode it as type Y, and it might work. Might not, but might. You can easily imagine messages that are isomorphic on the wire. E.g.

message X { optional string foo = 1; } message Y { optional X foo = 1; }

Encoded buffers of X will decode as Y just fine.

<comment><USER52> Sure, but it has more of a schema than JSON does :-) That's why I said "you can be REASONABLY sure you've got a valid message" (if it decodes successfully). It might not be the actual message you THINK it is, but at least all fields are going to be there.

Then again, if you're accepting arbitrary bytes off the wire and blindly assuming they're going to be the correct proto as long as they decode, you're... most stubby services I worked on. :-)

</comment></comment></comment></comment><comment><USER53> I like protobufs. I've only ever used them as a serialization format for save and map files for some C# games that I tinker away at, but I've been pretty happy using protobuf-net[1]. File size is smaller than JSON produced by JSON.NET or using the built-in binary serializer, much less XML. And it couldn't be much easier to use.

[1] [https://github.com/mgravell/protobuf- net](https://github.com/mgravell/protobuf-net)

</comment><comment><USER54> I'll give you one example where the protobuf design helped out a lot: GWT Protobuffers. The GWT Compiler has a version of Protobufs that transpiles all of the protos into Overlay types of naked JSON objects.

You get a statically typed Java interface, but underneath it a raw JSON object, and deserializing the proto is basically JSON.parse() + a cast.

e.g. something like this

MyProto foo = MessageDeserializer.deserialize(serializedProto); console.log(foo.getFieldFoo());

compiles to this JS:

var foo = JSON.parse(serializedProto); console.log(foo[1]);

</comment><comment><USER55> > _Fields with scalar types are always present. Even if you don 't set them. > Did I mention that (at least in proto3) all protobuffers can be zero- > initialized with absolutely no data in them?_

I don't this complaint. You have to initialize data with _something_. proto2 had to "solve" this problem in C/C++/Go by making everything a pointer. How is dealing with null the "sane" case against zero-initialization?

<comment><USER6> To be fair, in C++ you could use std::optional or something else that's type- safe.

Go has its own philosophy about zero values which is controversial, but at least with pointers you do get to express optional values. And it's not like you can't work around the ergonomic awkwardness that arises:

      type Post struct {
        Title *string
      }

      func (p *Post) GetTitle() (string, bool) {
        if p.Title == nil {
          return "", false
        }
        return *p.Title, true
      }

      func (p *Post) SetTitle(s string) {
        p.Title = &s
      }

Hardly _elegant_ , but this is Go.

Go also run into the same issue when encoding and decoding JSON. Most languages do distinguish between empty string and a missing string, but not Go, which makes it hard to validate anything. I wrote a code generator for JSON Schema [1] recently, which applies validations while deserializing, and has to resort to a rather low-tech trick to do so:

      type Post struct {
        Title string `json:"title"`
      }

      func (v *Post) UnmarshalJSON(b []byte) error {
        var raw map[string]interface{}{
        if err := json.Unmarshal(b, &raw); err != nil {
          return err
        }
        if _, ok := raw["title"]; !ok {
          return errors.New("field title: must be set")
        }
        type plain Post
        var p plain
        if err := json.Unmarshal(b, &p); err != nil {
          return err
        }
        *v = Post(p)
        return nil
      }

(Clearly there are slightly more refined and faster ways to do this, but not that easily without importing third-party code, which I wanted to avoid.)

C and Go are in the minority here. This doesn't come up in languages like Rust, Swift, TypeScript, Nim, Haskell, OCaml, C#, F#, or Java >= 8, all of which have some sort of optional support (algebraic data types or built-in). For example, TypeScript:

      interface Post {
        title?: string
      }

Or Rust:

      struct Post {
        title: Option<&str>
      }

[1] [https://github.com/atombender/go- jsonschema](https://github.com/atombender/go-jsonschema)

<comment><USER56> so, just for fun, here's a way to do that check without double unmarshalling and allocating maps and such for everything under your struct (also does a check for _extra_ fields, but you could pull that out if you want):

        type Post struct {
        	Title string `json:"title"`
        }

        func (v *Post) UnmarshalJSON(b []byte) error {
        	dec := json.NewDecoder(bytes.NewReader(b))
        	required := map[string]struct{}{
        		"title": struct{}{},
        	}
        	tok, err := dec.Token()
        	if err != nil {
        		return err
        	}
        	if d, ok := tok.(json.Delim); !ok || d != '{' {
        		return errors.New("Expected object")
        	}
        	for {
        		tok, err := dec.Token()
        		if err != nil {
        			return err
        		}
        		if d, ok := tok.(json.Delim); ok && d == '}' {
        			break
        		}
        		switch tok {
        		case "title":
        			delete(required, "title")
        			err := dec.Decode(&v.Title)
        			if err != nil {
        				return err
        			}
        		default:
        			(*v) = Post{}
        			return errors.New(fmt.Sprintf("Unexpected field %s", tok))
        		}
        	}

        	if len(required) > 0 {
        		(*v) = Post{}
        		return errors.New(fmt.Sprintf("Missing %v required fields", len(required)))
        	}
        	return nil
        }

<comment><USER6> Thanks, that's neat! The challenge here is that I also need to validate values (e.g. support minimum/maximum), not just within structs, but also standalone values, which means an UnmarshalJSON directly on the type. I might end up doing something like your example, though. (Rejecting extra fields is on my list!)

<comment><USER56> the json.NewDecoder stuff does still trigger unmarshals of the types in the struct and such: i.e.

        package main

        import (
        	"bytes"
        	"encoding/json"
        	"errors"
        	"fmt"
        )

        func main() {
        	fmt.Println("Hello, playground")
        	var p Post
        	err := json.Unmarshal([]byte(`{"title": "test"}`), &p)
        	fmt.Println(err, p)
        	err = json.Unmarshal([]byte(`{"title2": "test"}`), &p)
        	fmt.Println(err, p)
        	err = json.Unmarshal([]byte(`{}`), &p)
        	fmt.Println(err, p)
        	err = json.Unmarshal([]byte(`{"title": "long test"}`), &p)
        	fmt.Println(err, p)
        }

        type Post struct {
        	Title ShortTitle `json:"title"`
        }

        func (v *Post) UnmarshalJSON(b []byte) error {
        	dec := json.NewDecoder(bytes.NewReader(b))
        	required := map[string]struct{}{
        		"title": struct{}{},
        	}
        	tok, err := dec.Token()
        	if err != nil {
        		return err
        	}
        	if d, ok := tok.(json.Delim); !ok || d != '{' {
        		return errors.New("Expected object")
        	}
        	for {
        		tok, err := dec.Token()
        		if err != nil {
        			return err
        		}
        		if d, ok := tok.(json.Delim); ok && d == '}' {
        			break
        		}
        		switch tok {
        		case "title":
        			delete(required, "title")
        			err := dec.Decode(&v.Title)
        			if err != nil {
        				return err
        			}
        		default:
        			(*v) = Post{}
        			return errors.New(fmt.Sprintf("Unexpected field %s", tok))
        		}
        	}

        	if len(required) > 0 {
        		(*v) = Post{}
        		return errors.New(fmt.Sprintf("Missing %v required fields", len(required)))
        	}
        	return nil
        }

        type ShortTitle string

        func (st *ShortTitle) UnmarshalJSON(b []byte) error {
        	var tmpst string
        	err := json.Unmarshal(b, &tmpst)
        	if err != nil {
        		return err
        	}
        	if len(tmpst) > 6 {
        		return errors.New("Title too long!")
        	}
        	*st = ShortTitle(tmpst)
        	return nil
        }

</comment></comment></comment></comment></comment><comment><USER57> If protobuffers are wrong, then I don't want to be right.

</comment><comment> d--b Now to the real question: what is the best alternative?

</comment><comment><USER18> One of the most surprising features of Proto mentioned in this thread (at least for me/my background) is the desire to forward on a message you receive. Possibly a message you can't read/parse, or don't want to. And you want to forward that whole payload to the next person.

What kind of systems have this architecture? Why would you forward on a message and not know its contents 100%?

<comment><USER58> > What kind of systems have this architecture?

The Web, for example.

> Why would you forward on a message and not know its contents 100%?

In the web architecture you can observe following cases:

* proxying, which itself is an umbrella term for a lot of different things, such as:

        * request routing

        * filtering

        * access control

        * logging

     * caching

     * load balancing

For example, a very typical scenario for load balancing is "sticky sessions". Basically, a reverse proxy (e.g. nginx) needs to inspect a HTTP request and analyze certain headers to find the server to forward request to (which already has user's session). Note that nginx can just pass headers without understanding what they mean. And it simply cannot understand the meaning of query part of URL or message body.

So, apparently, protobuf gives you a lot of HTTP's flexibility in a binary RPC format. So e.g. you can make a cache server which will cache responses without understanding them, so you can evolve your backend without updating cache server each time you add a new field to a response.

What I don't understand, however, is why they couldn't control that behavior using compiler options. Sometimes you want to be lax with validation, sometimes you don't. Why not give programmer a choice...

</comment></comment><comment><USER59> Protobuffers always seemed like an interesting approach but every time I've tried to use in a prototype I've ended up deciding against the added overhead of including them. And then I use JSON because I simply wanted versioned serialisation when messaging or persisting data.

</comment><comment><USER60> Versioning was the biggest disappointment for me. I just want a middleware layer that can handle clients of different versions reliably. Surely everyone has the same problem.

<comment><USER36> >Surely everyone has the same problem.

Yes, we do.

But the appropriate answer in most cases is "it's up to the business logic". Trying to solve it at the transport layer is futile.

</comment></comment><comment><USER61> I never understood debate over technology that just works

You may not prefer it, but it works. Do something better, and make people use it.

<comment><USER18> The disagreement is over what "just works" means. Obviously the author things Protobuf isn't suitable for some use cases. They don't work in his view.

</comment></comment><comment><USER62> Wow, some minor complaints on language formality issues, then discredit the thing as a whole.

But in the end, the author probably does not understand PB solves the inter- language data sharing problem, which makes all its complain secondary and inconsequential.

It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can talk)

<comment><USER49> > But in the end, the author probably does not understand PB solves the inter- > language data sharing problem, which makes all its complain secondary and > inconsequential.

Huh? It's a serialization format. This problem has been solved many times in many different ways.

The author is pointing out that Protobufs were designed poorly, and they didn't have to be.

<comment><USER62> > This problem has been solved many times in many different ways.

Mind provide examples of such tools.

<comment><USER63> Plain old JSON, with types defined.

<comment><USER40> Way too slow even at moderate scale.

</comment></comment></comment></comment></comment><comment><USER64> What I was reading? It is easy to badmouth someone without knowing the context. It is not uncommon to end up with a flawed design if you find yourself in circumstances like short deadline, uncertain requirements, lack of resources, small user base, no QA and so on that are outside of the developer means to change.

</comment><comment><USER65> I think attacking a widely used open source piece of software rather than trying to help improve it behind the scenes without drawing attention to yourself will always be less work.

</comment><comment><USER66> Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin

<comment><USER67> That's the core of the author's argument. Protobuffers optimize for something besides usability and maintainability, because Google cares more about incremental performance than developer-friendliness. Which is a fine thing to care about at Google's scale, but maybe others' calculations should be different.

<comment><USER3> That doesn't seem like the author's main argument - they say: "Protobuffers were obviously built by amateurs because they offer bad solutions to widely- known and already-solved problems."

</comment></comment></comment><comment><USER68> Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

<comment><USER69> JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts

<comment><USER70> How would graphql help reduce the latency of API calls between services?

<comment><USER7> I'm no GraphQL expert/fan but one example is requesting only the data you need -- you can cut down on what you have to send over the wire which is at least guaranteed to help _some_.

Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.

</comment></comment></comment></comment><comment><USER71> Easiest way to send data over the wire.

       struct mytype s;
       s.field1 = something;
       s.field2 = something else;
       send(socket, &s, sizeof(s), 0)

or, using a language with a good type system like Haskell

       data MyData = MyData Int Int deriving Generic
       instance Storable MyData

       alloca $ \buf -> do
         let d = MyData field1 field2
         poke buf d
         send socket buf (sizeof d)

<comment><USER72> This may be easy, but it's _wrong_. Endianness issues are just the start. Information leaks due to padding are a big deal. And it straight up doesn't work if nontrivial data structures are involved.

<comment><USER71> Endianness issues are something every programmer should be aware of when sending data over the wire. I'm sorry I didn't insert the htons, htonls in my C code.

In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want).

I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for dealing with binary data that are almost universally more straightforward than proto bufs.

<comment><USER72> htons, etc are very 1980s, and I'll make a fairly strong claim: they should never be used in new code, with a single exception. The reason is that an int with network endianness simply should not exist. In other words, when someone sends you a four byte big-endian integer, they sent four bytes, not an int. You can turn it into an int by shifting each byte by the relevant amount and oring them together. And a modern compiler will generate good code.

The sole exception is legacy APIs like inet_aton() that actually require these nonsensical conversions.

<comment><USER71> You could also use the functions with explicit bit widths, like bswap64 and bswap32.

<comment><USER72> No, you should not. By the time you have a wrong-endian uint64_t or whatever, you've already done it wrong.

<comment><USER71> After re-reading your comment above, I'm actually confused. You think you should never store a big-endian int? That is ridiculous. Some architectures are big-endian. You should not be using custom bitswapping as part of application code, because you cannot know the endianness of your architecture.

The ntoh* functions are the right approach, and your claim is not only strong, it's wrong. The ntoh* functions exist to transform network byte-order to host byte-order. Depending on your architecture endianness, their functionality will change.

<comment><USER72> Let me try saying it differently. The following code is poorly written:

        char *buf = ...;
        uint32_t word = *(int32_t *)buf;
        uint32_t host_word = ntohl(word);

Because you just type-punned the read from buf. (In fact, this code is UB.) You could write it a little better like:

        char *buf = ...;
        uint32_t word;
        memcpy(&word, buf, 4);
        uint32_t host_word = ntohl(word);

Although IIRC there is or at least was still some disagreement as to whether this might be UB. You could use a union to make it definitely not UB.

But none of these variants are sensible, and, in fact, they don't even translate to most safer languages than C. The correct way to write this code is:

        char *buf = ...;
        uint32_t host_word = ((uint32_t)buf[0] << 24) |
            ((uint32_t)buf[1] << 16) | ((uint32_t)buf[2] << 8) |
            (uint32_t)buf[3];

On any recent compiler, this will generate as good or better code, and it doesn't make pointless assumptions about the representation of uint32_t on the platform you're using.

So I stand by my claim: well-written modern C code should not contain any "network-order" values. They should contain bytes, vectors of bytes, and numbers.

<comment><USER71> My C code didn't include mention of ints though, so I'm wondering where you got that from.

Your first example is UB and again, is not something my example depended on.

Your final claims are overly cautious. It is perfectly fine to use uint32_t in this way. Uint32_t is defined as a 32-bit unsigned integer. There is a bijection between network order 32-bit unsigned integers and host order integers, and ntohs is the bijection. It is no different than storing any other value. It is certainly not wrong.</comment></comment></comment></comment></comment></comment></comment></comment></comment>


Show HN: Naive Bayes classifier for text categorization in five steps - <AUTHOR> https://towardsdatascience.com/implementing-a-naive-bayes-classifier-for-text-categorization-in-five-steps-f9192cdd54c3 <comment><USER2> This is not a bad explanation but when doing this practically it can be useful to take log() of the probabilities so that you work with sums of logs rather than multiplying small floats.

[http://getpopfile.org/docs/faq:bayesandlogs](http://getpopfile.org/docs/faq:bayesandlogs)

<comment><AUTHOR> Thank you for the insight John, I have included your remark on the article.

</comment></comment><comment><USER3> From the article:

        For an English spam classifier that
        considers all the words in the English
        language, the number of the words (n)
        is approximately 171,476.

That's a remarkably precise number to be preceded by the word "approximately".

<comment><AUTHOR> Agree, that was odd, it now reads:

"the number of the words (n) is approximately 170k"

Thank you for the remark.

</comment></comment><comment><USER4> Nice article, very glad to read it. Keep up the good work.

<comment><AUTHOR> Thank you Victor, I enjoyed your JS articles too!</comment></comment>


Ask HN: Do I need consent to track usage data? - <AUTHOR> I'm currently building an iOS app in my free time and I would like to collect some anonymous data such as which features are being used, how long the app is open for, etc.  Ethics aside, do I need to inform the user of this? <comment><USER2> IANAL, but at minimum you should have a clear privacy policy that outlines what you do collect, why you collect it, and how you use it. You should also offer, to the best of your ability, a way for the user to opt-out if possible.</comment>


Show HN: Shoot ping pong balls in real time at roboempress - <AUTHOR> https://letsrobot.tv/robocaster/roboempress/robot/7060011 <comment><AUTHOR> She's got a shield and sword to fight the onslaught. ~200ms latency, running on golang, node.js, react.js, python. Aim, shoot, also talk (text-to-speech)

</comment><comment><USER2> Doesn't seem to work on iOS.</comment>


Test apps over 50+ countries over real 2G 3G 4G networks: HeadSpin - <AUTHOR> https://www.producthunt.com/tech/headspin <comment><AUTHOR> How do we test if the mobile apps and mobile websites work seamlessly over 2G 3G 4G networks across all key countries, sitting from a chair without any code changes? HeadSpin allows mobile apps to work faster across 50+ countries on 2G 3G 4G networks and prevents your head from spinning with the 'buffer wheel'.</comment>


Txtr reader - an open ebook reader(Kindle alternative?) - <AUTHOR> https://developer.txtr.org/Introduction

<comment><USER2> Kindle's value is not the hardware; the hardware is average. The value is being able to make any book you want show up on the screen in 60 seconds.

I suppose you can use Bittorrent to get the books, but that is inconvenient and not nice to the authors. I agree that Amazon's DRM is annoying, but it is very easy to remove if you are morally opposed. (Sometimes I am in the mood to read on the computer, so I break the DRM for that.)

The hardware itself is also hackable. Jesse Vincent has written some software for the Kindle that lets you view any ebook on it; not just Amazon's.

</comment><comment><AUTHOR> A review: [http://www.crunchgear.com/2009/10/15/new-e-reader-txtr- germa...](http://www.crunchgear.com/2009/10/15/new-e-reader-txtr-germanys- answer-to-the-kindle/)

And according to txtr's site, it is going to cost about 319 Euros - <http://reader.txtr.com/en/>

<comment><USER3> It won't be a Kindle alternative at that price - 319.00 EUR = $474.370 USD as of today.

</comment></comment><comment><USER4> Isn't the bebook already open source? Also costs less. I've been happy with mine so far.</comment>


Stallman on Steve Jobs - <AUTHOR> http://stallman.org/archives/2011-jul-oct.html#06_October_2011_%28Steve_Jobs%29

<comment><USER2> Comments like this drive me away from FSF, despite heavily using open source software each day of my life.

</comment><comment><USER3> Already in discussion here - <http://news.ycombinator.com/item?id=3083349>

</comment><comment><USER4> For someone who claims his philosophy is altruistic and is trying to make technology better for all human beings, he sure has a lot of hate.

</comment><comment><USER5> Orthodoxy is a smaller jail than any software system.

</comment><comment><USER6> What a joke, he either bears a grudge or is hopelessly misinformed. Or both.

</comment><comment><USER7> Disgusting.

</comment><comment><USER8> What a dick.</comment>


Amazon warehouse worker in New York has died of Covid-19 - <AUTHOR> https://www.theverge.com/2020/5/5/21248427/amazon-warehouse-worker-coronavirus-death-new-york-covid-19 <comment><USER2> Amazon almost has 1 million employees so it would be strange when Amazon would not be hit by any corona casualties.

Article doesn't mention the age of the unfortunate employee. It is probably best for Amazon to send everyone above the age of 50 to home until further notice.

<comment><USER3> Numbers are tricky things. Amazon doesn't have a million employees in Staten Island. This is the facility where the organizer of a walkout to protest unsafe conditions, Chris Smalls, was fired. That backstory is part of the reason why this particular death draws attention.

"So far, Amazon has closed only one facility in the US, a returns-processing center in Kentucky, and only after it was ordered to do so by the governor." I found this very interesting.

Amazon performs essential distribution of goods, but perhaps there is a good reason that it has drawn criticism for the care it takes with the health of its employees.

<comment><USER4> The death rate in NYC is over 2 per thousand people, and Amazon employs well over 1000 people in NYC. The law of large numbers basically guaranteed they wouldn't stay at zero forever.</comment></comment></comment>


Why Innovate in Utopia? - <AUTHOR> http://techcrunch.com/2010/10/28/singapore-why-innovate-in-utopia-2/

<comment><USER2> I spent about a week there for a conference. Yeah there's definitely the gleaming bits, but the city also has little India, and the Arab district. They're a bit more teeming and chaotic, I found. Overall the place was definitely a touch soulless, but it's not quite as extreme as these articles going around make out. It was more fun than I expected. The food sure is good, too.

</comment><comment><USER3> As beautiful as Singapore may appear on the outside, it's a pretty draconian place to actually live in (at least it would be for me). Also, this notion that people need to suffer in order to find the motivation innovate is a ridiculous claim.</comment>


Ask HN: advice needed on how to overcome objections to my new Bitcoin product - <AUTHOR>

I have developed a software application that literally creates Bitcoins from nothing.
Well strictly speaking it looks for arbitrage opportunities on the cex exchange, and sometimes when the market conditions are just right, a couple of simple trades conjures bitcoin out of thin air.
It works, for real.  My software conjures money essentially out of thin air.
BUT I have a couple of really thorny problems and I'd like to ask for the help of HN.
1: - no one believes me.  They think it's a scam.  And indeed it SOUNDS scammy.  How the heck to counter this?
2: - people don't trust me. I'm an ordinary middle aged man with a family who has been developing software for more than 20 years.  I know I'm trustworthy, but in the bitcoin world, no one trusts anyone. How do I get people to use my software when they are super paranoid about being hacked etc because it seems that everything bitcoin gets hacked.
FYI extra information: my software does not make any great quantity of bitcoin in any given trade - we're talking fractions of a bitcoin each time the opportunity comes up.  However I think there's plenty of room to grow this across exchanges and limits on any one users trading limits the amount that can be made.
Any advice welcome.  The software is called "coinfinder" and the site is www.bcoinex.com
Thoughts?  Advice?
Anton
One thing to note: for the purposes of easy explanation, I have said "bitcoin" above but its actually trading gigahashes which are used by cloud mining to create bitcoin.  A future version of my software will automatically exchange the gigahashes for bitcoin so it will be native bitcoin finding. <comment><USER2> I've seen a number of scams revolving around forex/sports arbitrage, and it would warm my heart if you could appropriate their marketing techniques for a genuine product. There was one in particular which I read all the copy for in sickly fascination. I couldn't find it again, but if I do I will post it to this thread.

They often have a ticker of out-of-date arbitrage opportunities, to demonstrate their capabilities. Your trial is superior to this in one way, and inferior in another (which is key.) You offer real moneymaking opportunities, proving the value. But potential clients, as you're well aware, do not yet trust you. And running your code requires trust. I think you should take a leaf from the con men's book, and show out-of-date opportunities on your site.

The scammers give an explanation of what arbitrage is, and what "separates" their system from a get-rich-quick scheme. They invoke images of fast cars and big houses. They are, of course, exaggerating to prey on the naive and desperate, but the principle holds for a real product. The wikipedia page you link to is sterile. If it were me, I'd write an explanation which dealt directly with gigahashes, and which sought to excite your customers and inspire them to imagine their pile of BTC growing every six seconds.

These next thoughts are more my own. If your software is any good, people are going to reverse engineer it; why not post the mathematics behind your system on your site? If people can see the basis behind your program, they've a reason to trust it. Even if they don't understand the principles, they'll be reassured to see that there are indeed principles. I'd put up all the personal details you're comfortable sharing. If people can read a bio blurb and see your face, and feel you've made yourself vulnerable, they're much more apt to make themselves vulnerable to you. (Of course, if you put up too much information the vulnerability ceases to be metaphorical.)

Additionally, while I understand you can't compromise your competitive advantage, I'm curious to learn more about your system. Is six seconds an arbitrary number, or did you arrive there empirically? Are you running the tool yourself? How much have you made? What would be the impact if many people started running your tool, and (if their clocks sync up) take advantage of the same opportunities? Would that degrade its utility?

<comment><AUTHOR> @fabulist, thank your for taking a look and giving real feedback. it is very much appreciated.

in no particular order, the questions you have raised.

Why 6 seconds. cex has a 600 requests per 10 minute limit on their API otherwise they ban the caller. i make sure that I am under that

I already know that others are using triangular arbitrage, in the stock market and I suspect they are also doing it on cex, otherwise I would be getting large numbers of profit opportunities rather than say 20-60 a day i am currently getting. So others doing something similar cant be controlled.

out of date opportunities i am thinking of doing. i can simply report every arbitrage i find to let people see what is happening. Although currently the trial which doesnt need people to even sign up or use cex shows it live as it happens.

I think you are also right into going into the math/detail. originally i have been thinking about keeping it fairly "high" level because most won't want to learn the math, but it may give them a greater sense of it being a real opportunity

thanks again fabulist

<comment><USER2> No trouble, I'm experiencing similar problems with trust trying to get my pentesting company started. The skepticism is frustrating, if understandable. :)

From what you've said, I'm inferring that when one utilizes an triangular arbitrage opportunity it is expended. If thats so, I would advise extreme caution. This could make scaling painful, as your customers would be using the same algorithm and finding the same opportunities, but only one of them would get the payout. The result would be that the value of your product decreases with the number of customers. Perhaps you could adjust your code so that it queries CEX no more than 600x/10min, and ever 6s on average, but the timing has a random element. This would prevent your clients from syncing with each other. YMMV.

</comment></comment></comment><comment><USER3> The obvious question is why don't you use your software instead of selling it?

<comment><USER4> I second this question. If you can make Bitcoins out of thin air, then why are you selling it. How do you make more profit selling it than doing it yourself? Is there anything shady or risky about your process?

<comment><AUTHOR> yes, i am running it myself and making a profit, but as i mentioned (in another post) it is a small amount. currently it is about $1 a day with a very small risk. it may grow, but that depends on volume or volatility growing at cex. the profit is directly tied to volume and volatility.

in terms of the risk. triangular arbitrage is in "theory" safe. in that it ONLY trades when there is guaranteed profit. in practice what happens is that other automatic trading bots can place the trade faster than your bot which means you have to wait for the trade to complete. the other risk is that there can be a network problem. the triangular arbitrage needs 3 trades to complete, if only 2 complete and the network goes down, you can be left scrambling to fix that last trade. both the risks seems to be fairly managable from the 1 month or so that i have been successfully going.

so why am i trying to sell it? if i can get $50 straight up rather than wait i will do it. also, "quant trading" from the real world stock market is way way advanced on what this app does, so i have plenty of new trading bots to create and in some fantasy land of mine i would love to be the guy that created the first set of software for it on the bitcoin market

thanks for the input cryptolect

<comment><AUTHOR> I might also add, that to give an analogy. if bitcoin mining is so good, why do cex sell cloud mining and not keep it for themselves?

As with them the answer is that it is more profitable to build a customer base and profit from them and let them then make a small profit on the mining

<comment><USER3> Mining is not marketed as risk free though. I have personally made some money through automated inter-exchange arbitrage but I ended up concluding that the risks outweighed the rewards at current time (maybe intra exchange is less risky).

<comment><AUTHOR> most of the inter exchange arbitrage opportunities are eaten up by the fees and because there can be a time delay across exchanges it also adds risk. The intra trading (especially in my case ) has those risks minimized. there are no fees on cex and the trades happen very fast.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> @olalonde,

I am using it. it works quite well, but it is not big money. it makes as much as the volume and volatility of cex. So that works out at about a $1 a day right now. if cex grows, or if there is more volatility then it will increase.

right now i am starting to look the coding for other exchange.

maybe its not something bitcoiners are going to accept because it seems too good to not be a scam or it doesnt seem like there is enough in it to make it worth while.

currently the feedback i get is that people think its a scam

I am thinking of doing a speech at the local bitcoin events but that doesnt seem too scalable

<comment><USER2> HN actually use replies, rather than @tags. Below olalonde's comment, you'll observe theres a small "reply" link. I can understand how it'd be easy to miss though.

Paul Graham often advises people to "do things that don't scale" in order to get to a large scale. Doing a talk sounds like a cool idea, and if it get onboard your first few customers (or additional customers if you already have some), its a win.

</comment></comment><comment><AUTHOR> Just an update on the product launch.

I've got a handful of sales coming in which is great. Mostly driven by a post on the bitcoin forum.

People there are certainly raising the trust/hacking issue. I am being open and honest with them and trying to communicate in a responsive manor. Maybe honesty, responsiveness and time will be the key.

</comment><comment><USER5> Here is a great example of what not to do:

[https://coingeneration.com](https://coingeneration.com)

<comment><AUTHOR> lol point taken.</comment></comment>


Unity2020 – A Plan to Save Our Republic - <AUTHOR> https://articlesofunity.org/ <comment><USER2> The idea here is a "unity ticket" which includes both a center-left and a center-right candidate. But who of them is the president and who the vice president? The latter is a purely symbolic role, unless the president dies, resigns or gets impeached. Given that only one person can be president, a truly balanced ticket is impossible.

Everything is incredibly vague. There are no policies, politicians are supposed to pursue strategies not policies. What strategies? Not answered either.

Cynically, I'd suggest that this is about getting email addresses of naive idealists, maybe to pull something like Jill Steins recount crowdfund campaign.</comment>


List of words text-to-speech engines can't say correctly - <AUTHOR> https://wiki.therofl98.co/wiki/List_of_words_the_text-to-speech_engines_can%27t_say_correctly <comment><USER2> I was hoping to see 'live' on the list.

> I live in US.

vs

> I went to a live concert in US.

The Alex voice on macOS used to confuse them a lot.

Ah, and 'ML paper' read as 'millilitres paper'

</comment><comment><USER3> BTW, are there any available services but very easy to use for non-programmer? I know Amazon polly and Wason, wason maybe better they got a free demo page which you can use, but for Amazon polly, u have to pay and know how to code and use their API, any other very good alternatives?

</comment><comment><USER4> A Bug Bounty comes to mind.

This sort of stuff could be crowd sourced better even without a bounty.

Why not dump part of the billions $ spent for the visually impaired on web design instead into long lasting solutions like better text-to-speech.

I guess less opportunists to virtual signal is the real answer.

So you'd need a way to allow people to virtual signal to fix this, which also possible, and in fact might be also a way to earn $.</comment>


SQL Database Hacks Using AS and ORDER BY - <AUTHOR> https://www.morpheusdata.com/news/2016-06-16-sql-database-hacks-using-as-and-order-by <comment><USER2> I thought this would be about some weird exploit. Then I thought it might be the other use of hacks - an unintended use.

It's neither, it's describing the intended use of two very basic keywords in sql.</comment>