Why are we templating YAML? - <AUTHOR> https://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml.html <comment><USER4> My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language.

I think this approach is a byproduct of thinking about infrastructure and configuration -- and the cloud generally -- as an "afterthought," not a core part of an application's infrastructure. Containers, Kubernetes, serverless, and more hosted services all change this, and Chef, Puppet, and others laid the groundwork to think differently about what the future looks like. More developers today than ever before need to think about how to build and configure cloud software.

We started the Pulumi project to solve this very problem, so I'm admittedly biased, and I hope you forgive the plug -- I only mention it here because I think it contributes to the discussion. Our approach is to simply use general purpose languages like TypeScript, Python, and Go, while still having infrastructure as code. An important thing to realize is that infrastructure as code is based on the idea of a _goal state_. Using a full blown language to generate that goal state generally doesn't threaten the repeatability, determinism, or robustness of the solution, provided you've got an engine handling state management, diffing, resource CRUD, and so on. We've been able to apply this universally across AWS, Azure, GCP, _and_ Kubernetes, often mixing their configuration in the same program.

Again, I'm biased and want to admit that, however if you're sick of YAML, it's definitely worth checking out. We'd love your feedback:

\- Project website: [https://pulumi.io/](https://pulumi.io/)

\- All open source on GitHub: [https://github.com/pulumi/pulumi](https://github.com/pulumi/pulumi)

\- Example of abstractions: [https://blog.pulumi.com/the-fastest-path-to- deploying-kubern...](https://blog.pulumi.com/the-fastest-path-to-deploying- kubernetes-on-aws-with-eks-and-pulumi)

\- Example of serverless as event handlers: [https://blog.pulumi.com/lambdas- as-lambdas-the-magic-of-simp...](https://blog.pulumi.com/lambdas-as-lambdas- the-magic-of-simple-serverless-functions)

Pulumi may not be _the_ solution for everyone, but I'm fairly optimistic that this is where we're all heading.

Joe

<comment><USER5> This is a great analysis, but it's missing a fundamental point: why do we have a problem with these approximations of a programming language or just using a programming language to template stuff?

Because your build then becomes an actual program (i.e. Turing complete) and you have to refactor and maintain it! This is the common problem of using a "programming language as configuration" (e.g. gulp?)

Dhall solves exactly this problem: [https://dhall-lang.org](https://dhall- lang.org)

It has the same premises of Pulumi, but without the Turing completeness (I don't know if/how Pulumi avoids that, but if it does it should be part of the pitch), so you cannot shoot yourself in the foot by building an abstraction castle in your build system/infrastructure config.

We use it at work to generate all the Infra-as-Code configurations from a single Dhall config: Terraform, Kubernetes, SQL, etc.

And there is already an integration with Kubernetes: [https://github.com/dhall-lang/dhall-kubernetes](https://github.com/dhall- lang/dhall-kubernetes)

<comment><USER6> I don't get the problem with using a turing complete language to generate configuration. There's nothing wrong with maintaining and refactoring a program, that's a natural process for any program. If you don't want an infinite loop, don't write one, as you wouldn't in any other program. You can choose as much or as little abstraction as you so wish.

Give me a real language any day over dhall or jsonnet.

<comment><USER7> FWIW jsonnet is a "real" language. It's a dynamically typed, lazily evaluated purely functional programming language).

<comment><USER6> Fair enough. I should have said "general purpose language" rather than "real", which makes for flame-bait.

<comment><USER7> I once built a mandelbrot fractal renderer which emitted a data-URL encoded PNG string to stdout in BCL (a spiritual predecessor of Jsonnet @ Google).

Yeah, I know what you mean. It lacks generic input/output, you cannot read write arbitrary files and perform arbitrary network requests etc.

I do like that restriction in the context of managing configuration systems, because it allows you to build hermetic evaluations.

With kubecfg we added the ability to import from URLs, which I wish was available out of the box in jsonnet.

</comment></comment></comment></comment></comment></comment><comment><USER8> I know I'm in a minority, but I really dislike YAML... I recently did a lot of Ansible and boy, at the beginning, I was just struggling a lot. Syntactic whitespace kills me.

I don't like it in Python either, but for some reason, when I write Python, it's a lot easier. Maybe YAML is just a bit more complex (and Python has better IDE support..?)

<comment><USER9> > Syntactic whitespace kills me.

Okay, I'm gonna be the asshole in the room, but how hard is it to just use consistent indentation? I can't count how many times I've heard people complain about significant whitespace in languages.

Not only is it not difficult to begin with, but every code editor and IDE will show you where there's a syntax error in your YAML. People are free to dislike YAML, even for its significant whitespace, but how does it "kill you"?

Look at this example from the article:

```

something: nothing

      hello: goodbye

```

This is pure sloppiness, and anyone who has trouble carelessly adding pointless bytes to code, no matter the language, is sloppy. I don't understand why people criticize YAML and Python because "whitespace is hard".

P.S.: There's a similar configuration language called ArchieML, which is similar to YAML but doesn't have significant whitespace.

[http://archieml.org](http://archieml.org)

<comment><USER10> Three big things that annoy me even though I'm happily writing Python:

\- "cut and paste and edit" is broken. You can't autoformat the pasted code into the right place, you have to go back and fix the whitespace. Since whitespace is semantically significant, this can introduce bugs.

\- visually identical whitespace may not be textually identical whitespace. Unless you go around breaking the tab key off your colleague's keyboards you'll trip over this. Especially (again) if you paste. Occasionally seen in merges too.

\- editors can no longer give you 100% correct indentation.

<comment><USER11> > \- "cut and paste and edit" is broken. You can't autoformat the pasted code > into the right place, you have to go back and fix the whitespace. Since > whitespace is semantically significant, this can introduce bugs.

Depends on how your editor is configured / it's feature set. Which makes me wonder how editorconfig would handle this when enabled. It seems like a insignificant issue to me, you can auto-PEP8 the code before pasting it. You should probably be following PEP8 anyway (as far as spacing is concerned at least).

> \- visually identical whitespace may not be textually identical whitespace. > Unless you go around breaking the tab key off your colleague's keyboards > you'll trip over this. Especially (again) if you paste. Occasionally seen in > merges too.

I turn on show all whitespace on my editors regardless of programming language. I've been burned by Sublime Text not just figuring out the already defined whitespace ruleset for a file by what it's using and just shoving in it's own defaults. I wish _all_ editors would base whitespace on what the file's structure looks like, if there's mixed spaces, give me a warning.

> \- editors can no longer give you 100% correct indentation.

I don't understand this, it sounds like you've got your editor configured poorly or something? But it goes back to how unintuitive the nice editors can be. You can use editorconfig to define the indentation project wide, then any editor should pick it up, of course if you define PEP8 at a minimum it guarantees spacing settings.

I'm not sure if PyCharm covers a few of those cases, since I use it so seamlessly I don't usually have complaints.

</comment></comment></comment></comment><comment><USER12> I didn't see this mentioned anywhere else, so another alternative (that I've seen and really like conceptually, but haven't used so far) to all this wildness with YAML and JSON -> [https://github.com/dhall-lang/dhall- lang](https://github.com/dhall-lang/dhall-lang), and for kubernetes specifically -> [https://github.com/dhall-lang/dhall- kubernetes](https://github.com/dhall-lang/dhall-kubernetes)

<comment><USER13> god those examples are ugly - commas at the beginning of a line? mismatched brace styles?

<comment><USER14> It's a haskell thing. The main advantage is that each line is independent. You can comment out a line or add a line at the end without modifying anything else.

<comment> basil-rash Each line is not independent: you cannot comment out the first line. A better approach is to allow trailing commas. (I suppose you could allow leading commas, does Haskell support this?)

<comment><USER15> Unfortunately no.

Allowing trailing commas, like Python does, would be really great. Unfortunately trailing commas already mean something: (a,b,) is a function that still takes 1 argument to make a triple. It's called "TupleSections".

</comment></comment></comment></comment></comment><comment><USER16> If you have been around long enough you still remember the world that was excited about XML and templating it using XSLT. As a hindsight it was a horrible world.

Even though YAML is not optimal, it is a human friendly compromise between too verbose XML and machine only JSON. It lacks native templating, leading to funny constructs e.g. with Ansible files. However human kind has made progress and will make progress further, so it is just a matter of time until someone comes up with sane "native templated YAML" and all projects will adopt it.

<comment><USER17> > If you have been around long enough you still remember the world that was > excited about XML and templating it using XSLT. As a hindsight it was a > horrible world.

I actually really like the _idea_ behind XSLT: machine-friendly, human- tolerable, structured data + declarative rules for turning that data into a display, or a report, or whatever else.

The execution was horrible though: incredibly verbose, lots of overcomplication due to XML weirdness/asymmetries (e.g. attributes vs elements vs text, namespaces, ...); mixtures of different languages hidden inside each other (e.g. XPath hidden in attributes); etc.

I would really like to see what this could look like if done in a more minimalist, lispy fashion (normal code-is-data stuff in Lisp is _similar_ , but I think term-rewriting is a more appropriate evaluation mechanism for such rules)

<comment><USER18> XSLT was one in a litany of domain specific languages (ant, apache rewrite rules, latex macros, etc.) that evolved towards turing completeness because that's what the problem space demanded.

In most if not all of these cases an existing and well designed turing complete programming language would likely have better served them.

<comment><USER19> There was a post on HN a few weeks back to the effect that it's rather easy for Turing completeness to emerge accidentally. I wish I remembered more specifics so I could find it again.

<comment><USER17> Was it [http://beza1e1.tuxen.de/articles/accidentally_turing_complet...](http://beza1e1.tuxen.de/articles/accidentally_turing_complete.html) by any chance?

</comment></comment></comment></comment></comment><comment><USER20> I saw this title and immediately knew the article would be about Helm. I don't think anyone wants to use Helm. People use it for a set-and-forget thing that they don't care about (who cares that it's called impressive-leopard- kubernetes-dashboard, after all.)

kustomize is much more sane for your own stuff: [https://github.com/kubernetes-sigs/kustomize](https://github.com/kubernetes- sigs/kustomize)

It is actually a little bit too magical for my taste, but I continue to use it because it hasn't done anything stupid. I have one file that maps logical names to images in a container repository. If I create a service called "foo" pointing to selector.app.label="foo" in the base, then in production it's called foo-prd and the label magically updates to foo-prd for the selector. It actually understands what it's generating, and while they might have taken it a little bit too far, it's far better than just dumb text replacement.

<comment><USER21> > I don't think anyone wants to use Helm.

Here is why everyone should use Helm:

Helm 2.0 introduced package as a first-class concept for Kubernetes and created the standard to distribute applications, thanks to Helm thousands of people could discover and collaborate on cloud-native deployments of the open source software [https://github.com/helm/charts/tree/master/stable](https://github.com/helm/charts/tree/master/stable) published and managed by organizations and contributors all over the world.

Helm 3.0 keeps innovating, it adopts the most forward-thinking approach to package management and Kubernetes config management by using higher level domain specific language based on Lua to create expressive package management system:

[https://sweetcode.io/a-first-look-at-the- helm-3-plan/](https://sweetcode.io/a-first-look-at-the-helm-3-plan/)

Helm is also backed by CNCF[1] and is the best choice so far for organizations to create a reproducible CI/CD pipeline in a Kubernetes cluster.

[1] [https://www.cncf.io/blog/2018/06/01/cncf-to-host- helm/](https://www.cncf.io/blog/2018/06/01/cncf-to-host-helm/)

<comment><USER20> That is a lot of buzzwords.

I don't really trust Helm to do anything that's actually useful in the long term. It will get something running very quickly, but whether or not it's maintainable, I am yet to be sure of. For example, very early on, I installed the helm chart for prometheus. Now I want it to live in the kube-system namespace because I am tired of seeing its resources in the default namespace. For some reason, I highly doubt that changing values.yaml to change the namespace is going to do anything other than give me a fresh instance of prometheus running in another namespace. It's not going to use the already allocated storage volume to satisfy the persistent volume claim in the new namespace. It's not going to update the other stuff in my cluster to refer to prometheus-pushgateway.kube-system.svc.cluster.local. It's not going to update my Grafana dashboards to refer to the new namespace, even though I installed Grafana with Helm! So what did I really gain? Helm isn't giving me the ability to manage the long-term lifecycle of third-party software. It just explodes some API objects all over my cluster and lets me delete most of them automatically. That's all it does.

I get why Helm is popular. You can get some piece of software running in Kubernetes with minimal effort. I would have never successfully made some random complex piece of software work correctly in Kubernetes on day 1, especially using something that assumes you deeply understand the core API objects like kustomize does. What that boils down to is that Helm doesn't go far enough, and in its current state, just encourages people to make mistakes early.

</comment></comment></comment><comment><USER22> As others in this thread have said: I ask this question all the time, except s/templating/using/.

YAML is insanely over complicated; it's as bad or worse than XML for config files, and it doesn't even have the nice streaming mode. Not to mention that it's a bit of a security nightmare (seriously, who put pointers into the YAML spec?).

And, on a more subjective note, YAML is just confusing: between all the significant whitespace and the random single character symbols that no one ever remembers what they do, I never get a YAML document right on the first try.

Templating it really does add a whole new level of headache too.

<comment><USER23> >it's as bad or worse than XML for config files

XML works very well for config files. It's schema-optional (but is there), well-specified, human-readable, has plethora of supporting technologies (making things like templating easy), and is well supported by every language.

At the very least it is way better than JSON.

<comment><USER24> It's missing one important part for config files. It's tedious to write by hand.

<comment><USER25> XML is not tedious at all with the right tooling. For example a tool like Visual Studio IntelliSense proposes only elements and attributes valid in the context, automatically close tag, format the file and complete opening tags too so it makes editing XML file a breath.

<comment><USER26> The tooling bar for config files is set firmly at notepad.exe

If your config file requires more tooling than that you fucked up.

<comment><USER27> I mean, with Visual Studio I can even the change the code of the application I am running on the fly, depending on what exactly I am configuring :)

That doesn't make XML an easier format to maintain.

</comment></comment></comment></comment></comment></comment><comment><USER28> File-based configs are a troublesome abstraction: they package unrelated concerns into a rigid document whose form must take a particular, application- dependent shape, and the assembly and disassembly of that document essentially becomes an API where key-value pairs are mixed with complex glue code. The application has to do this internally, but anyone who's generating their configs are also doing parts of this externally.

Templates try to bandage over that by drilling down the abstraction to key- value pairs themselves. And imperative constructs that sneak into templating languages are an artifact of wanting to gain expressiveness without losing the benefits of declarative form -- but really, the two are at odds.

YAML is a red herring -- we had the same headaches with XML a decade prior. The problem is always that there's relationships among the data (or even multiple instances of the config) that we care about, but that the structure of a single config file at rest cannot model.

Databases -- let's say, an SQL one -- are actually among the better solutions, because they allow the universe of config items to live in structured places without overspecifying the exact form the data must take when serialized into a file. Then, data can be normalized where it makes sense to avoid repetition and introduce propagation. An SQL database gives all the tools needed to accomplish this, using mostly declarative code.

Databases in a KV sense are often used for configuration, and SQLite's rise has increased richly structured configs that are specified at a higher level than what's typically done with other serialization formats, but the full approach has not caught on outside big enterprise systems and complex applications. Which is a shame, because it's hardly more complex than the current awkward pairing of a full serializer and a templating engine.

<comment><USER29> SQL as a configuration format is not that bad of an idea

</comment></comment><comment><USER30> Wait, what?

I feel this article is missing the _bigger_ problem - one that for some reason just cannot die.

The problem is that of _gluing strings together_. YAML is not an unstructured text file, it's a tree notation. Whatever "templating" or "generation" mechanism you want to use, it needs to respect the tree nature of the language it operates on. It needs to respect semantics.

Gluing strings together is literally what causes SQL Injection to exist. It caused countless of defacements on the web, and countless of broken websites. I would think we've learned our lessons, but for some reason, I see these template languages still alive and kicking.

<comment><USER31> The article goes on to talk about Jsonnet, which takes the exact approach you describe - it generates JSON by aiming to be a "templated JSON" where the templating involves generating semantic objects, not strings.

Here's an example (adapted from some real-world code) where I specify the k8s cpu limit in one place, and then look up that info in several other places to avoid needing to change multiple values later:

        {
          local container = self,
          requests: {cpu: 5.5, memory: "2G"},
          limits: container.requests + {memory: "4G"},
          environment: [
            {
              name: "NUM_THREADS",
              value: std.toString(std.ceil(container.requests.cpu)),
            },
          ],
        }

Note how I can patch the container.requests object with an alternate memory limit, and how I can calculate an expression for the NUM_THREADS value in order to automatically set it to ceil() of the requested cpu.

(edited for nicer formatting of the code)

</comment></comment><comment><USER32> There have been times I've wanted to templatize my configuration but I don't want to do it with text-based templates but templates within the configuration files syntax (be it yaml, toml, or something else). Not sure what this is called, I've been calling it "structural templating".

So far the only things close to this are

\- Azure pipeline's syntax: [https://docs.microsoft.com/en- us/azure/devops/pipelines/proc...](https://docs.microsoft.com/en- us/azure/devops/pipelines/process/templates?view=azure-devops) \- Something called Jasonette: [https://docs.jasonette.com/templates/](https://docs.jasonette.com/templates/) \- Something called Jsonnet: [https://jsonnet.org/](https://jsonnet.org/)

Azure Pipeline's approach I think is closest to what I've been looking for.

Anything else in this space?

<comment><USER33> I prefer [https://github.com/taskcluster/json-e](https://github.com/taskcluster/json-e)

Has the advantage/disadvantage that it's still valid json/yaml

I wrote a scary command line wrapper for it:

[https://wryun.github.io/rjsone/](https://wryun.github.io/rjsone/)

Has libs for Python, Go, and JS, and there's a bazel interface.

<comment><USER32> This looks really nice. I'll have to give this a try at some point to see how I feel about it vs Azure Pipelines. From my quick look, this looks more general purpose at the cost of more verbosity.

</comment></comment></comment><comment><USER34> Dhall-lang ( [https://dhall-lang.org](https://dhall-lang.org) ) is another, somewhat interesting, attempt to solve this program: it comes with a non- Turing complete programming language, so you can bring some abstraction to your configuration files without having to worry about things like infinite loops.

</comment><comment><USER26> The real question is why are we using yaml at all?

<comment><USER35> Baffles me. I don't like any language or file format where whitespace matters. Even Haskell bothers me in this regard. To me white space shouldn't add cognitive load. I want to look at the symbols not the formatting of the antisymbols to understand what is going on.

Write JSON and use your editor tools to format it with nice indentation, and you are sweet!

That said Yaml makes an excellent format for reading, but not for writing.

<comment><USER36> White space matters with Python but I rarely run into issues with indentation in Python. But YAML on the other hand, I have nothing but nightmares.

<comment><USER37> I have the same experience. I just think think it is possible to design easy- to-write indentation-sensitive formats but YAML is not. For example is always baffles me that

        a:
        - <USER2>
        - <USER3>

has a list inside an object but the list is not further indented. There's in fact a hierarchy relationship but absolutely no indentation.

<comment><USER38> You can indent it though, but the hyphen already means a list item.

That is a preferred syntax for many that are used to YAML though.

<comment><USER39> The hyphen is associated with the a though. In what's posted, a is a list, not a map - the type of the parent is being declared by that hyphen.

</comment></comment></comment></comment></comment></comment><comment><USER40> Maybe we should be exploring using Dhall instead of YAML.

>One of the clearest signals I've gotten from users is that Dhall is "the YAML killer", for the following reasons:

>Dhall solves many of the problems that pervade enterprise YAML configuration, including excessive repetition and templating errors

>Dhall still provides many of the good parts of YAML, such as multi-line strings and comments, except with a sane standard

>Dhall can be converted to YAML using a tiny statically linked executable, which provides a smooth migration path for "brownfield" deployment.

Source: [http://www.haskellforall.com/2019/01/dhall-year-in- review-20...](http://www.haskellforall.com/2019/01/dhall-year-in- review-2018-2019.html?m=1)

</comment><comment><USER41> LOL we are doing the same with k8s. We deploy to any environment, where each environment is a different k8s namespace. We even have a namespace for each developer. The variables are things like the image name (the tag is effectively a git commit id, or sometimes different for not-committed yet stuff). Beyond that just about nothing really needs to be a variable, but we do have different RAM amounts.

We have a couple ways that we template this out, but mostly we literally just do this in bash:

        sed -e "s/\$CI_COMMIT_SHA/$CI_COMMIT_SHA/" kube-deploy.template.yaml | kubectl -n $ENV apply -f -

(Where CI_COMMIT_SHA comes from gitlab) ENV comes from our gitlab CI file.

That all being said, the extent of our k8s integration is lots of stuff like that. We could write a JS file that creates a JSON k8s template, but honestly, that would be more work and more learning than we had to for what we're doing. Why would we do more just because we want to avoid templating in a YAML file?

</comment><comment><USER42> I think they are missing the real selling point of JSON. It's basically interoperable with JavaScript objects.

That means you write it, send it, store it, operate on it, etc. with little or no modification.

The author says "converting between the two is trivial" which may be true, but the developer overhead is less trivial. And it will always be JSON in the client - JS doesn't support YAML objects.

<comment><USER43> Except that you cannot encode pretty valid Double.NaN or Date(), which is showstopper for many.

I remember mongodb started with json, but switched to in-house bson pretty early because of the json limitations.

<comment><USER13> new Date().toJSON()

<comment><USER43> When you parse back the result of this, you are not getting Date back.

</comment></comment></comment></comment><comment><USER44> Helm reminds me of the 90s level of webtech. I.e. php cgi files mixing html, logic and includes.

<comment><USER20> I believe people are back to doing that for web stuff. Open up any modern web stuff and it looks like "const widget = <div>foo</div>".

Not making this up! [https://reactjs.org/docs/introducing- jsx.html](https://reactjs.org/docs/introducing-jsx.html)

<comment><USER45> The difference, at least for Vue single file template components is that it's 3 separate areas in one file:

        - <template /> (Templated HTML)
        - <script /> (JS OR Typescript)
        - <style /> (CSS OR SASS/etc)

Whereas in old PHP files you could mix it in anywhere and the files were a big mess. Including inline SQL into your view templates which is hardly a good separation of concerns. While a Vue component can be separated into separate files, at least as one it all represents one isolated piece of the interface.

[https://flaviocopes.com/vue-single-file- components/](https://flaviocopes.com/vue-single-file-components/)

</comment></comment></comment><comment><USER46> Off-topic: The whole discussion is about deploy-time configuration management, but our problem is more about run-time configuration management.

We have done the classical memcached+database custom solution, but I was wondering if there is any accepted library/tool to change application run-time behavior. We have tried consul KV store [1], but does not quite fit in our environment.

My ideal solution would be a webapp with some text editor (think codemirror). Changes in this text file would push the configuration data to a running application.

[1] [https://learn.hashicorp.com/consul/getting- started/kv](https://learn.hashicorp.com/consul/getting-started/kv)

</comment><comment><USER47> Who knows.

It makes me throw up a little in my mouth every time I see hundreds of lines of YAML to configure something like Traefik with Kubernetes. The worst is when people say they prefer that because "I don't have to write a config file for my backend". That's true but instead now you have extremely verbose configuration mixed in with other verbose configuration.

But in YAML's defense I think it's more of a problem with the tools that use it more so than YAML itself. Ansible is a great example of how amazing YAML can be to manage complex configuration in a concise way.

<comment><USER48> I agree that simple YAML can be nice as a quick and clean tool, but Ansible is an example of everything _wrong_ with how YAML is used. Layering program flow constructs like loops, variables, templates, references etc. are exactly the sort of abuses that make YAML feel awkward.

</comment></comment><comment><USER49> YAML is a data stream, not a program. Please do not shove programs into data.

Your data does not need to be "expressive", it just needs to provide input to a program. If your data files need to be complex, you need a program to generate them for you.

I've danced the dance of ini -> json -> yaml -> weird hybrid -> embedded logic, and it ends with "program that asks for what the thing you want looks like and generates data files". Industrial software design figured this out ages ago.

<comment><USER30> And you end up with program compiled to a config anyway, so with a proper toolchain it means your real config _is_ the program.

People keep increasing complexity unintentionally precisely because they don't realize that _code = data_. There's no real distinction. Code is data is code.

You will end up having Turing completeness _somewhere_ , it's just a matter of choosing (or blindly selecting, like most people do) where. For a popular product, it eventually gets embedded in the configuration language, turning it into half-assed programming language (see most web-related templating). For less popular products / more enterprise'y settings, you can probably get away with embedding the Turing-complete part _in your bureaucracy_. That is, I can't code my config to make it do what I want, but I can pay you to get developers to write some code and export it to the config language as a keyword. There's a spectrum to this, and tradeoffs galore.

But ultimately, YAML is nothing but a tree notation. Tree notation is enough to represent high-level programming languages. Lisp without parenthesis, if you might, or Python, if you squint your eyes.

<comment><USER49> "Data" to me is the least complex input from a human, whereas "code" is more complex and necessitates a lot more work to make sure it's correct and bug- free.

If you embed "code" in "data", you made your thing way more complex and subject to software design patterns. But in software operation, we already have to contend with highly complex systems, so we want to remove as much time and effort and complexity as possible from the instrumentation.

To put it another way: if you had to run a nuclear reactor, do you want to instrument it by constantly writing new code, or turning a dial? I'd rather turn a dial. That means I have to develop the code for that dial ahead of time, but in the end, actually using it will be safer.

</comment></comment></comment><comment><USER50> My approach to this these days is:

* all configuration, without exception, is XML.

* all configuration may be generated from any other format imaginable, but it's sure as fuck going into the Big Main Godlike Application as XML.

Separation, interfaces, etc. Disclaimer: I work in .NET almost exclusively. The .NET configuration APIs generally work, as long as you _only ever use them for reading_ ; treating config as something the application itself can fiddle with is a fast route to madness.

</comment><comment><USER51> I feel like Ansible generating YAML with Jinja templates is really a sweet- spot, with idempotency and reusability.

I find it pushes me to write plain YAML files for variables and defaults (Ansible), while allowing strong templating of generated files (Jinja) and letting the result be readable (YAML). By readable I mean minimum programming bloat (code spread on many lines just to write a for loop) and minimum extra syntax that clutters the screen (brackets and quotes). It also lets me write very little custom code (aside from variables, obviously).

If I had to use a more "powerful" YAML-like replacement, it would mix all these into files, written differently by people with different styles, and it would have bloat all over the place.

The main issue I have with helm is that values.yml is not templatable by default so you have to generate it if you want reusability.

YAML does one thing and does it well, it's readable and bloat-free. Maybe we need more tools like "kubectl explain" to know the syntax though.

</comment><comment><USER52> Reading the commentary on whitespace, my thoughts immediately jumped to the C preprocessor. Even though it's built into the language, it has the same sorts of problems as a template engine has with generating YAML: the preprocessor just wasn't enough aware of the syntactic structure of the language to make it easy to generate anything of significant complexity.

I'm not proud of this (and like to think I could come up with something better these days), but this code was a bit of a nightmare for that reason:

[https://github.com/mschaef/vcsh/blob/master/vm/number.c#L256](https://github.com/mschaef/vcsh/blob/master/vm/number.c#L256)

Lisp macros do better, but they have the problem that the macros (and their potentially unusual evaluation rules) can easily just blend in with ordinary function calls.

</comment><comment><USER53> In the Clojure world we use EDN (extensible data notation) which is a subset of Clojure. [https://github.com/edn-format/edn](https://github.com/edn- format/edn)

You can extend it, convert it to JSON if necessary, and it is easy to read.

</comment><comment><USER54> Helm charts are great, and accomplishes the task, but things get really weird and complex if you're not careful. Oh man, the indentation and _helper.tpl nightmares. It's not obvious at all. And there's still a lot of repetition for each chart; writing a deployment or whatever is so verbose in K8s.

On the other hand, Ansible uses yaml and there it works great. I feel like Ansible uses yaml in a way that's easier to understand and the way it was meant to be written. With Ansible, you're writing configuration, not templates of configuration. I don't think a layer on top of Ansible like Helm is for K8s wouldn't make sense.

</comment><comment><USER55> I'd rephrase the question: "Why are we templating YAML with text-based templating tools?"

I and @akx wrote a templating tool called Emrichen that's specifically designed for producing YAML and JSON from YAML templates:

[https://github.com/con2/emrichen](https://github.com/con2/emrichen)

In contrast to other template systems, Emrichen templates are not just "based on YAML", they _are_ YAML. YAML tags like "!Var varname" are used to perform things like variable substitution, loops etc. Variables can be of any JSON type, not just strings, and the template is evaluated top–down.

</comment><comment><USER56> Over the last week I created a tool processing YAML and JSON files using Jsonnet. It's called [ycat]([https://github.com/alxarch/ycat](https://github.com/alxarch/ycat)) and is inspired by `jq` but uses Jsonnet for processing. It can also be used just as `cat` to concatenate JSON/YAML files. It's still young but very usefull, especially for handling complex kubernetes configurations.

</comment><comment><USER57> Wholly agree with this which is why I've been experimenting with a project similar in goals to jsonnet as a side project of my own. I didn't know about jsonnet when I started or I'd probably have just used/contributed to that instead.

Why template yaml when you could just generate it? Or generate json, or toml, or xml, or environment variables...

<comment><USER58> If you are generating it, why use it at all?

<comment><USER6> Because more often than not that is what the target system expects (k8s most notably).

</comment></comment></comment><comment><USER59> Jsonnet is nice, but not actively developed, unfortunately. I highly recommend you to support UCG [0], which is written in Rust (Jsonnet has two implementations, which is one source of the slowness).

[0]: [https://github.com/zaphar/ucg](https://github.com/zaphar/ucg)

</comment><comment><USER60> His main argument is that you need to template differently for different environments (dev/stage/prod) and cloud regions (us-west/us-east/emea/apac).

This is actually a solved problem, and you shouldn't be doing it in your YAML/JSON templates. You should be using an external parameter store to do this, and using a single template for everything.

See [https://aws.amazon.com/blogs/compute/query-for-the-latest- am...](https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux- ami-ids-using-aws-systems-manager-parameter-store/)

This is a simple use case: I want to deploy the latest AMI (Amazon Machine Image) in any region, so I always get the latest patched Linux base image to run my application on. I don't (and shouldn't) want to update my YAML/JSON every time a new image is published.

So, why are people having to go to these crazy templating macro lengths? Just store the changing bits in an external config/parameter store like etcd and let your infrastructure as code templates remain unchanged.

<comment><USER20> Based on the appearance of Helm, this is probably referring to Kubernetes, which really does have per-environment things that can't be represented in any form other than namespaces. For example, you have an app running (that's a deployment in k8speak). You want network traffic to get to this app, so you set up a load balancer. The load balancer config needs to know the name of your app (or more precisely, a selector for "pods" that are created by the deployment), which will change between environments, so now there is that common variable that has to be updated in both places. That's a simple example but is why people are templating their k8s configs. Yes, you could work around it by giving every environment its own namespace and using the same set of objects in every environment (differing only by namespace, which should probably be in the file... but since you can't edit the files, you can pass it in to kubectl probably), but there are other cases where even that doesn't work.

You do need some way of saying "this is the base configuration and this is what we change for staging and production". Helm is a way to do that, and a popular one, but it's pretty ugly. Hence this article.

<comment><USER60> This is how you do it in AWS CloudFormation:

Parameters: AMI: Type: AWS::SSM::Parameter::Value<String> Default: /aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id

That gives you the latest, fully patched base OS image no matter which of 19 regions you launch it in.

Even hardcoding that in a K/V store is going to get outdated unless you manually update it. Parameters like this are great because you can simply write your code once and never have to update unless you're adding new functionality. All base parameters and external systems (APIs, etc) are parameterized and never need to get updated, except by your SaaS partners that update them for you.

</comment></comment></comment><comment><USER61> YAML is the bastard offspring of XML. A bunch of ways to write semantically identical stuff is bullshit. Let data files be data files, and build them using languages actually suited for the job. JSON is plenty complex for 105% of the use cases of YAML, with much fewer downsides.

</comment><comment><USER62> When I joined Google, I realized that it was kinda surprising I hadn't seen something like GCL in the outside world (not to say it doesn't exist, just it's ubiquitous enough for me to hear about it). This seems to be an example of that hole being filled.

</comment><comment><USER63> Why is it so hard to write the meat of the argument in the first paragraph? So much setup...

</comment><comment><USER64> I've recently settled on typescript object literals for creating complex config files.

Adv:

\- It's still quite declarative.

\- supports expressions.

\- supports merging objects using the spread `...` operator. This enables breaking up large configs files into smaller files.

\- supports type constraints.

\- serialises to JSON easily.

<comment><USER65> Same here... just using the same language as the rest application, usually a file with constants or a class/struct/module/whatever.

the only reason config files in another language might be required is when you require configuring the application after it is compiled to native binary code.

Obviously this doesn't apply to all these applications written in scripting languages.

I do wonder how you create type-safe config files though. I currently have a `development.ts` and a `production.ts`, where the `production.ts` is only loaded when `node_env === production`. `production.ts` contains blank/default values and the file is overwritten on the server with production secrets.

<comment><USER64> > I do wonder how you create type-safe config files though

The types help prevent erroneous config declarations.

e.g

```

enum LogLevel {

        Info = "info",
        Debug = "debug"

}

const MyConfig = {

        logOptions: {

            level: LogLevel.Debug

        }

};

```

And since we serialise `MyConfig`, the config has to be transpiled by Typescript.

</comment></comment></comment><comment><USER66> I use j2cli to template my YAMLs, pretty powerful to template stuff without using ansible. You just need to right environment variables. And you can put if logic if you want.

</comment><comment><USER67> We cannot now be far off some ansible [1]-like tool using LISP s-expressions for its configuration language.

[1] or any of that ilk

</comment><comment><USER68> A consistent observation has been that all config languages tend to become turing complete over time.

</comment><comment><USER69> Next, inheritance for JSON! Look how well it worked for CSS.</comment>


Yahoo Now Requires Your Phone Number - <AUTHOR> http://imgur.com/NMZZOzr I tried to sign up for Yahoo mail, but decided against it because they required my phone number to sign up. I hope this isn't the trend. Also, I'm wondering why they are requiring more information from their users when certain things are going on right now in the U.S. <comment><USER2> What is this!? I don't even... What in that picture says anything about e-mail!?

<comment><AUTHOR> Oops, I meant phone number!</comment></comment>


Flask vs. Django: Worksheet to choose the right framework - <AUTHOR> https://wakatime.com/blog/25-pirates-use-flask-the-navy-uses-django? <comment><USER2> One of my more favorite reasons to use Django would be - [http://www.django- rest-framework.org/](http://www.django-rest-framework.org/) \- whose discoverable API browser is awesome and saves so much time in building out a great REST API with pagination, related resources, and so many other things (plus you can post from the API browser). A bit less so the built-in admin features, which are also nice at times. Possibly also the migrations system. I don't use Django templates, but the rest of it are pretty much great.

I almost started a very major project with Flask, but am glad I did not for how much work DJRF and South had saved.

<comment><USER3> [http://python-eve.org/](http://python-eve.org/)

<comment><USER4> For those who want a little context in-thread:

"Eve is an open source Python REST API framework designed for human beings. It allows to effortlessly build and deploy highly customizable, fully featured RESTful Web Services.

Eve is powered by Flask, Redis, Cerberus, Events and offers support for both MongoDB and SQL backends [*]."

<comment><USER5> I'll admit I have no idea about Eve, but having used Tastypie, the beauty of Django Rest Framework is its consistency with Django. The Serializers are very similar to forms, and the API views are very like Djangos generic class based views.

<comment><USER6> You could also use a Django Form to validate the data with Tastypie[1].

[1] - [https://django- tastypie.readthedocs.org/en/latest/validation...](https://django- tastypie.readthedocs.org/en/latest/validation.html#formvalidation)

</comment></comment></comment></comment></comment><comment><USER7> I always fall back on my bastardized version of Greenspun's Tenth Rule.

In most real-world projects started using Flask - you end up reimplementing half of Django badly by the time you're finished.

<comment><USER8> Interesting, I take the opposite approach: As your project starts to grow, you end up ripping out the "batteries" of django and implementing other modules. The most notable one would be SQLAlchemy.

<comment><USER9> Yup. I switched to Flask mostly because I always ended up ripping Django apart:

\- Jinja instead of Django templates.

\- Custom (and sane) class-based views instead of Django's disgusting defaults.

\- Custom class-based settings.

\- There's no way to define app-only middleware in Django. Flask blueprints make it trivial.

However, 10 years later and no framework has come close to Django's true strength: built-in admin.

<comment><USER10> Interesting. Almost every experienced Django dev I know throws out the admin on their projects nowadays.

<comment><USER11> In favour of what?

</comment></comment></comment></comment></comment><comment><USER12> For Python users looking for a very slim framework to build your API on, I'd highly recommend falcon [https://falcon.readthedocs.org/en/latest/](https://falcon.readthedocs.org/en/latest/)

It's built by the folks at Rackspace and it's super fast because you can build it using Cython and because it's just a very thin layer over wsgi. We switched over to it from Django and saw a 10x improvement in response time.

<comment><USER13> Django/Rails/etc are kitchen sinks. they are not optimized for "building an API", or speed for that matter. it's kind of silly to expect them to perform as well.

<comment><USER12> 100% agree, but since they're so ubiquitous, I'm sure quite a few people out there do so without thinking twice.

<comment><USER13> you're right, i hope i didn't sound argumentative. i should clarify my own point too, and add that just because they are not "built for it", or are slower, doesn't mean they are inherently a bad choice either. it's really just what you need to optimize for.

so many ways to skin a cat...

</comment></comment></comment></comment><comment><USER14> This part seemed odd:

        Flask serves JSON responses slightly faster than Django.

        Flask: 60k
        Django: 12k

A 5x increase is slightly? Yes there is another slide zoomed out showing other frameworks much faster, but still, I'd take 5x over a poke-in-the-eye with a sharp stick if I cared at all about performance.

<comment><USER15> Those numbers are meaningless without knowing what the code is doing.

<comment><USER16> It says on the site, but it's basically just a "hello world" json response, so its just comparing the overhead for each web request.

</comment></comment></comment><comment><USER17> The questions and answers, for anyone who wants a summary:

1\. Have you been coding for more than 5 years? (yes:'Flask', no:'Django')

2\. Have you coded in Python for more than 3 years?(yes:'Django', no:'Flask')

3\. Do you like reading Python code? (vs reading docs)(yes:'Flask', no:'Django')

4\. Will you have an API? (yes:'Flask', no:'Django')

5\. Will your site primarily have dynamic HTML pages? (yes:'Django', no:'Flask')

6\. Will you use Service Oriented Architecture (SOA)? (yes:'Flask', no:'Django')

7\. Using an SQL database? (yes:'Django', no:'Flask')

8\. Want new devs to already know your conventions? (yes:'Django', no:'Flask')

</comment><comment><USER18> > Complicated SQL queries are a piece of cake with Django's built-in ORM.

I disagree here or maybe "complicated SQL query" means something entirely different to me. I've found the ORM to be great for straightforward to reasonably involved queries. However the moment you start trying to improve the database performance, you need to drop to SQL. It's another story that SQL is a lot more difficult to maintain and not very DRY across modules.

</comment><comment><USER7> This bit:

> Flask > Django for APIs. For example, a nested api resource in Flask is

> @app.route('/users/<string:user_id>/teams/<string:team_id>', methods=['GET', > 'POST'])

> while with Django REST framework this is impossible.

> (drf-nested-routers helps but breaks unless you use only the most basic > Django REST framework features.

Is this really true? For a start I've managed to build a complex app without needing nested routers (although they are cool). Second - I've used DRF extensions nested routers without finding any problems.

For me needing an API is a point in favour of using Django for a lot of use cases - you get an elegant declarative syntax and lots of batteries included.

<comment><USER19> It's not true at all. I've done nested routers in DRF plenty of times.

        router = DefaultRouter()
        router.register('users/(?P<user_id>\d+)/teams', UserTeamViewSet)

You can then access user_id via self.kwargs['user_id'] from within the viewset. And the team ID is implicitly added by the router to support list (and update), and retrieve (delete, put, patch) methods. user_id is also passed as a kwarg to the dispatched method (create, retrieve, etc...)

</comment></comment><comment><USER20> > Pirates use Flask, the Navy uses Django. WakaTime is a pirate

I don't understand this. According to Netcraft, the US Navy uses Active Server Pages and the UK Navy uses ASP.NET. While there could be Django somewhere in there, there's likely also someone in the US Navy using Flask. I wasn't able to find a source which says that US Navy has decided to use Django for most of its web development - I think it would be pretty surprising for the US Navy to switch to Python over a more traditional enterprise solution!

More likely this is some pop culture culture reference I don't get. Anyone care to explain?

<comment><USER21> [http://www.folklore.org/StoryView.py?story=Pirate_Flag.txt](http://www.folklore.org/StoryView.py?story=Pirate_Flag.txt)

<comment><USER20> Ahh! Yes, that's a perfect match for the given context. Thanks!

I question the assertion "Django is older and larger, but Flask has a more active community according to GitHub." Surely the number of stars and watchers don't mean that those people are active in the project. According to the github pulse reports for the last month:

> Flask (March 23, 2015 – April 23, 2015): Excluding merges, 19 authors have > pushed 23 commits to master and 69 commits to all branches. On master, 16 > files have changed and there have been 340 additions and 43 deletions.

> Django (March 23, 2015 – April 23, 2015): Excluding merges, 124 authors have > pushed 189 commits to master and 608 commits to all branches. On master, > 2,395 files have changed and there have been 25,689 additions and 84,146 > deletions.

By those numbers it seems that Django has a much more active community, in the sense of making code changes. There are other types of activity than code commits, like support, conferences, etc. I don't get the sense that Flask is more active there either.

Edit: If it is meant as an allusion to the Mac/Apple][ era at Apple then activity is a poor measure - there was much more activity and general interest in the Apple ][ during the Mac development.

<comment><USER22> Part of the problem with comparing commits is the Django is more comprehensive. There is a larger "surface area" to make changes to.

<comment><USER20> The question really is, what's a useful way to determine which of two projects has a "more active community according to GitHub"? Is the number of stars/watchers a good enough proxy to use for decision making?

The projects I use are typically quite small, and my code is on Bitbucket, so I have effectively no experience in this.

</comment></comment></comment></comment></comment><comment><USER23> I've been using django pretty much since 1.0 came out. Every project I start these days thats a web project, I use django.

The main reason why I use django is because o the ORM and the admin. If I were to do a project in Flask, I'd have to build mt own admin, and i'd have to use SQLAlchemy. I had to use SQLAlchemy once for a project and I hated it.The django ORM is much more straigh forward an easier to use, in my opinion. In my 10 years of using django ORM, I have had to "break out of the ORM" maybe once.

<comment><USER24> How do you manage with just one primary key column allowed by the ORM?

<comment><USER7> Some people argue that composite primary keys are an anti-pattern. They might be relationally pure but they make other things a lot harder.

I have a hunch that uuid's or other non-semantic primary keys are a good example of of 'practicality beats purity'.

</comment></comment></comment><comment><USER25> I wish they added bottle.py [1] to the mix...

I'm a huge fan of bottle.py, it's simpler/cleaner, but does lack of fucntionalities and plugin. And you'll notice that it is much higher on the the JSON serialization chart at 13% (flask and django are at 5.7% and 1.1% respectively).

1: [http://bottlepy.org/](http://bottlepy.org/)

</comment><comment><USER26> Perhaps a bit off topic, but what's the reason Flask hasn't had a release in almost 2 years?

<comment><USER27> Probably because most of the commonly used components are maintained separately.

<comment><USER26> Sure, but there's been almost 700 commits. There must be something in there worth a release.

</comment></comment></comment><comment><USER28> What about Python 3 adoption? Django feels a much safer bet in this regard.

</comment><comment><USER29> "It's a tie! Read about Django vs Flask to make your own choice."

Thanks, this really helps a lot! LOL

</comment><comment><USER30> What about Pyramid?

<comment><USER31> Personally, I like pyramid as a middle-ground between the two. More to get you started quickly a la Django; less to get in your way as you grow like flask. It tends to hit my needs right on the spot.

</comment></comment><comment><USER32> I can't access the site. Is it down? Blocked?

Anyone have a mirror?

<comment><AUTHOR> Alternative url:

[https://github.com/wakatime/wakatime- blog/blob/master/posts/...](https://github.com/wakatime/wakatime- blog/blob/master/posts/25-pirates-use-flask-the-navy-uses-django.md)

The page loads for me just fine.</comment></comment>


Photocracy.org = Revolutionizing photo-sorting? - <AUTHOR>

Hi everyone, For my Senior Thesis, I'm working on implementing pair-wise comparisons for sorting photos, in particular for national perception and cross-cultural identity (I'm an East Asian Studies Major). As of now, there are limited implementations of pair-wise comparisons for text or photo (bix.yahoo.com as one of the few).
We're preparing to launch on March 16th and are hoping to get as much activity as possible after that point (hypothesis/assumption: the more people in the crowd, the better the resulting data set). We launched a text-based version for student government issues with a fair amount of success in the Fall of 2008 ("WDYWM").
Any and all suggestions for spreading the word or making improvements are very much appreciated!
Josh Weinstein <comment><USER2> "These photos have won" is far below the fold -- I didn't see it until it was over. When it is over, the middle part of the screen is blank -- no game over, no thanks, nothing else to do. You need to follow up with the user to keep them engaged.

I don't know much about China, or these sample photos, though I got the impression that most of it was "popular symbol vs present-day life".

I am wary of what actual insights you can get with this kind of sociological game... seems to me pretty fuzzy, especially since you don't have any data on the users. "Pairwise" and statistics don't mean it's good science.</comment>


McNealy: I would have run HP if they'd asked me  - <AUTHOR> http://www.networkworld.com/news/2011/100811-mcnealy-i-would-have-run-251755.html?hpg1=bn

<comment><USER2> I don't know too much on his history tbh, but didn't Sun go to hell when he was CEO?

<comment><USER3> fta:

      *McNealy, 56, co-founded Sun in 1984 and was its top executive for 22 years, making him one of the longest-serving CEOs in Silicon Valley history.*

he both built it and oversaw its fading</comment></comment>


George Soros on the Coming U.S. Class War - <AUTHOR> http://www.thedailybeast.com/newsweek/2012/01/22/george-soros-on-the-coming-u-s-class-war.html <comment><USER2> There's been a class war here for a long time. The difference is that it isn't one-sided any more.</comment>


Mazviita Chirimuuta: Color is a dance between your brain and the world - <AUTHOR> http://nautil.us/issue/26/color/ingenious-mazviita-chirimuuta

<comment><USER2> I really dislike this article. It stretches the "What if your blue is my red?" concept too far, and makes it sound far more dramatic than it really is.

Let me clear something up first. Our senses are ways of acquiring information. What we sense (i.e. colors, different pitches) are just the categories with which we sort them. As long as we can agree on some common definition (i.e. wavelength for a color), they have done their job.

The talk about the subjectivity of color is misleading. The only thing that is subjective, is how our brain "parses" the information of these categories (i.e. my red vs. your red). But, because we can't share the brain's abstraction of color with one another, the discussion is meaningless.

Thus, while there is an aspect of colors that is subjective, it is impossible to think about outside of your own, and thus the discussion of color in terms of wavelength is the only plausible discussion.

\-----

My problem with the article lies in the entirely inadequate account from the interviewee. Regarding her theory of color, she says: "... we need a way of theorizing subjectivity in such a way that we'll just acknowledge that there are parts of our experience and our perceptual knowledge of things that are generated by the particular ways that we interact with the world." Which basically says "Our brains will parse the image differently, we need a way to discuss the results," and is pretty much useless. Then she claims that color is a combination of our interpretation of color and the properties of objects.

Two possibilities: this might be just wrong, as she is conflating the color in the sense of wavelength and color in the sense of properties of objects that reflect light. These are clearly distinct.

Alternatively, she is referring to the brain's interpretation of a scene given the information from the eyes. That is, given the "bitmap data" the eyes take in, the brain attempts to "parse" the scene, and adjusts colors for light sources. Well great, mystery solved, but we knew this already. So what exactly is her "theory" that she is contributing?

To my understanding, the entire article is an attempt to confuse different meanings of the word color.

</comment><comment><USER3> > When I asked Mazviita Chirimuuta why philosophers were so crazy about color, > she smiled, as if she sensed my question had an implied criticism of > philosophy's penchant for chewing more than it bit off, to paraphrase what a > wit once said of Henry James.

Too pretentious for my blood.</comment>


Insights on the First Three Years of the Right to Be Forgotten at Google - <AUTHOR> https://elie.net/blog/web/insights-about-the-first-three-years-of-the-right-to-be-forgotten-requests-at-google <comment><USER3> The article omits what I think is probably the most important datum: _' 0.2% of requesters accounted for 20.8% of all accepted delistings. Many of these frequent requesters were individuals using law firms and reputation management services.'_ That's a total of about 214,000 URLS deleted on account of 1000 individuals. The most prolific requester requested 5,768 URLs relating them be removed. It's unstated how many of these were approved.

The sort of individuals using law firms and reputation management services to try to erase information about themselves from public access, are generally going to be the individuals whose information has the most probable public utility. But they're also the individuals who can most threaten Google with lawsuits and other difficulties should they not comply.

The internet is something that will live on long after we're all dead, and it is creating a living chronicle of the times. The more this chronicle is interfered with, the less valuable this information will be. Einstein's 'marital demands' [1] (not what you might think) of his wife would almost certainly fall within this 'right to be forgotten' yet they now provide a all too rare glimpse at the less 'sculptured' life of one of the most important figures in history. This yin and yang, good and bad, nature of people helps provide a more balanced, and less idealistic view of actors in the past - <USER2> present.

Perhaps the thing becoming ever more clear is that search is something that needs to be decentralized. It's all too easy for the player in control to exploit the data for their own personal benefit, and they also end up working as a single point of failure for access to said information.

[1] - [http://www.openculture.com/2013/12/albert-einstein- imposes-o...](http://www.openculture.com/2013/12/albert-einstein-imposes-on- his-first-wife-a-cruel-list-of-marital-demands.html)

<comment><USER4> Maybe Google can "unforget" after people die.

<comment><USER5> Does the right to privacy end with death?

<comment> Paul-ish What harms could happen to someone who is dead?

<comment><USER6> You can most definitely harm the family of the dead person.

<comment> Paul-ish They should send their own request in. The request should be evaluated with respect to the living individual's circumstance, without regard to the deceased.

</comment></comment></comment></comment></comment></comment><comment><USER7> To me, the most concerning thing is the level of government officials requesting delisting. Democracy requires transparency. If you've committed crimes, had financial judgements against you, or even been accused of sexual misconduct, I want to know about it.

Powerful or high network individuals or corporations requesting delisting should have a much higher bar.

<comment><USER8> _had financial judgements against you,_

You had me with the other two, I don't know about this one. In the absence of criminal financial malfeasance, or deliberately poor mishandling of instruments an official had some responsibility over as an individual acting in the public trust, I can't think of a compelling argument for why _this_ needs to be the public's business.

I can only presume, but I'm with pause to put words into your mouth that this is probably want you meant?

<comment><USER9> I'm not quite sure whether "accused of sexual misconduct" is that great either. Anyone can accuse anyone of anything, and especially on the internet, these things have a way of getting a life on their own. And especially in the case of public figures, these things can be an easy way to blackmail them/bring them down.

Not that I'm in favour of Google acting policeman, but these things are just pet peeves of mine that I think we should be careful with.

<comment><USER10> Given two headlines:

\- "foo suspected of sexual misconduct" @t=0

\- "foo proved to be innocent in sexual misconduct case" @t+1 mo/yr/whatevs

Guess:

\- what people will consciously or subconsciously remember when scanning through Google results

\- which one will get the most clicks/views/pagerank at various points in time

That is, if they ever continue and look for the second one. And even if it were the last one, many (most?) people will be subconsciously biased anyway by the shock effect of the foo <-> sexual misconduct tie, which obliterates the "innocent" part.

<comment><USER11> People aren't proven innocent (at least in the US), they're found "not guilty", which is significantly distinct

</comment></comment></comment></comment></comment><comment><USER6> I am a pretty big supporter of Right to Be Forgotten as a thing, and thought this was an excellent summary of how it's gone so far! When you add the 89% of the users being "private individuals", and that the next biggest category, 4.4%, is minors, it seems most of the requests definitely fall within the purpose intended.

Without going into the details, 43% delisting rate seems likely quite fair, it doesn't seem like it's being over-implemented by any means, but a good number of requests are indeed being approved, this is a lot better than I expected, to be honest.

<comment><USER12> I'm the exact opposite: I believe that history should _never_ be rewritten nor forgotten. The increasing "transientness" of information is, quite frankly, extremely scary --- when future historians looking back may see only a whitewashed representation of what things were really like.

Besides, Google forgets enough even without being told to:

[https://news.ycombinator.com/item?id=16153840](https://news.ycombinator.com/item?id=16153840)

<comment><USER13> Why should history not be forgotten? It used it be that way all the time before we managed to build information storage large enough to capture this huge amount of data. And at what cost does it come that we never forget? Not all too long ago (1993-1997), there was a large trial against 25 people for alleged child abuse in germany. (1) All were eventually cleared, one person died while in prison before the trial ended. It was one of the largest case of false accusations for child abuse. The names were all over the press. Should those people not have the right to have that memory fade? To have history be forgotten at some point?

> Besides, Google forgets enough even without being told to:

That was not the case when the laws were drafted and enacted.

(Wormser Prozesse, sorry, german only [https://de.wikipedia.org/wiki/Wormser_Prozesse](https://de.wikipedia.org/wiki/Wormser_Prozesse))

<comment><USER14> Should those people have the 'right' to be remembered, falsely, as child abusers? Should anything and everything every written about the case be expunged? Should individuals be responsible for purging their diaries of such information too? Why not? Someone might upload it to the Internet in the future and, because everyone else officially 'forgot', who will remain to remember that this is officially prohibited information?

</comment></comment></comment></comment><comment><USER15> I have been following Elie and his team's work for a while now, and enjoyed this read, but I am curious as to why it was posted on his personal website as opposed to Google's Research Blog ([https://research.googleblog.com](https://research.googleblog.com)). He is one of four authors and this was clearly not a personal effort of his. Not to mention, it would reach a larger audience if shared by Google directly.

<comment><USER16> Hey :)

We did have an official post today: [https://www.blog.google/topics/google- europe/updating-our-ri...](https://www.blog.google/topics/google- europe/updating-our-right-be-forgotten-transparency-report/) that is focused on the improved transparency report rather than the research paper and I am sure it will reach a larger audience. You will recognize the infographic that I borrowed from it :)

When we make a paper public I have the habit to share a short summary (sometime with a delay thus) of what the paper is about on my personal blog. Kurt, Luca and Yuan helped with the summary on a personal capacity which is why they are co-author of the blog post.

Bottom line this is not the official post, we do have one, it is just a summary a few authors of the paper wrote in personal capacity to share some of the finding we found the most interesting.

<comment><USER15> I didn't see the Google Blog post - thank you for pointing me in the right direction! And great work.

</comment></comment></comment><comment><USER17> I've been wondering: How will Right To Be Forgotten be implemented in blockchain products, where forgetting is supposed to be impossible? Am I misunderstanding something fundamental about blockchains?

<comment><USER18> Indeed, this is an excellent thing to consider when you are using immutable databases. GDPR is another such requirement that people are now having to struggle with. Essentially, you need to be able to rewrite the DB with the new information. For merkle chains that are in the public record (like Google's scheme to to publish certificate requests so that CA shenanigans are obvious to anyone that looks -- the name of which escapes me at the moment), it's not going to be reasonable. It's not so much the merkle chain, but the fact that the data is _already_ distributed. Getting everybody to resign the chain would be pretty hard, but getting them all to delete the data is unreasonable... "Pretty please. We got asked by a judge, you know!"

So essentially, you need to make sure you don't put that kind of information in that kind of DB.

(P.S. It's a bit of a shame that you're getting down voted for such a good question, but I suggest you substitute "merkle chain" for "blockchain" when you want to ask technical questions of this sort.)

<comment><USER19> I believe the program you're looking for is Certificate Transparency. To be honest, write-permanently databases have been around for a while (for example pgp keyservers that peer and distribute keys relatively quickly). Basically, it is a problem unsolved except by having seriously smart people who do this stuff for its own sake, hacking at the data directly, and are distributed across and able to effectively communicate from many jurisdictions.

</comment></comment></comment><comment><USER20> So are there any search engines that ignore RTBF?

Edit: Public ones, I mean. I'm sure that major TLAs do.

</comment><comment><USER21> We need to be careful with our words. There is no "Right to be forgotten". Framing[1] denotes the battlefield terrain, if you accept the frame you've already lost. A better title might be: Insights on the First Three Years of European Censorship of Google.

1 [http://www.nytimes.com/2005/07/17/magazine/the-framing- wars....](http://www.nytimes.com/2005/07/17/magazine/the-framing-wars.html)

<comment><USER22> _There is no "Right to be forgotten"._

In the context of Google's search results, according to the European Union's "Data Protection Directive" that's based on an interpretation of the OECD's definition of privacy, yes there is. You can't just declare there isn't such a thing as the Right to be Forgotten as fact. You may choose not to recognise it, but there definitely is such an _idea_ that many people believe is a right.

In a wider context there are many countries where case law implies there is a Right to be Forgotten too, including the US. [https://en.wikipedia.org/wiki/Right_to_be_forgotten](https://en.wikipedia.org/wiki/Right_to_be_forgotten)</comment></comment>


Show HN: How to Get into a Programming Bootcamp - <AUTHOR> http://www.codecampguide.com/

<comment><AUTHOR> Author here. I graduated a programming bootcamp in 2013. After graduation I worked for a startup for about a year and half. I had a lot of fun and made a bunch of money. Now I'm traveling thorough Asia. I've coached several friends on how to get in and decided to put the time and make a guide out of it.

<comment><USER2> That is cool. Are there specifc bootcamps that you think people should avoid? Have you thought of running your own bootcamp inside of just writing a guide?

<comment><AUTHOR> There are a few camps that come to mind. But it's possible they have changed since.

However, I do have some tips on how to choose a good one. Generally the bad bootcamps will fail at the things I list here. [http://www.codecampguide.com/blog/2015/3/23/choose-the- right...](http://www.codecampguide.com/blog/2015/3/23/choose-the-right- bootcamp)

I'm currently traveling around the world so I have no desire to run my own camp. Would rather help people as best I can from my laptop :)

Feel free to reach out if you have any other questions!</comment></comment></comment>