Overnight Brain Stimulation Improves Memory - <AUTHOR> http://www.jneurosci.org/content/38/33/7314 <comment><USER2> Read this before self-experimenting:

"Popular electric brain stimulation method used to boost brainpower is detrimental to IQ scores"

[https://www.sciencedaily.com/releases/2015/05/150505152140.h...](https://www.sciencedaily.com/releases/2015/05/150505152140.htm)

</comment><comment><USER3> Looks like they use Transcranial Alternating Current Stimulation (tACS).

The military has been doing tDCS (direct current) experiments for a while for increased learning & performance.

I had a friend who used to swear by tDCS and did it very frequently. He attributed all kinds of mood & cognition improvements to it, including substantially reduced need for sleep, improved eyesight (he was born partially blind), etc. He also complained about migraines a lot and liked to experiment with all kinds of supplements, so YMMV.

Tried tDCS myself a handful of times, not for improved learning but for improved focus. If applied correctly it definitely has an effect that I'd consider better / stronger than a cup of coffee. I wouldn't feel comfortable doing it regularly though.

There's a subreddit that used to be pretty active: [https://www.reddit.com/r/tDCS/](https://www.reddit.com/r/tDCS/)

And here's a good Radiolab podcast, including DARPA Sniper experiments, etc: [https://www.wnycstudios.org/story/9-volt- nirvana/](https://www.wnycstudios.org/story/9-volt-nirvana/)

<comment><USER4> Some of the posts on that subreddit are seriously concerning. Burns on your head/arms seem to be common. One company in particular, "Foc.us", seems particularly dangerous.

<comment><USER5> Interesting tidbit: the foc.us manual says: "If you see white flashes (known as phosphenes), adjust the position of the foc.us headset away from your eyes"

<comment><USER6> This seems like a really stupid thing to experiment in. Wait a few years and find out you permanently damaged your brain for a "caffeine-like" boost.

</comment></comment></comment></comment><comment><USER7> We already do this it's called dreaming.

</comment><comment><USER8> Interesting to consider if an unintended common cause of tACS is better encouraging lymphatic cleaning processes, especially where evidence is accumulating of the role of the dura mater.

[https://www.nature.com/articles/d41586-018-05763-0](https://www.nature.com/articles/d41586-018-05763-0)

</comment><comment><USER9> Can someone kindly explain this to me? I feel bad asking, but, I honestly don't understand the abstract =(

<comment><USER10> By no means my area of expertise, but a quick glance suggests:

The experimenters set up a system that could detect slow-wave (SW) fluctuations in brain activity. They used trans-cranial magnetic stimulation (electro-magnets on the scalp) to "augment" this SW activity. This seems to have improved memory, measured by some specific learning task they set for the participants.

Trans-cranial magnetic stimulation is becoming popular in trans-human type circles. Apparently it's becoming feasible to do at home with cheap off the shelf components. However, of course, the long term effects and side effects are poorly understood.

This particular experiment used 16 participants (only three of whom were female, for some daft reason) so we can't draw any firm conclusions from this.

<comment><USER11> Not magnetic, electric it looks like. I'm not sure how I feel about running electrical current through my brain.

<comment><USER12> ECT(Electroconvulsive therapy) is already a thing - it is used for various treatments for depression and seizures. It consists of small electric currents passing through the brain which is very similar to brain stimulation, the only difference is that he first one is a medical procedure while the latter is, as of now, more like a brain hack. I wouldn't recommend anyone doing it without knowing the risks involved.

<comment><USER13> I don't think it's accurate to describe ECT as using small electric currents. The very name says that the procedure induces convulsions.

From Wikipedia: "ECT is often used with informed consent as _a last line of intervention_ for major depressive disorder, mania, and catatonia. ECT machines have been placed in the Class II category (special controls) by the United States Food and Drug Administration (FDA) since 1976." (Emphasis mine.)

<comment><USER6> It sounds like hard rebooting a mind.

</comment></comment></comment></comment></comment></comment><comment><USER14> Is there some aspect of this that's quite new?

I'm reading Why We Sleep (Matthew Walker), published 2017-09 - an HN recommendation from a while back - and I've just read the chapter where he describes an experiment that sounds exactly like this (as in, same approach, same outcomes).

</comment><comment><USER15> Binaural beats / audio-based brainwave manipulation seems a much, much safer approach!

<comment><USER16> is there any evidence that this can actually work?

<comment><USER17> Anecdote: I self-experimented with binaural beats some years ago. I did not notice anything regarding concentration, but I did find that tones starting in the normal awake/aware range and sliding linearly down to the slower, deep sleep range over 15-20 minutes would reliably cause me to fall asleep.

I ultimately layered that audio behind a guided meditation relaxation track I found online and used the combination as a sleep aid for a few years.

</comment></comment></comment><comment><USER18> I want to buy one of these devices but it seems risky and difficult to do so.</comment>


Original Grim Fandango design document released [pdf] - <AUTHOR> http://www.doublefine.com/themes/site_themes/default/freestuff/GrimPuzzleDoc_small.pdf <comment><USER2> Ah, the one game that should've had a sequel but did not. I would've traded Monkey Island 4 for GF2 gladly.

PS The tears of joy box on the very last page was a great touch.

<comment><USER3> How would they do a sequel? By the end of the game, everyone's dead.

<comment><USER4> At the start of the game, everyone's dead.

</comment></comment></comment><comment><USER5> The good old adventure game genre, seemingly dead today except for perhaps Psychonauts, but the home of so many great games.

My favorite is Star Control II; perhaps a bit more action-oriented than a "pure" adventure game, but still in many ways true to the concept. And, like many adventure games, brilliantly hilarious at times (Zoq-Fot-Pik, anyone?)

<comment><USER6> SCII is legendary! One of my all-time favourites too.

<comment><USER5> And I forgot to mention, for those who don't know already, you can get the game for free for a modern computer as _The Ur-Quan Masters_ ( <http://sc2.sourceforge.net/> ), due to the open-sourcing of the original game's code. It runs natively on Windows and Linux (and I assume OS X), and comes complete with all the audio of the 3DO version, too.

<comment><USER7> I discovered it not much time ago and it resulted in a sleepless night... very good game

</comment></comment></comment></comment><comment><USER8> The best game ever created.

<comment><USER9> Fantastic game. I actually thought Psychonauts was even better. Tim Schafer is a very rare gem.

</comment></comment><comment><USER10> [http://www.google.com/products?q=grim+fandango&btnG=Sear...](http://www.google.com/products?q=grim+fandango&btnG=Search+Products&hl=en)

It looks like the game is still selling for $30+ in retail. I thought it would be a lot cheaper by now.

</comment><comment><USER11> I must be apart of the minority, but I still need to play this game. Added to my list of "things to do after I've made bank".

</comment><comment><USER12> this game changed my life

<comment><USER13> How?

</comment></comment><comment><USER14> I never played the game, but this design doc is truly fascinating. Def in my top 5 HN finds.

</comment><comment><USER15> nostalgiastart

I was into Day of the Tentacle and Full Throttle but I never bought GF.

/endnostalgia

<comment><USER16> Day of the Tentacle was one of favorite adventure games ever, but there was really something special about Grim Fandango. Off the top of my head, I can't think of any other game that combined superb art direction, music, ambiance, story and memorable characters to the same extent as Grim Fandango.

I don't think it's too much hyperbole to say that Grim Fandango was the swan song of an entire genre.

Now I'm getting sad :(

</comment></comment><comment><USER17> THE greatest game ever.</comment>


Lightmod – A beginner-friendly, all-in-one tool for full stack Clojure - <AUTHOR> https://sekao.net/lightmod/ <comment><USER2> Five years after Bret Victor's Inventing on Principle[1], it's crazy that there are not more coding environments that give new users a great out of the box experience with live reloading like this.

I started learning Clojure recently and had a bad experience configuring a development environment on the Mac. Leiningen threw "illegal reflective access" and "Unable to open cgroup memory limit" warnings that appeared to be related to an incompatible version of the JRE. I ended up fixing it (JRE downgrade, then upgrade after it was patched in Leiningen), but it was still a less than smooth introduction to a functional language compared with Elm, Elixir, or even Haskell.

The advice from many Clojurians to use Emacs is fine if you happen to use it anyway, but also seems particularly hostile to beginners.

Lightmod compiled all the demo examples without error for me.

I really want to explore Clojure and ClojureScript to build game servers and web games, and this project looks like a great way to get people using the language faster.

[1]: [https://vimeo.com/36579366](https://vimeo.com/36579366)

<comment><USER3> I ran into that bug too. It's the first bug that I've encountered with Leiningen after using it for more than five years. Sorry that that was your first experience with the toolset, but it's far from the normal experience.

Also, if you're happy using an editor other than Emacs, then go for it. I use it but only because I've been using it for quite some time, and other than the fact that I like the way emacs indents Clojure source there's nothing special about it in the way that I use it. Visual Code has nice support for the language so that's a good option if it's more up your alley.

One area where I see Lightmod being really helpful is setting up the initial project. Learning what's going on w/cljsbuild, figwheel and then how to combine that with Clojure in the same project file isn't the simplest thing in the world to say the least.

<comment><USER2> Thanks for the tips and reassurance – it's good to hear that using Visual Code is acceptable; it's what I'm more comfortable in, and I'll persevere with it!

<comment><USER4> It would be really useful if VSC supported the Emacs indentation style: it's what most people use in real Clojure code.

Paredit would be brilliant too ([http://bit.ly/paredit](http://bit.ly/paredit) if you're not familiar)

<comment><USER2> It looks like there's an extension for paredit, but not sure how it compares to Emacs:

[https://marketplace.visualstudio.com/items?itemName=clptn.co...](https://marketplace.visualstudio.com/items?itemName=clptn.code- paredit)

For formatting, I've been using this, which supports cljfmt:

[https://marketplace.visualstudio.com/items?itemName=avli.clo...](https://marketplace.visualstudio.com/items?itemName=avli.clojure)

And the rainbow brackets extension has made code more legible while I've been learning:

[https://marketplace.visualstudio.com/items?itemName=2gua.rai...](https://marketplace.visualstudio.com/items?itemName=2gua.rainbow- brackets)

</comment></comment></comment></comment></comment><comment><USER5> There's a good cognicast[1] (podcast) with the author of this editor, he talks about the motivation of making such editors.

[1] [http://blog.cognitect.com/cognicast/130](http://blog.cognitect.com/cognicast/130)

</comment><comment><USER6> Difficulty with setup certainly is a huge pain point with me using Clojure, the amount of time I've spent wrestling with different libraries were just not _working_ is way too high, and is probably the number one reason I don't use Clojure today. Clojure desperately needs something like Anaconda to simplify setup, this looks like it might help.

<comment><USER7> Really? Which libraries? Did you try using leinengen? One of the strengths of Clojure imo is how well everything JUST works as opposed to python, ruby, etc.

<comment><USER6> I tried using leinengen and boot, each had it's own issues. I remember spending many frustrating hours trying to get the live reloading to work.

I feel like a lot of the Clojure ecosystem is quite hacky, and if you can hack around in clojure you can work around many of the problems, but if you're a clojure novice each tiny hiccup is minutes to hours spent trying to figure just what is the problem.

</comment></comment></comment><comment><USER8> Not to be confused with LightTable[1], an editor with Clojure live coding support.

[1] [http://lighttable.com/](http://lighttable.com/)

<comment><USER9> Is this still actively maintained? I haven't kept up with the project but my impression was the original creator moved on and left the project in the cold

<comment><USER2> [http://witheve.com/deepdives/lighttable.html](http://witheve.com/deepdives/lighttable.html)

The maintainer decided that a more fundamental change to programming was required to improve the experience as radically as they hoped.

They're writing a new language and literate programming environment called Eve:

[http://witheve.com/](http://witheve.com/)

It's early days, but the language has some very interesting ideas so far.

<comment><USER10> Eve is - was, when I last checked - a poor reiteration of Smalltalk with a bit of CWEB bolted on top. You can get much more mature implementation if you just download Pharo, with the added benefit of Pharo being native, not browser- based.

Chris is one of very few people in tech that I genuinely dislike, because he managed to get me to believe in him and in LighTable, only to dump LT and make me feel stupid. LT was an ambitious project, promising to finally deliver something better than Emacs for working with dynamic languages and there was a chance of it being completed in this decade. Eve is a moving target and after two years it's not any closer to being a practical tool.

Personally, I need a modern, powerful editor, preferably polyglot, with tight integration with languages runtimes and Emacs-level scriptability - and that's what LT promised. It could have been a real, immediate improvement in how I work with code. Switching to Eve development is adding a gorilla and a whole jungle in fire to the banana we (LT backers) originally needed. Not to mention, Chris failed to complete a simpler project, I see no reason to believe he'll do a good job on Eve.

/rant

</comment></comment></comment></comment><comment><USER11> Highly appreciate Zach's Nightlight editor - it's one of the best Clojure editors around in terms of design, simplicity and ease on usage. [https://sekao.net/nightlight/](https://sekao.net/nightlight/)

</comment><comment><USER12> For those looking for an out-of-the-box web app platform for Clojure you can checkout Solvent:

[http://www.codesolvent.com/static-assets/gif-studio/lang- tes...](http://www.codesolvent.com/static-assets/gif-studio/lang- test/shot.webm)</comment>


Japan approves first human-animal embryo experiments - <AUTHOR> https://www.nature.com/articles/d41586-019-02275-3 <comment> La-ang [In 2017, Nakauchi and his colleagues reported the injection of mouse iPS cells into the embryo of a rat that was unable to produce a pancreas. The rat formed a pancreas made entirely of mouse cells. Nakauchi and his team transplanted that pancreas back into a mouse that had been engineered to have diabetes, The rat-produced organ was able to control blood sugar levels, effectively curing the mouse of diabetes1.] WOW :O

<comment><USER2> A good day for diabetic mice!

<comment><USER3> Was it the hitch hikers guide that said, "As a result of their time on Earth, mouse scientists had developed the means to cure all mouse diseases, allowing their civilization to survive in perpetuity."

</comment></comment></comment><comment><USER4> I hoped technology would reduce animal exploitation. It does in some areas, like cell farming to create food without animals. And synthetic models for pharmaceutical research. Maybe this branch of research can evolve to eliminate animals as well.

<comment><USER5> There will still exist a fascinating dichotomy, up until we can predict result without either simulation or animal models. An ideal engineered organism would be a delightfully weird homunculus, and a sufficiently advanced simulation could be deemed to be a living creature too.

Perhaps it is best to make sure that lab technicians feel a great deal of empathy for the animals they work with.

<comment><USER4> Have to be careful, though. Too much empathy and you'll quit your job.

<comment><USER5> I would hope instead that they refuse to do certain procedures. Then if someone else gets asked to do the same procedure in hope of less empathy there should be alarms going off.

The lab technicians will after all be the experts on the subject; not academics on an ethics board.

</comment></comment></comment></comment><comment><USER6> These damn ethics committees are just slowing down much needed progress. Let's not pretend that DARPA and other black military scientific experimentation sites around the world have, and are, operating under the ethical radar and keeping their findings classified and to themselves and highly doubtful to the betterment of mankind. Let's just call a spade a spade and loosen the restrictions on the scientific community that actually operates with the public good in mind.

<comment><USER7> Let's move fast and break things, including our ethics.

Counties have done unethical experiments on humans before, and maybe they do it now. Why should we hold ourselves to a standard, if other groups do not? I couldn't possibly think of a reason why the scientific community would hold themselves accountable, rather than think they know best and act as if they were superior to the larger community they work to improve. I mean, that mindset has never gone wrong.

Personally, I like seeing a community doing some attempt at self regulation. Maybe it's not perfect, but it's better than nothing.

<comment><USER8> _Let 's move fast and break things, including our ethics._

Why not? That's how we got vaccines, isn't it? (Stated completely without any ironic intent.)

</comment></comment></comment><comment><USER9> As usual, Japan is ahead of things. Maybe in not-so-distant future we'll see human DNA modification. It's the only way towards, as devolution will dull us otherwise with all these bad, useless habits too available.

</comment><comment><USER10> I understand (and agree with) all these moral related questions and why so many downvotes in this topic. What is interesting, when one nation develops this tech and their people will live, say, 150 years, will these morals survive in other countries?

<comment><USER11> Like iPhone manufacturing, the West will buy the finished product before it allows the experimentation under its roof.

<comment><USER10> Might be. But it is quite different from iPhone manufacturing. It actually could more dangerous than nukes. West will just buy this you say, but why it is not possible now to just buy several intercontinental ballistic missiles? There are people with enough money and desire to buy this.

<comment><USER12> > but why it is not possible now to just buy several intercontinental > ballistic missiles?

It probably is. You'll have to buy them from North Korea in secret though.

</comment></comment></comment></comment><comment><USER13> Good for mice i guess. Bad for rats.

<comment><USER14> Excellent analysis.

<comment><USER13> In my defense, it's not really my area.

</comment></comment></comment><comment><USER15> When I heard Alex Jones on Joe Rogan saying that there was human-animal experiments being made I thought he was exaggerating.

In my opinion this kind of research crosses some ethical lines that shouldn't be crossed. If one considers an embryo as an independent life, what species the embryo belongs to? Wouldn't be the ones in this experiment half humans with human rights? After all, the scientist doesn't know all, and the cells he put in could evolve to a chain reaction of the hybrid developing human conscious.

Just my opinion tho, as I know nothing about genetics and biology.

<comment><USER16> I saw you had been downvoted and I upvoted you, despite disagreeing very strongly with your opinion, because you stated your opinion without vitriol and while I might disagree I'll still fight for your right to say it as long you do so in a way that isn't hurtful.

I think the line of inquiry into replacement organs can be researched ethically but the how of the research may cross ethical lines if not done carefully. Given diseases we have, I think we need to do the research and find an ethical way to make this kind of organ production happen, if we want to extend human lifespan.

<comment><USER15> I knew I would be, but its fine. Thank you for your words! I share the same belief regarding freedom of speech.

About the research, I hope I am wrong and the advances that came out of it help save lives and extend the human life. I just wish they take those ethical questions in consideration too.

</comment></comment></comment><comment><USER17> It beats organlegging (which is a real thing now) but we should be able to regrow our own organs and limbs.

Becker, "The Body Electric" [https://en.wikipedia.org/wiki/The_Body_Electric_(book)](https://en.wikipedia.org/wiki/The_Body_Electric_\(book\))

But then he died and most research dried up, but now there's Levin: [https://ase.tufts.edu/biology/labs/levin/](https://ase.tufts.edu/biology/labs/levin/)

</comment><comment><USER18> How does the human part affect cognition? A stem cell for example is before it "decides" to be a specific cell right? If that is the case in terms of embryo development, then how do they coax only specific cells in embryos to only express themselves in parts of the body excluding the brain. Complicated but I really am not understanding some basic principles here.

<comment><USER19> Yes you are missing basic parts. There are certain genes that produce proteins that then bind to other control areas of the DNA that affect how much other genes are expressed. These last genes translate to proteins that make e.g. a pancreatic cell what it is. But this is a tree of increased specialization on which embryonic stem cells become e.g. blood stem cells which then specialize to rbc, white platelet producers.

</comment></comment><comment><USER20> Yeah America has all the moral debates about genetics and experiments with embryos. But the bottom line is other countries are going to move forward no matter what America does.

<comment><USER21> While I disagree with the US's stance, ignoring your morals because other countries are doing something would be a race to the bottom.

<comment><USER22> _> ignoring your morals_

If those morals say people should die instead of getting the organ transplants they need, maybe we are better off ignoring such morals.

<comment><USER23> The moral is that embryos deserve the same rights as people, so you're really just killing one person to save another

<comment><USER24> No it is not.

Viability matters here. I realize you see that differently. Realize others will ALWAYS differ on this. ALWAYS.

Frankly, this will absolutely be done. There probably will be wars over it too.

And it still will be done.

And then it will be done sans an embryo, using stems.

Right now, powerful, wealthy people are paying full attention. The one thing they cannot buy will end up for sale. More time.

Buckle up. It is going to get super ugly.

<comment><USER25> He's not saying the idea is accurate, he's simply stating the idea. No need to argue unless you don't think he's representing the viewpoint accurately.

<comment><USER24> I think it is accurate, and we all see that how we see it.

There is not a wrong, just grave differences, IMHO, unresolvable ones. At least for a considerable time.

That was the intended point.

People are going to really struggle over this area of tech.

Not so much an argument as it is, look out. Incoming!

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> Isn't it highly likely this is already being done by various other nations already in secret labs?

</comment><comment><USER27> Alex Jones Heavy Breathing

</comment><comment><USER28> Why all comments are being down-voted?

(Personally... I think this tech is amazing and I can't wait for it to be available worldwide!)

<comment><USER29> You mean the South Park references that add no contribution to this scientific article? Or people broadcasting their moral opinions into the void?

I don't see any good comments downvoted.

<comment><USER30> Yeah for example the super-profound and constructive top comment that just quotes the article and adds "WOW :O".

That's quality content, unlike a the South Park reference drawing a simile between serious research and the (humorously) exaggerated potential disasters of combining human DNA with other species.

Obviously the second was made by some common peasant lacking the intelligence to engage with the audience here. The first, clearly being the work of an intellectual powerhouse the likes of which we stand in awe of, has rightly collected the praise and votes they deserve.

</comment></comment></comment><comment><USER31> Finally, Japan can make good on the real life creations from their anime movies.

<comment><USER32> _Ed...ward..._

</comment></comment><comment><USER33> Poor rats. Poor mice.

</comment><comment><USER30> man-bear-pig origin story confirmed

</comment><comment><USER34> The first thing that came to my mind was "Edward...".

</comment><comment><USER35> Yes, it's pretty crazy how well this already works.

A toast to the manpig that shall grow my future replacement liver!

<comment><USER36> We must all stop manbearpig.

<comment><USER37> Not here, please.

</comment></comment></comment><comment><USER38> grow human organs inside rats and? how does that work? i don't understand. meanwhile we are entering the sixt mass extinction event caused by a massive climate change. what good is this technology when you have failing crops, extreme weather, no electricity and mass immigration at your doorstep?

<comment><USER39> Presumably, the eventual goal is to grow full-sized organs inside some larger animal. Then the organ can be harvested and transplanted to a human who needs it. Per my understanding, medicine does a lot of work with rats because they're cheap and easy to work with, so any new technology of this type would be tried on rats first.

By the way, my neutral explanation above doesn't mean that I approve of doing such things. Honestly, I'm quite torn about the issue.

<comment><USER40> I'm not remotely torn about it, taken to the finish, it's a human organ with my dna grown in a pig and then transplanted into me.

We already eat bacon and use large mammal heart valves in surgery.

I mean we wear the skin of cows, grease things with their rendered fat and use them for glue.

This seems like a strange point to be torn on.

<comment><USER41> Remember that a bio ethicist has years of education on smart ways to block stuff like this.

If they didn't block this stuff they wouldn't have a job.

</comment></comment></comment></comment><comment><USER42> I worry about allowing Japan, a country that has refused to fully own up to the biomedical atrocities it committed with Unit 731, to lead the way on these kinds of experiments. I would feel much more comfortable if there was international ethical oversight.

<comment><USER43> 1945 was a long time ago and a very different Japan... We don't still force Germany to have caps on its army.

<comment><USER22> _> We don't still force Germany to have caps on its army._

You pacified the German psyche so thoroughly that the Germans do it themselves now.

<comment><USER44> It wasn't pacification. It was a nation collectively coming to terms with their brutal past and taking to heart the phrase "never again". If "not bombing non-knee-bending nations on a whim" equals "pacified" in your mind, then I hope you never hold a position of power in any government.

<comment><USER45> I'd imagine Japan's attitudes have changed the same, as well.

<comment><USER44> I'd be interested in knowing if that's true, and if so, to what extent. My knowledge of Japanese culture - especially modern prevailing attitudes towards the Japanese government's actions in the second world war - is very limited. I'm not qualified to speak to that.

<comment><USER46> Historical revisionism is alarmingly prevailing in Japan, and I see Japan is slowly turning into nationalism and militarism.</comment></comment></comment></comment></comment></comment></comment>


Let's Stop Talking About "Backups" - <AUTHOR> http://www.joelonsoftware.com/items/2009/12/14.html

<comment><USER3> It's not us you need to tell Joel, it's your business partner. And if this is your way of telling him, isn't it a little passive-aggressive to do it in a blog post?

<comment><USER4> I think that a lot of people think they have backups, but they've never restored them, so I thought it would be a good practice if everyone starting thinking in terms of "have we restored" rather than "are we backing up." Of course, there's no question that the thought process started when Jeff Atwood's personal blog was lost, but don't think for a minute that the only way I communicate with him is by blogging... we talk all the time, over skype, over email, over FogBugz, and sometimes, when there's something other people can learn, in public on the internet.

<comment><USER5> The weird part is not that you made a blog post about this issue now.

The weird part is that you didn't mention the 400 pound gorilla in the room. ("I've been thinking about this for a long time, but it really hit home recently when...")

To my mind, that's why the post seems odd or passive aggressive. (It's a relatively short post as well. It just feels clipped somehow. The reader inevitably says, "What's _not_ here?")

<comment><USER6> It may be difficult to remember now that Twitter is all the rage, but essayists often aim for that "timeless" quality. You want the essay to seem as relevant three years from now as it is today.

The web has more than enough content that feels stale after a week. Better to aim for something with a slightly longer shelf life.

<comment><USER7> Yea, but how is "my friend/business partner recently lost his entire blog due to poor backups, and it got me thinking..." not 'timeless?'

<comment><USER8> Not poor backups, poor restores. You're missing the point of the article!

<comment><USER9> It's also poor backup. They thought the backup was being done by their hosting company while in reality it was failing silently.

<comment><USER10> A good backup is one that can be restored.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> I find it disturbing how many people decided to comment or up-vote on Joel- Atwood angle, or insult each other.

Remember: great people discuss ideas, normal people discuss events, shallow people discuss other people.

</comment><comment><USER12> What I find someone charming/quaint about Joel's Posts on "Operations" over the last six years that I've been reading him, is that he is, slowly but surely, discovering the "Art of Operations" - albeit at a glacial pace.

Most People who work in Production Operations environments of any scale, discover what has taken Joel the better part of a decade, in the first two- three years of their career.

I almost feel like that Airplane passenger sitting beside Brooks Jr - <USER2> saw him reading his book, "Mythical Man Month" - and asked the guy (who had no idea who he was sitting beside) what he thought of the book - The gentleman responded that it was basically a summary of things he knew already. Joel is a giant in the industry, but he does have a tendency to discover/restate the obvious.

"It's not backups, but the restores that matter" - is kind of the mantra of every single person who has ever been responsible for backups.

Then, you go to _any_ class on running a production environment, and you discover things like RPO, RTO, Dress-Rehearsals, etc.. and the whole "It's restores that matter" begins to look quaint.

<comment><USER13> the "Art of Operations" suggests a book or similar that you are referring to - is there one, or am I reading too much into some double quotes?

<comment><USER14> Yes there is a book: The Practice of System and Network Administration, Second Edition. This is the best Operations book I've ever bought. Worth every penny.

<comment><USER15> Because Google isn't links:

[http://www.amazon.com/Practice-System-Network- Administration...](http://www.amazon.com/Practice-System-Network- Administration-Second/dp/0321492668/)

</comment></comment></comment></comment><comment><USER16> This is good food for thought. Let's also add to this a concept from a different realm: Everybody has at least two dns servers listed in their /etc/resolv.conf, right? The reason is that in case one of them goes down, there is the other one.

So this seems like a good lesson to take about backups. Mebbe three? One by your hosting provider, one at tarsnap, one on a separate dat tape, one on a usb stick?

A good point is though that even something as big as a dat tape looks pretty small by the standards of what we need to back up today.

</comment><comment><USER17> Also this, similar take on the horror of backups gone wrong: <http://www.penny-arcade.com/comic/2005/8/10/>

Don't blindly rely on your partner to do it... Trust, but verify.

</comment><comment><USER18> It's helpful to boost signal for this message. It's an old message, but planning and practicing system restores can be as expensive in terms of equipment and manpower as actually making the backups. This leads to _a lot_ of neglect.

</comment><comment><USER19> ... shouldn't it be common practice to test your backup system to make sure that the restore procedure meets the requirements of the client (company, etc?)

The IT company that I work for creates a backup system based on the requirements of our clients and then demonstrates the whole backup and restore procedure to make sure that it falls in line with what our client actually wants. It's really not difficult to do. Sure, some of the restore procedures may be slower (depending on other requirements, such as cost,) but the client knows that will be the case and signs off on it.

</comment><comment><USER20> Common sense and obvious. In the 1980s I worked on a large DARPA project where a huge hit was taken because our admins never tried to restore from backups. It is the kind of lesson that is (hopefully) learned with just one bad experience.

This is another reason why I like EC2 deployments: it is fairly easy to take your backups (automated deployment scripts, application, data) and spin up another copy of your whole system (except for flipping the DNS). Make sure those EBS-backed EC2 AMIs are really bootable and functioning :-)

<comment><USER11> What happens if Amazon runs out of machines, has a system-wide corruption that was noticed too late or decides to be dicks about something or other?

<comment><USER20> Good questions.

I would think that if they are making money with AWS then they will keep buying more servers.

I usually trust S3 for backups and restores but periodically back my own data off of AWS to local storage.

I expect both Amazon and Google infrastructure services to experience outages from time to time. However, they have far more resources and expertise than I do to provide scalable services for a low cost.

<comment><USER11> I was thinking about making a set up where my S3 backups are automatically mirrored to rackspace or something like that. That would be a really neat setup.

</comment></comment></comment></comment><comment><USER21> I tend to be a little paranoid about backups, so I have a few different disks backing up my main, desktop, machine. But I also use one of my backups to sync data to my laptop, not quite a full "restore" due to a big size difference in the respective drives. But, generally speaking, the two machines are in sync and I can be sure that at least one of my backups works reasonably well.

</comment><comment><USER22> This whole ordeal is getting me motivated to actually buy a cloud backup service (personal use, not business use). I was thinking of Carbonite or Backblaze. Anyone have any experience with those?

<comment><USER23> I used carbonite in early 2006 and the software was horrible. Eventually, I tried to uninstall and the process failed. I got a half installation that wouldn't work and couldn't be removed. I didn't try too hard after that, because I was planning to format and start over.

</comment></comment><comment><USER24> Full disk image backups are a good solution for this problem. No worries about partial backups or a complex restoration process. It's totally inefficient but storage is cheaper than man hours.

<comment><USER25> I'm assuming you're talking about some kind of atomic file-or-block-level backup such as LVM snapshots? Large files such as databases can change while reading them over a long period of time, so a standard disk image or file copy wouldn't be reliable for a live system.

<comment><USER26> zfs send/receive is amazing. I wish other filesystems had it

</comment></comment></comment><comment><USER27> As if I didn't lose enough sleep pondering backups already.

</comment><comment><USER28> Testing your (off-site) backups is an obvious first item in many lists <http://watsec.com/article/49>

</comment><comment><USER29> What he's saying is, "We failed miserably at having good process and procedures. Because of this we are going to lecture others, and point the blame at everybody but ourselves, in hopes that they'll stop pointing out how much credibility we lost over this."

</comment><comment><USER30> Pointless write-up about linguistics. He says "restore" is the important thing, not the "backup". Well, duh.

<comment><USER31> Not really. That's actually one of my favorite "health check" questions: when was the last time you restored?

Most places have very reliable backup procedures. Most of those have very poor restore procedures - I'd say about half fail when put to the test.

<comment><USER30> To me that is like saying "You can put money on your bank account but you should make sure you get it back". Of course you have to test your backups.

</comment></comment></comment><comment><USER32> backups are for suckers. keep the data on a few different spinning disks. if you can solve data synch between two sites, just keep your data synched.

it's much better to ask yourself how long to replicate your existing system then how to back up. pxe boot to a kernel that you can install over the network with, bcfg2 to get the thing up to spec, start copying data.

a _lot_ of machines can be back and configured in 5 minutes.

that said, i'm not you. i don't have terabytes of data to do statistics on. maybe there are other horrible details i'm forgetting. fast rebuilding is a pretty awesome strategy for a lot of cases.

<comment><USER6> _maybe there are other horrible details i'm forgetting_

Yes.

Why should I bother to write this? I'll outsource the task to the authors of _High Performance MySQL, Second Edition_ , page 475:

 _Backup Myth #1: "I Use Replication As a Backup"_

 _This is a mistake we see quite often. A replication slave is not a backup. Neither is a RAID array. To see why, consider this: will they help you get back all your data if you accidentally execute DROP DATABASE on your production database? RAID and replication don't pass even this simple test. Not only are they not backups, they're not a substitute for backups. Nothing but backups fill the need for backups._

<comment><USER33> This is the incremental backup script I use on my Linux box at home, a quick- and-dirty imitation of what Time Machine does. Obviously $HOME/backup is a different physical disk. Feel free to improve on this.

\</comment></comment></comment><comment>--

#!/bin/bash

HOME=

date=`date "+%Y-%m-%dT%H:%M:%S"`

rsync -aP --link-dest=$HOME/backup/current /home

$HOME/backup/back-$date

rsync -aP --link-dest=$HOME/backup/current /etc $HOME/backup/back-$date

rm $HOME/backup/current

ln -s back-$date $HOME/backup/current

#see if the disk is getting full

FREE=`df -lk|grep sdb1|awk -F" " '{print $5}'|awk -F"%" '{print $1}'`

#alert me if the backup disk is getting full.

T=80

if [ "$FREE" -gt "$T" ]

then

        df -lk| mail $myaddress -s"disk alert $T% capacity"

else

        echo "backup disk is less than $T full"

fi

<comment><USER34> You should think about keeping an offsite backup if possible. A second disk won't necessarily help if your computer gets dropped in a pool, house catches on fire, etc.</comment></comment>


The Great Solvent North - <AUTHOR> http://www.nytimes.com/2009/02/28/opinion/28tedesco.html

<comment><USER2> Reconfiguring the American banking system to look like the Canadian model seems excessive. The current problems were caused by a bug in the financial system regulations: banks and people who purchased real estate were permitted to use too much leverage. Instead of a rewrite, maybe we should patch the bug by reducing the level of permitted leverage. This has already been done for banks, but it could also be done for real estate purchases in a fashion similar to the way that Fed regulation T limits leverage in stock investing. The great depression was kicked off by excessive leverage in stock investing, and reg. T was part of the patch applied to fix that problem back in the 30s. The fix seems to work pretty well since the internet stock bubble didn't endanger the financial system. In fact, the regulations that require stock brokers to consider whether an investment is suitable for their customer should probably also be mirrored in new regulations for mortgage brokers as well.

<comment><USER3> Its just a matter of hiring enough lobbyists to get the leverage limits changed. I think as we see the true consequences of the current crisis Canada's system will look better and better.

</comment></comment><comment><USER4> "Yes, Prime Minister Stephen Harper's government may buy as much as 125 billion Canadian dollars (about $100 billion) worth of mortgages, increasing banks' capacity to lend. But this is small change compared with the scale of Washington's bailout."

Well, considering we have 1/10th your population, that would translate proportionately into 1 trillion, not exactly small potatoes. And keep in mind, we are about 2 years behind the US in the housing bubble, so the domestic non- performing loan problem is just now starting to kick in here. So I think we Canadians should perhaps wait a while before we get too full of ourselves.</comment>


For Sale: Original Swiss Bank Deposit Safe Money Swimming Pool - <AUTHOR> http://www.jamesedition.com/extraordinaire/729198

<comment><USER2> I hate it when a site redirects visitors with a mobile device straight to the home page without any regard for the link they were actually using...

<comment><AUTHOR> Sorry, I didn't notice. I hate that too!

</comment></comment><comment><USER3> This would be perfect for Kim Dotcom.

</comment><comment><USER4> What is this exactly???

<comment><AUTHOR> A Scrooge McDuck-like pool of money made from Swiss 5-cent coins …</comment></comment>


Bernie Sanders Suspends Presidential Bid - <AUTHOR> https://www.bloomberg.com/news/articles/2020-04-08/sanders-suspends-presidential-race-leaving-nomination-to-biden <comment><USER2> Shit.

</comment><comment><USER3> Anyone know who his VP pick will be? I know this won't happen but it'd be funny if he picked Obama.

<comment><USER4> Iirc only people who would be eligible for the presidency can be elected as VP, which would disqualify Obama. This is the same reason Bernie couldn't have chosen AOC as his running mate

<comment><USER5> There is no such restriction. Article II never specified any limit on how long someone could serve.

The 22nd Amendment limits only who can be _elected_ to the office of _President_. Since a VP becomes President through devolution, not election, it doesn't limit who may serve as VP.

<comment><USER6> Isn't there a 10 year limit as well though? Especially for this case where someone serves as president without being elected?

That being said, I'm not sure either if the restriction in the parent exists.

Edit: a reference I found afterwards [https://en.wikipedia.org/wiki/Term_of_office#Federal](https://en.wikipedia.org/wiki/Term_of_office#Federal)

<comment><USER5> Not exactly. The 22nd Amendment would prohibit the _second_ election of someone who had served for more than two years without having been elected.

For example, if Gerald Ford had been elected president after he finished Nixon's term, he would be ineligible to be elected a second time, since he initially served for a little over two years and five months.

That doesn't change the fact that the 22nd Amendment only restricts the _election_ of a president. If someone has already served in the office for ten years, they could still be the VP who upon whom the presidency devolves, because there is no constitutional restriction on such eligibility.

</comment></comment></comment></comment></comment><comment><USER7> Well, damn, I guess that means that Trump will have a second term then.

I mean, its unlikely that someone who voted for Trump last time will be voting for Biden this time. I think the Democratic party is missing the boat on this one. They're so busy pushing someone "presidential" that they are forgetting who the actual target audience is, Trump voters.

<comment><USER8> > I mean, its unlikely that someone who voted for Trump last time will be > voting for Biden this time.

Why do you say that? Voter turnout during primaries was higher than 2016, and Biden was doing well. Even though it may seem like Bernie is more popular on the internet, Biden is more popular among people that actually vote (older people).

<comment><USER9> Trump is anti-establishment. Biden is as establishment as it gets. Trump is pro social security and Medicare. Biden wants to cut it. They're polar opposites in so many ways

</comment></comment></comment><comment><USER10> This is substantial enough to warrant an exception to HN's usual preference for avoiding politics. You could (for example) discuss the way the covid-19 outbreak intersected with the primary or something like that if you don't want to talk about the candidates.

<comment><USER11> Substantial to you, not this site.

</comment></comment><comment><USER12> And on April 8th 2020 Trump was reelected.

<comment><USER13> Care to explain why? You have no evidence as to why that would be, I assume.

The viability of Sanders' campaign is a serious question and is why he wasn't nominated. If there had been resounding support for him as a candidate, he would have been nominated...

I understand that his campaign and supporters will make some argument about how it's the DNC holding him down, or some conspiracy, or whatever, but the reality is that he isn't/wasn't/won't be viable in a general election against Trump.

Socialism, or whatever Bernie wants to sell himself as does not sell to the broader electorate that is going to have to be captured in November.

<comment><USER12> Purely from an observational opinion I don't see Biden as electable. Sanders would of also had a tough time, but running a populist vs populist versus a career politician vs populist I think the former has the strongest chance.

I also think Bernie's approach to healthcare during the times we are in now would hold a lot of favor.

I think Biden is either senile now or turning senile quickly (based on observing him speaking publicly the last 4 months) and won't fair well in debates or campaigning as a result. He's doddering, did not do much as VP, and doesn't have a history of strong stances that he can hold up and swing voters or independents.

I'm not for or against any of the candidates and this is obviously opinion but I think Trump just won re-election.

<comment><USER13> It's fairly easy to imagine the attacks that Trump will land against Sanders, which would work to discredit him through fear.

I don't think you'll see that with Biden. Sanders' campaign would quickly fall apart under the weight of those attacks.

Also, I'm tired of hearing this senile line. It's a form of ageism. He's never been a talented public speaker, specifically when he's not feeding off the energy of a crowd. Just go back and look at his old speeches.

If you are looking for a young, vibrant candidate, neither Sanders or Biden is your ticket, anyway. Sanders is older than Biden.

<comment><USER12> It's not ageism when all three candidates are very old. If you look back to when Biden was VP and how he speaks now there is a noticeable difference.

I think both Biden and Sanders would have had a hard time campaigning for the reasons you listed. I believe Sanders would have stood a better chance again because of the populist vs populist approach. I believe Sanders supporters are as vocal as Trump supporters and you dont have that with Biden supporters at all.

You can't lobby much against Biden because he doesnt stand for much (again obviously opinion, and Im not trying to start a fight or argument) as compared to Sanders. They have vastly different goals in mind for a presidency.

<comment><USER13> Ageism is still ageism, even if the other people you are comparing are of advanced age, too...The literal definition of ageism is thinking someone can't do something due to age.

</comment></comment></comment></comment></comment></comment><comment> java-man why is this submission [flagged]?

<comment><USER14> > why is this submission [flagged]?

My guess is that it's _purely_ politics, and there's a large number of users who just don't want to have purely political discussions.

HN sometimes has decent threads about political policy on non hot-button issues, but threads about political personalities are unlikely to go well and are thus not worth the trouble.

<comment><USER10> HN can be very arbitrary about this.

This submission's topic is one step away from this one at over 1800 points a few years ago: [https://news.ycombinator.com/item?id=12907201](https://news.ycombinator.com/item?id=12907201)

<comment> java-man And probably as relevant. I think it will have a great effect on the tech scene as well.

</comment></comment></comment></comment><comment><USER15> Odds on Trump reelection is 1/1 on most betting site. Maybe the safest bet ever with a good return

<comment><USER16> A cursory search of the top couple of betting sites proves this statement to be complete BS.

<comment><USER15> Betfair gives 1.06/1

[https://www.betfair.com/exchange/plus/politics/market/1.1281...](https://www.betfair.com/exchange/plus/politics/market/1.128151441)

Skybet is down to 10/11

[https://m.skybet.com/politics/us-presidential- election/event...](https://m.skybet.com/politics/us-presidential- election/event/19989820)

Oddscheck on 1/1

[https://www.oddschecker.com/politics/us- politics](https://www.oddschecker.com/politics/us-politics)

Betway is 0.8/1

[https://sports.betway.com/en/sports/evt/1279338](https://sports.betway.com/en/sports/evt/1279338)

Paddy is 13/8!!

[https://www.paddypower.com/politics/2020-us-presidential- ele...](https://www.paddypower.com/politics/2020-us-presidential-election)

<comment><USER10> Most of these are a coin toss. Clinton had better odds against the same person in 2016.

</comment></comment></comment></comment><comment><USER17> This is a crisis moment.</comment>


Ask HN: What parts of Internet infrastructure are owned by Google? - <AUTHOR> <comment><USER2> How do you define "Internet Infrastructure"?

They obviously run a lot of popular services, and host others services, but most infrastructure they run is mostly in support of those. Their 220.127.116.11 DNS service and the Certificate-Transparency logs they run are maybe the most "core" infrastructure, but 22.214.171.124 going down doesn't affect other DNS resolvers and others run CT logs too.</comment>


Chrome Store Foxified – Install Chrome Extensions in Firefox - <AUTHOR> https://addons.mozilla.org/en-US/firefox/addon/chrome-store-foxified/ <comment><USER2> wake me up when they reverse all that and I have firefox true extensions on chrome.

remember kids, without the power of the old extensions and xul, we would never have had firebug or dalvik debugger, the forefathers of todattoday dev tools.</comment>


Ask HN: Are all YC startups encouraged to get acquired? - <AUTHOR>

What is the path to liquidity you're encouraged to take as a YC startup? Are you ever encouraged to build for the long haul and create something profitable that pays investors dividends? Is it assumed that all YC startups will be acquired and is that where the emphasis is placed? Is there any focus on building revenue and keeping costs down, or is the focus on growth and future funding rounds? Is the option of not raising future rounds but rather self funding through revenue discussed much?<p>I'm genuinely curious about the state of startup incubators. Thanks. <comment><USER2> 1) It's in YC's interest for their startups to aim as high as possible. Ideally billion dollar acquisitions or IPOs. Even if less than 1/10 can achieve that goal and the rest die it's still financially beneficial if they all shoot for it. YC makes very little money off small exits.

2) YC doesn't want to stand in the way of founders getting rich. They accept that many of the YC founders will take life-changing acquisition offers and don't try to stop them. They hope at least some will take a little cash "off the table" along the way and then go for The Big Win.

3) YC doesn't get paid until a "liquidity event". In practice that means either an acquisition or an IPO. There's currently no acceptable method for distributing dividends or similar.

I think that's about all there is to it.

</comment><comment><USER3> I don't know the answer to this question, but if I were an investor I'd be looking to create a business that showed a lot of long term promise.... Then I'd want to sell that promise. So acquisition or a big stock sale (IPO).

Think of an incubator as a tree nursery. They handle the high risk portion of getting the seeds to grow into a sapling, and determine if it will bear fruit. But fundamentally they don't want to be a fruit producer, and they aren't interested in running an orchard. They are selling the promise of fruit for years to come to someone else.

<comment><USER4> > _"Think of an incubator as a tree nursery."_

A nursery will plant 3000 seeds. Pot up the 2000 that survive the first year to one gallon and sell half as one gallon to recover their costs. Later they will pot the remaining 1000 up to three gallon and sell many of those. Very few will be sold as 45 gallon trees.

</comment></comment><comment><USER5> While operating costs can be a reason startups are forced to shut down, I'm sure its not _the_ reason startups close shop. I would assume advice is optimized for growth and traction to make something people want. After all, nobody is in this game to build a business slow and marginally (although there's nothing wrong with that).</comment>


Show HN: ION, a JSON alternative – Versatile, compact, fast, binary data format - <AUTHOR> https://github.com/jjenkov/iap-tools-java <comment><USER2> Or you could use an actual RFC standard[1] that has numerous high-quality implementations[2].

[1] [https://tools.ietf.org/html/rfc7049](https://tools.ietf.org/html/rfc7049) [2] [http://cbor.io/](http://cbor.io/)

<comment><AUTHOR> We have already look at CBOR and MessagePack. They miss the ION tables for compact arrays of data. Here is a link to our comparison to other data formats. [http://tutorials.jenkov.com/iap/ion-vs-other- formats.html](http://tutorials.jenkov.com/iap/ion-vs-other-formats.html)

<comment><USER2> I don't really see how ION tables are an improvement over arrays of arrays, e.g.:

        {
          "headers": [ "a", "b", "c" ],
          "rows": [
            [ 1, 2, 3 ],
            [ 4, 5, 6 ],
          ]
        }

Furthermore, ION appears to require you to know the length of your data up front, whereas you could use CBOR unspecified length arrays to stream data from your database without precalculating the table length. It seems like quite a niche format, though. Most data is not truly tabular.

Also, in the table it claims "Yes" under support for "Cyclic references", and yet further down the page:

> ION has support for expressing cyclic references between objects. At this > point this support is not 100% finalized.

So surely this should be "Yes(*)"?

<comment><AUTHOR> First of all, lots of results sent back from backend services (or databases) are arrays of objects. So no, tables are not a "niche" format - tables are heavily used.

Second, an array of arrays could mean anything. You have not semantics telling whether the arrays are independent or if the first array is an array of columns for the following arrays. ION Tables add that semantic information.

Third, yes, we could encode everything as text or as raw bytes and leave it up to the user to make sense of it. But that is exactly what we are trying to avoid with ION. We want to give devs a decent standard data format to use, that doesn't require a lot of data encoding choices up front. The encoding options have been thought through already, and sensible choices already made which you can just follow.

Fourth, you can nest ION tables inside ION tables, and thus create a more compact representation of an object graph. Using JSON / CBOR you would need a lot of nested arrays inside arrays to emulate that. Possible, but not exactly pretty.

<comment><AUTHOR> Yes also, the TLV nature of ION means that you have to buffer the ION data while writing it, in case you don't know the full size of the embedded ahead of time. However, this is really only a problem for very big messages. HTTP has worked like that for a long time already, and HTTP servers have proven capable of scaling pretty well, won't you agree?

Additionally, if a message does not include its length up front, you are just trading faster write time for slower read time. A node receiving dynamically sized data then has the problem of knowing how much data to allocate for the full message, plus the receiver has to inspect the data as it comes in to see where it ends. With an ION message the receiver knows within the first 4-5 bytes (typically) how big that message will be, and can thus copy the following bytes directly into the perfectly allocated memory area, without having to examine them any further. Since data is most often read more times than written (e.g. data written to a file), we felt that making a tradeoff that favors read speed over write speed made sense.

</comment></comment></comment></comment></comment><comment><USER3> On one of these pages, the claim is made that Protobufs is not self- describing, and therefore cannot be used for "network applications". It seems that "self-describing" here means that the format includes key names, instead of compressing them by using numbers like protobufs does. I can't understand why having field names is going to make a difference for anyone. Once you are setting up a system to deal with a specific format of data, why not just include a protobufs schema?

<comment><USER4> Here's a use case where protobuf is terrible because it isn't self-describing: write a wireshark plugin which parses and pretty-prints protobuf messages for human consumption.

You can't, because such a plugin would have to have a-priori knowledge of the schema in use.

<comment><USER5> You mean something like this, which does exactly that?

[https://code.google.com/archive/p/protobuf- wireshark/](https://code.google.com/archive/p/protobuf-wireshark/)

Sure, you don't know the NAME of the field, but you can see the ID of it.

</comment></comment></comment><comment><USER6> Binary format makes me believe it's not human-readable. How doesn't this compare in size to gzipped JSON? JSON overhead is fairly small (some quotes, colons, brackets and keys) - it's no XML.

<comment><AUTHOR> Yes, binary formats are not easily readable in a text editor. But, it is actually possible to convert ION to an XML format and back again without loss of information (we have not implemented this yet). This should make it easier to read messages during debug - especially because you don't need to know the schema for the given message to conver it to XML.

Regarding GZipped JSON, it is true that GZiped JSON is small. But, due to the CRIME and BREACH attacks it is not recommended to compress data sent over encrypted connections (TLS).

If you look at our performance benchmarks page you can see a list of serialized length comparisons. As you can see, as soon as you send a few objects in an ION table, the difference is big. More than what you normally can gain with GZip (except perhaps for String).[http://tutorials.jenkov.com/iap/ion-performance- benchmarks.h...](http://tutorials.jenkov.com/iap/ion-performance- benchmarks.html#serialized-length)

Furthermore, GZip only helps with transfer time, and actually slows down parsing time. If you look at our performance benchmarks you will see that ION parsing time is a lot faster than JSON. Additionally, if you really, really want high speed you do not parse ION (or JSON) into Java objects. You process the data directly in its binary form. If you look at our read-and-use benchmark you can see just how big a speed difference that gives. ION is designed for being processed directly. JSON isn't as good for that purpose.

Finally, ION is designed for fast arbitrary hierarchical navigation. JSON is not.

</comment></comment><comment><USER7> Note for anyone who was as confused as I was: this is not the same thing as Amazon's internal typed JSON format, which is also called Ion.

<comment><USER8> Well, why would it be? That's internal.

Hasn't been published as far as I can Google.

<comment><USER7> I don't work at Amazon anymore so I thought maybe they had open-sourced it

<comment><USER9> Yes, a guy told us that Amazon has an internal data format called ION. We googled for it, but didn't find it, so we assumed Amazon wants to keep it internal.

<comment><USER7> Not that you are under any obligation to care, but this will now be very annoying for Amazon employees.

"Oh, you need to call that service using Ion, not Json. No, the other Ion..."

<comment><USER10> Not to mention if they try to use it internally, then might end up with package name conflicts..

</comment></comment></comment></comment></comment></comment><comment><USER11> > As you can see, an ION field can contain values that are up to 2^120 bytes > long. If you need to encode larger blocks of data than that, you would need > to break it up into multiple fields.

Har, har.

</comment><comment><USER12> I'm looking at: [http://tutorials.jenkov.com/iap/ion-vs-other- formats.html](http://tutorials.jenkov.com/iap/ion-vs-other-formats.html)

As the author of Protobuf v2 (the version that was open sourced by Google), I object to some of the "no"s in the protobuf column.

(Note: I no longer work on Protobuf, and I did not invent the format. I do work on and did invent Cap'n Proto.)

> Protobuf apparently isn't great at encoding raw bytes either (according to > their own website).

Protobuf can handle raw bytes just fine, using the "bytes" type. There is no special encoding done on bytes; parsing and encoding is done by memcpy(). I'm curious to know what part of the web site you interpret as saying otherwise. It's entirely possible that the web site contains confusing language, but a citation would have been a good idea here.

> Schema / Class Id > Self describing

The Protobuf libraries have extensive support for manipulating dynamic schemas and transmitting schemas over the wire. See the "Descriptor" and "DynamicMessage" APIs. This is mentioned on the web site:

[https://developers.google.com/protocol- buffers/docs/techniqu...](https://developers.google.com/protocol- buffers/docs/techniques#self-description)

> Even if these compact objects do not contain any property names, they are > still self describing enough that you can see where fields start and end, > plus their data type, without an external schema. You cannot do that with > Protobuf (as far as we know).

You absolutely can do that with Protobuf. This is what the "protoc --decode_raw" flag does, and it should be clear enough from reading the encoding.

[https://developers.google.com/protocol- buffers/docs/encoding](https://developers.google.com/protocol- buffers/docs/encoding)

> Cyclic references

While it's true that Protobuf doesn't support these, I hope you've considered the denial-of-service vulnerabilities they tend to create if the receiver is not expecting them. Please ensure that cyclic references are only allowed in cases where the app opted into it.

Relatedly, overlapping references / backreferences ("Copy" in your table) potentially leads to an amplification attack where a small message on the wire turns out to be much, much larger when traversed. If applications cannot defend themselves from huge payloads by setting a message size limit, then you'll need to give them some other way.

> All of the formats (except perhaps Protobuf) supports arbitrary hierarchical > navigation of the encoded data, without first converting it to objects.

Protobuf supports this, and in fact should be an unqualified "Yes" rather than "Yes(*)" like the others. Protobuf encoding is very similar to ION's. Sub- messages are length-delimited, which seems to be exactly the advantage you're claiming that ION has.

Note that none of these formats support random access in the way that Cap'n Proto does.

In summary, I believe Protobuf deserves a "yes" in: "Raw bytes", "Good at raw bytes", "Schema / Class Id", "Arbitrary hierarchical navigation", and "Self describing".

<comment><AUTHOR> If that is really true, then we will of course update the comparison page. However, we have put it together from what we were able to find in Google Protocol Buffer's own docs + stack overflow + googling. It is entirely possible that we made mistakes.

Sending schema over the wire is not a good solution for anything else than point-to-point communication. An intermediate node would need every single schema transmitted along with every single messsage, or have another way to keep the schemas cached. That becomes complicated.

The Protobuf documentation says very clearly that you cannot see when one message ends and another begins. Then a protobuf message is not fully self describing. This might be easy to add, but it doesn't have it (according to Protobuf's own docs).

We have looked at Cap'n Proto - but late in the process where we had already looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty much just a binary struct. That is pretty close to what we wanted to do with ION, except we wanted it to be compact on the wire too. We have seen that Cap'n Proto has a compaction mechanism, but we have not yet had time to analyze and compare it to ION's.Cap'n Proto with compaction would be very similar to ION - on a conceptual level.

However, we need to make space for some IAP specific fields coming later in the process (like cache references, column stores and more). Stuff that is IAP specific. That is why we chose to roll with our own encoding in the first place.

<comment><USER12> > Sending schema over the wire is not a good solution for anything else than > point-to-point communication.

OK, let's back up a moment. I am not entirely sure what "Schema / Class Id" in your table means. Your table claims Protobuf doesn't support it, but the text below is unclear on what you think Protobuf doesn't support.

You frequently use the term "self-describing", but this could have two meanings:

1) Like JSON, where the names of all fields appear in the message, so that a human can read the message easily without external information.

2) Limited self-description in which field values can be identified and parsed, but their names are not available (perhaps replaced by numeric tags or indexes).

Protobuf can support (1) by including the schema in the payload. I agree this is not commonly useful.

Protobuf supports (2) natively, by virtue of being a TLV format (just like ION).

Re-reading the page, it sounds like you are assuming the Protobuf format cannot be deciphered at all without the schema, but this simply isn't true.

If you meant something else, please explain.

> The Protobuf documentation says very clearly that you cannot see when one > message ends and another begins.

I wrote that documentation. It doesn't mean what you think (my fault, perhaps). What it's saying is that the top-level message is a series of tag- value pairs with no explicit indication of where that series ends (on the assumption that you already know, e.g. based on EOF). Thus, if you concatenate two whole messages without adding any delimiter then it will look like one big message containing all the fields from both. However, each field within the message _is_ clearly delimited and sub-messages are length-delimited therefore skipable.

> We have looked at Cap'n Proto - but late in the process where we had already > looked at quite a lot of formats. From what I can see, Cap'n Proto is pretty > much just a binary struct. That is pretty close to what we wanted to do with > ION, except we wanted it to be compact on the wire too. We have seen that > Cap'n Proto has a compaction mechanism, but we have not yet had time to > analyze and compare it to ION's.Cap'n Proto with compaction would be very > similar to ION - on a conceptual level.

ION is a TLV encoding like Protobuf. Cap'n Proto is fixed offsets + pointers. These are vastly different styles of encoding that enable different modes of use. You can certainly debate which is better but I don't think it's correct to describe the formats as "pretty close", unless you consider all binary formats to be "pretty close" to each other.

<comment><USER9> You are right, the term "self describing" as used in our docs could be more clear. Being self describing means that you do not need a schema to make sense of a stream of data of that format.

However, there is also a degree to which a data format can be self describing. A CSV file is reasonably self describing because you can see where one field ends and the next begins (at the comma / separator), and where one record ends and the next begins (new line). With a header line of column names a CSV file becomes more self describing, as you now also have a name indicating the semantic meaning of fields in that column. If a CSV file could somehow contain a specification of the data type of each column, it would be even more self describing etc.

This is what we are trying to achieve with ION. If you need speed, you can omit most of the meta data like property names etc. If you need messages to be self describing, you can add a lot of meta data (like class / schema names + version, property names etc.).

I apologize for having written incorrect documentation. If you wrote those docs for Google Protocol Buffers, part of that is on you. They are not exactly crystal clear ;-) (our doc's aren't either - still working on them!)

Thank you for clearing up that Protobuf fields can be distinguished in a stream of Protobuf fields, even without schema. That was unclear to me before now. By the way, that is pretty clear in Cap'n Proto - your invention right? So - better docs already!

And - thank you for clearing up the difference in the encoding of Cap'n Proto. Any link to where I can read about that encoding style in more details?

<comment><USER12> > Any link to where I can read about that encoding style in more details?

Hmm, I'm not aware of any literature other than what's on the Cap'n Proto web site. You can of course find the Cap'n Proto encoding documented here:

[https://capnproto.org/encoding.html](https://capnproto.org/encoding.html)

The format is, of course, a lot like how in-memory data structures are laid out in C (fields of a struct have fixed offsets; variable-size fields are behind pointers). Unlike native pointers, though, Cap'n Proto's pointers are designed to be relocatable and easy to bounds-check, and they contain just enough type information for the message to be minimally self-describing (so that you can e.g. make a copy of a particular sub-object without knowing its schema).

</comment></comment></comment></comment></comment><comment><USER13> i have found that most of my huge json data is from uniform recordsets. there's a great json-compatible encoder for such cases that stores them in a format that's CSV-esque:

[https://github.com/WebReflection/JSONH](https://github.com/WebReflection/JSONH)

<comment><AUTHOR> That might work for a web app, but not for mobile apps, or backend to backend communication.

<comment><USER13> anything that used json before would benefit from JSONH. things that abused json before, would be better off switching to a different format that is less restrictive than json, like ION or msgpack [1]

[1] [http://msgpack.org/index.html](http://msgpack.org/index.html)

</comment></comment></comment><comment><USER14> Is there a spec? Is this a Java API for it?

See also: [https://github.com/edn-format/edn](https://github.com/edn- format/edn)

<comment><AUTHOR> Yes please check:

Specification: [http://tutorials.jenkov.com/iap/index.html](http://tutorials.jenkov.com/iap/index.html)

Benchmarks: [http://tutorials.jenkov.com/iap/ion-performance- benchmarks.h...](http://tutorials.jenkov.com/iap/ion-performance- benchmarks.html)

Tutorials: [http://tutorials.jenkov.com/iap-tools- java/index.html](http://tutorials.jenkov.com/iap-tools-java/index.html)

Here is also a recent Infoq.com article [http://www.infoq.com/articles/IAP- Fast-HTTP-Alternative?utm_...](http://www.infoq.com/articles/IAP-Fast-HTTP- Alternative?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global).

<comment><USER15> the "message structure" page is 404ing. all the rest hints to a designed-by- committee buzzword thingy.

<comment><USER9> Here is the Message Structure doc [http://tutorials.jenkov.com/iap/iap- message-structure.html](http://tutorials.jenkov.com/iap/iap-message- structure.html) . We are not as far with the core IAP protocol as we are with ION, but we will get there.

Designed-by-committee - really? You pull that?

</comment></comment></comment></comment><comment><USER16> HTTP ERROR 404

Problem accessing /iap/message-structure.html. Reason:

        Page not found

<comment><USER17> Pfff 404, that's so HTTP. Why aren't you using IAP?

<comment><USER9> That is coming... but browsers aren't supporting IAP (yet)!

</comment></comment></comment><comment><USER18> [https://media.giphy.com/media/1M9fmo1WAFVK0/giphy.gif](https://media.giphy.com/media/1M9fmo1WAFVK0/giphy.gif)

</comment><comment><USER19> Is there an ION vs. Thrift comparison?

<comment><AUTHOR> Not yet. We have been asked to compare ION to Flatbuffers, Cap'n Proto, Thrift, Avro, Transit, BSON and several other encodings. However, writing the benchmarks and going through the features systematically is a lot of work, so we have not yet had the time to go through them all.

</comment></comment><comment><USER20> I think the most interesting difference to the usual serialization formats might be the copy and reference types. I'm a little bit undecided whether they might be a brilliant idea or not. The decision whether to support copy or not puts some extra effort in the serializer and deserializer, but the total result is the same that you can have as without a copy field mechanism. The support of cyclic references makes a big change, because you can't directly model them with technologies. You might also have trouble using these data structures in some programming languages or libraries (e.g. if you are only using immutable types or want to use only value types). However for some kind of data it seems to make sense to support cyclic data, as GraphQL and Falcor have also added support for that.

I also don't see that many use cases for the table structure. I have deployed thousands of RPC APIs into production, and I can't recall having the need for it. And even if you need it, using an object with 2 arrays in it would be just fine.

I also looked through the IAP documentation (btw. bad name => ipod accessory protocol) because it's quite related to what I'm working on. I think that the shown basic communication patterns are correct, but from the documentation I can't really get a feeling what I could expect from an IAP library. Would it be some low level messaging system (like MQTT, ZeroMQ, etc.) or would higher level communication patterns (request/response, notifications) also be built in. There are no predifined message formats for RPC listed in the documentation which would outline that. The WAMP specification ([http://wamp.ws](http://wamp.ws)) e.g. makes it clearer what I could expect from such a protocol. I'm not sure whether we need a new low level messaging protocol or if the work should be more focused on adding higher level semantics on top if it.

E.g. I think some pattern that I really need in my domain is remote object synchronization, which means the status of an object on the server gets automatically pushed towards all interested client and is continously updated during changes (=> e.g. to build something like Firebase). Of course one can built something like that on top of basic messages by defining subscribe and update messages in the API, but I'm wondering if it's worthwhile to add something like that directly in the protocol. On the one hand this is also a special case of the subscription pattern which is also listed here, on the other hand it can not directly be implemented with the subscription possibilites of many message broker systems, because they won't send you the current state of an object after subscription but will only forward you a message after the value changes for the next time.

The connection and sequence definition in IAP looks a little bit redudant to me on the first look. I really think there is a need for message ordering and you must support it. The question for me is then if you don't need message ordering, why not put the message into a seperate channel and let channels/streams always be ordered (like in HTTP/2)? Overhead for channel creation? Or to setup channels during creation either as ordered or unordered and keep that for the lifetime of the channel?

<comment><AUTHOR> Mathias, you also don't need a binary protocol. XML would work. JSON too. But binary is faster and more compact. Same with the table construct. You can work around not having it, but now ION has it built in. You don't make the mistake of serializing an array of JSON objects because you are busy. The objects are serialized as an ION table - not a list of ION object fields. If you ever send an array of objects across the wire with ION (IAP Tools), you will be using the table mode automatically. You save bandwidth and parsing time automatically. Who don't want that - even if you don't need it?

Regarding Copy and Reference, the support for them is still not very good (= not automatic). But imagine your service executes an SQL JOIN query, and in that result a lot of objects are repeated (e.g. same zip + city for a lot of objects). The Copy field can be use to include the zip + city fields just once, and after that refer to them later with a Copy field. That is shorter than including them again. These two fields still need some work to have full support, but we are working on it.

Right now ION is the most well-defined part of IAP. The network protocol itself is still not 100¤ finalized. But, now that we are close to being done with ION (we still have extra fields to add as extended types), we can move forward with the IAP core protocols and semantic protocols. If we do not define a standard semantic protocol for remote object synchronization, IAP will be designed so that you can plug in your own semantic protocol to meet that need.

</comment></comment><comment><USER21> I consider no need for a schema a plus of MessagePack. But the Nos are all red, haha.</comment>


Quick History of Apple Mac OS X - <AUTHOR> http://www.ihash.eu/2013/12/quick-history-of-apple-mac-os-x/

<comment><USER2> Next was great, revolutionary in UI technology & design.</comment>


Why Functional Programming? It's the Composition - <AUTHOR> https://tech.iheart.com/why-fp-its-the-composition-f585d17b01d3 <comment><USER2> The title is truly misleading. I was expecting a mildly interesting post on FP and got a great one on composition with monads !</comment>


Studying the mind is the most pertinent necessity - DAndre-Johnson http://www.sciencedaily.com/releases/2015/07/150702073603.htm

<comment> DAndre-Johnson Hello Hacker news community. Working on my new web series called 'THE GYM'-The difference between mental and physical fitness' has opened my mind to the possibilities studying the mind can afford us as a human race. My series focuses on mental health in relation to physical fitness. I will be posting on youtube to spread the message and open peoples eyes about mental health. The link posted above intrigues me, due to its relation to memory recollection. Thank you for your time. Information is power.</comment>