How I wrote and published my novel using only open source tools - <AUTHOR> https://medium.com/@gabrielgambetta/how-i-wrote-and-published-my-novel-using-only-open-source-tools-5cdfbd7c00ca <comment><USER4> It seems odd to me that he would write and publish a novel using open source tool and then write and publish about that using a closed source platform. I wonder why he chose medium (I wonder why anyone chooses medium over their own site)

<comment><USER5> Probably for simplicity of use, as always. Although it isn't _that_ simpler than building your own blog, at least for a tech aficionado. Wordpress if you don't know web development, one of the hundreds of blog engines otherwise.

I've used Jekyll on Github pages for quite a bit and it suited me perfectly. Being able to write a post in markdown and `git push` to deploy is all the simplicity I need.

<comment><USER6> Github is a closed platform too

<comment><USER7> you don't need github pages to deploy jekyll. While source code of github pages itself is not available, since it supports a open standard, which can deployed to a webserver/CDN directly without major changes, I would consider it to be open.

<comment><USER8> Just to follow this to its logical conclusion, the webserver may be running Apache or Nginx on Linux, but there are likely to be proprietary blobs in the drivers and the firmware is probably closed. Plus, much of the hosting company's networking hardware is probably running proprietary software.

IANANE (I Am Not A Network Engineer) but I imagine with open firmware like Coreboot and networking distros like Cumulus and pfSense (BSD) and the whole Open Compute Project it might be possible to run a completely open-source hosting company. Well, no. Your UPS's and other auxiliary hardware are probably running closed OSes. Darn.

<comment><USER9> Cumulus contains a large portion of closed-source code, including all the driver code provided by Broadcom, which is where much of the interesting stuff happens. (I assume their Mellanox products are the same, though I'm not familiar with them)

I'm not aware of any efforts to write open source drivers for any high speed ethernet chips, though I haven't looked much. Intel's (formerly Fulcrum) switch chips have open data sheets (but not the reference code, simulators, etc) so it would at least be plausible to start there.

</comment></comment></comment></comment></comment></comment><comment><USER10> Nice writeup.

I'm currently writing a (technical) book in markdown, and I know I am going to pay for it later, but I don't care because I'm able to write quickly without anything getting in the way. I'm able to use git to commit changes and a host of tools to preview the work.

Also, using markdown now will force me to edit it again when putting a final version together for a real editor.

For the skeptics: I don't see how using markdown is any worse than using a typewriter. Its an interim medium that allows for quick capture of structure and thought.

<comment><USER11> If you use LeanPub, you can fully publish both ebook and paperback versions from the Markdown source. And for editing, if you have an editor who's willing to not use Word, you can use a tool like Authorea to track changes, and then incorporate those back into the markdown source.

I really don't get the hate on Markdown; it's IMO the fastest way to get words from brain into text, more so than rST and other formats, because of its simplicity. And for slightly more involved formatting, with LeanPub, you can put down what you need, then fix formatting by previewing the published copy until it looks just right.

<comment><USER12> I never understood this argument about rST. The syntax for the same MD features in rST is only /marginally/ different, certainly not harder. I use both, depending on the context (which forces me to use one or the other). Most of the time you don't even notice the difference, because both have a really good baseline.

However MD falls short for writing technical documentation, which is why pretty much anyone has to dope the "standard" (GH flavored MD, and so on). The syntax for the extensions is not any better than what rST offers. rST generally allows a more human-readable writing style, whereas MD with extensions has a lot of very verbose inline formatting.

In fact, rST _CAN_ have a slight edge in my mind as a human readable document if you use the appropriate formatting, whereas MD with extensions is easier for a program to parse.

But, I repeat myself, for the same set of MD features, rST is hardly different. If you don't believe this, use pandoc and try to transform a vanilla MD document into rST.

Now pick any rST page from sphinx documentation, and move back to MD. Hardly any better. sphinx is often criticized for being "hard to write" or illegible, transferring the blame to rST, but for the same set of formatting MD is not better. sphinx allows a whole slew of extra features which require a lot of annotation, and MD doesn't save the day.

What's really annoying if having to switch between the two. We have projects which have a good majority of text documents written in MD and then API references in rST. I hate that. There's constant mismatch.

Just pick one :(

</comment></comment></comment><comment><USER13> I use a similar flow for my own books, except Perl scripts instead of Pandoc. Starting with a markdown-style text format is so nice.

One thing with LibreOffice is that you can generate documents, they are just XML zipped up. So my tools produce ODF files with proper formatting (ready for CreateSpace), automatically. The only thing I can't automate is producing the table of contents; for that I have to open the document once and update the ToC, then export the PDF.

My tools are on github ([https://github.com/booksbyus/mkbook](https://github.com/booksbyus/mkbook)) and there's an example book using them ([https://github.com/booksbyus/scalable-c](https://github.com/booksbyus/scalable-c)).

Also to note:

* gitbook works really well as a front-end to show Markdown books off a git repo and collect edits. See [https://www.gitbook.com/book/hintjens/social-architecture/de...](https://www.gitbook.com/book/hintjens/social-architecture/details).

* If you write your own filters (text to whatever) you can do really interesting things. It's worth learning Perl just for this kind of work.

* CreateSpace is excellent, once you have cracked how to produce perfect PDFs for the cover and interior. Highly recommended.

* Amazon's author tools are overall good. A bit clunky sometimes (e.g. lots of separate logins), yet they work and can produce decent income for a self-published author.

* LibreOffice produces excellent PDFs. Really top class, once you've figured out the details. The way I use it in the mkbook tool is by templating the format; i.e. design a book by hand, then slice out the various pieces of XML, and then generate the body of the book text using the template. This gives me 100% control over the results.

* When you automate this, it can work really quickly, as in a few hours to to make a publishable draft, and perhaps 30 minutes to release a new version of the text to CreateSpace.

* Get a professional cover designer. It isn't that expensive and makes a big difference.

<comment><USER14> Recommendations for finding a professional cover designer? That's one of the things missing from the original article that I was interested in.

<comment><AUTHOR> The cover for my novel, which may be the most professional bit about my novel, was done by an artist I found through 99designs. I loved the process, I had over a hundred designs to choose from, tons of variants of these, etc. And I'm super happy with the result!

</comment></comment></comment><comment><USER15> Congratulations on a great accomplishment. For what it's worth, I found your post on how you reverse engineered Dan Brown [0] far more interesting than this one. Very cool!

But it seems writing was the easy part; how do you plan to market the book? After four years of writing, surely you want to make sure it gets maximum exposure. Do you have any concrete plans here?

Did you ever consider shopping the book around to publishers, or were you always set on publishing it yourself? Any comment on the fee structure of Amazon?

[0] [https://medium.com/@gabrielgambetta/how-i-wrote-my-first- nov...](https://medium.com/@gabrielgambetta/how-i-wrote-my-first-novel-during- my-daily-commute-e1d02c9447b9#.6i2bynt95)

<comment><USER16> > how do you plan to market the book?

that's what he's doing here.

<comment><AUTHOR> You're not entirely wrong, but not for the reasons you think :) I hope this post raises the profile of the book a but, but I'm much, much more interested in getting the book into the hands of more people, than in selling more copies. In other words, I want more readers, not more customers.

I can't just post a link to the book because of Amazon's terms, but as I offered at the bottom of the post, I'm giving free copies to anyone who sends me an email.

</comment></comment></comment><comment><USER17> I'm glad this all works well for the author, but still ...

 _Why Markdown?_

I know that it is widely used, but that's only because GitHub pushed it so much.

 _AsciiDoc_ is superior in almost every aspect. For larger texts I can strongly recommend it. The toolchain is mature, there are many modules for everything, even included UML from text description, if you need it.

Moreover, in addition to HTML output, AsciiDoc has proper PDF and LaTeX output as well.

Really, Markdown is only good within its niche, while AsciiDoc has a mature toolchain and has a proper extension system, so that new features are additional modules for the main tool, rather than incompatible forks of the Markdown processor.

There is the original AsciiDoc tool, but for modern use I recommend Asciidoctor.

[http://asciidoctor.org/](http://asciidoctor.org/)

<comment><USER18> Markdown was popular _long_ before Github.

The reason for using Markdown is simple: pandoc. Pandoc is just that damned good.

<comment><USER19> Pandoc can also process reStructuredText [1], which looks similar to Markdown, but has more features and a single non-ambiguous specification. Me and my co- author rejected Markdown for our book about internet memes [2][3] after a short evaluation, mainly due to ambiguities [4] and lack of features.

[1] [https://en.wikipedia.org/wiki/ReStructuredText](https://en.wikipedia.org/wiki/ReStructuredText)

[2] The book can be downloaded for free: [http://internetmeme.de](http://internetmeme.de)

[3] About the writing process: [https://news.ycombinator.com/item?id=12311546](https://news.ycombinator.com/item?id=12311546)

[4] [http://roopc.net/posts/2014/eval-stmd/](http://roopc.net/posts/2014/eval- stmd/)

<comment><USER18> I know. Personally, I prefer reST, but Markdown is more approachable than reST for a lot of people.

I wouldn't say reST is all that similar looking to Markdown though: they have very different philosophies, and really only coincide on the fact that they're used for marking up plaintext documents.

<comment><USER19> What do you mean with "more approachable" ? Markdown surely is more popular – but which constructs differ much in usability?

<comment><USER18> Off the top of my head, the complaints I've gotten have referred to directives, the link syntax, roles, not being able to nest inline markup.

None of this bothers _me_ , and directives and roles the source of much of reST's extensibility and power, but apparently those are the things that make reST a bit intimidating compared to the likes of Markdown.

<comment><USER19> Please elaborate on the complaints. What about the directives do users consider problematic?

<comment><USER18> The two nicest things about directives once you get the point behind them are that they're _consistent_ and _flexible_ : once you know the syntax for one directive, you know most of the syntax for the rest.

The problem with that is that the syntax is comparatively heavyweight. Part of this is due to reST's alignment rules and partly down to the generality of directives.

Let's take 'image' as an example. The simple way to embed an image in reST is this:

        .. image:: foo.png

And in Markdown:

        ![](foo.png)

And with some alt text:

        .. image:: foo.png
           :alt: Foo!

And in Markdown:

        ![Foo!](foo.png)

I've had people struggle with image directives simply because it doesn't sink in that the `:alt:` attribute has to be aligned with the directive name, whereas the rough equivalent in Markdown has no such issues.

OTOH, once you know how to embed an image in reST, you're 90% of the way to knowing how to embed a code block, whereas in Markdown you need to learn a little bit more syntax. The downside of that is that this:

        .. code::

           print("Hello, world!")

Is more heavyweight than:

        ``
        print("Hello, world!")
        ``

And people prefer the latter as it's less to type to the former, even if the former introduces no special-purpose syntax.

This is where Markdown's ad hoc nature has benefits over reST's more structured nature: if you want to embed an image in Markdown, you learn how to embed an image; in reST, you learn the directive syntax so you can embed images. That gives Markdown and reST different learning curves. Markdown gives you lots of little easy-to-learn tools, but requires that you be constantly learning new tools as you go along, whereas reST gives you fewer, slightly more complex tools you have to learn up front, and all the new stuff you learn later is based off of those. Moreover, Markdown has less impact on the text due to its ad hoc markup being more compact, unlike reST, which takes the route of being more general at the cost of being more verbose.

Somebody starting out confronted with either reST or Markdown will look at both and see that the latter looks more like plain text than the former, and thus there is less of an initial barrier to entry. That's why the likes of MkDocs exist in spite of the likes of Sphinx existing long before MkDocs and its ilk did.

TL;DR: Markdown makes easy stuff super easy and hard stuff possible; reST makes the easy stuff a little harder than Markdown, but lets you lift mountains.

Edit: s/codeblock/code/

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER20> I saw this a couple of weeks ago: [https://www.youtube.com/watch?v=FtieBc3KptU](https://www.youtube.com/watch?v=FtieBc3KptU)

I really like how he has no idea how to pronounce stuff, but he still managed to bend emacs after his will.

<comment><USER21> This is great, and hilarious. A good place to get an idea of the video from is about 13 minutes in until 15 minutes.

<comment><USER20> And actually quite inspiring. He got me to set up abbrev mode properly and actually start using emacs for _everything_ text related (even this comment, using "it's all text" for firefox).

I set up it's all text with a bunch of different file endings depending on what language I use, so that abbrev mode works correctly. It's a godsend.

</comment></comment></comment><comment><USER22> Hey, OP! I admire the fact that you took all this on yourself, and also from reading the linked blog post[0] it seems like you took the hacker's approach to the planning and writing of it too.

Honestly, I'm not sure it's a book I'd read - not a big Dan Brown fan - but I am deeply envious that you've completed such a huge undertaking :)

[0] [https://medium.com/@gabrielgambetta/how-i-wrote-my-first- nov...](https://medium.com/@gabrielgambetta/how-i-wrote-my-first-novel-during- my-daily-commute-e1d02c9447b9#.y4os6qg6e)

<comment><USER23> ... then OP, could you share this structure? "Soon the structure underlying The Da Vinci Code and Angels and Demons and The Lost Symbol was laying bare before my eyes. I could see why the stories worked. I had reverse-engineered Dan Brown."

<comment><USER24> Multiple mcguffins, drip-feed expository dialogue and prose, cliff-hanger in 30 minute reading time chunks. Always keep at least one good mystery open. Start in media res, slow the pace for the second act, which is where you consolidate to accelerate to a frantic conclusion where you draw seemingly unrelated threads together. Essentially, give the reader a reason to keep reading, and to keep coming back.

It's not Brown's recipe, it's how pretty much all compelling fiction hangs together.

<comment><AUTHOR> Pretty much, plus mini-cliffhangers at the end of each chapter. And tons of chapters. My novel has around 70, so roughly 5 pages on average. It helps keep the pace, and because of the psychological need to read "just one more chapter", it makes it a "page turner" or "a book you just can't put down".

</comment></comment></comment></comment><comment><USER25> You _can_ do it that way, but using something that's made for the job makes life a lot easier. Between Scrivener for writing and Vellum for compiling to ebook formats there's no reason to spend your time futzing around. If you want to format for print, either pay someone or learn InDesign because nice print layout is tougher than it appears.

Then again, a lot of the self-pub strategy these days is specifically geared around getting as much out as quickly as possible, so that's not necessarily an invalid approach.

In any case, I prefer to spend most of my time writing rather than screwing about with tooling.

<comment><USER26> > learn InDesign

Ahem, or learn the open-source Tex/Latex, from which InDesign took inspiration. If you can format a beautiful scientific/mathematical textbook for print using Tex, you can format anything. And Pandoc is a great adjunct.

But yes, it's not for people who wish to avoid screwing around with tooling. It's for perfectionists who want absolute control over layout/notation.

<comment><USER25> Writing a novel in LaTeX is not the best approach if you're looking to write a lot of text and don't have to worry about notation. But Neil Stephenson famously wrote Cryptonomicon in Emacs and LaTeX, so hey, it works for some people.

Edited to add that InDesign is much easier for back matter, front matter, laying out images, covers, etc. . . . And that's coming from someone who LOVES LaTeX. I set my old software company with some sweet templates and all of our proposals were typeset in LaTeX. It was awesome. But if you're writing a novel for print I really wouldn't recommend it.

<comment><USER27> I don't think you are supposed to write in LaTex anymore than you are supposed to write in InDesign

<comment><USER25> Spoken like someone who has never encountered `auctex-mode`

</comment></comment></comment></comment></comment><comment><USER28> I have a friend that writes porn using open source tools. Her stuff is on AMazon as well. She also told me how their numbers work. A book at this author's rank sells maybe 1 book a month. That's not good. Anyway...she uses openoffice to write her docs, then uploads to kindle. She authored some kinky shit like Control - A Billionaire Romance series that got high sales and turned on lots of folks. If you're into that sort of thing, like bondage and such.

<comment><USER29> It's kind of impressive just how high sales you get with not even that good smut fiction. ( _coughs_ 50 Shades. _coughs_ )

I've picked the wrong job.

<comment><USER28> Yeah...me too. Wish I wrote her books. Her name is M.P. Lodi in Amazon.

<comment><USER19> If she has written a blog post or something about her experiences, please link to it.

</comment></comment></comment></comment><comment><USER30> Let me recommend "working copy" for fantastic GitHub integration - <USER2> it's a git client on iOS (and android?) and it has a good enough editor that I can write .rst and git push and I have published an article - this went up today - [http://www.mikadosoftware.com/articles/alwaysgoodships](http://www.mikadosoftware.com/articles/alwaysgoodships)

<comment><AUTHOR> Oooh, that sounds great. I'll give it a try. Thanks!

Any idea whether it has good support for large files (e.g. a 300k character novel)? I've tried a ton of iPad editors but most of them can't handle large files, or don't have good scrolling support so it's a PITA to go to the end of the text.

<comment><USER30> I have never tried anything so large. I suspect it's just pretty much iOS basic editor (whatever that is)

Why not divide up the novel into chunks (chapters?) - I do this and stitch the chapters together before running through docutils - it's trivial and let's me divide and conquer (so far it's an unfinished manual for Dev Leads and managers)

<comment><AUTHOR> Yeah, that's a good point. I'll give it a try!

</comment></comment></comment></comment><comment><USER31> I used Latex to write my own book, in case anyone cares: [http://www.zerobugsandprogramfaster.net/](http://www.zerobugsandprogramfaster.net/)

If I had it to do again, I would consider using just tex and Makefiles.

</comment><comment><USER32> I'm a HUGE fan of using softcover.io to build documentation for runbooks and other technical documentation. You can write the text in Markdown (with some LaTeX and other goodies), and it automatically converts it to both HTML as well as epub, mobi, pdf. I like using it because you can make mobile documentation that can be pushed out to Mobile devices (and optionally managed by Apple's tools) so your on call resource has the ability to pull up the documentation about the alert that they are receiving on their mobile device so they know how to remediate/mitigate that issue.

</comment><comment><USER19> Here are some lessons I learned writing the German-language O'Reilly book about internet memes [1]:

• reStructuredText [2] looks superficially similar to Markdown, but is vastly more useful: It has more features and a defined specification without ambiguities. Many RST processors do not follow the specification in edge cases, but at least it is clear how the document should be layouted. The author of Pandoc [3] fixes bugs very quickly. The Sphinx document processing system [4] can export to HTML, PDF, LaTeX, Epub etc. and can check included URLs automatically (we did not use it for the final version).

• Using Git is a good idea even if you are writing alone: The web interface for our repository and Sparkleshare – a DropBox-like interface for Git [5] – helped immensely with reviews. Gitstats [6] helped me to find out that crunchtime before a deadline always led to at least a few days of writer's block, which enabled me to gauge what amount of text I could write in a sustained manner.

• If possible, save everything relevant to the book in your repository. One example: I investigated the origin of the first lolcats ("Happy Cat" [7], it came from promotional materials of Russian distributors for the "Happy Cat" cat food brand), but have lost some of the emails related to the investigation. Similarly, footnotes in our printed book are now exhibiting link rot. Both of this could have been prevented by importing all source materials in the repository for our book and making sure that all the web pages we mentioned are archived in the Wayback Machine [8].

• The file format of the text matters for interoperability, but which text editor you use does not. Do not learn a new editor for the project if you are already comfortable with one. I used GNU Emacs while my co-author used GNU nano; we never had any problems due to that.

• Me any my co-author worked remotely for the most time. Each evening we had a short chat about what each of us finished during the day, what we were working on, and what plans we had for the next few days. At least for me, telling someone else what I did boosted my motivation; having such a routine process helped me to overcome writer's block.

• There seem to be at least two different writing modes. I think a long time about what I want to write and then write the text, making a few cosmetic changes later. My co-author writes a rough draft and then refines it with each pass. This means that my text is always of the same quality, but he will always meet the deadline. Using Git, we found out he wrote around twice the amount of text as I did, as he re-wrote almost everything at least once. We had no problems with this because we clearly allocated who was responsible for each chapter in the beginning.

• Avoid changing your workflow mid-way or towards the end of the project if you can avoid it, it probably leads to more stress than it is worth. Thanks to publisher shenanigans, we had to send in our texts as multiple ODT files and got it back as one layouted PDF that we had to annotate and send back. One problem was that there seemed to be no software to merge two PDFs with annotations. I quickly hacked up something with Ghostscript only to be told much later that we unintentionally overwhelmed the publisher's pipeline as each one of our around 1000 annotations to the PDF text was apparently faxed to the layouter.

• Get a proper advance. We did not want one and I think it was a failure. You will most likely not make much money with your book – even if it is good and about a popular topic. Three years after the book is published, only around 15 copies are sold each quarter; I suspect it might be one university course using it as classroom material.

• Include a section in your contract about free licensing in case the book is not available. We told O'Reilly we would only accept the contract with a provision that the book would be CC BY-NC-SA and the licensing would switch to CC BY-SA (same as Wikipedia) as soon as O'Reilly stops selling the book. Now that the German section of O'Reilly went under the book is still sold, but by another company that uses the O'Reilly imprint [9].

• Get your hands on as many specimen copies as you can get away with, those are some of your best promotion material in terms of effort vs results. O'Reilly did not do much to promote our book, but they did provide us with copies.

• Do not neglect your loved ones because of work. I regret not having spent more time with the ex-girlfriends I was together with during the time I wrote the book.

[1] The book can be downloaded for free on [http://internetmeme.de](http://internetmeme.de).

[2] [https://en.wikipedia.org/wiki/ReStructuredText](https://en.wikipedia.org/wiki/ReStructuredText)

[3] [http://pandoc.org/](http://pandoc.org/)

[4] [http://www.sphinx-doc.org/](http://www.sphinx-doc.org/)

[5] [https://www.sparkleshare.org/](https://www.sparkleshare.org/)

[6] [http://gitstats.sourceforge.net/](http://gitstats.sourceforge.net/)

[7] [http://knowyourmeme.com/memes/happy- cat](http://knowyourmeme.com/memes/happy-cat)

[8] [https://en.wikipedia.org/wiki/Wayback_Machine](https://en.wikipedia.org/wiki/Wayback_Machine)

[9] [https://en.wikipedia.org/wiki/Imprint_(trade_name)](https://en.wikipedia.org/wiki/Imprint_\(trade_name\))

<comment><USER33> Since shutting down my own Internet presence for good, which also took down technical documents I had written years ago that - to my great surprise - <USER3> found were actually cited a lot according to Google Scholar - I'm thinking maybe linking to the archive.org link is better than linking to the actual URL? Any thoughts? My documents are still available under the orginal URL there, so if all those footnotes and citations pointing to one of my docs pointed there they would still work. It also has the added advantage of being able to link the concrete version as it was when you included the link.

Example for the first link in the parent comment, pointing to the (right now randomly selected) 26 March, 2016 version of the webpage:

[https://web.archive.org/web/20160326165458/https://en.wikipe...](https://web.archive.org/web/20160326165458/https://en.wikipedia.org/wiki/ReStructuredText)

TL;DR I recommend to always cite sources using their archive.org document version.

<comment><USER19> A HTTP GET to a URL prefixed with [http://web.archive.org/save/](http://web.archive.org/save/) archives the content behind that URL.

I use the conkeror web browser [1] and have included the following code in my $HOME/.conkerorrc/config.js and execute "wbsave" for many documents (even my own) whenever I send the URL to someone else.

      define_webjump("wbsave", function(url) { return "http://web.archive.org/save/"+url; } );

[1] [https://en.wikipedia.org/wiki/Conkeror](https://en.wikipedia.org/wiki/Conkeror)

<comment><USER34> Probably best to use HEAD instead of GET to save downloading the result.

</comment></comment></comment></comment><comment><USER35> Most novels in the history of the world have been written using exclusively open-source tools.

<comment><USER36> I'd be careful with that assumption. Tthere aare over one million novels written every year now.

The novel as a literary form dates only to the 1500s. Cervantes' _Don Quixote_ is generally considered the first novel.

I suspect a quite large share, possibly majority, ofnovels have been written since the word processor era.

[http://www.forbes.com/sites/nickmorgan/2013/01/08/thinking-o...](http://www.forbes.com/sites/nickmorgan/2013/01/08/thinking- of-self-publishing-your-book-in-2013-heres-what-you-need-to- know/#75c05e3929f4)

</comment></comment><comment><USER30> A little more interesting was how he reverse engineered Dan Brown novels (see [https://medium.com/@gabrielgambetta/how-i-wrote-my-first- nov...](https://medium.com/@gabrielgambetta/how-i-wrote-my-first-novel-during- my-daily-commute-e1d02c9447b9#.248qy79fb))

I would love to see his spreadsheet on what makes a Dan Brown novel - it's like distilling literary evil.

<comment><AUTHOR> _" How I distilled literary evil"_ sounds like an amazing title for a follow- up article :D

</comment></comment><comment><USER37> I'd like to know what people are using for writing non-fiction. Mainly I'd like to know how people keep track of their research and cross referencing it from their editor.

Here's what I have:

* Many words typed up in One Note

* Highlights and annotations in iBooks, Kindle, and Instapaper (although Instapaper allows export to MD)

* Notes from PDF exported to Markdown using the Highlights app[0]

* Images

* Tons of shit to read in pinboard and other places

Here's the kicker, I travel a lot (every week) so I need most of my tools to work well offline/slow wifi and preferably on an iPad/iPhone.

What do people use for organizing notes and research (literature of all sorts), cross referencing, organization, and first draft?

0: [http://highlightsapp.net/](http://highlightsapp.net/)

<comment><USER38> 1\. I use Devonthink Pro for notes / research: [http://www.stevenberlinjohnson.com/movabletype/archives/0002...](http://www.stevenberlinjohnson.com/movabletype/archives/000230.html).

2\. Many use Scrivener for writing, but I have never quite managed to agree with it properly, so I just use Word.

3\. I use Bookends: [http://sonnysoftware.com/](http://sonnysoftware.com/) for citations. No one it seems really likes their citation manager, but I like this one best.

See also [http://blogs.plos.org/neurotribes/2011/06/02/practical- tips-...](http://blogs.plos.org/neurotribes/2011/06/02/practical-tips-on- writing-a-book-from-22-brilliant-authors/).

<comment><USER37> 2\. Scrivener is great, wish it could do .md natively 3\. Yes I hate Papers. Citation managers seem to only be useful for academic papers anyway.

</comment></comment></comment><comment><USER39> I also wrote a book called the Final Six Days. If you visit his Golden Legacy book page on Amazon you will see my book in the "Others Have Viewed" section. It's mostly due to me checking on his and other novels.

I was planning a Medium article about my (similar) experience. I'm not sure why he'd chose markdown when epub is just structured HTML. If you're a software dev just stick with HTML. Also, to anyone planning a novel: get ready for annoying inconsistencies in display across the various Kindle + iOS + android devices. Fonts not displaying correctly, sizing is off, things not centering, some CSS selectors not being applied at all. It is IE/Netscape circa 1997 all over again.

<comment><USER40> > I'm not sure why he'd chose markdown when epub is just structured HTML.

Markdown's a lot simpler/faster to read and write than HTML, and books rarely need the extra features HTML provides. When you do, you can always do inline HTML in Markdown.

> Also, to anyone planning a novel: get ready for annoying inconsistencies in > display across the various Kindle + iOS + android devices. Fonts not > displaying correctly, sizing is off, things not centering, some CSS > selectors not being applied at all.

Oh yeah. It's even worse if you write a technical book.[1]

[1]: [http://journal.stuffwithstuff.com/2014/11/03/bringing-my- web...](http://journal.stuffwithstuff.com/2014/11/03/bringing-my-web-book-to- print-and-ebook/)

<comment><USER39> I fought getting fixed-width fonts to work for weeks across all these devices. Believe it or not, despite being a sci-fi novel it does have code snippets :)

</comment></comment></comment><comment><USER41> Can somebody show him how to take a screenshot on Linux please?

<comment><AUTHOR> I know how to take screenshots on Linux :D Taking pictures of the screen felt more "authentic", in the same way that I took pictures of the printed book instead of pasting a PDF.

</comment></comment><comment><USER42> I have started to write a novel using almost exactly this setup. It's encouraging to hear it worked out for someone. So far I've been using plain markdown and just rendering it to HTML for a pretty view, but with plans to turn it into e-pub later. As mentioned in the article, the controlled sync is what makes this beat Google Docs. I set up Vim and Mercurial on my phone, so I can sync and edit the text anywhere. Fun tools to play with!

</comment><comment><USER43> Go back to the early 2000s. I'm kind of obsessive about data portability, and the lifespan of a novel is closer to 30 years than the 3-5 months typical of business documents. Alas, I'm selling novels to Big Six publishers who like to chew on a Word file. So ...

Writing tool of choice: vim. Markup format: Perl's POD (plain old document) format, because (a) I can mess around with the formatter if I need to (old Perl hand), (b) it can export to a variety of formats, (c) it has the minimum necessary set of directives -- headings, lists, emphasis -- that I need (novels are typographically simple unless you're playing Stupid Layout Tricks, hello Alfred Bester and Samuel Delany) and (d) Markdown didn't exist back then.

Version control: a novel with a single author is piss-easy. I used rcs, because it was ubiquitous, easy to understand, and to the extent that I was keeping projects on multiple computers I was fine using rsync to keep a directory tree updated; remember, a novelist working alone has no need for multiple user IDs, no forking of multiple versions, virtually _never_ any need to diff or merge -- just the ability to roll back to a previous point if I took a wrong turn in writing (in other words, undo checkpoints with names). No cloud services, of course. (Dropbox didn't come along for a few years.)

Build: makefiles and some homebrew perl scripts FTW. Type "make", check out latest draft and generate HTML, RTF and (eventually, via an external toolchain) Word files I can ship to my editors.

Back in those days, copy edits showed up as a bunch of paper print-outs with red ink on them and you mailed them back to production after you added your own chicken scratchings. (If you were smart you scanned/photocopied the pile first, for insurance: this saved my ass on two occasions when CEMs went missing in the post, thank you for nothing, Royal Mail). Page proofs ditto. So I was able to use my homebrew toolchain happily for several years during which I wrote "Singularity Sky", "Iron Sunrise", "Accelerando", "The Atrocity Archives", and the first three Merchant Princes novels using it.

Then the publishers began moving to Microsoft Word tracked changes for processing copy-edits, and annotated PDFs for the page proofs, and it was all over bar the shouting; Word tracked changes suck, but trying to check the changes on a _large_ document in a third party word processor like LibreOffice sucks even harder (one sleeper bug that nobody triggered before can cost you a couple of months work), forcing me into Word for the post-writing workflow. And then a Better Way came along for writing books in the shape of Scrivener, for which there is nothing remotely like an open source equivalent and which is well worth the price. Alas.

Which is why my open source novel writing days are over. (But if I had to do it again these days? Markdown, Pandoc, and Git, probably with all the fancy Vim plugins I could lay my hands on.)

<comment> david-given At some point I'd be really interested to read a writeup on Scrivener and your writing workflow.

<comment><USER43> Sure: here you are!

[http://www.antipope.org/charlie/blog- static/2012/07/writing-...](http://www.antipope.org/charlie/blog- static/2012/07/writing-a-novel-in-scrivener-e.html)

<comment> david-given Awesome --- thanks!

</comment></comment></comment></comment><comment><USER44> FWIW - Scrivener is a plausible novel-writing tool, mentioned a couple of times in these comments. It took a Long Time, but their iOS app was released a couple of weeks ago.

Scrivener by Literature & Latte [https://appsto.re/us/jqx95.i](https://appsto.re/us/jqx95.i)

</comment><comment><USER45> You probably don't want to give away specific numbers, but in generally, how is the profitability of selling on Kindle versus paperback? The price difference looks to me to be less than the cost difference, thereby making Kindle more profitable. Is that accurate?

<comment><USER46> >You probably don't want to give away specific numbers, but in generally, how is the profitability of selling on Kindle versus paperback?

Not OP, but I've sold thousands of ebooks, but only a few dozen paperbacks.

Paperbacks are for people with distribution networks and warehouses. As a self-publisher, we are relying on electronic distribution to cut the overhead costs. Profit per unit is about the same between the two, but we can sell more ebooks because the cost to the customer is only a bout a 1/3 compared to print.

Mostly print copies are for promotion. Handing them out as free prizes/gifts are a great way to build readers.

</comment></comment><comment><USER47> There handful of proprietary software I actually pay for because I personally think they are by far the best tools of the trade. For creative writing I think I have found the best solution in form of Scrinver.

It has been simply the best tool I have used to write my novels.

</comment><comment><USER48> This is pretty cool. But I would be interested to know how the author would have handled other requirements. Embedding images, other scripts, typesetting different column layouts fonts, spacings, etc.

<comment><USER36> Almost none of that matters in _a novel_. Which has little structure other than sentences, paragraphs, and chapters. Many won't even include italic text (though some might).

If your output is simple, stick to simple tools. Tool tyranny -- decisions you're _forced_ to make, because your tools dictate them -- is a significant cognitive overhead.

(Incidentally, this is why I found Lyx actually _harder_ to use than LaTeX. Lyx forced me, _from the beginning_ , to consider the document format and structure, whilst LaTeX lets you just start writing paragraphs and sorting out the rest _later_.)

The more extended answer: if you've got other shit you need to do, you address it when the time comes.

Markdown, specifically, allows for inline inclusion of other formatting languages, particularly LaTeX and HTML. It also has native support for images, tables, lists, etc.

You've got the option for Roman, bold, italic, and monospace text. If your document needs more than this, address it in layout.

In the workflow mentioned here, the author actually answers your question: images, etc., were added (for the cover) in LibreOffice and through ePub management tools, including a Python script to reorganise the output file as needed.

Again, the advantage of Markdown (or rsT, or AsciiDoc, or ...) is that _the markup gets out of your fucking way_. Instead of obsessing over format, _you write your fucking story_ (or article, or essay, or rant, or ...). Then deal with the layout _later_.

</comment></comment><comment><USER49> LibreOffice has a plug-in for generating ePub files. You can do the whole job in one step.

Not only that, with LibreOffice, your book can have _pictures_!

<comment><USER36> Markdown: ![arbitraary text](image-uri)

</comment></comment><comment><USER50> I like this idea of this.

Would be great to have a website site that shows entire open source workflows and the optimal software/ approach. Anything like this exit?

</comment><comment><USER51> Ultimately, any text processing toolchain that uses ReGex for parsing will inherently be difficult to extend. And slow. And perhaps even broken. The disconnect lies in the human-machine readability gap. The only way to do it right, I believe, is custom text formats, scanned via structured strings into FSM grammers.

</comment><comment><USER52> You do use a mac though...

</comment><comment><USER53> Interesting to see that open source DTP is still a challenge in 2016.

</comment><comment><USER54> "self published" _

</comment><comment><USER55> ガンベッタさんは、良く頑張った、ね。

</comment><comment><USER56> vim, pandoc

</comment><comment><USER57> no git?

</comment><comment><USER58> I did this back in 1990. Using Emacs and TeX/Metafont. And so did many, many other people.

</comment><comment><USER59> > Ex Google Zürich

Badge of Honor

<comment><AUTHOR> Relevant, because [https://medium.com/@gabrielgambetta/how-i-wrote-my-first- nov...](https://medium.com/@gabrielgambetta/how-i-wrote-my-first-novel-during- my-daily-commute-e1d02c9447b9)

</comment></comment><comment><USER60> "To create this file, I used Pandoc again, this time to convert the Markdown files to ODT (OpenDocument Text), which can be read by LibreOffice."

Ouch. WYSIWYGs are terrible.

<comment><USER61> I think he says he changed styles only, which is closer to the WYSIWYM ("what you see is what you mean") idea used by Lyx, for example.

I agree that WYSIWYG type of editors are terrible, specially for big documents, but limiting them to the use of styles, is manageable (in my experience at least).

</comment></comment><comment><USER62> Meh.

I wrote two novels in emacs.

(now I just need to find some time to finish the final proof read and actually PUBLISH the damned things).

<comment><USER63> You managed to call one of your novels "recursion". Proof that the medium colours the work.

</comment></comment><comment><USER64> Is it inappropriate to point out that most likely there were a lot of closed source binary blobs involved in making his computer actually work?

<comment><USER65> Does pointing that out advance the discussion in a useful way? Does it help having a constructive debate? Does it make anyone's day or life better? Are you just posting in the hope of getting some upvotes for making a funny, snarky short comment? Is your comment actually relevant to the discussion? Does the fact that you ask that question indicate that you already know the answer?

<comment><USER64> I did not write that from my Lemote, channeling Stallman, nor I have any general enmity towards closed source software (that I both use and write). I just don't like inaccurate proclamations of purity.</comment></comment></comment>


Protecting privacy in genomic databases - <AUTHOR> http://news.mit.edu/2016/protecting-privacy-genomic-databases-0809 <comment><USER2> Argh. The amount of headache we must go through to lock down this information is ridiculous compared to the risk. While this might be an issue for Bill Clinton or Kanye West, the vast majority of people won't be terribly adversely affected by breaches of their medical records.

A much better solution than encrypting every byte of data and vainly trying to secure the impossible (to say nothing of purposely corrupting the data) is just step up enforcement: punish people who leak, sell, and buy private health information more assiduously. Data breaches will happen; the social contract is our best defense against them.

</comment><comment><USER3> I've worked with Biobankers/Bioinformaticians on this type of problem before and none of them wanted us to got the Differential Privacy route. Why? The vast majority of genomic diseases are rare diseases (with 10s-1000s of carriers). Destroying information is not an option with small sample sizes. For huge GWAS studies, it could be an option. But in the general case, security is preferable over privacy when a choice needs to be made.

<comment><USER4> Differential Privacy has its limits for small data sets. Luckily, more research is on the way to run arbitrary calculations on multiple datasets that remain private in its entirety (basically homomorphic encryption, but efficiently) [1]. Algorithms that work reasonably well are simple statistical tests and ridge regression [2].

I think these innovations are rare instances where technology can really solves a social problem.

[1] [https://www.microsoft.com/en-us/research/microsoft- researche...](https://www.microsoft.com/en-us/research/microsoft-researchers- enable-secure-data-exchange-cloud/) [2] [https://www.microsoft.com/en- us/research/wp-content/uploads/...](https://www.microsoft.com/en- us/research/wp-content/uploads/2016/06/SDE.pdf)

</comment></comment><comment><USER5> No scientist will ever use a database like that and no government agency will fund it. Data provenance is absolutely vital for getting your work published and good data accessible is infinitely more valuable than garbage accessible to everyone. I say this as someone who is dedicated to open data and open science. Privacy is something that science has to deal with. There may be a set of questions that can still be answered with degraded data, but I bet that if such a system were built scientists would still try to used the degraded data to answer questions which it could not answer, wasting their time and federal money.

</comment><comment><USER6> So they want to improve privacy by corrupting data...seems like a bad solution.

<comment><USER7> GWAS isn't precise in any case. Adding a few bits of entropy isn't going to materially impair the quality of results.</comment></comment>


Chekhov: "Cultured people must, in my opinion, satisfy the following conditions" - <AUTHOR> https://medium.com/the-polymath-project/anton-chekhov-how-to-become-a-cultured-person-f4bf9e8889f3#.otca76tbz <comment><USER2> I am an American living in Russia. When I was studying Russian I read a number of Chekhov's short stories and got a feel for who he was.

I later came across one of his quotes: "В человеке должно быть все прекрасно: и лицо, и одежда, и душа, и мысли" (Everything in a person should be excellent: his face, his clothes, his soul and his ideas). I sometimes discussed this phrase with Russians, eventually coming to be believe that "face" in this context meant that you should do all you can to appear clean and groomed, not necessarily that you need to be beautiful. I liked this quote and it can be a starting point of interesting discussion when I talk to Russians about Chekhov.

There's a funny story about when Chekhov went to Tomsk (now a town of about 500,000 people in Siberia.) In a letter, he wrote "Tomsk is a very dull town. To judge from the drunkards whose acquaintance I have made, and from the intellectual people who have come to the hotel to pay their respects to me, the inhabitants are very dull, too." In recent history, the inhabitants of Tomsk erected a bronze caricature statute [1] of Chekhov which makes him look funny. The local residents consider that touching his nose is good luck, so the statue has a really shiny bronze nose. I personally saw it and even touched his nose.

[1] [https://www.tripadvisor.com/Attraction_Review-g665310-d35959...](https://www.tripadvisor.com/Attraction_Review-g665310-d3595947-Reviews- Chekhov_Monument-Tomsk_Tomsk_Oblast_Siberian_District.html)

<comment><USER3> > Everything in a person should be excellent

Probably nitpicking, but I'd translate "прекрасно" as "beautiful" here.

<comment><USER4> I think "exellent" is an excellent choice of the word (beautiful ≈ красиво).

<comment><USER5> "excellent" makes it sound like Bill & Ted doing the translation, though. 'first-rate' is a common choice for that quote which is also unsatisfying - without the unwanted English connotations but also without any of the desirable Russian ones.

<comment><USER6> It must be a very small subset of English speakers for whom the word "excellent" can only be read in a Bill & Ted accent, so I suggest this is bogus argument.

<comment><USER5> Yes, it must be and yet 'excellent' is a really poor translation for this.

</comment></comment></comment></comment></comment></comment><comment><USER7> One needn't click on affiliate links to buy Chekhov's works. Project Gutenberg has them: [http://www.gutenberg.org/ebooks/search/?query=chekhov](http://www.gutenberg.org/ebooks/search/?query=chekhov)

<comment><USER8> Shameless plug, Chekhov's complete short fiction for free, at Standard Ebooks: [https://standardebooks.org/ebooks/anton-chekhov/short- fictio...](https://standardebooks.org/ebooks/anton-chekhov/short- fiction/constance-garnett/)

We took the Gutenberg transcriptions, compiled them into one ebook, ordered by publication date, lightly modernized, edited, corrected, and completely proofread them. Check out the Github repo to see the changes (which are a lot, since it's a huge amount of writing): [https://github.com/standardebooks/anton-chekhov_short- fictio...](https://github.com/standardebooks/anton-chekhov_short- fiction_constance-garnett)

While I don't speak Russian, I've found Garnett's translations to be extremely readable. A lot of her contemporaries seem to have agreed.

<comment><USER9> Standard Ebooks looks to be a very interesting project. Is there a story behind the project that details how you got started and who are the main contributors?

A 'Show HN' [0] or a write-up on indiehackers.com [1] would be awesome and could you more contributors.

I subscribed to your RSS feed [2] that announces your new publications but it lacks a URL for me to download the Ebook right away.

Also, is there a way for omnivoracious readers like me to download all your Ebooks in a mega .zip file at one go?

[0] [https://news.ycombinator.com/show](https://news.ycombinator.com/show)

[1] [https://www.indiehackers.com/](https://www.indiehackers.com/)

[2] [https://standardebooks.org/opds/all](https://standardebooks.org/opds/all)

<comment><USER8> Thanks for the suggestions! The project is very small, not for profit so indiehackers.com would probably not be interested. It's all volunteer-based so if you're interested in contributing work somehow then drop me a line.

The OPDS feed is mainly for reading systems and libraries, not for human consumption. However it does actually include direct links to epub files, but you typically have to view the feed source--i.e. the raw XML--to see them. Humans should subscribe to our mailing list,[0] where we do actual release announcements as they occur.

[0] [https://groups.google.com/forum/#!forum/standardebooks](https://groups.google.com/forum/#!forum/standardebooks)

<comment><USER9> > _The OPDS feed is mainly for reading systems and libraries, not for human > consumption._

Understood. I was using a standard RSS reader to parse the feed.

I added the OPDS feed to my FBReader app on Android [0] via its Network Catalog feature [1]. Et voila, I have your entire catalog at my fingertips. Thanks. :-)

> _It 's all volunteer-based so if you're interested in contributing work > somehow then drop me a line._

I can definitely see myself volunteering. Thanks for the invite. :-)

[0] [https://play.google.com/store/apps/details?id=org.geometerpl...](https://play.google.com/store/apps/details?id=org.geometerplus.zlibrary.ui.android)

[1] [https://fbreader.org/content/documentation/own- opds](https://fbreader.org/content/documentation/own-opds)

</comment></comment></comment></comment></comment><comment><USER10> I notice this theme again and again in every "change yourself" book, show, class, lecture. Be it weight loss, improving mental health, curing addiction, becoming a better writer/photographer/programmer, the theme all comes down to a battle against yourself. Fighting the desire to snack, fighting the desire to lay on the couch instead of going to the gym, fighting the desire to veg and watch netflix instead of watching another lecture in the MOOC you're taking.

You'd think it'd be easier to control ourselves. I have trouble understanding how we can _want_ something, but some other part of us can seem to _not want_ that thing, or just be extremely short sighted to the point of being an obstacle to our "long-sighted self."

<comment><USER11> The more I think about it and learn about the functioning of the brain, the less I believe in the unitary 'self'. I once read the suggestion that what we think of as the 'self' is really just the brain's public relations department.

<comment><USER12> You might enjoy a book called Quantum Psychology (Robert Anton Wilson).

He lays out some interesting concepts about the way we think and one thought experiment in particular, observing all the 'thinkers' within and how they can multiply at will i.e consider the following innocuous statement:

"I observe that I am happy"

Who is happy and who is the observer?

<comment><USER13> Eckhart Tolle may say the observer is your conscience and who is happy is your mind. That's a guess though. :)

</comment></comment></comment></comment><comment><USER14> I suspect a subtle mistranslation here. Chekhov seems to be talking about decent people, or maybe good people, not cultured ones.

In English, it is quite possible to be cultured but bad. You know all kinds of stuff about literature, music, art, history and science (and therefore you are cultured) but you cheat on your wife and your taxes (and are therefore bad).

<comment><USER15> Agreed with treve, what does culture really mean? If you read ALL the literature, listen to ALL the music, view ALL the art, can recite ALL the history and do ALL the science... and still conclude it's OK to cheat on your wife and taxes, the argument can be made that none of this culture was internalized, and hence you are not cultured.

Of course things are rarely so black-and-white, but that's the gist of it.

<comment><USER16> You can understand something and possess the qualities of it and still choose to ignore them

</comment></comment></comment><comment><USER17> This translation is great but I think it's still interesting to read into the connotations of Chekhov's original words:

[http://www.anton-chehov.info/754-n-p-chexovu.html](http://www.anton- chehov.info/754-n-p-chexovu.html)

 _> you are drawn away from it, and you vacillate between cultured people and the lodgers vis-a-vis_

 _> тебя тянет от нее, и тебе приходится балансировать между культурной публикой и жильцами vis-а-vis._

Here "cultured people" are originally "культурная публика", literally "cultured public".

 _> Cultured people must, in my opinion, satisfy the following conditions:_

 _> Воспитанные люди, по моему мнению, должны удовлетворять след<ующим> условиям:_

Here "Cultured people" are originally "Воспитанные люди". More like well- brought-up.

<comment><USER18> In English people used to say 'well-bred' but I can't quite squeeze that into the title above.

<comment><USER19> IMHO well-bred has a meaning of specific ancestry, of aristocracy of one sort or another, of which the Russian "воспитанный" has none - it's rather well brought-up, well-behaved, conforming to the norms and customs of the polite society.

</comment></comment></comment><comment><USER18> Here's a lovely little article about Chekhov that has been languishing on my "post this someday when HN is slow" list.

[http://lithub.com/anton-chekhov-a-post-modernist-way- ahead-o...](http://lithub.com/anton-chekhov-a-post-modernist-way-ahead-of-his- time/)

</comment><comment><USER15> I'm struck by how much this echoes Confucius and the Analects and how they define the so-called gentleman or noble man (not to be confused with nobleman, and translation is hard). Given how unlikely it is that Chekhov had exposure to either, it's incredible to observe the universality of these qualities considered admirable.

If people haven't, do recommend checking out The Analects! Caveat: there are quite a few excerpts that need to be taken in historical perspective, or can be easily misinterpreted (e.g. advice to be slow to embrace new that was in context of centuries of chaos and warring, or statements on women that interpreted naïvely clearly contradict other statements on mothers).

<comment><USER20> Can you give some examples of those "statements on women that interpreted naïvely clearly contradict other statements on mothers?"

I've read bits and pieces of Analects, thought it was ok, but felt that is was very much reflective of the perspectives of the time. Most of it felt like stuff that was obvious and other stuff felt not applicable to the world today. IIRC there was some stuff about traveling abroad / being a good guest in a foreign country, and I did think that part was useful.

I don't really recall reading stuff about women though. I wouldn't expect an enlightened perspective from a man in that time period though, no matter how kind the interpretation. It wouldn't invalidate the whole work, it would just show that the author was wrong about at least one thing, which is not an issue (unless we're talking about holy books where it's not acceptable to acknowledge mistakes).

<comment><USER15> Agreed on all counts!

"The Master said: 'Girls and inferior men are hard to get along with. If you get familiar with them, they lose their humility; if you are distant, they resent it.'" (Chapter 17)

This sounds pretty bad. My reading of it is that he meant that relationships between men and women are particularly tricky (bearing in mind heteronormative standards at the time, etc. etc.), so one has to be careful not to misrepresent them or else everyone suffers. This can be generously reinterpreted to general use ("don't toy around with people and treat them straightforwardly"). Realistically... I'm happy to take it as a mistake on his part and just extract something worthwhile from it even if it's not what he meant.

With regard to enlightened perspective or lack thereof, there's an interesting question. I'm no expert on Confucius himself and how he regarded women, but his filial piety towards his mother is widely lauded (as was Mencius').

Hope that helps!

<comment><USER20> Thank you for giving that additional context! Certainly good food for thought.

<comment><USER15> You're quite welcome, glad it was interesting!

</comment></comment></comment></comment></comment><comment><USER21> _They do not use their smart phones in the presence of others._

<comment><USER22> On the timeline of humanity, our pocket computers are brand-new. It will likely be a couple of decades before we sort out the social norms of using them. However, I think that we will fall towards using these devices constantly in all social settings because the young people growing up won't have the nostalgia of time before smartphones-- there was never a time where you and your friends sat at a table and talked completely free of digital interruption. Go to a coffee shop or restaurant near a high school around the lunch hour...

<comment><USER23> It's also possible the current physical manifestations of these devices will change quite a bit in the near future. I imagine in 20 years of less we'll mostly be using some eye based (glasses, lenses, whatever) display with AR and gesture control.(likely with a gesture "shorthand" for AR control with limited movement for when you don't want to wave your hand in front of you).

At that point, it will be both much more and much less invasive, as it will likely be fully integrated. You can use it to enhance your conversation by adding context, distract from your conversation by utilizing it for something unrelated, or ignore it entirely. Sort of like your memory and imagination now (because that's what it will be at that point, a slower but much fuller extension to your current mind).

<comment><USER22> Thats an intriguing prediction of the future.

<comment><USER24> Why intriguing? That's the only way I see it going, as well.

The Internet is already an extended memory. Other people's minds are an extended processing power, as well.

It can only get closer and more tightly integrated to our brains, unless society as a whole chooses to get away from that.

This is why privacy concerns are real - what we do today about them will impact the future massively. Will there be a central database of people's private thoughts and memories or will they remain private?

</comment></comment></comment></comment></comment><comment><USER25> This letter appears to be a drawn out detailed set of insults to his older brother. Maybe cultured isn't the best translation. Is this what we in the US would have called "classy" back then?

<comment><USER26> Classiness does not preclude being insulting. It simply calls for a higher degree of skill in the insulting. History is full of classy insults. Diplomacy is nearly made of them.

[http://allowe.com/laughs/book/When%20Insults%20Had%20Class.h...](http://allowe.com/laughs/book/When%20Insults%20Had%20Class.htm) \- just a quick hit on "classy insults" that seems to fit what I'm saying here.

<comment><USER27> There's also something to be said for long-form writing with snail mail enabling this. A very quick wit is required to make such insults in real time, as opposed to behind a pen with a day or two of thought.

</comment></comment></comment><comment><USER28> > no great writer has not been touched in some way by his stories.

Don't trust people who say things like this, btw.

</comment><comment><USER29> Dunno. I'm reading Chekhov's letters and it doesn't look to me he was up to his own standards.

<comment><USER30> > it doesn't look to me he was up to his own standards

Would anyone pass your test?

<comment><USER29> I haven't made any test. What do you mean by it?

</comment></comment></comment><comment><USER31> I have a set of rules I give my children. I started with them as kind of a joke because they would always flip out about stuff, so naturally, rule #1 is:

1) Be calm

2) Be kind

3) Don't steal

4) Always attempt your best

I've always liked the thought that "don't steal" is the ultimate morality lesson. Killing someone is stealing their life away. Adultering is stealing someone else's love or relationship. And of course, stealing in its basic meaning.

<comment><USER32> I've noticed that younger kids seems to take events extremely seriously

One thing that helped me keep this in perspective, is that every event makes up a significantly larger fraction of their existence than that event makes up of my existence

I'm 36, my oldest daughter is 6

occasionally she'll get really upset when seemingly minor setbacks happen (a craft she's working on doesn't turn out the way she wants it too seems typical), while we do try to work on her handling her frustration better

its helpful for us, to remember that 1 messed up craft out of 6 years of life experience, is 6 times a bigger deal than 1 messed up craft out of 36 years

</comment></comment><comment><USER33> I used to be cultured, at home, at work, and in public.

But, as I hit my late 20s (circa 1998),I became the opposite at work, and that approach has gotten me much farther ever since.

Play the game.

</comment><comment><USER34> Reminds me a lot of Desiderata.

[https://en.wikipedia.org/wiki/Desiderata](https://en.wikipedia.org/wiki/Desiderata)

</comment><comment><USER35> My favorite, is his point about "squeezing out the slave". He foresaw well how society can be highly tolerable of being subservient to dictatorship.

</comment><comment><USER36> Seems more like a description of middle-class virtues.

</comment><comment><USER37> Yawn, in addition to becoming cultured, I fear this advice will also make you incredibly boring.

</comment><comment><USER38> Oh, how I laughed, to finish this ode to humility and there at the bottom, this little blurb which seems to be working overtime in demonstrating as many of the just-disparaged traits as possible.

>Want more? Join over 15,000 readers getting The Open Circle, a weekly dose of my best ideas. I'll also send you 200+ pages from my private notebooks and 24 of my favorite books. Get it here.

<comment><USER18> Yes, but please don't post snarky dismissals anyhow. How often do we get to discuss Chekhov?

<comment><USER39> I've never heard of him and probably won't hear of him again, so I guess somewhere less than once a decade.

</comment></comment></comment><comment><USER40> And all this striving towards Chekhov's idea of cultured allows one to achieve what? I'd venture that for most it would amount to smugly patting themselves on the back for having achieved such a fine state of serenity. If this version of cultured means just another way to feel superior to others, I'm happy to be uncultured.

<comment><USER41> It allows one to achieve a view where they don't judge others in an attempt to speak for them as you have done here. While you can imagine someone patting themselves on the back for being aesthetic, the reality is that is only occurring vividly in your own mind, and not out here. If someone was indeed patting themselves on the back out here for these things, that is a separate problem created of their own judgement. Speaking for other's actions leading to judgement is itself viral irrationality, as doing so only spreads the behavior and irrationality further into the aggregate.

For the nerds here, I present this as a recursive pattern which spreads by being computationally inefficient. Software bloat, maybe?

As an aside, when I talk about a "dissonance" button for social media sites, your comment would be a perfect example of when to push it. That's not to say your _leading question_ and _comment on others future irrational actions_ shouldn't be allowed, but I think that it's important to call out content that is potentially "messy" in nature due to it's own innate ability to spread to others without your intervention (after saying it, of course). Maybe limiting replies to such "comments" would be one way to increase the overall aesthetic.

<comment><USER42> Your comment is barely comprehensible to me. Can you condense this to simple English? What are you even saying in half of your post?

> It allows one to achieve a view where they don't judge others

So you're telling me that being cultured allows one to not judge others, but while clearly having the effect of outwardly being smug to others?

That's basically what I got out of your comment and the parent post.

Your aside also implies that his comment basically shouldn't be allowed despite your explicit comment that it's "not to say."

This seems all rather smug and rude.

<comment><USER41> One side-effect of stating one's internal truth is that others may take it how they like. If you decide to take what I'm saying as smug and rude, then that's your prerogative to speak for me in an internal frame, inside your mind. Out here in the external frame, what appears to be a deterministic reality by all accounts, I'm telling you directly that I am not smug about this and I do not consider my actions rude given I'm not blaming you, personally, in my original comment. (The state of rudeness can only be evaluated by the entity which receives the comment, not someone who speaks for them out of turn.) In fact, it is my intent to help others achieve a state of understanding that allows their own assertion of my claims here. If Boothroid has something to say about me or my comments here, then that is their prerogative. I would encourage them to keep speaking for my internal frame to a minimum, however, and make observations instead of judgement. Only each of us may speak for our own internal frames.

I will note that there is blame in my comments here now, which is being directed toward someone who is clearly speaking for another's thought processes - that would be you speaking for me being smug and Boothrid in my "being rude" to them, when I'm saying otherwise here publicly. This behavior _can_ contribute to a cyclic behavior of increasing blame (from the irrationality spreading) if one decides to ignore what another is saying and continue speaking for their internal frame.

Tending toward the aesthetic encourages letting go of personal judgement because once you stop judging yourself, you stop judging others. There's also a deeper cause for letting go of judgement, which involves understanding what all of _this_ really is. You may think you are are separate entity and definitely have a right to claim separation from the whole, if that's what you want. OTOH, I think you are one part of a larger unidentified, unified entity: a global consciousness frequently mistaken for "God" by religions.

However, me stating that internal thought to you here is direct blame if you don't believe that, given I (would and am) pointing out publicly that you are part of a global entity (which itself is an irrational thought). Again, spreading irrationality by broadcasting irrational logic is not ideal, so I really shouldn't do it. However, I did it here to prove a point. ;)

</comment></comment></comment></comment><comment><USER43> The bit about lying really struck a chord with me. I have been a compulsive liar for most of my life (25 now). Just this past Friday I lied to my skip level manager about the progress of a project.

I've lied about the most mundane things to even lying to my family and friends about graduating college. At this point I feel as if I've been struck by a disease. Worst of all is that I lie to myself the most with respect to how much it is affecting my life, and how dangerous it is for me to continue on this path. I've personally ran out of ideas for how to internally fix this.

<comment><USER30> First, sorry it took so long to respond. The way HN works (if you don't know) is that messages from new accounts sometimes are automatically killed and therefore nobody can respond (or even let you know there's a problem). I actually asked support to unkill this one, and they were all for it.

...

That must be very hard to carry around with you, 24/7/365\. Obviously you don't want to be a liar, but behavior is not so easy as people seem to imagine. They think, 'if you don't want to do it, just choose not to do it'; but they aren't looking at reality, or even in the mirror. If you look around, human beings don't work like that - or we'd all be happily married, well- adjusted millionaires, in happy workplaces, with no drugs, no hate, no crime, no politics, etc. You are not alone in carrying this human burden.

Get some professional help; you are hardly the only person with this problem and many problems like it. A good therapist has heard it all before (and much worse) and can help you; get some recommendations if you can, and try a few to find one you feel personally comfortable with (from what I understand, rapport with the therapist is the most important factor) and otherwise meets your needs. You wouldn't trust your business to an amateur, and this is far more important - find a professional and hire the best person for the job.

Maybe you've reached the point with your problems that I think most people encounter as they mature (if they are honest with themselves). I once heard a songwriter put it his way (paraphrased): 'When I wrote songs at 20 years old, I was the hero in all the stories; I was learning and growing so rapidly that I never considered that I wouldn't outgrow all my problems. But now I'm older and some of these problems haven't gone away; the easy ones are all solved but some others have been very hard to move past. And I find that, in real life, sometimes I'm the villain.' I know I reached that point, too.

I wrote the following in response to someone else and at least a few people found it encouraging; maybe it will be encouraging to you too, but I'm not a professional. Find one! Take care of yourself!

[https://news.ycombinator.com/item?id=13870928](https://news.ycombinator.com/item?id=13870928)</comment></comment>


Ask HN: YC out of touch in China/SE Asia? - <AUTHOR> I am not sure if I am the only one seeing this through but I feel that we are in midst of changing times in terms of the market shift. YC startup playbook is very relevant for valley (meaning a much more mature market such as US or EU). But what we are seeing with China to SE Asis and then India starting to follow defies most of the - how to start a company theory. There are quite a few models - marured to moid and even early stage - from wechat, alipay, grab, mobike, pinduodup and I won't mention a host of new one's emerging in ecom/fintech space. None of these map anywhere near YC theory of how to start a company.
This is not a criticism.
I am just asking if what I am seeing is what non valley founders are realising also - I am speaking from first-hand experience working in China/SE asia for past 3 years and recently Vietnam, where I met many founders.
Finally - what I really hope for is if someone can suggest good reads/books on growing scaling companies in Asian environments. The english literature is sparse on the topic and usuals of sam altman to peter thiel honestly does not fully apply in the global / cross border dynamics. World works somewhat differently for people starting company in Asia now and I kid you not - markets are huge and many huge names will continue to emerge beyond Tencent and Alibaba or Flipkart.
I am just turned off by how much good valley advise is now out of touch on what is happening in asia - circa 2018. <comment><USER2> As a partner at YC, I can say that we are actively working on how YC can help founders in China. One difference I've noticed with Chinese startups is the relative speed at which they move. If you thought SV moved quick...it's nothing compared to Beijing! Re: what SV advice is transferable, we're still very much figuring this out. It's not 100% but I highly doubt it's as limited as I've heard some BJ investors and founders imply :)

Good resources on China and startups: \- 996 Podcast by GGV [https://996.ggvc.com/category/podcast/](https://996.ggvc.com/category/podcast/) \- Sinica Podcast [https://supchina.com/series/sinica/](https://supchina.com/series/sinica/)

Daily/TC style news \- Axios China daily email [http://link.axios.com/join/signup-all](http://link.axios.com/join/signup-all) \- Pandaily [https://pandaily.com/](https://pandaily.com/)

<comment><USER3> I'd be interested in reading a blog article that describes in what way Chinese startups move faster, I feel there is a lot to be learned here.

<comment><USER4> I think it is mostly that competition is insane. Not that competition is low in Silicon Valley, but you are still able to carve out some niche for yourself. In China not so much, or at least it is a lot harder. People will come in and do what you do, what your supplier does and get the same deals as you, on every level of your business.

</comment></comment></comment><comment><USER5> SE Asian here. Some things I've noticed:

* Talent is extremely rare. The ratio is about maybe a tenth as much in good areas vs somewhere like NYC. Put it this way - our interview process for _senior_ engineers involves asking them to build a linked list, and it involves a lot of hints.

* The unicorns of the region are app+logistics, things like Grab, Lazada, ofo.

* Or things that cut through heavy bureaucracy - alipay, iMoney, iPay88, WeChat. Stripe and similar companies have tried but it's hard to pierce the bureaucratic layer.

* Scaling is a very different game. You simply can't hire a hundred of the best people. You can get maybe 2-3 excellent people, and the rest will have to be trained from scratch. This makes moving fast difficult.

* However, living costs are low... founders in SEA except Singapore can live off $10k/year.

So from all these points, it's extremely dangerous to move fast. A lot of local startups overextend into multiple countries then run out of money or can't manage.

The SE Asian game is similar to the Vietnam War. One has to move slowly and carefully. Higher tech may not defeat a well entrenched local, and companies like Uber have bled trying.

I think the startup game in SE Asia and China is moderate tech. You won't have Google or Apple, but you'll have Tencent, LTE, and Huawei. The biggest tech companies in SEA are things like telcos and job listing sites.

Asians are not sharks but bottom feeders - they can survive at lower depths, under more extreme conditions. Ideally startups in SEA would be more like MailChimp, Reddit, Postman, Runcloud. Where it's not a race, but the unit economics matter.

</comment><comment><USER6> Can you zero in on a few specific pieces of YC advice that are most wrong for early-stage companies in Asia? I think it'd be more productive to have a discussion around some specific issues.

YC has much less experience in Asia than the US, and we'd like to learn.

<comment><USER7> YC seems very oriented to Software and other businesses where iteration costs are low. In hardware you have to spend some money to test product market fit. The only way to not lose your shirt is to follow a waterfall approach with more traditional planning. From an external view it seems like the antithesis of the YC approach.

<comment><USER6> YC started with software companies, but a substantial fraction of current companies are building hardware. See [https://blog.ycombinator.com/yc- winter-2018-stats/](https://blog.ycombinator.com/yc-winter-2018-stats/) and the links to company descriptions.

Waterfall means different things to different people. I think the key to success is getting several iterations in front of the customer, and being able to make major changes based on their feedback. That does usually mean some duplication of effort, but that's better than building something complete that nobody actually wants.

Also, there's a new funding deal for biotech companies that provides more money: [https://blog.ycombinator.com/yc-bio/](https://blog.ycombinator.com/yc- bio/).

<comment><USER8> I think the issue for biotech isn't just you need more money -- you do to get initial POC (the most analogous concept to product / market fit), but can actually get to exit with less than tech co's nowadays -- but that you have to spend your initial money right. product / feedback cycles are orders of magnitude longer / more expensive, so you need to do more planning to make sure you do the right experiments. this is the hard lesson that the experienced biotech VCs have learned, that generalist VCs have struggled with and why you dont really see many generalist tech VC backed biotech co's succeed (except stemcentryx)

more money without better planning = more money wasted

</comment></comment></comment></comment><comment><USER9> Cutthroat competition of well-funded second-movers is a whole different ballgame. It's funny that you mention Peter Thiel. His "globalization" pretty much describes this.

I think you'll find relevant advice in more traditional managerial business literature. You want operational excellence above all.

</comment><comment><USER10> What do you feel works in Asia, but isn't done or works elsewhere?

Also what doesn't work in Asia that YC and others do?

</comment><comment><USER11> I would rather China/SEAsia grow their own networks rather than YC come in and impose their way of thinking onto a different culture and market. I get a sense that this notion of American exceptionalism is waning an that people, especially in China, don't look to America for leadership the same way as in the past.

And I think there is a general sentiment that China is well on it's way. I suspect YC needs China more than China needs YC, or at least I think they're headed towards that future. I've noticed a lot of really talented Chinese immigrants and students who have worked in Silicon Valley for several years moving back to China to work for Tencent, Baidu, JD, Alibaba etc.

I'm sure "Trump's America" is also a factor in all this.

</comment><comment><USER12> It's propaganda. They don't have the cultural tools for that (yet).</comment>


Exclusive: Facebook to Offer Data Storage - <AUTHOR> http://www.rev2.org/2007/09/16/facebook-offers-data-storage/

<comment><USER2> TFA is credulous and deceptive.

Can something be an 'exclusive' if it's been on a well-trafficked wiki for over a month?

Also, if they'd looked closely, by 'data store' Facebook mean a loose database of objects (consisting of key-value properties) and typed object associations.

That's very very interesting, but not as a bulk data drop competing with S3, which is how it's spun. I mean, you could fragment and base64 your photos into the 64K text blob properties offered by this 'Data Store', but that's clearly not its design goal.

</comment><comment><USER3> Someone should make a web based RDBMS on top of one of these data drops.</comment>


EPH On Schneier On Chinese Hackers.  - <AUTHOR> http://www.0x000000.com/?i=612

<comment><USER2> Actually, Hacker Webzine on Schneier on Chinese Hackers

<comment><USER3> To be precise: mhb on ajbatac on hacker webzine on Schneier on Chinese Hackers.</comment></comment>


The Problems and Promise of WebAssembly - <AUTHOR> https://googleprojectzero.blogspot.com/2018/08/the-problems-and-promise-of-webassembly.html <comment><USER2> Good analysis as always from Project Zero team. Multithreading and SharedArrayBuffers will be very interesting moving forward.

Future of WebAssembly

[https://blog.scottlogic.com/2018/07/20/wasm- future.html](https://blog.scottlogic.com/2018/07/20/wasm-future.html)</comment>


Facebook Publications - <AUTHOR> https://www.facebook.com/publications

<comment><USER2> Lets not forget the incredible articles in the Tandem and Digital archives. Including the famous 5-minute rule paper.

HP, Digital, Tandem archives [http://www.hpl.hp.com/hpjournal/journal.html](http://www.hpl.hp.com/hpjournal/journal.html)

5-minute rule paper [http://www.hpl.hp.com/techreports/tandem/TR-86.1.pdf](http://www.hpl.hp.com/techreports/tandem/TR-86.1.pdf)

</comment><comment><USER3> I would love to see more companies opening up technical research documents (regardless of how formal they are).

As academia becomes more stagnated it makes sense for companies to open up and spread knowledge to drive their particular ecosystem forward.

Not to mention it's probably an excellent recruiting tool.

<comment><USER4> A lot of the big tech companies do this. Yahoo and Microsoft would be particularly active in terms of publications. Google are more secretive but do publish. IBM obviously publish a lot as well. Others that come to mind Ebay , Linkedin, HP / Compaq (Weh), various big online chinese companies, and so on.

[http://labs.yahoo.com](http://labs.yahoo.com) [http://research.microsoft.com](http://research.microsoft.com) [http://research.google.com](http://research.google.com)

<comment><USER5> Microsoft research is especially amazing. Not only do they publish papers on their systems like Google and Facebook, they also do a lot of basic science on a very diverse set of topics.

The research talks are also worth checking out: [http://research.microsoft.com/apps/catalog/default.aspx?t=vi...](http://research.microsoft.com/apps/catalog/default.aspx?t=videos)

The latest talks range from "Culture differences between US and China" to "Distributed Optimization via Alternating Direction Method of Multipliers".

<comment><USER6> [http://research.microsoft.com/pubs/167719/whyfromnigeria.pdf](http://research.microsoft.com/pubs/167719/whyfromnigeria.pdf)

</comment></comment></comment></comment><comment><USER7> Pixar also publishes some of their investigations - [http://www.pixar.com/companyinfo/research/deb/](http://www.pixar.com/companyinfo/research/deb/)

</comment><comment><USER8> Hate to bring Twitter up here. But this makes one even more puzzled about where Twitter's R&D costs of $90 million goes! Sure there are open source stuff they put up on [http://twitter.github.io/](http://twitter.github.io/) and there's bootstratp. But none of it is the sort of cutting edge research that the likes of LinkedIN and FB seem to be doing. Looking up posts on their blog with "research" tag also returns dismal results.

<comment><USER9> There's a tax credit for claiming your work is R&D:

[http://en.wikipedia.org/wiki/Research_&_Experimentation_Tax_...](http://en.wikipedia.org/wiki/Research_&_Experimentation_Tax_Credit)

This creates an incentive for companies to find ways to report their work as R&D. (Not implying that Twitter is defrauding or that their work isn't R&D, just stating why it might be useful for a company to "invest in" R&D, especially if that is work they wanted done anyway.)

</comment></comment><comment><USER10> I was excited at first because I thought Facebook was building a social network for researchers to speed up the scientific process/peer review. Then I visited the site. Oh well.

<comment><USER11> I'm building a social writing site for researchers and scientists using Markdown syntax and LaTeX at markdawn.com — someone might be interested if they see this news.

</comment></comment><comment><USER12> I think research in big companies is an interesting activity, both for the companies themselves and for the research world that can receive input from "real" world problems.

I was in research before (IT security) and I often found interesting results from companies such as IBM, Intel, Microsoft and Google.

What I think is kind of strange here, is that there's not a section on security/privacy in the page... well, but perhaps I'm biased ;)

</comment><comment><USER13> This looks like an intelligent initiative from Facebook and proves it is well on its way towards transparency and would love to see a lot more of papers/publications coming up about their tech. This would largely benefit the academia.

In my view, I love Microsoft Research.

</comment><comment><USER14> [https://www.facebook.com/publications/593599364008099](https://www.facebook.com/publications/593599364008099)

Dealing with how families interact on Facebook looks to be particularly interesting.

</comment><comment><USER15> I'm surprised by how relatively little machine learning research they have. Microsoft, Google, IBM and Yahoo seem much better represented at the core ML conferences like ICML and NIPS.

</comment><comment><USER16> For a second there, I thought Facebook was going to take on the academic publishers.

</comment><comment><USER17> Can someone download them and put nicely on one page?

</comment><comment><USER18> Not what I hoped, but at least you don't need to log in.</comment>


Show HN: F-dotfiles, GNU stow with healthy organization guidelines - <AUTHOR> https://github.com/Kraymer/F-dotfiles <comment><AUTHOR> I decided to have a dotfiles repo a few months ago. Started with a @holman fork but was rebuted by the bootstrap/install scripts limitations.

Using stow resolved that.

Then I stumbled upon questions that did not have obvious gold standard way to do it :

\- how do you differentiate folders meant to be stowed from others?

\- how do you store config files saved at different locations depending on platform without duplicating content?

\- how to deal with files with sensitive information?

So I used some guidelines to deal with theses cases, the result is F-dotfiles.</comment>


Please, Let Me Pay - <AUTHOR> http://log.maniacalrage.net/post/3948497854/please-let-me-pay

<comment><AUTHOR> "Most users of free services are douchebags."

It's a blunt way to put it but my experience has shown this to be generally true.

</comment><comment><USER2> "I'm of the opinion there's no reason to give away a product for free if you want it to succeed."

That depends on how you define success. Not everyone measures everything in monetary terms.</comment>


Chongqing, the world's most surveilled city, 168 cameras per 1000 people - <AUTHOR> https://www.scmp.com/tech/policy/article/3031390/chongqing-worlds-most-surveilled-city-these-residents-are-happy-trade <comment><USER2> Chongqing is a very interesting city to visit, very hilly. Great food as well.</comment>