Help Make the 19th Century Searchable - <AUTHOR> https://blog.archive.org/2020/08/21/can-you-help-us-make-the-19th-century-searchable/ <comment><USER2> Commercially available OCR is amazingly bad. The errors it makes are crazy. For example, if the top of a 'd' is missing it might read it as an 'a', while a human can see that it's identical to all the other 'd's on the page except that a bit is missing, while it looks nothing at all like an 'a' _in any of the fonts used in that book_. Or the OCR sees a random blotch and guesses it might be a comma, although it's in completely the wrong place and has the wrong size to be part of the text.

I think perhaps the developers took a wrong turn when they started trying to improve OCR with language models rather than font models. Humans can accurately transcribe a _printed_ text without knowing the language. They do that with a mental model of font metrics and so on. In fact, a human transcribes more accurately (though more slowly) when they don't know the language because they don't erroneously "autocorrect".

I wonder if Google has in-house OCR that works much better than commercially available OCR. Google has OCR-ed at least 25 million books. You can't download the complete texts, but you can see snippets. Perhaps someone would like to publish a paper assessing the quality of Google's OCR and comparing it with commercial software. (Probably someone has done that already; I'm just bad at finding papers.)

<comment><USER3> Have you evaluated the latest version of Tesseract (developed by Google)? They added an LSTM-based OCR engine starting with version 4 and the improvements over version 3.x are startling.

<comment><USER4> Also, ocrmypdf [0] is a great Tesseract-based tool which makes it easy to add ocr layers to raster pdfs. Plus, it takes care of optimizing the resulting file (compression, etc). I used it on several old academic papers and have been pleased with the results.

[0]: [https://github.com/jbarlow83/OCRmyPDF](https://github.com/jbarlow83/OCRmyPDF)

</comment></comment></comment><comment><USER5> We recently analysed a dataset from an 1859 geomagnetic observatory in Rome [1].

We found the document (the observatory yearbook written in 1859) via Google Books. As I don't speak Italian, I initially typed out passages into Google Translate in order to find the information I needed. Google Books has a view text option, but the format of the page and font often made it garbled (when pushed through translate in any case). Decent OCR likely would have made my life a lot easier.

There is a recent trend in space weather research to study extreme geomagnetic storms that happened in the 19th and early 20th century, and is aided partly by all of the scanned documents from that era available on the likes of Google Books, The Internet Archive and HaithiTrust. Better OCR would be a great help.

Although even having access to all of the documents is already incredible!

[1] [https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/201...](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019JA027336)

</comment><comment><USER6> Some months back an article was posted here about the Library of Congress's newspaper digitization technology:

[https://arxiv.org/abs/2005.01583](https://arxiv.org/abs/2005.01583)

[https://news-navigator.labs.loc.gov/](https://news-navigator.labs.loc.gov/)

What they have done already is great:

[https://chroniclingamerica.loc.gov/newspapers/](https://chroniclingamerica.loc.gov/newspapers/)

I've used it quite a bit and they have some of the best OCR results I've ever encountered when it comes to scanned newspapers. But it's just the tip if the iceberg. Historical newspapers seems to be one of the largest corpuses of material yet to be digitized.

<comment><USER7> Thank you so much for the chroniclingamerica link. That is seriously cool. I heard my personal "Dangerous Rabbit Hole" alert when I clicked on it, but what the heck, it's Saturday. I'm diving in and definitely won't be out before dinner.

</comment></comment><comment><USER8> Sounds like a great project. There are many remarkable documents from that century ... explorers in many realms, science pioneers making great strides.

Re their statement: "What we do not have is a good way to integrate work on these projects with the Internet Archive's processing flow. So we need help and ideas there as well."

... maybe there are still people from the Gutenberg project around, they used to be handling human-transcribed stuff at quite a volume. (Personally I'd rather type stuff in than 'repairing' bad OCR, that constant back-and-forth is just aggravating. I'd be glad to dig in on something personally interesting, like say a little-remembered expedition or research in an area of interest ... they could arrange stuff by subject that way and then put up wish-lists for volunteers. History is full of forgotten amazings.)

</comment><comment><USER9> I would start by either improving the OCR, or piggyback on GCV/Amzn, who have better tools. For instance, what if I drag-and-drop their sample image to this demo page [https://cloud.google.com/vision](https://cloud.google.com/vision) ?

All text is recognized, and as far as I can tell, there are no errors.

There are a lot of difficulties with older text, but I would start with low- hanging fruit such as trying to use better tools. On top of that, you have the problem of making sense of the layout, fixing common typos, etc.

<comment><USER10> I tried the GCV demo as well and got one mistake "prevent:" instead of "prevents" as it is in the text. But that's the only I could find, which puts it several categories ahead of the txt file.

I'm not sure though whether the IA can get Google to OCR it for them for a budget they can afford. Likely they'd want OCR solutions that have a one-time cost, so volume based SAAS offerings won't work.

</comment></comment><comment><USER11> Why not just do a SETI@HOME style thing where you get a slice of a newspaper article, a human being types it up, each is compared to one or two other people typing the same thing up, when they all match that piece is done and you move on.

I guess that would take a tremendous amount of time, but it would be really cool to get an old newspaper article to look at, and I bet a lot of people would find interesting things to talk about.

You could have a "comment thread" for people who added the piece, kind of like adding a digital layer to history.

<comment><USER12> That sounds exactly like the Distributed Proofreaders project, which is part of Project Gutenberg:

[https://www.pgdp.net/c/](https://www.pgdp.net/c/)

<comment><USER11> ...I love HN. Thank you for pointing me to this.

</comment></comment></comment><comment><USER13> Valiant effort! Could you potentially post this on [https://scistarter.org/](https://scistarter.org/)?

</comment><comment><USER14> While the effort is noble, I think it should be applied to all mankind written documents, not only 19th century.

<comment><USER15> Don't let the perfect be the enemy of the good. XIX english is pretty close to XXI english, XIV not so much[1]. I make frequent use of [https://gallica.bnf.fr/](https://gallica.bnf.fr/) but the older the document the more unfamiliar the dialect. In german the older scripts (let alone XX forms such as Sütterlin!) are likely to be even less OCR/search friendly.

[1] and good luck with Beowulf: [http://www.bl.uk/manuscripts/Viewer.aspx?ref=cotton_ms_vitel...](http://www.bl.uk/manuscripts/Viewer.aspx?ref=cotton_ms_vitellius_a_xv_f132r)

<comment><USER14> Just like middle age Portuguese isn't like modern Portuguese, and we also have stuff like Linear B and whatever else exists since there are written documents.

</comment></comment></comment><comment><USER16> I love this!

Related to the overall culture tech, most of the scientific revolution occurred in NeoLatin which isn't taught, mostly remains untranslated and untranscribed. For instance: Descartes first book, which dealt with music theory and human emotion.

</comment><comment><USER17> I can read the text in the snippet without issue, but I clicked through to some of the full images in the web viewer and couldn't read the text at all. I tried downloading a few PDFs and still couldn't read them.

Not sure if I'm just not getting the full quality somehow or if the image quality just isn't there for OCR to ever work?

<comment><USER18> Text is readable for me in the viewer. Did you use the magnifying glass icons in the bottom right to zoom in?

<comment><USER17> Ah. I used pinch to zoom on my mobile - it didn't have the magnifying glass icons. That works, thanks.

</comment></comment></comment><comment><USER19> I remember the first time I visited a website in 1856.

</comment><comment><USER20> Can someone help me understand the greater vision of this project?

I do like the idea and efforts from a technical point of view. Tinkering with OCR on unusual (or old) languages. But that's not the goal of this project as far as I'm concerned (it's a byproduct?) Archiving every single news entry for the sake of completion sounds more like obsession than purpose. We're creating so much information that it will be even harder to separate garbage from valuable information (you have to spend time reading the useless stuff before you can justify whether or not it's valuable to you).

Information overload IS a problem and by adding more information to an already saturated ecosystem I don't see the vision here but would like to understand :)

It almost seems like a hording problem but for the digital natives. People accumulate a lot of stuff but rarely can they actually appreciate what they possess as time & perception is a very limiting factor.

An article that would shed some light would be highly appreciated.

<comment><USER21> They're not generating new information though, they're aiming to make old information available. I'm sure that it's quite helpful for historians to be able to search & read old newspapers, it gives more details about what people read about and often establishes a more specific time line.

We're probably not that good at recognizing which bits will be of interest to future generations, so archiving everything (to a point ... but I believe that newspapers are well within reasonable) sounds like a good idea. Plus you never know what you discover when you make things available.

<comment><USER22> Not to mention that from 1800 to 1930 the world population doubled (1 billion to 2 billion, passing the 1.5 mark just around 1900).

While this pales in comparison with the 6 billion humans added since, that's a significant change - particularly for most likely available recorded sources - at a time of monstrous evolutions to major world powers/empires, expansion into vast new areas of the world (namely North America) of essentially the British empire (while at the same time the East India Company ceased to exist by the end of the century for contrast), some abolition of slavery becoming a reality in places (1833 for the British), and what arguably kickstarted much of the mental frameworks for our entire lives: the first two industrial revolutions (for example: democratization of once-monastic school system while adopting the year-of-production type of mental model for its promotions).

1804 is the first locomotive. 1859 is The Origin of Species by Darwin. 1861 is Maxwell equations. 1869 is Mendeleev's period table. And so on and so forth[0]. Measurement devices also improve in reliability and efficiency, leading to many of the early recordings we can now look back at when it comes to the consequences of the explosion of human activity with regards to the environment.

It's quite a fantastic century to keep a trace of, frankly.

[0] [https://en.wikipedia.org/wiki/19th_century#Science_and_techn...](https://en.wikipedia.org/wiki/19th_century#Science_and_technology)</comment></comment></comment>


US breakup: Myth or reality? - <AUTHOR> http://rt.com/politics/panarin-usa-collapse-economy-905/

<comment><USER2> Ya know, I'm usually a fan of alternative history and "what if" kinds of futurism, but these people can't be serious. A breakup of the United States, let alone one that sees the US divvied up between Canada, China, the European Union __and Mexico isn't even remotely possible in the next year or even in the next decade. Even with a 50+ year time horizon you have to be pretty imaginative to come up with a plausible course to a scenario remotely like this--and that's still ridiculously unlikely.

 __\- Also, what does "European Union" even mean here? The EU is composed of individual member states. Is the former US Northeast ruled directly and collectively from Brussels in this scenario? Is that an independent nation that has joined the EU?

The author, and the "analysts" he links to are either trolls or fools. Possibly both.</comment>


New Project Intend to Simplify Web App Creation - <AUTHOR> http://www.workinmind.com/

<comment><USER2> I actually just had a go and this could be really good for fast prototyping and communicating ideas with designers. A bit like Balsamiq but with more emphasis on the functionality rather than the visuals.

</comment><comment><AUTHOR> It could be used as prototyping tool for now. As long as it has the ability to visually define server side behavior, it could be able to create real web app :-)</comment>


BPEL4People - <AUTHOR> http://www.oasis-open.org/news/oasis-news-2008-02-14.php <comment><AUTHOR> Saw this on twitter. While I can't deny that "BPEL4People" has a poetic ring to it, I thought the 3rd paragraph could be right out of The Onion.</comment>


Tapes.fm [beta] - A Music Collaboration Service - <AUTHOR> http://tapes.fm/

<comment><USER2> nice, webish version of jampipe, but i'm more interested in getting bits vs building in browser

<comment><AUTHOR> I haven't gone too far into it yet, but the workflow seems to be select a tape for remixing -> download track(s) -> add/modify track(s) -> upload and update the tape to match what you had in your own multitrack.

So you do get the bits, putting it together in the browser is essentially so others can listen to it and do remixes of their own.

<comment><USER3> Yes this is exactly how it works.

tapes.fm wants to make the music making process more "social" and "effective" and not change the way how you actually make music.</comment></comment></comment>


Google Retools Its Flu Prediction Engine After Getting It Wrong - <AUTHOR> http://blogs.wsj.com/digits/2014/10/31/google-retools-its-flu-prediction-engine-after-getting-it-wrong/

<comment><USER2> I had a chance to talk to a guy who worked in the pharmaceutical industry and it's amazing how much effort is put into predicting the flu season. Hundreds of millions of dollars are on the line not just predicting how many folks will get the flu, but also what strains will be the issue.

<comment><USER3> Trying to predict the strains makes perfect sense, the flu shot is only ever effective against the specific spread of strains that it's created against. I presume there's some fall off of effectiveness or large increase in cost associated with creating a flu shot that targets more strains.

</comment></comment><comment><USER4> I find it interesting that when the Google predictions match the CDC estimates poorly, the conclusion must be that Google got it wrong. But the CDC estimates are based on patient data that has been filtered through doctors' judgements, and those doctors only got to see the people that made use of health-care services. I think it is a quite reasonable possibility that Google's predictions were on the mark, and the CDC estimates were where the problem was.

Of course, this is a science article for the general population. I imagine there is significant analysis that were not being told about. Perhaps there are very good reasons to believe that it is Google, and not the CDC, whose estimates fail to match reality.

Still, I have to wonder.</comment>


Why Did It Take So Long to Shut Down Silk Road? - <AUTHOR> https://medium.com/editors-picks/86d97b0dc928

<comment><USER2> The article talks a ton about the utility and reasons of popularity behind Silk Road - however with the exception of saying "..uh.. Tor slash encryption" never actually answers "Why Did It Take So Long to Shut Down Silk Road?"</comment>


Sinkholed – A DNS Horror Story: How I Lost and Regained My .IN Domain Name - <AUTHOR> https://susam.in/blog/sinkholed/?hn <comment><USER2> didn't you submit this earlier this morning already?

<comment><AUTHOR> Yes, I did. However, I had screwed up the title then and I was unable to edit it, so reposted it with the fixed title. Sorry about the spam.</comment></comment>


Google's Javascript PDF Viewer is Embeddable - <AUTHOR> http://googlesystem.blogspot.com/2009/09/embeddable-google-document-viewer.html

<comment><USER2> For HN as an alternative to scribd's vaccuum maybe?

<comment><USER3> scrolling/zooming even works! die flash, just die.

<comment><USER4> Text copy wont work.

I really dont see anything new here. This is exactly the same stuff they have at Google books. Except now its in an iframe .

Flash implementations are obviously superior as of now.Also, try building a decent browser game in javascript and html5.

<comment><USER5> Actually games that don't involve too much motion tweening (think sudoku, tic tac toe etc.) can be done pretty well with js and html5.

<comment><USER4> any live examples?

<comment><USER6> This is old but this version of arakanoid was always pretty impressive.

<http://www.schillmania.com/arkanoid/>

Note it does cheat and use flash for SFX

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> It also works as its own page: <http://docs.google.com/gview?url=>

Does anyone know if firefox extensions can register as handlers for MIME types, or if they're restricted to diddling with chrome / pre-existing pages? It'd be really easy to write a WebKit plugin to do it, but not very useful since Safari ships with a flawless PDFKit-based viewer.

I want to get this working in Firefox, but NPAPI plugins are really gross. It looks doable that way, as it appears that a naked nsplugin still gets a DOM and everything, so I could just immediately set window.location.href and be done with it.

<comment><USER7> It sounds like you're looking for window.navigator.registerContentHandler:

        https://developer.mozilla.org/en/DOM/window.navigator.registerContentHandler

...except a website can only register themselves as a content-handler. You could probably hack up a bookmarklet to do it, if you can't find an XSS vector, but I can't seem to make it work myself.

<comment><AUTHOR> So... close...

It'd be easy to make a greasemonkey-style extension that calls this on *, but it wouldn't work for links to other domains (which I'd bet are the majority of PDF links)

<comment><USER8> That would already be very useful though, for things like online papers and datasheets, which as a rule sit on the same server as the page you're looking at.

<comment><USER9> It's becoming more common to use a separate domain for static content like this to be cached more effectively (and to have no cookies etc sent), so it may not always be the same server as the page you're looking at.

</comment></comment></comment></comment></comment><comment><USER10> It's good to know that there are alternative ways to embed PDFs, but I'm a little confused by the reaction in the comments. Is this a big improvement over simply using the object tag?

<http://blog.flashcolony.com/?p=244>

<comment><AUTHOR> Yes, for a wide variety of reasons:

      * Before version 8, Adobe's plugin was an epic piece of shit on all platforms
         - old versions are on hundreds of millions of computers!
         - it's still really crashy on Linux (especially in nspluginwrapper)
         - The only OS X browser they support is Safari, in which it is unnecessary
      * Google re-hosts the file for you
      * It is ridiculously snappy with no main-thread-blocking initialization
      * It will never crash your browser
      * Your users don't have an ingrained hatred towards it
          - Rebranding the free Acrobat as "Adobe Reader" was the stupidest thing
            they've ever done (ever hear a user say 'my Adobe is broken'?)

<comment><USER11> I can't tell if you're being facetious in your last line, but I used to work at a place requiring, in their own words, "Adobe must be installed on every computer" long before they changed the Acrobat name.

</comment></comment></comment><comment><USER12> It's images! Is that really better?

It's not text layed (laid?) out in a cool way with HTML.

It's a png image of each page of the pdf. It's quick sure, but images? Not text? It just seems wrong - why even make pdf's, if all you want is an image of the text?

<comment><USER13> Generating an image from the PDF server-side is the only way to faithfully perserve the layout of the PDF.

For some PDFs, Google will give you the option to view it as HTML (it seems to just be the PDFs returned in search results?). It's usable for mostly-text documents, but that's about it.

</comment></comment><comment><USER14> I hope somebody makes a Firefox addon which lets me right-click on a PDF link and select "Preview" using the Google previewer.

<comment><USER14> There's this, but it sucks:

<https://addons.mozilla.org/en-US/firefox/addon/10737>

You click a link and it first does some communicatin with Google Docs to get some kind of security token, so it takes 2-3 seconds for the new tab to come up.

EDIT:

A kickass version, just uploaded it to the Firefox Addons site:

<https://addons.mozilla.org/en-US/firefox/addon/14226/>

<comment><AUTHOR> I think that addon is worse than you describe: doesn't it add the files to your logged-in Google Docs account, and do so by uploading said files from localhost?

<comment><USER14> You're right, I just checked my Google Docs and it added the files. The hacked version just opens the URL and passes in the PDF url as a HTTP param.

<comment><USER15> You mean the second link in your granparent post, right? Thanks for that -- works for me. I was looking for something like this just the other day.

</comment></comment></comment></comment></comment><comment><USER16> This is very cool. My only gripe is that if they are going to fake their own selection, then they should override command-a (select all). I tried this, and now I have google's selection AND the entire page selection, which I can't get rid of.

</comment><comment><USER17> I hope that it will be extended to TeX, PostScript and ODF as well.

The viewers for the first two formats are lagging in the UI department, it would be really nice to be able to access them online.

</comment><comment><USER18> I presume one can't use this viewer in your own project (e.g. as a file viewer in a web app).

Does anybody know of similar project but that can be used in your own web app?

<comment><USER19> Quick Googling suggests that <http://vuzit.com/> would work.

You might also be able to hack Thindoc in such a way that it worked for your purpose.

<comment><USER20> Don't know why you're downmodded.

I use <http://www.ajaxdocumentviewer.com/> on my own personal site. It works the same way as Google (but has ads), and handles .doc files.

I'd much rather use the Google app if it could handle .docs :(

</comment></comment></comment><comment><USER21> I wish their Chat Window on Google Spreadsheet were embeddable , too.

</comment><comment><USER22> Scribd must love this

</comment><comment><USER23> Do you think the same thing exists for Word documents?

</comment><comment><USER24> Can anyone confirm if this works in Internet Explorer?

<comment><USER25> It does. Even current versions of IE can show PNGs in iframes, believe it or not ;-)

</comment></comment><comment><USER26> Very smooth, but why does it lack a search box?

<comment><USER27> The full page version has a (pretty good) search function. I'm not sure why they don't have it the embedded one. Even if you just got taken to the results in the full version.

<comment><USER26> ACtually the full page version highlights the results but doesn't navigate to them. Also it seems to be substring search, not an indexed one.

</comment></comment></comment><comment><USER28> Here is a bookmarklet I just wrote to look at all the links on a page and open any ending in ".pdf" in google's viewer. (if you have firefox set to open tabs for windows, it will open them in new tabs) Feel free to share and improve.

\- - - (select entire text and make into a link) \- - -

      javascript:(function(){dl=document.links;for(i=0;i<dl.length;++i){if(dl[i].href.substr(-4)==".pdf"){window.open("http://docs.google.com/gview?embedded=true&url="+dl[i].href);}}})()

\- - -

<comment><USER5> I made this into a greasemonkey userscript. It searches for PDF links on a page and sets them to open in Google's pdf viewer.

Userscript at: <http://userscripts.org/scripts/show/57453>

Tested on Minefield latest nightly, winxp, greasemonkey.

<comment><USER29> Thanks! It needs to not work on <http://docs.google.com> so that you can click the "Download" link if you need to...otherwise you are stuck in a loop. Also, I noticed that the search on the full-screen page isn't always available (ie <http://bit.ly/4tf1xt>)

<comment><USER5> About the search not being available, I think it's something to do with google's implementation, not the script. The other, I'll fix ASAP.

Edit: Fixed version available now.</comment></comment></comment></comment>


Technology Preview: Signal Private Group System - <AUTHOR> https://signal.org/blog/signal-private-group-system/ <comment><USER2> Again, in the theme of "features every group messaging system had already, but Signal didn't, because they hadn't figured out a way to implement it without turning Signal's central servers into a database of who's talking to who about what". Signal didn't even have _user profiles_ until recently, for the same reason. Here, they've slightly expanded the state of the art in MAC-based anonymous credentials to accomplish their goal.

One interesting aspect of this is that Signal gets to do this, because they have immense goodwill with the cryptographic research and engineering communities; though it's no guarantee of soundness, they have the advantage of having the feature designed, implemented, and ultimately reviewed by cryptography engineers that aren't generally/economically available to other messaging projects.

This is either a reason you _love_ Signal (raises hand) or _can 't stand_ Signal. My take is, if you're in the latter group, that's fine; I use Slack, too.

<comment><USER3> Honestly, the one and only feature I'm missing in Signal that would let me use it and recommend it to everyone without reservations (rather than exclusively for ephemeral-only communication) is the ability to keep identity and full message history when moving to a new device.

Today, on iOS, you _can 't_ move your Signal history to a new device, and on Android you can only do so by manually making an encrypted backup file and writing down a 30-digit passcode, completely separate from the normal Android process of moving to a new device.

People keep long histories of messages, going back a decade, containing pictures and memories that aren't stored anywhere else. Message history is valuable data.

This doesn't seem like a "new cryptographic research" problem, this seems like a "well-established crypto (encrypted files) plus integration with standard device backup/migration" problem.

I really like Signal, I think they're doing things very well, and I wish I could use it without being constantly at risk of data loss. And this doesn't seem like an uncommon request, from what I've found.

Is there something I'm missing that makes this a _hard_ problem? Or is it just a problem that nobody has prioritized?

<comment><USER4> I actually going in the other direction with Signal

I turned on timer (1 week) for all of my conversation.

Nothing stays more than a week and I do not keep any backup.

It's not for security or privacy reasons. I feel like I don't need a full history of all my conversations with everyone from the beginning of time.

This fits more to the real life model of having a conversation with someone. I don't record my conversations with people so why do I need to do it in chat apps?

My Whatsapp is the same. Don't need all the massive amount of chat history...

<comment><USER3> Interesting that that model works for you, but that doesn't mean it works for everyone.

Persistent history that lasts many times longer than the lifetime of any one device is a required feature to fully replace chat apps that have such history.

<comment><USER5> Whatsapp doesn't have that feature when you switch between iOS and Android however.

</comment></comment></comment></comment></comment><comment><USER6> Very nice (seriously!). Now, please let people use the platform without needing a valid phone number. The one major issue I have is that. Phone numbers are the new SSN, just like SSN is being misused by traditional businesses, phone numbers are also misused thse days (due to how you generally can be tracked down to a physical area for antifraud and how "everyone" has a cell phone) to uniquely identify users.

I don't get why users can't be addressed by both phone numbers and a "signal id", if you opt-in to use a phone number for addressing, your phone will be verified and signal will resolve it to your signal id. If you opt out people will need your signal id to address you and you can't use it for SMS. What are the challenges with that?

If I have a signal private group system, signal can find out a ton about me and my associations with others using only that information. Many other messaging platforms do not nees this very sensitive information from me to function. And it does not support a desktop only app even if you give them a phone number and verify you control that number.

I am always reminded of General Hayden (Former NSA chief) was saying how they love PGP at the NSA because they can sniff metadata and know who talks to who, it lets them easily find who has something to hide so they can target them. Not that I have the NSA in my threat model but I am very sensitive to unnecessary metadata being generated

<comment><USER7> What you're asking for is exactly how Telegram works, you can add someone with a phone number or by username, but if you add someone via username they don't see your phone number. Of course, Telegram chats are not encrypted by default, and there is some controversy over the encryption protocol.

[https://telegram.org/faq#q-if-someone-finds-me-by- username-m...](https://telegram.org/faq#q-if-someone-finds-me-by-username- messages-and-i-reply-will-they)

<comment><USER8> Last time I checked, the only option to login was using a phone number. And at least the web client only has the phone number as login. I do not want to give them my phone number. Full stop. They can tie my account to my email, to my domain, to a chosen username, whatever. But if your service requires a phone number to use it, it's not something I will use.

<comment><USER7> It's still true. Telegram still uses your phone number to login, even if you never give your phone number to anybody. At least usernames are an option unlike Signal and WhatsApp.

I dislike it too, but understand the reasoning behind spam prevention and account authentication.

</comment></comment></comment></comment><comment><USER9> I really want Signal to succeed. Or rather, I want anything that has decent cryto and is not FAANG to succeed.

The problem is not which messaging app I want to use, it's which messaging app my friends are using.

That said, if I had to choose, I think Matrix has a slight edge in my books because it's a protocol rather than a silo. Even though Signal is private and open source, they are hostile towards people running their own Signal builds on company servers, and unwilling to federate with other servers.

Essentially, you run the official Signal app on the official Signal servers, or GTFO.

<comment><USER2> Matrix and Signal aren't comparable from a security perspective. Because Matrix is a protocol rather than a silo, many (most?) of its implementations don't even support E2E, and because Matrix has its roots in an ecosystem where E2E was a nonstandard add-on, Matrix will never be as safe as Wire or Signal.

<comment><USER10> Matrix project lead here; fwiw we're aiming to turn on E2E by default for private rooms by end of Jan. It's not really a non-standard add-on; it's in the core of the protocol and has been designed for from the outset. It's a pain in the ass to get right in a decentralised world though, hence the delay in forcing it on for everyone.

p.s. support for ephemeral msgs was released on the server in RC yesterday.

<comment><USER2> The way these conversations are structured, I'm always going to come across like I'm rooting for Matrix to fail, which is not at all the case. Like I said, I use Slack more than any other group messaging system, and while Slack does have some security assets that Matrix lacks, nobody can say that it has a more coherent encryption story. I wish Matrix all the best; I just don't think it's reasonable to suggest it as an alternative for people who need secure messaging that reliably works in groups of people.

</comment></comment></comment></comment><comment><USER11> I really hope Signal takes over the world from Whatsapp. I hate Whatsapp and yet I am forced to use it due to all my friends / family / parents use of it. I try to fight but is hard and currently FB mess and WA are the only ones with a consistent reliability of delivering notifications promptly, while rest of chat apps either are too hard to use for non-computer people or they lose notifications. I mean, c'mon Microsoft!!! Is it really that hard to make Skype reliable again?!!

</comment><comment><USER12> What I really want from Signal is the ability to use it as an application transport. In particular, I want to authorise certain people to request my phone's location.

At the moment I share it with Google so I can share it with friends or family, which sucks.

<comment><USER13> Maybe ask the writers of Mr. Robot. They figured it out!

<comment><USER11> Please. I hope that's sarcasm. Because if it's not, then may I steer you clear of that annoying series and into Breaking Bad + its follow up Better Call Saul?

<comment><USER13> No, I'm being totally serious.../s

</comment></comment></comment></comment><comment><USER14> The point of this seems to be hiding the identity of users when they fetch or modify group attributes - but why? The user's identity is otherwise known to the server: they know that account X was accessed by IP address Y, and that IP address Y fetched metadata for group Z, therefore account X is in group Z. They can also figure out group membership by tracking clusters of messages that are sent simultaneously. What am I missing?

<comment><USER15> They do not know that information unless they're monitoring on-the-server in the clear. They simply know they're talking to Signal servers. This new feature protects in the case of Information Requests because they themselves don't know who is in what groups.

<comment><USER14> You either have to trust the server operator not to keep logs, or assume they are keeping logs and _do_ know what groups you're in. In either case, what is this fancy crypto scheme actually buying you?

<comment><USER2> The fancy crypto is what allows them not to (effectively) keep those logs in the first place. If you build this feature without the fancy crypto, then even if you say you're not logging this stuff, you (effectively) are, because your system depends on durable access to that metadata in order to function. This is, for instance, the major security difference between Signal and Wire.

One strong indication you have that Signal isn't logging this stuff is that they had to wait until they were able to advance the state of the art in anonymous credentials in order to implement group access control at all.

<comment><USER14> To verify the claim that my group message content is protected, I can examine the Signal client. I don't have to trust the server operator whatsoever; I can independently confirm there is no way for them to see the content of my messages.

In contrast, I cannot verify this new claim that my group memberships are protected. I have to trust them.

I think you are basically saying: 'well, they built all this crypto that is only useful if you believe they're not logging, so I believe they're not logging.'

<comment><USER2> No, what I'm saying is that without the cryptographic protections, a messaging provider _can 't_ claim not to be logging, because their serverside logic requires the log.

Prior to doing this cryptographic work, Signal simply went without having these features at all.

<comment><USER16> You're both right. The other poster's claim is that we cannot verify the code running on the server. They could support this new scheme AND just store every thing in plain text. We can only verify the interface is the same (because that's what we're using).

But you're also right it would be a long con to go without these features for so long, develop state of the art cryptography to add them securely and privately, then not use that.

<comment><USER17> > They could support this new scheme AND just store every thing in plain text.

Could they?

I am not clear that this is possible. I thought the entire point of "Alice provides a zero-knowledge proof to the server that she possesses an AuthCredential matching some particular entry" is that the server learns nothing about Alice other than her possession of a matching AuthCredential. Indeed, the paper says: "Because of the zero-knowledge property, the server has assurance that the user possesses such an auth credential without learning the UID certified by the credential, or other information that might link this use of the credential to other uses or to credential issuance."

It would be nice if someone more knowledgeable could confirm whether it is indeed possible for Signal to compromise user privacy while using this scheme. Is SheinhardtWigCo right when they write, "In contrast, I cannot verify this new claim that my group memberships are protected. I have to trust them."?

<comment><USER18> SheinhardtWigCo's central idea is that if someone receive a packet over the network it has an IP address in it, and that's the sender's "identity" and so the Signal servers can't avoid knowing Alice's "identity" when she does this, and then they can collect such data to try to re-assemble group membership in terms of IP address "identities".

For example let's say a packet arrives from 10.20.30.40 [[ all IPs used are from 10/8 as examples I am aware that Signal probably rejects packets claiming to be from an RFC1918 network ]] which contains proof that group #1 member #4 has authorised adding a new member #8

SheinhardtWigCo believes this tells us that this identity (10.20.30.40) is a member of this group, group #1 and they suppose that Signal's server could in fact store this, and then perhaps later tell some Spooks a list of such members of group #1 and it could do this on a vast scale, so that it would be able to say for any "identity" (IP address) the list of all identities (IP addresses) which seem to be members of groups which that identity is also a member of.

Now, I don't think Spooks would find that very useful, but there you go, that's what SheinhardtWigCo thinks is a big problem here.

[ Edited to clarify early paragraph ]

<comment><USER17> I see, thanks. I was confusing "logging" for logging the association between AuthCredential and UID rather than logging IP addresses. For what it's worth, Signal does allow connecting over Tor.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> Using a throwaway for obvious reasons... I am grateful these are being worked on because they are extremely needed for some use cases.

I have been part of a group organizing protest in Beirut and I was surprised there was no clearly go to app that provided the security features we need.

We started off with WhatSapp because that's what everyone used before security became a concern. We then moved to Signal mostly to get auto-deleting messages. We then ran away to Telegram because there was no way to kick a compromised phone outside of a Signal group.

We considered using Wire which seemed to have what we needed but the interface was a bit clunky and it did not run well on all the phones of the group... We are currently evaluating an considering Keybase.io which seems to have all the feature too, but not sure how it will handle about a hundred people in the group...

 _If anyone has ideas about which apps are recommended for that (or has additional useful things) please help_ , the main things we need are:

\- Encryption E2E is nice to have but not a deal breaker.

\- Possibility to kick a user from the group, deal breaker ( a thug stole someone's phone in the protest once and another time we got a message saying someone's security code changed then they became inaccessible) both incidents ended up ok but there was no way to kick the person out of the group and proceed while clearing things out with signal.

\- no old history kept of the conversation. Either auto-deleting messages set to short duration like signal, or if not possible we can survive with an admin at home deleting old messages constantly and clearing the chat for everyone in sensitive situations ( like telegram allows)

\- Free. For various reasons, some people can't buy apps no matter how cheap.

\- easy to use. Most protesters are not too technical.

\- possibility to display sender and group but not the content of messages in the notifications.

\- having an easy way to add password to the app itself. (nice to have)

\- making screenshots inconvenient to take (just nice to have).

\- Not tied to phone numbers also really nice to have but not mandatory.

Our main threat is riot police and pro government thugs taking protesters phones and forcing people to unlock them or running away before the phone is locked then snooping around. Very rarely are people alone when this happens so we almost always get a notification that X is compromised, so we clear chats and kick them out of the group before their phones are really compromised.

I don't think the government is running sophisticated deep packet inspection. I don't think our group has been infiltrated but that is always a possibility.

We are also trying to find some free device management solution to remotely track / lock and maybe wipe phones when they get taken.

Sorry for the wall of text... just though now might be a good time to ask...

<comment><USER20> This list could be a helpful starting point:

[https://en.wikipedia.org/wiki/Comparison_of_cross- platform_i...](https://en.wikipedia.org/wiki/Comparison_of_cross- platform_instant_messaging_clients)

You can sort the table by clicking on the column headers. The "E2EE group chat" column should be useful.

<comment><USER21> That list seems somewhat out of date, at least for OMEMO based group chat.

</comment></comment></comment><comment><USER22> I noticed this from the paper:

> Note that a user who has acquired a group's GroupMasterKey and then leaves > the group (or is deleted) retains the ability to collude with a malicious > server to encrypt and decrypt group entries. We deem this risk acceptable > for now due to the complexities in rapid and reliable rekey of the > GroupMasterKey.

Does this mean that the server and a deleted user can always collude to get the deleted user readded to the group? Also, is there no provable audit trail of who added or deleted whom? Unless I'm misunderstanding, it seems like deleting a user is therefore enforced only via server trust, but please correct me if I'm wrong.

<comment><USER18> Yes, this means the server and a deleted user could collude to re-add them, or anybody of the deleted user's choosing to the group, or to remove selected people from the group (the server doesn't need collusion to remove random people from a group)

No, the members of the group would be able to see that the deleted user is back, or whatever else has happened to the list. Signal's server isn't responsible for deciding who gets the group messages, only for storing the agreed list in encrypted form. So members don't need to trust that the server did as it was told.

Certainly if you have a group where you suspect a member of colluding with the Signal server to betray the group you should probably NOT remove that member but instead take the extra trouble to explicitly form a new group (without that member obviously).

<comment><USER22> Got it. I was thinking that for bigger groups, it might be hard for members to keep track of who got deleted when and by whom, so it might be easier for a deleted user to slip back in without attracting notice.

Your point that the deleted user and the server can collude to add a rando to the group seems like a bigger deal, since it would be harder to catch.

To make the same point more critically, if the members need to constantly recheck the mapping of group name to membership list (to stop server cheating), then the scheme might not be buying much.

</comment></comment></comment><comment><USER23> Overall cool stuff. It feels like this has implications for auth/authz schemes in general, like a variant of kerberos, or a way to do auth/authz for a ssh like service - maybe even a way to anchor trust (in user principals and service principals - like ssh keys and/or certificates)?

If we replace "the signal server" with "the authentication/authorization service ("the AD service" / the organization's internal certificate authority")...?

Maybe I'm just needlessly afraid of the complexity of managing a real world certificate authority (keeping it secure, keeping it running, keeping as much as possible off line..).

</comment><comment><USER24> Is it a coincidence this came up on HN at the same time Telegram is getting dissed on HN? Can't help but think it was coordinated...which is sad for HN

<comment><USER25> People don't like Telegram because it's a centralized thing and maybe not trustworthy. I'm not sure if I'd trust the Telegram founders, and their commitment to open source seems questionable to me (no server, outdated clients).

People advocate for Signal because it's arguably the least offensive of the available e2e options. Also the founder for Signal has a long history of doing good work in this area.

<comment><USER26> > People don't like Telegram because it's a centralized thing and maybe not > trustworthy.

Just like Signal.

> I'm not sure if I'd trust the Telegram founders, and their commitment to > open source seems questionable to me

Meanwhile Moxie Marlinspike's _opposition_ to free software is evident. You use the client he dictates or fuck off. There's closed source software that respects freedom more than Signal.

<comment><USER25> [https://github.com/signalapp/Signal- Android](https://github.com/signalapp/Signal-Android)

Can't I build and use this if I want? It looks very open, but I haven't tried building my own client.

<comment><USER26> Free software is not about being able to compile and code Moxie Marlinspike allows you to use. It's about freedom that he opposes:

[https://github.com/LibreSignal/LibreSignal/issues/37#issueco...](https://github.com/LibreSignal/LibreSignal/issues/37#issuecomment-217211165)

Yes, Signal's licensing complies to the letter of how we define free software. But that is irrelevant as it violates the spirit.

</comment></comment></comment></comment></comment><comment><USER27> I think I know the answer already, but just in case: is there a way to use Signal to communicate with users using Whatsapp? IOW, can I receive Whatsapp messages in Signal?

The only reason I use Whatsapp is because it's what all my contacts use. It's _everywhere_. It's _the_ de facto standard for text communication. And I hate the app. I hate its guts.

I read that whatsapp implemented the signal protocol, does that mean anything with respect to being able to communicate with people using a different app? Because I was hoping so, but I can't find a way to see my whatsapp messages in signal.

<comment><USER11> Yup, you guessed right. By default that's a no. However, you can do a cow- helicopter by using a 3rd party that will be a proxy between your signal account and your whatsapp account. Hell, you can do whatever you want between any 2 services with a 3rd party. Problem is, you still need an account on both ends.</comment></comment>


Churn Buster Gets Acquired - <AUTHOR> http://churnbuster.io/acquisition <comment><USER2> That's awesome news -- I (well, AR) had been a customer since very early days in the life of the product. It integrated pretty painlessly and basically gave us an accounts receivable department without having either myself or my sales rep having to spend lots of time calling $29 a month accounts to see if they were soft-canceling or just having transient billing failures. The ROI on it was stupendous -- I remember saving $100 in MRR in the first two days on a $50 a month plan.

Anecdotally, there seems to be a wave of mini-acquisitions going through the small-SaaS-osphere this last year or so. I think it's an interesting development, partially do to the obvious reasons (money for founders is nice), partially as it frees up founders to move onto their next level, partially since these seem to be better digested than acquisitions in VC land, and partially just because it is interesting to see this side of the fence maturing a bit.

<comment><USER3> That's really great to hear, Patrick.

We're pretty excited about how we structured the deal, and it's definitely not the way things are done in Silicon Valley. Even down to the expectations of our investors. We're setting out to grow a profitable business with the goal of early and lasting distributions for investors...not with the goal of an eventual mega-exit.

Happy to answer any detailed questions specifically about the structure/acquisition process (negotiation/due diligence/lawyers/etc), something you know plenty about :-)

Also of note, we didn't use an escrow account, or a website broker. This probably saved about $20k-$30k. We had an advisor, JD Graffam of SimpleFocus, that helped coach us along the way based on his experience acquiring products like Ballpark App.

<comment><USER4> I'd be interested in this. How was the deal structured? What did your due diligence phase look like? What sort of multiple of revenue range is this in?

<comment><USER3> Can't specify any financial details (sorry!), but the whole process lasted about 3 months.

Legal fees were reasonable using a local attorney that another startup introduced us to (with experience in M&A).

Due diligence included reviewing ROI reports (how well the product works for clients), financial reports of the company (where does revenue stand, how fast is it growing), current customer lists, past customer lists, all support conversations (how happy are customers, how fast are support requests handled), and gathering listings of accounts in-use that would need to be transferred.

We decided on a purchase price before entering due diligence, but saved equity offerings until after the deal had closed. At that point, we brought in our investors, and offered Andrew a priced option to invest (at a higher valuation than the purchase price based on the expanded team and advisors).

If you don't have guidance from someone with experience, you should probably use a broker. And if you don't know the seller super-well, you should probably use an escrow company for the transfer of assets. In our case, we knew we were working with a stand-up guy based on our friendship with Andrew, and how wonderful he was throughout the entire process.

Let me know if I can answer anything else! Matt

</comment></comment></comment></comment><comment><USER5> Hey all, I'm the creator of Churn Buster.

Over two years ago I was working on Stripe web hook emails for a client and spending way too much time tuning, tweaking, and maintaining them. Every little improvement was yielding even greater conversion rates and more money in the bank. At that point, I suggested that instead of paying me so much to maintain the in-house dunning system on their behalf, I could create a service that did the same thing but better, and charge them $200/mo instead. The plan was to subsidize the cost of development by selling the product to friends who were having the same problem. I've been working on it and growing it ever since.

I'm really proud of some of the incredible companies and brands I've had the opportunity to work with in the last two years, (some of them are listed on the new product homepage,) but I'm also really pleased to now be able to put the product into the hands of Matt, Joelle, and Ken and let them take it's growth from this point forward.

Happy to answer any questions anyone may have!

</comment><comment><USER3> Thanks for sharing, Michael!

Matt here, new CEO of Churn Buster. We're so freaking excited about working with Andrew, and our other investor Rob Walling (getdrip.com).

Andrew has really built something special here and I think his excitement shines through in his announcement video.

If you have any questions about the acquisition/round of funding (it was pretty unique) or about churn, failed payments, etc. just post 'em here!

</comment><comment><USER6> Wow, the website of churn buster explains NOWHERE how churnbuster actually works. Just pointing to tweets, and saying you have "algorithms" doesnt cut it. This is a crap landing page (for tech savvies). But maybe only marketing guys are landing at that page.

<comment><USER7> patio11 summarized churnbuster pretty well here:

>> [Churnbuster] integrated pretty painlessly and basically gave us an accounts receivable department without having either myself or my sales rep having to spend lots of time calling $29 a month accounts to see if they were soft-canceling or just having transient billing failures.

</comment></comment><comment><AUTHOR> I posted this as I'm just constantly amazed at what a dynamic and big ecosystem that Stripe has created around itself.

</comment><comment><USER8> I've known Ken for many years, and I can say without doubt Churnbuster's exactly what he's great at. I've also followed what Matt/Joelle have done for awhile and think quite highly of the whole team. I'm sure they'll do some great stuff with this product—i'll definitely be referring it to many people in the future!</comment>


Prepare for new copyright crackdown: GDP to treat artistic works as investment. - <AUTHOR> https://webcache.googleusercontent.com/search?output=search&sclient=psy-ab&q=cache:0LycQXx4rCUJ:http://www.ft.com/cms/s/0/52d23fa6-aa98-11e2-bc0d-00144feabdc0.html%2B%22Brent+Moulton,+who+manages+the+national+accounts+at+the+Bureau+of+Economic+Analysis,+told+the+Financial+Times%22&gbv=1&sei=IrWHUYzYC8Xp0QH14YGQDQ&hl=en&ct=clnk

<comment><AUTHOR> Details here: <https://www.bea.gov/scb/pdf/2011/06%20June/0611_artistic.pdf>

</comment><comment><USER2> Mods, please fix the completely bogus title. This is about cooking the books on inflation, not copyright.

Everyone, enjoy feeling 3% richer today!</comment>


If Facebook Threatens the Web, then WordPress Saved the Web - <AUTHOR> http://marcgrabanski.com/articles/wordpress-saved-the-web After reading Tim Berners-Lee's writings on how Facebook threatens the future of the web it occurred to me just how in the reverse argument is also true...projects like WordPress actually saved the web up until now by saving the intellectual property of millions of content publishers. <comment><USER3> We're never going to convince regular people to run their own Diaspora server. I mean, look at this:

[https://github.com/diaspora/diaspora/wiki/Installing-and- Run...](https://github.com/diaspora/diaspora/wiki/Installing-and-Running- Diaspora)

Even Wordpress is increasingly becoming a huge liability for someone who isn't a sysadmin to run. Telling a non-geek friend to use a self-hosted Wordpress install to host their personal blog strikes me as incredibly irresponsible in the same way that jailbreaking/rooting a non-geek's smartphone would be doing them a huge disservice. Nobody should be self-hosting unless they know exactly what they're committing to. I'd much rather they use a hosted platform with a good export mechanism than use wordpress, see their site hacked using a zero- day export, and completely lose faith in "openness".

Data portability is something that average people can understand, something that sites can implement fairly easily, and something that's compatible with how regular people use computers. If geeks push self-hosting, they risk completely losing their credibility with regular people in the same way that the FSF has. Data portability might not be as sexy, but it's something that we have a chance of seeing through.

There's a reason why people used closed systems like Facebook, and it's not ignorance—it's convenience and quality. People aren't going to move away from these systems for the sake of "openness", but they might be convinced to push companies to implement true data portability if we can convince them why it's important.

<comment><USER4> Maintaining a self-hosted WordPress installation is awfully easy since they added the one-click updating for plugins and WordPress itself. I agree that a _neglected_ WordPress install is a liability, but it takes negligible effort to keep later versions of it running smoothly. Even "normals" can handle clicking an update button once in a while when they log into their dashboard to check comments, stats, etc.

<comment><USER3> But "normals" just aren't always going to do that. If Wordpress installations automatically updated without any user intervention, then _maybe_ it would be better, but I still think it's dangerous to get non-geek friends to use a self-hosted Wordpress unless they really understand server administration—not just following an installation guide, but how permissions work, how to back up databases, how to properly vet add-ons, etc.

Plus, a lot of the Wordpress exploits have been zero-day. Even if they are checking their Wordpress dashboard every day, or even every week (and that's probably not a good assumption to make), their installation could be silently compromised before an update was even available.

I think geeks tend to assume that regular people understand and care about even the most basic (by our estimation) best practices. You could argue that that's their fault, not ours, but I think there's a certain lack of pragmatism involved in thinking that regular users can responsibly administer a Wordpress install. I'm sure contrary examples exist, but there's a lot of horror stories as well.

</comment></comment></comment><comment><USER5> > Nobody should be self-hosting unless they know exactly what they're > committing to.

it is just a FUD. Self-hosting can be done in a range of options - <USER2> with your personal webserver on your own home or collocated computer (geek) to the pages personalized with your domain name on some common hosted and managed _open_ platform like wordpress, etc... (regular, non-geek person)

The issue isn't about technicalities of hosting or data portability. It is about content ownership. What FB does is "all you base are belong to us".

My related post on similar issue about giving away reviews to sites likes Yelp <http://news.ycombinator.com/item?id=1898945>

Edit : this post is actually a reply to <http://news.ycombinator.com/item?id=1943861>

</comment><comment><USER6> I think Facebook/Twitter are basically just blogs with inbuilt blog readers, and a beefed up blog roll. Isn't that why Tumbler seems to be taking off suddenly?

Funny how much of a difference subtle details of implementation can make.</comment>


Node.js Has 1 Gb Memory Limit - <AUTHOR> http://code.google.com/p/v8/issues/detail?id=847

<comment><USER3> It would be more correct to say that V8 has a 1G memory limit. But node.js actually gives you these nice things called Buffers which, according to the node.js manual, are "similar to an array of integers but corresponds to a raw memory allocation outside the V8 heap" (see <http://nodejs.org/api/buffers.html>).

Buffers are used all over the place in node, which should mitigate the degree to which this is actually a problem. Even if this bug report stays a wontfix, it won't affect my decision to use node because of Buffer support.

<comment><AUTHOR> Sure, Node is a nice platform, did not imply otherwise.

For a certain category of apps it helps to know the limits beforehand, though (I've described my use case in another comment). I think this particular limit deserves to be more widely known.

</comment></comment><comment><USER4> It's probably worth mentioning that the GC cycles start to take upwards of one second at around 500mb of RAM usage with lots of small objects.

This limit doesn't stop you using NodeJS, but it's definitely something newcomers should be made aware of before they start writing database servers or making heavy use of a naive in-memory cache through an object literal.

Hopefully when the new GC gets rolled out we'll really be able to let loose with the RAM usage.

<comment><USER5> Funny how all VMs go through this. Remember when JVM GC pauses killed websites - and how over the years this was fixed by better - non pausing - <USER2> strategies.

<comment><USER6> That's very easy to explain. A low-pause GC requires a concurrent garbage collector. Writing a correct concurrent garbage collector is VERY tricky, so you usually start with something much simpler. Concurrent GCs typically also add overhead to the mutator (the user program), so it tends to be a trade-off of high throughput vs. short maximum pause times. As an example Java's G1 collector ("Garbage First") took a team of experts about 5 years to get correct.

OTOH, writing a standard (single-threaded) Cheney-style copying GC can be done in a few days. Actually, the more time consuming part for me was to get all the pointer information to the GC.

</comment></comment></comment><comment> eric-hu Relevant snippets:

"diego.ca...@gmail.com, Sep 19, 2010

... In my case, I've been forced to suddenly stop my work ... with node.js because it cannot handle more than ~140K websockets concurrent connections ..."

"erik.corry, Nov 2, 2010

The limit for 64 bit V8 is around 1.9Gbytes now. Start V8 with the \--max-old- space-size=1900 flag."

<comment><USER7> I don't understand why he has to stop the project just because ONE node.js process cannot handle more than 140K concurrent connections. If the goal is to handle millions of concurrent connections, spreading the connection load out to multiple servers is a must.

I'm doing something similar with node.js now to handle millions of concurrent connections. I use multiple node.js servers to handle it. Not only it helps in even out the load, it has the nice failover property. One node down doesn't mean the whole app is down. Clients of the down node migrate their connections to the rest of the nodes.

<comment><USER8> If you have 140M active users, handling a fleet of 1000 servers is not an easy operation, considering you need to manage inter-cluster communications (i.e. when sender and receiver sockets fell on different servers).

I've built Erlang/OTP websockets clustered server, which can handle 3M per node (giving you have enough RAM). Here you handle all your users with "only" 47 servers.

There is a big operational and scaling difference between cluster of 1000 servers and cluster of 47 servers.

<comment><USER9> I'm a bit curious about this. If you have 140M concurrent users (since this was the original complaint) and you are NOT prepared or capable of servicing/monitoring/maintaining 1000 servers, that seems like a fatal flaw in your server management and analytics processes.

Certainly 47 servers vs. 1000 is far nicer. But at 140M concurrent users levels (there must only be a few handfuls of sites with these types of concerns), not having a team prepared to oversee 1000 servers seems like folly.

<comment><USER8> The problem is not only operational, but also a technical one:

Nodes in the cluster need to communicate with each other and with other systems, like databases, message queues, monitoring servers, etc.

You can aggregate data per node, so the less servers in the front-end cluster you have, the less load on the back-end servers.

There is also financial problem: many organizations can afford 50 servers. But not many can afford 1000 servers.

<comment><USER10> There's no reason why you can't have many node processes per server. This also raises the fault tolerance per server.

<comment><USER8> Facebook chat has only 38 gateway servers. I guess, that only possible b/c not every active facebook user, uses Chat.

<comment><USER11> Perhaps they should consider adding more, it's permanently screwed up in some way. Messages not being sent or the other person not receiving them.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> There's a general point to be made here: this is the sort of thing that happens when you have "closed development open source". Admittedly, there seem to be very few people in the world capable of delivering a world-class GC.

</comment><comment><AUTHOR> To add my own perspective: I was running some analytics batch job on Node and hit this limit, had to add multi-stage processing to accommodate it. Using --max-old-space-size=1900 did help a bit.

In case you're wondering, here's a test case: <https://gist.github.com/1148761>

I was told by one of the V8 developers that the new GC is pretty usable now, so worth giving it a shot if you need more than 2 Gb.

</comment><comment><USER13> What rude and demanding bug reports.

<comment><USER14> Welcome to Web 2.0, where people that think they're programmers try to interact with people that _are_ programmers. The results are often ... depressing.

<comment><USER15> A minor re-formatting.

> Welcome to Web 2.0, where people _who_ think they're programmers try to > interact with people _who_ are programmers. The results are often ... > depressing.

I was confused by your initial framing of the sentence, though my not being a native speaker would have played a part.

<comment><USER16> Hope this helps: "[T]his distinction applies only to _which_ and _who_. The alternative _that_ is found with both human and non-human antecedents." [http://en.wikipedia.org/wiki/English_relative_clauses#Human_...](http://en.wikipedia.org/wiki/English_relative_clauses#Human_or_non- human)

<comment><USER15> That's news to me. Language follows speakers, especially influential speakers, and the list of speakers using _that_ for human antecedents includes Shakespeare and Mark Twain, so I guess it's all right now, even if it were an error at some point(grammar isn't static - it has been changing ever since it came into being).

Now I am wondering about why I thought _that_ isn't to be used for human antecedents. Was it an error earlier or it changed recently? If it was an error earlier, Shakespeare using it doesn't fit.

<comment><USER17> You probably thought that because it's uncommon in good writing and you're taught not to do it. It's probably against style guidelines for major newspapers, for example, and it sounds elegant to educated native writers.

</comment></comment></comment></comment></comment></comment><comment><USER18> I guess the point of posting it here is to gather folks with pitchforks and torches demanding immediate attention to the issue. For a few months V8 team has been working hard on improving the GC ([http://code.google.com/p/v8/source/browse/branches/experimen...](http://code.google.com/p/v8/source/browse/branches/experimental/gc)) which was the major limiting factor here. I don't have an ETA of when it's going to be merged, but the new GC branch is in a pretty good shape.

</comment><comment><USER19> This all boils down to using the right tool for the right task and knowing how to organize your code so you don't run in to limits in the platform components that you use.

In this case, the obvious solutions are to either use 'buffers' (think of them as extended memory from the old days) or to use multiple instances on the same machine or spread over several machines.

If you write your code in such a way that you end up handling all your users in a single process then you will sooner or later run in to some limitation.

</comment><comment><USER20> This has been long known and looks to no longer be a problem soon reading the comments of the OP. It hasn't been a serious issue as long as you know about it going in.

</comment><comment><USER21> They're running up against a V8 limit.

<comment><AUTHOR> Yep, but the reason we care about it is exactly because it affects Node.

<comment><USER21> The reason you care about it is because it affects Node. I'm more interested in V8 than Node to be honest.

<comment><USER22> No doubt. This comment really struck me as off-tone:

 _"Sorry, Google, but your open-source project is important to much more of the world than just >your browser< now; and while this may not be an issue for the zones of impact >you< care about (said browser.), it's a >huge< issue for much of the area where V8 is important in general in the modern (post-Node) world."_

You know Google is biting their tongue at what they really want to say to complainers like this. You really want it working for Node? Roll up your sleeves and get to refactoring. Hell, they even explained how to fix it.

<comment><USER14> The moral of the story is to never be nice to anyone. Nobody complains about the memory limits of IE's javascript engine as they pertain to server-side applications.

</comment></comment></comment></comment></comment><comment><USER23> If there's anything I've learned in my software career, is that problems can be fixed. It sounds like this problem can, or already has been fixed. NodeJS is a wonderful platform to work with, nonetheless.

</comment><comment><USER24> nice FUD headline

<comment><AUTHOR> Why? It's been a real bummer for me to find that out (the hard way).

<comment><USER15> _Why? It's been a real bummer for me to find that out (the hard way)._

Knowing about the limit is hurting you, or you have really faced this constraint in a project? As mentioned elsewhere, node has buffers which are allocated outside of v8 heap, and the user data will largely be unaffected by the memory limit if you are using buffers.

<comment><AUTHOR> I've tried to explain it in another comment:

> To add my own perspective: I was running some analytics batch job on Node > and hit this limit, had to add multi-stage processing to accommodate it. > Using --max-old-space-size=1900 did help a bit.

So yeah, it did affect me, although not on a typical web project. It wouldn't be a problem at all if I knew about this limit from the start. Thus this is a warning for others.

(BTW another limit I was told about is that objects can't have more than a million keys. Thankfully I did not hit that one.)

<comment><USER25> Well, if your data set is big enough to hit that limit then you'll probably need to horizontally scale it out anyway...

Or rewrite in C

</comment></comment></comment></comment></comment><comment><USER14> "C Has 16 Exabyte Memory Limit"</comment>


Send e-mails from spreadsheet using OpenOffice macro - <AUTHOR> http://asserttrue.blogspot.com/2015/03/send-e-mail-from-openoffice-spreadsheet.html

<comment><USER2> Send e-mail with formulas: Demo

[https://radiant-bastion-1152.herokuapp.com/i3pqpufosc](https://radiant- bastion-1152.herokuapp.com/i3pqpufosc)

Many people want to send e-mail with a spreadsheet. Requests get posted on spreadsheet forums all the time. The same question is posted again and again, week after week. So I decided to create some formulas for it.

Not sure is a big pain point, not sure people would pay, but happy to finish it off if anyone can think of a way to cover costs.</comment>


How can C Programs be so Reliable? - <AUTHOR> http://tratt.net/laurie/tech_articles/articles/how_can_c_programs_be_so_reliable <comment><USER3> People often write in higher level languages because they want lots of bad code fast. Almost all business applications are CRUD apps (create/retrieve/update/destroy) with some business logic, and they're generally written in C#. The app may crash when you click the wrong button, but the app is cheap to develop and the programmers are easily replaceable.

Of course I'm generalizing, and a lot of C# programmers write great and reliable code. The point is that they often don't -need- to write great code, because lousy code is good enough for all business purposes (except when the software is your product, which is rarely the case).

The second issue is that people who choose C for their projects tend to (a) understand low-level concepts, (b) care about speed / memory usage / reliability / dependencies, (c) don't consider development time that important. That you create more reliable software that way is obvious - <USER2> same programmers would create reliable software in C# (or similar language). But there aren't many situations in which development speed doesn't matter, speed and memory usage don't matter, dependencies don't matter but, for some reason, reliability is very important.

<comment><USER4> I come from the systems research world. I've seen people choose C not for the reasons you mention, but just because it's what they know best. This is not always the best thing to do.

One instance is a colleague who needed to do data post-processing, and just did it in C because it was most familiar. A language like Perl or Python would have been a better choice, and saved him time in the long run, since string manipulation in C is tedious and error prone.

<comment><USER5> Your colleague may be interested in pcre (perl compatible regular expressions) see <http://www.pcre.org>

But I agree, C isn't the best choice for that task.

</comment></comment></comment><comment><USER4> The author has a unique perspective since he - somehow - skipped learning C until now. He programmed assembly before, and from his other work he clearly is familiar with dynamic and more modern languages.

Consequently, his perspective on C is that of somehow who is new to the language, yet also understands both the fundamentals of what his code will compile down to, and the higher level facilities that later languages and programming models abstracted away.

<comment><USER6> I'm still not convinced that there's anything particularly magical about C here. Reliable programs are written by people who:

* understand the problem domain

* know the implementation language _and its supporting library_

* pay attention to detail

Admittedly, some languages fit some problem domains better than others, but 90% of the time, picking the language you're personally most familiar with will be as good a choice as any.

<comment><USER4> The author comes to a similar conclusion.

</comment></comment></comment><comment><USER7> This misses two points I consider important:

\- A whole lot of C code is old. Old, actively maintained code, tends to be more reliable than new code.

\- C tends to be employed in relatively predictable sub-systems. Something like a device driver has a relatively predictable set of states relative to a GUI application. There aren't that many paths. C code for GUIs, in my experience, tends to be at least a buggy as code in other languages.

<comment><USER8> You seem to forget that in embedded systems, C is still the most used language. So most of the new systems developed these days have new C code. And embedded systems are everywhere.

Also a big part of embedded systems have to be reliable for years in hostile environement without external interventions. so I wouldn't say that this is easily predictable subsystems.

<comment><USER9> Hmmm, I've worked in the embedded world for the last 10 years, and I can tell you that the bulk of embedded code is far from hardened. It appalls me the number of times that mobile phones, set top boxes, etc crash. Typically, reliability is low because it is hard to run decent unit tests in an embedded environment. You're obliged to write a type of simulator that you can run on a PC, but the simulator never has the exact same behaviour as the target system, so you can never truly verify correct code behaviour.

<comment><USER10> Yes, this is true: A lot of embedded code is poorly tested. Even more is poorly designed and written by poor programmers (like most software.)

Some people have to deal with the machine, and C is a good model of the machine. Even reliable systems have some core in C (or an equivalent) that provides an abstraction where higher level abstractions can be expressed.

Testing isn't the ultimate solution; some people are just able to use the machine better and produce better code. Testing crappy code doesn't really help. A subset of developers produce tools where others can produce systems with less risk by providing appropriate abstractions.

</comment></comment></comment></comment><comment><USER11> Considering it's the first (alpha) release of the first significant C program he's written which is for a new (and therefore little used) programming language, and the page of Converge tools has no mention of testing tools, and the page itself has error messages on it, the claim that it's "not riddled with bugs" seems a touch, erm, cheerful.

</comment><comment><USER12> The thing about C is that you are generally very aware of the side effects. Aside from the libraries you use, the (data) structures are yours. When you set something to NULL, you know damn well that that means to you.

As the author alludes to as well - in C you're made more aware of the error conditions you can handle and the ones you can't. So you can code to a level of robustness... Exceptions in Java are all well and good, but I haven't seen many implementations that _do_ anything with IOException except for cascade it.

This works really well for programs/modules that can fit in the head of a single programmer. When you go beyond that it gets pretty messy - which is where some of the advantages of metaphors like OO start to help... Course, there is the argument that modules should never get that big, but that's another debate.

<comment><USER13> "As the author alludes to as well - in C you're made more aware of the error conditions you can handle and the ones you can't."

You mean like when a function silently returns -1 to indicate failure and then you wonder why your program returned a wrong result (if you're lucky enough to even notice)? In the bigger part of most of my programs I want a big, flashy, loud, total failure by default if anything goes wrong.

<comment><USER14>
      if ((result = foo()) == -1) {
         fprintf(stderr, "!$*!$&^!$*!!!!\n");
         exit(1);
      }

<comment><USER6> If _exit()_ is too abrupt, you can always use _setjmp()_ and _longjmp()_ to set up a non-local jump to an error handler.

<comment><USER15> setjmp/longjmp is essentially what exception handlers do for you. I love C for its simplicity, but I'm not certain that leaving exception handling out of the language was a good idea. There are a lot of people who would agree with me there, including (I believe) a few Bell labs veterans who wrote the language in the first place.

<comment><USER16> The nice thing about exceptions beyond setjmp/lngjmp (in C++ at least) is the destructor semantics which you can use to guarantee that resources are cleaned up on error. There are better ways to handle such things, such as scoped resource allocation, but exceptions do an ok job.

<comment><USER6> You can actually use setjmp()/longjmp() to implement that kind of thing, it turns out. Here's but one example:

<http://www.on-time.com/ddj0011.htm>

I'm almost tempted to make an analogy with scheme's (call-with-current- continuation) here, but I think that might be pushing it.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> > What I realised is that neither exception-based approach is appropriate when > one wishes to make software as robust as possible.

The author misses the main point of exceptions: they let us separate data processing code from error handling code. This is why we are more productive in languages that have exceptions and our code is easier to maintain.

C requires us to handle errors throughout our program, tightly coupling the data processing code with the error handling code. With apologies to Mr. Spencer, I would declare that:

"Those who don't code in languages that lack exceptions are doomed to reimplement them, poorly."

<comment><USER9> I have to disagree. From my perspective it is an error to think that error handling can be handled separately from the main code path. The classic example that I use to demonstrate this is to point to all of those tedious discussions that programmers have as to whether something is an exception or just normal behaviour of a system. Straight away for me that's a red flag that a non-real distinction is being made.

I honestly don't see any advantage to exceptions over C-style return codes, with one important ...euh... exception: the boiler plate for exception handlers can be well handled by modern IDEs. All the other supposed advantages seem to be just waffling to me. Take the whole 'Oh, but exceptions make handling errors the default!' kind of argument (several examples on this thread already). Yes, sure, you _do_ have to write exception handlers for all errors in a language such as Java. But my experience is that if I'm writing use-once-and-throw-away in C, I'll just not use the return code. In Java I'll just stick a whopping great big try/catch around the whole app, and be done with it. If I'm trying to write stable code that's going to be around for a while in C, I check the error codes returned by a function every single time, which gives me around about as much work as when I am using Java, and actually handling different exceptions correctly.

All of which means, for me at least, that exceptions don't add anything to a language, but they do make the language just a little bit harder to learn (remembering exactly how any given language has implemented exceptions, and which resources can still be safely used when is a pain, as each language tends to have subtle differences that can bite you).

</comment></comment><comment><USER18> After having read a lot of these types of language debates, the only conclusions I can safely arrive at are 1) that every language has its advocates and 2) some people are highly productive in their preferred language, much more so than the _average_ programmer would be in their preferred language. But the real question, imho, is how do average programmers compare? How will the same app written in C++ and Java compare, when written by non-superstars? The question merits some empirical research.

</comment><comment><USER19> The problem with C is they left too many things completely on the wild. Strings and memory management are all laissez-faire and everybody does whatever they think is right.

A typical performance issue is calling malloc/free all the time, it can be avoided but there are no standard ways.

I hope some great features of C++ get some day backported to C. But don't bet money on that :(

<comment><USER20> The later C standards add additional features. However, I'd argue that there is no such thing as a "great feature" in C++: almost everything in the language is a mistake, either intrinsically or in combination with other 'feature's.

It's a blind pig with fifteen legs, trying to put on its own lipstick while riding a unicycle.

<comment><USER21> Spoken like someone who truly doesn't understand C. You even managed to confuse it with C++ (you might as well have said "Java" in the place of C++, it's about as close to C as C++ is)

Well done.

 _edit: Ignore this post. I misread. I'm an idiot. Sorry about that_

<comment><USER22> I think you misread the comment you were responding to, and the comment that it was responding to.

<comment><USER21> Yes, yes I did.

</comment></comment></comment></comment></comment><comment><USER19> Programs compiled from C don't change much when run, while dynamic applications vary significantly on every run due to the large environment (e.g. GC.) This is a bliss for debugging. I've never seen anything like gdb on any other language. You can backtrack, set and change things, watch for expressions, and even run one line of _compiled_ code at a time to see what's breaking.

Also C programs usually are made to run many times and stay alive, so the attitude of the developer tends to be more careful.

</comment><comment><USER23> As a person who spent few years doing C programming, and then got trapped into the Ruby realm, I must say that the main difference is focus. If you have to focus about memory management, strings, you think much more about the code you write.

Theoretically, high level programming should enable you to focus on the abstractions much more, but somehow it doesn't work this way.

</comment><comment><USER24> See also: No Silver Bullet

</comment><comment><USER25> On the downside, most bugs ever discovered are most likely in c code. Especially memory leaks, buffer overflows etc.

</comment><comment><USER26> _If, as in the case of extsmail, one wants to be robust against errors, one has to handle all possible error paths oneself._

That's not really a bad thing, as the author points out.

One thing I've always felt a sense of dread about in C# and Java (last Java I did was back in 2001) is that I never truly knew what errors were lurking with their exception handling. It would be really nice if all error possibilities were listed in the documentation so I could pick precisely what to handle.

<comment><USER3> It's even worse. For top level functions it can throw the UNION of the exceptions thrown by the functions it calls. Therefore, by changing a low level function's exception signature the exception signature of all higher level functions changes too. That always worried me, but it's doesn't seem to be a big deal in practice.

<comment><USER27> It's a huge deal with checked exceptions in Java. You have to change the signature of every caller, and so on, when you change a low-level function. Abstraction leakage galore; it's why many libraries just throw a single generic exception type whenever anything goes wrong (which defeats the purpose of exceptions) or subclass RuntimeException (which defeats the purpose of checked exceptions).

Unchecked exceptions don't seem to be a problem, because oftentimes you don't care what specifically went wrong, you just need to know that _something_ went wrong and abort appropriately.

</comment></comment></comment><comment><USER28> Do they? Mature C code: yes. Modern one: no, many times affected by using very bad algorithms...</comment>