MongoDB Gotchas and How To Avoid Them - <AUTHOR> http://rsmith.co/2012/11/05/mongodb-gotchas-and-how-to-avoid-them/

<comment><USER2> Very good summary of what to look out for. Here are a few others that I ran into back when I was still entertaining the idea of using Mongo in production:

1\. The keys in Mongo documents get repeated over and over again for every record in your collection (which makes sense when you remember that collections don't have a db-enforced schema). When you have millions of documents this really adds up. Consider adding an abstraction mapping layer of short 1-2 character keys to real keys in your business logic.

2\. Mongo lies about being ready after an initial install. If you're trying to automate bringing mongo boxes up and down, you're going to run into the case where the mongo service says that it's ready, but in reality it's still preparing its preallocated journal. During this time, which can take up to 5-10 minutes based on your system specs, all of your connections will just hang and timeout. Either build the preallocated journal yourself and drop it in place before installing mongo, or touch the file locations if you don't mind the slight initial performance hit on that machine. (Note: not all installs will create a preallocated journal. Mongo tries to do a mini performance test on install to determine at runtime whether preallocating is better for your hardware or not. There's no way to force it one way or the other.)

<comment><AUTHOR> Thanks; #1 can be a problem for large collections...due to the extra space. I think I've seen some pre-rolled abstraction layers for this, but never used a open-source one myself.

#2 - usually the journal should be pretty quick to allocate - I've not experienced this problem directly myself.

I'll add some extra bits to the bottom of the post with your notes.

<comment><USER3> There is a ticket for compression of both keys and values <https://jira.mongodb.org/browse/SERVER-164>

The Mongo team seem somewhat reluctant to implement it.

<comment><USER4> It's probably not a clear win in most use-cases.

<comment><USER3> Reducing the working set is a big win. Mongo's behaviour when the working set is larger than RAM is really bad- <https://jira.mongodb.org/browse/SERVER-574>

Compression could have drawbacks on documents that get updated frequently. But it will be extremely useful on documents that get created and rarely/never change, coincidentally what I mostly have.

It would also greatly help if keys are compressed or indexed in some way since it could be done transparently.

You may recall the Mongo team being reluctant to make the database function well in single server setups, but they did address that with journalling.

</comment></comment></comment></comment></comment><comment> T-R An excellent and practical article. I do want to emphasize one thing, though, since I feel like the article almost seemed to downplay its significance:

 _MongoDB does not support joins; If you need to retrieve data from more than one collection you must do more than one query ... you can generally redesign your schema ... you can de-normalize your data easily._

This is a much larger issue than it seems - nested collections aren't first class objects in MongoDB - the $ operator for querying into arrays only goes one level deep, amongst its other issues, meaning that often-times you _must_ break things out into separate collections. This doesn't work either, though, as there are no cross-collection transactions, so if you need to break things into separate collections, you can't guarantee a write to each collection will go through properly. (Though, I suppose if you're using the latest version, you _could_ lock your whole database)

<comment><USER5> Absolute showstopper for anything more than keeping track of simple stats or storing comments. Denormalizing has limits. Invariably apps will come to the point where you need to do joins and that is when you start cursing your decision to go with NoSQL. My experience is that NoSQL (including Mongo) is not a replacement to traditional RDBMS, but if you use NoSQL complementary to an RDBMS, primarily for real time performance, works quite beautifully. That said, there may be quite a few simple web app use cases that do not need RDBMS at all.

<comment><USER6> it's true you almost always will find yourself needing to join cross collection/table... I believe the recent support to integrate with hadoop should help this: <http://www.mongodb.org/display/DOCS/Hadoop+Quick+Start> when your reason for needing joins is for reporting (often the case for say financial reporting)

Also, the postgres integration (linked/discussed) here on HN

</comment></comment></comment><comment><USER7> There are some good things here, but on a systems level there are huge oversights that are absolute showstoppers on production systems. Maybe there is a level of Mongo proficiency above MongoDB Master? I hope so.

1) Make sure to permanently increase the hard and soft limits for Linux open files and user processes for the MongoDB/Mongo user. If not, MongoDB will segfault under load and when that happens, the automatic recovery process works incredibly slowly. It's a bit tricky to get this right, depending on your level of sysadmin knowledge. 10gen doesn't emphasize or explain the issue very well in their docs: "Set file descriptor limit and user process limit to 4k+ (see etc/limits and ulimit)" That probably makes sense to just about 0.1% of the people setting up MongoDB: [http://www.mongodb.org/display/DOCS/Production+Notes#Product...](http://www.mongodb.org/display/DOCS/Production+Notes#ProductionNotes- GeneralUnixNotes)

2) Make sure to disable NUMA. This 10gen documentation note is a great example of clear documentation: "Linux, NUMA and MongoDB tend not to work well together ... Problems will manifest in strange ways, such as massive slow downs for periods of time or high system cpu time." Massive slowdowns and mysteriously pegged cpu usage on production database systems are definitely 'strange'. I would probably choose stronger and more precise language, but 10gen clearly knows what they're doing: <http://www.mongodb.org/display/DOCS/NUMA>

tl;dr If you have problems with MongoDB, you aren't using it right. Read the documentation more carefully, and then when that doesn't work, hire an expert.

<comment><AUTHOR> Yep, I managed to miss a few things; I'll add these shortly.

<comment><USER7> Nice work. :-) This list is getting pretty comprehensive.

<comment><AUTHOR> Thanks; If you think of anything more, let me know!

</comment></comment></comment></comment><comment><USER4> I'm one of the people that like to make fun of MongoDB from time to time, but that's mostly from proximity producing contempt.

Nevertheless, a rundown of the gotchas and how to avoid them _based on experience beyond simply running apt-get install mongodb_ is one of the most useful pieces on MongoDB I've seen of late.

The only new-news for me was that SSL support isn't compiled in by default. That's pretty irritating. I wonder if that applies just to 10gen's packages or also to distribution provided mongodb packages.

<comment><USER8> Disturbingly 10gen uses SSL support as a reason to use their subscriber packages. Sure you can build it in yourself if you compile the package, but it's disappointing that SSL support is one of the carrots that they use for their premium offerings.

Edit: Link to applicable docs on how to compile in/use. <http://docs.mongodb.org/manual/administration/ssl/>

<comment><USER9> The reason binaries aren't distributed by default have to do with US Export Controls -[http://en.m.wikipedia.org/wiki/Export_of_cryptography_in_the...](http://en.m.wikipedia.org/wiki/Export_of_cryptography_in_the_United_States#section_2)

<comment><USER10> SSL support is in the free distribution but you must built it yourself. One reason would be export controls; another is that creates a dependency on the SSL library for those who don't use SSL, which we found awkward (especially if doing all platforms; the subscriber build just does the popular ones).

So it's available; albeit there is an intent to have a subscriber build with some extra features that are heavily enterprise biased in their usefulness.

</comment></comment></comment></comment><comment><USER11> Any one else noticed a striking similarity to PHP - every feature is broken somehow?)

I thing this will be a good slogan - 'We are PHP of storage engines.'

<comment><USER12> If by 'broken' you mean 'has taken over the internet'...

<comment><USER13> if by 'taken over' you mean 'pooped all over'

</comment></comment></comment><comment><USER14> One of the more useful Mongo articles I've seen here. You might want to clear up "You cannot shard a collection over 256G" however. The limitation is that if you have an unsharded collection that grows over 256GB you cannot make it a sharded collection. The way it's written now makes it sound like sharded collections can't grow over 256GB (at least to me) which isn't true.

<comment><AUTHOR> Thanks - I can see how that reads wrong, updating now.

</comment></comment><comment><USER15> Good to see some constructive advice on how to configure mongo, instead of just bashing it.

Even if it's not your favorite technology, sometimes you end up in a position where the rest of the company is using something, and you need to work within those constraints. It's important to understand the technologies you're building on, their configuration options, and to understand the best practices way of working with them.

This, by the way, is not restricted to mongo.

<comment><USER16> totally agree!

</comment></comment><comment><USER17> OP knows his stuff. I met him at a hackday and learnt an insane amount from talking to him at dinner. I'm keeping this post bookmarked for reference. Great stuff.

</comment><comment><USER18> I see the "32-bit vs. 64-bit" issue appear in many rants about MongoDB. There are two types of people that fall off the 2GB cliff. a) People who say "what just happened... oh, I get it... 32-bits, memmapped files... I'll switch to 64-bit" b) People who say "WTF.. #MongoHate.. going to blog about how @#$#%! a DB this is"

Some people understand the tools they work with. Some people know just barely enough to throw things together and don't tolerate it when something doesn't work out of the box. Worst of all, this second group tends to be very vocal on the interwebs.

I'd almost like to see 10gen not publish the 32-bit package at all. Source is still there. If you want 32-bit, cool, compile it. But forcing the user to compile the 32-bit version assures at least a minimum bound of technical proficiency (an "I understand what I'm doing, why it's not the default and what the limitations are").

<comment><USER19> It's interesting, because from my observation a lot of the crowd who popularised systems like mongo _were_ those people who weren't willing to think* . Learning the relational data model + tooling was too complicated. Now Mongo has a big ol' list of caveats you ought to understand before you can start chucking data into it too.

I'm a total RDBMS nerd, and it's amazing to me how few people truly care about their data storage. They just want it to work - and, I suppose, it's hard to blame them for that.

*Not that I mean to say that this is the only reason to use a NoSQL DB - it doesn't seem an uncommon one, though.

<comment><USER20> I have to admit that a lot of the joy I get from using MongoDB is during dev.

While you still have to think about your schema, it does mean that you're not constantly writing and removing migrations (rails), while an application is still evolving.

<comment><USER12> In Symfony2 (php) migrations are created by comparing the new schema with the old one. Is that not the case for Rails? What do you mean 'constantly writing migrations'?

<comment> grey-area Rails does not specify the mapping of models to database schema, so it requires specification of migrations instead to document changes to the database that go along with any code changes. So migrations are explicit commands (in a pretty simple dsl) to add columns etc. spread out over many files as the application evolves. This means each schema change requires adding a migration file with those changes in it, rather than modifying a master schema or mapping. There is a schema.rb file but it is created/modified automatically.

There are trade-offs to each approach but it is probably one of the areas that Rails could still improve by looking at other ORMs - I'd prefer to see the schema specified along with constraints etc for each field at the top of each model to make it explicit and self-documenting, and perhaps doing away with migrations altogether.

</comment></comment></comment></comment></comment><comment><USER21> "However due to the way voting works with MongoDB, you must use an odd number of replica set members."

So what happens if I have 2 sequential failures? Suppose I have a replica set of size 5 and the master fails? The remaining 4 would elect a new master from amongst themselves, right? But then what if this next master also fails? The remaining 3 nodes are still a quorum (3 > 5/2) and thus (theoretically) should be able to elect a master. But am I to understand that they won't be able to do so?

<comment><USER22> As I understand it, those 4 still think there are 5 nodes in the set (just that one is down) so you can still establish majority voting because the set size is 5.

</comment></comment><comment><AUTHOR> If I've missed anything from the article, feel free to let me know! :-)

<comment><USER23> This is one of the first articles I've seen where someone with a lot of knowledge posts a lot of realities about Mongo, thank you for that!

I'd love to read something describing the "perfect use cases for Mongo" from you :)

<comment><AUTHOR> I've got a post like that coming up soon; 'thought process and reasonings behind choosing a datastore'

</comment></comment></comment><comment><USER24> Here is one to add to the list - if you delete records and/or entire collections, you won't reclaim the associated disk space automatically. Once the space is allocated, it remains allocated and will be reused when more data is added later. If you want to reclaim the "empty space", you need to run a repairDatabase() which will lock the entire database while it's busy.

<comment><AUTHOR> or compact, which you can run on a secondary...good call - missed this one! Will add shortly.

</comment></comment><comment><USER16> It's interesting to get a rundown of Mongo's limitations from someone who clearly knows what they're talking about. Thanks.

<comment><AUTHOR> Thanks dude :-)

</comment></comment><comment><USER25> _The solution is simple; use a tool to keep an eye on MongoDB, make a best guess of your capacity (flush time, queue lengths, lock percentages and faults are good gauges) and shard before you get to 80% of your estimated capacity._

Any recommendations for such a tool?

<comment><AUTHOR> MMS (<https://mms.10gen.com/user/login>) from 10gen is pretty good - it's very mongo specific.

I've also used Munin (<http://munin-monitoring.org/>, there is a great plugin - <https://github.com/erh/mongo-munin>), CloudWatch (<http://aws.amazon.com/cloudwatch/>) and various in-house ones as well.

<comment><USER26> I've been using MMS for about a year now, and I've found their agent to be not-so-great. I keep getting random alerts about the agent being down and then up again.

Other than that, it's descent and free!

</comment></comment></comment><comment><USER27> I have recently wrote a similar blog post (same idea but different set of "gotchas" here: [http://blog.trackerbird.com/content/mongodb-performance- pitf...](http://blog.trackerbird.com/content/mongodb-performance-pitfalls- behind-the-scenes/)

</comment><comment><USER28> I have a suspicion that this seemingly popular sentiment about so many people hating MongoDB is untrue.

Or people are careless about what systems they put into production?

Also, awesome article!

<comment><AUTHOR> Thanks! Well, it seemed true enough for me to write the article - there have been a whole load of them on HN recently - [http://www.hnsearch.com/search#request/submissions&q=mon...](http://www.hnsearch.com/search#request/submissions&q=mongodb&sortby=points+desc..).

</comment></comment><comment><USER29> On the same token, albiet a bit off of the trail. Does anyone have any suggestions for effectively storing fields which can contain BIG5 (IE non utf-8) chars in them, but usually do not? IE Email subject lines or senders.

JSON is picky in this regard, and I don't want to convert the whole string to B64 etc encode/decode it going in and out, as I would like to retain regex search capability for the 99% of email titles and names which are not Chinese within mongo from my php application which lives on the front.

<comment><USER30> If you need to store non-UTF8 data, MongoDB has a binary data type:

<http://php.net/manual/en/class.mongobindata.php>

You can't do things like regex searches on binary data, but since MongoDB supports different data types within the same "column", you can just store some as UTF8 and some as binary, depending on whether the string has non-UTF8 characters in it.

<comment><USER29> Thank you for pointing me in this direction, I'll see if I can make this work in the application I'm building. Thanks again for the reply.

</comment></comment></comment><comment><USER31> SSL support is not so easy to set up if you are on Suse Linux Enterprise. There is basically no support for it. And for some reason it doesn't work for me.

But the thing I don't understand is, if people use replicasets, how comes they're not using encryption? It would be easy to sniff data off the instances. But yet, when I search on stackoverflow/serverfault, there are close to no people using SSL with Mongo.

<comment><AUTHOR> Laziness, assumption and the difficulty in supporting self compiled apps over standard packages. Also, most projects I've worked on have either been firewalled or on a seperate internal only network for non-web layers...so it's less of an issue. Also, there is a performance overhead.

<comment><USER31> By non-web you mean a network not accessible from Internet right? I wonder if I should drop SSL as well... not much of a gain from it, as the DB layer is in fact configured as you said.

</comment></comment></comment><comment><USER20> Really great list of gotchas.

I have been using MongoDB for a long time, unfortunately mostly this has been small applications, so you don't really get to test how MongoDB scales.

On that same note, I would love to see a list of gotchas for Riak (assuming some exist). I keep hearing recommendations for Riak, it would be nice to know how it fares in a large production environment.

</comment><comment><USER26> Here's two:

(1) There's no need to add a "created" field on your documents. You can extract it from the _id field by just taking the first 4 bytes.

(2) If you are storing hashes (md5 for example), you might want to consider storing them as BinData instead of strings. Mongo uses UTF-8 so every character will be at least 8 bits whereas you can get away with 4 bits per character.

<comment><AUTHOR> Great points but they are not really gotchas; I was trying to help people avoid big / 'obvious' / documented things when using MongoDB :-)

<comment><USER26> Sure :)

</comment></comment></comment><comment><USER27> I wrote an article comparing disk space usage between MySQL and MongoDB with some notes about RAM requirements and data compression here: [http://blog.trackerbird.com/content/mysql-vs-mongodb-disk- sp...](http://blog.trackerbird.com/content/mysql-vs-mongodb-disk-space-usage/)

</comment><comment><USER32> Thanks for the great article. A gotcha I have come across: Document keys can't contain the dot character. If you are storing a complex document (hash-of- hash-of-hash-etc..), you would need to recursively clean up and ensure that none of the keys contain any '.' char.

</comment><comment><USER33> "For setups that are sharded, you can use 32-bit builds for mongod" - I don't think this is accurate. Whether or not you are sharded has no effect on the limitations of a 32-bit mongod. Did you mean to say that you can use 32-bit builds for the mongos?

<comment><AUTHOR> I did indeed, typo; they don't store any data so usually don't have the same issue with 2G limits. I've updated the article.

</comment></comment><comment><USER34> I believe you mean 'sharding' not 'sharing': "Unique indexes and sharing"

And

"Process Limits in Linux" If you experience segfaults under load with MongoDB, you may find it's __beacuse __of low or default open files / process limits

</comment><comment><USER35> Bottom line: MongoDB is not an RDBMS and you shouldn't try to use it as an RDBMS. Something with trying to fit a square peg into a round hole. MongoDB requires a different mindset and if you're unable to adapt, then you should simply stay away.

<comment><AUTHOR> It's not really this at all; the point is - read the docs, research and understand the tools you are going to use. Choose the ones which fit best. Understand the trade-offs.

<comment><USER36> no the point is , while its "api" is great , you cant replace your RDBMS with a mongodb , while other solutions like redis or couchdb are "minimalist" they are better suited for what nosql db are for , high availability and scalling.

<comment><AUTHOR> Unfortunately that's not true: it depends directly on what you are doing with your RDBMS - there are many work loads which are better suited to MongoDB. Obviously there are also many which are not.

Unless I'm out of date, Redis and High Availability don't go together in the same sentence; awesome as it is, it's still a single point of failure.

<comment><USER37> > Unless I'm out of date, Redis and High Availability don't go together in the > same sentence; awesome as it is, it's still a single point of failure.

Clustering is a work in progress (<http://redis.io/topics/cluster-spec> , <http://redis.io/presentation/Redis_Cluster.pdf>), replication is available (<http://redis.io/topics/replication>).

</comment></comment></comment></comment></comment><comment><USER38> Also worth mentioning that performance is much more predictable when the data fits into memory (or the working set, but that may be harder to convey).

</comment><comment><USER39> I stopped reading when you said up to 1tb of data like that was a large number.</comment>


Microsoft Debuts a Windows Rival to Raspberry Pi - <AUTHOR> http://mashable.com/2014/07/29/windows-raspberry-pi/ <comment><USER2> To say this is a rival is a bit overstated. It costs $299 versus the Pi's $35. Sure, it has more features (1GB RAM, QC CPU, etc.) but the price tag immediately puts it in another class.

<comment><USER3> Yup, not only is it in another class, but there are at least two classes between them (eg. the $~50-~100 boards like the BeagleBone Black or the exynos 4xxx series up through the $~100-~200 range of the imx6 and exynos 5xxx), etc.

But to be fair to Microsoft, the purpose of this board isn't really anything like the purpose of the Pi, so comparing the two is really just flawed to begin with. It should really be compared to boards like tge Arndale ( [http://www.arndaleboard.org/](http://www.arndaleboard.org/) ) and in that context the price is actually quite reasonable even if you value the the Windows image license at $0.

</comment></comment><comment><USER4> 8.3x more expensive then the PI. Sure its several of orders of magnitude more power then the Pi's weak ARMv6 chip, but jeeze... Also the fact that it ships with secure boot (which can be disabled easily-ish[1]).

Also they suggest kernel debugging over USB, is this common? I know the Linux Kernel's debug and boot messages are normally dump out a serial port (or can be).

[1] [http://msdn.microsoft.com/library/windows/hardware/dn745910%...](http://msdn.microsoft.com/library/windows/hardware/dn745910%28v=vs.85%29.aspx)

</comment><comment><USER5> Actually it is Intel's child, and can use to develop hardware and drivers for Windows and Android.</comment>


Software - <AUTHOR> Gmail Promotions tag has a mail from Facebook saying I have a message from one of my classmate. I followed the link from email which said click to read it and I was taken to play store at the end to install the messenger app but I didn't.<p>Within seconds I received another mail from FB to review my login activity. Does this happen to users who uses it regularly too? <comment><USER2> Did you type your password of FB? This looks like a phishing scam, but it may be just a stupid app grow hacking. Are you sure that the app that wanted to be installed is the real FB-messenger app?

<comment><AUTHOR> I was already logged in to FB and android/chrome remembered it. I didn't have to type password. Yes, its the real messenger app. I was wondering what is the point of sending review account email in this i.e.after sending the mail about a message and when the user chosen to interact and follow that email to message flow. Maybe its not much.</comment></comment>


High Performance Browser Networking (2013) - <AUTHOR> https://hpbn.co/ <comment><USER2> This is an amazing book.

Almost all software engineers would benefit from a basic understanding of networking. Most networking books are in-depth and intimidating, especially the ones that get into the nuts and bolts of packet formats. This book on the other hand provides exactly what most software engineers need to know.

It costs nothing to try it either.

</comment><comment><USER3> I have the print version since it came out. Required reading for almost all software engineers.

</comment><comment><USER4> Gixy: Nginx Configuration Static Analyzer | [https://news.ycombinator.com/item?id=14315277#14315887](https://news.ycombinator.com/item?id=14315277#14315887) (May 2017)

>Klathmon: _take a look at the book linked at the bottom of that page which includes a pretty comprehensive section on improving RTT and TLS speeds in general_

</comment><comment><USER5> Yup. This is the one I keep under my pillow ;)

HTTP/2, Brotli, TLS 1.3. As well as high resolution perf tools for measuring all kinds of scenarios.

8 Web Performance Expert Insights for 2018

[https://www.cdn77.com/blog/web-performance-expert- insights-2...](https://www.cdn77.com/blog/web-performance-expert- insights-2018/)

</comment><comment><USER6> This book is really amazing, covers basics of 3/4/5G networking as well. I would say this book is mandatory reading for all engineers who deal with networking. It also covers enough in HTTP that you can get by without reading the entire spec.

If I were designing a course I would put the book near the beginning.

</comment><comment><USER6> Worth noting that this is a Progressive Web App (PWA) and you can read the entire thing in airplane mode!

</comment><comment><USER7> I can not recommend reading this book enough. I learned an insane amount.

I find myself using what I learned several times a week. Simply having a slightly more in depth understanding of some parts of networking benfited my thought process hugely when it came to working on problems in that area.

</comment><comment><USER8> Looks exciting!

What are the things I should look out (and around) for when I'm reading this (for example, specs that have changed drastically since this book was written, etc.)?

<comment><USER6> The online version is kept up to date. Ilya keeps a good eye on the Github Issue Queue here [https://github.com/igrigorik/hpbn.co/issues](https://github.com/igrigorik/hpbn.co/issues).

</comment></comment><comment><USER9> I am pretty sure this is the most useful IT book I own.

5/5</comment>


The City That Has Its Own Operating System - <AUTHOR> http://motherboard.vice.com/read/the-city-that-has-its-own-operating-system <comment><USER2> So where do we opt out? How do we keep the X letter agencies from mining all the data? Thinking about this, the whole smart city idea always seems like a game of Russian roulette in Pandora's Box to me. Once we start something like this, there is no way of going back.

I know that the main goal of this project is to reduce Co2 output and use energy more efficiently. See [http://www.greendigitalcharter.eu/wp- content/uploads/2012/05...](http://www.greendigitalcharter.eu/wp- content/uploads/2012/05/Smart-City-Bristol-Programme-April-2012-Briefing- Note.doc)

But why network an entire city? Arguably a smart power grid would be the first option instead of what appears like a wireless network in monitor mode. Although as of right now it appears as if the network is only open for researchers [http://www.theregister.co.uk/2015/04/02/bristol_smart_city_r...](http://www.theregister.co.uk/2015/04/02/bristol_smart_city_reserved_for_boffins_sorry_bumpkins/)

YOUTUBE VIDEO [https://www.youtube.com/watch?v=ROKydMx9EGs](https://www.youtube.com/watch?v=ROKydMx9EGs)</comment>


Ask HN: Personal Operating System for Work? - <AUTHOR> Hey HN, I'm working on an interesting piece of software for my Master's thesis, basically an extensible personal assistant for work. It can do useful things like search across all your internal files, generate notes and reminders from documents, and even attempt to learn your habits to surface useful suggestions and analytics.<p>Would something like this be useful as an open source project or even a commercial SaaS product? Does anything like this exist in industry? <comment><USER2> I am working on the same lines, creating an AI assistant which will pretty much do anything you want to do and you will do. I have been training the model on my UI actions on the computer and trying for my assistant to do the same using reinforcement learning. I haven't reached an appreciable working product yet it works great for me, it can screen cast, type passwords not only in browser but also on consoles. I am constantly adding features to it and I have had the same question as you for a long time.

<comment><USER3> Can you share more information about how it works?

</comment></comment><comment><USER4> Sounds like Clippy or BonziBuddy. Both received overwhelmingly negative reception.

<comment><USER2> Clippy/Bonzibuddy were before the time of deep nets so all you can expect from them is a bunch of predicate logic but not real learning.</comment></comment>


Gallery of Processor Cache effects - <AUTHOR> http://igoro.com/archive/gallery-of-processor-cache-effects/

<comment><USER2> This type of programming is incredibly useful when writing efficient code in low level languages (eg, linear algebra in C). When writing Ruby web apps, though, I haven't found a need yet to think about cache lines or associativity.</comment>


Reading Thoreau at 200 - <AUTHOR> https://theamericanscholar.org/reading-thoreau-at-200/#.WTsHY1LMx74 <comment><USER2> I'm verklempt at the joy of having discovered Thoreau in mid-life. Like many American students, I read Civil Disobedience and Walden in high school and then forgot all about either. Maybe I even absorbed some of the atmospheric disregard for these works. In the past few years, I read all of Dickinson (what a genius), and then all of Emerson's published writing. And, then picked up Thoreau last year. And I haven't been able to put him aside. Walden, all his essays, A Week on the Concord and Merrimack Rivers, his Cape Cod and Maine travelogues, now deep into his journals and letters. Thoreau -- Henry! -- is like this amazing friend. I can hardly imagine anyone who has dismissed him has spent any serious time with him. I have marked thousands of thoughts in his writings as beautiful, profound, puzzling, delightful, or surprising. He's much funnier than most people realize. He had a genius for "the examined life". I really cannot conceive who would be so arrogant as to judge such a person negatively. His prose _is_ American English at its best. I can't really care too much about the posted article but can't resist evangelizing my beautiful friend. If you love to read, read Thoreau.

<comment><USER3> I think that a great deal of the reverence one heard for Thoreau forty and fifty years ago came from careless reading of him. He is a remarkable writer, but with something very cold in his thought; I don't know that his qualities recommend him to the young.

</comment></comment><comment><USER4> Great essay. To me many of Thoreau's points seem like excellent ideas, the difficulty is putting it into practice. Everyone is aware that your work and property is not "you", but in practice, I feel many of my peers, myself included, naturally gravitate toward this behavior. Reading Thoreau is an excellent reminder that I need to have a cautious relationship with property, lest it own me.

</comment><comment><USER5> Do not seek so anxiously to be developed, to subject yourself to many influences to be played on; it is all dissipation. Humility like darkness reveals the heavenly lights.

[http://thoreau.eserver.org/walden18.html](http://thoreau.eserver.org/walden18.html)

</comment><comment><USER6> > In truth, films move them far more; they talk about The Matrix the way my > friends once discussed Hemingway or Kerouac.

It seems the author is talking about how the times have changed, since the context of the sentence talks about how "these days" (things are different from how they were). This sentence is really about the kind of friends that a person would keep, if that person were the kind of person who would become a professor of literature at Princeton.

This may be colored by my own personal experiences interacting with faculty at various universities. While I loved studying and taking classes, there was something about the institutions themselves and the culture that drove me nutty. It was something that most faculty never really understood when they gave me well-intentioned advice.

> The intricate prose of Walden is a tough read in the age of tweets, so much > so that several "plain English" translations are now marketed.

This caricature of modern society is one that you'd find perfectly preserved if you sent yourself 50, 100, or 150 years back in time.

The point about Thoreau is well-taken, but putting aside Ayn Rand for a moment (did he choose her to maximize controversy?), the popularity of writers is always fickle.

<comment><USER7> > fickle

Historical figures regularly vacillate between hero and goat. The Founding Fathers are currently heroes, and Edison is a goat.</comment></comment>


What Petraeus and Broadwell could have used instead of Gmail - <AUTHOR> http://www.washingtonpost.com/blogs/worldviews/wp/2012/11/13/what-petraeus-and-broadwell-could-have-used-instead-of-gmail/?tid=socialss <comment><USER2> There are plenty of secure email applications, but Petraeus couldn't use any of them.

In other words, Petraeus could not be _seen_ using those applications. Official secure email goes through official secure channels. Personal email went to through gmail / yahoo / etc.

If someone had found out Petraeus was using a different email app (and I'm sure they would have), it would have immediately raised suspicions as to why the Director of the CIA was using a secure email app that wasn't the gov's official secure email app.

I think the _safest_ thing for Petraeus would have been to only send short coded messages via gmail (e.g. "talk at 10 pm" but replace that text with code words) and then to conduct most of the communication infrequently and in person.

But, clearly, Petraeus wasn't thinking clearly ...

<comment><USER3> This is exactly it. For Petraeus, even a secure, encrypted email solution wouldn't have solved the problem. Because he would have been seen to have been using it. Even something that encrypted the emails in-transit would have been a problem, because then the FBI still would have discovered a large number of emails b/w him and the mistress. Even if she was a biographer of his, the volume of email (and timestamps) would probably have been a good clue that there was something else going on.

</comment></comment><comment><USER4> Frankly, their solution was good enough and could have escaped detection indefinitely, if only they had stuck through with the discipline required. It was an emotional, not a technical failure that exposed the whole thing.

<comment><USER5> Ironically, it was the same technique pioneered by "terrorists",

"An even more secure method of communication is that which involves using one- time anonymous public email accounts; two terrorists who wish to communicate to open 30 anonymous email accounts whose usernames and passwords are known by each side. To communicate, one terrorist creates a web-based email and instead of sending it, saves it as a draft online. The "recipient" then logs onto this account, reads this message and deletes. - [http://en.wikipedia.org/wiki/Internet_and_terrorism#Communic...](http://en.wikipedia.org/wiki/Internet_and_terrorism#Communication_.26_Networking)

</comment></comment><comment><USER6> It's funny how almost anything that Petraeus could have done to keep this under wraps would have made it look like he was actually committing high treason and supplying national secrets to unknown foreign nationals.

Encrypted code words to burner phones, encrypted emails to unknown recipients, using non-standard apps, using non-standard phones, using multiple phones/laptops/internet connections - almost any type of non-standard communication would've looked like he was committing high treason.

Affairs are like that, huh?

<comment><USER7> What I don't understand is all the attention to technical risks of exposure. Ultimately his physical activities would surely have raised suspicion anyway? I find it hard to believe that the CIA and FBI rely solely on intercepting electronic communications to catch spies.

<comment><USER8> Intelligence officers sometimes have clandestine meetings, even in DC. Sec. Gates had at least one such meeting with then President-Elect Obama, before it was decided he would remain SecDef in the Obama administration.

As for FBI and CIA catching spies, I point to Robert Hanssen, who was brought down by his own arrogance. As ironic as it sounds, he was brought down by a system he devised while working counter-intelligence at the FBI. Hanssen came up with a system for ranking foreign operatives by what they have access to, not by their rank. When they were searching for the KGB mole, they checked who could have accessed the compromised reports. In some cases it was actually a small number of people had the access, including Hanssen.

</comment></comment></comment><comment><USER9> The best thing Petraeus and Broadwell could have done was to either split up with their respective spouses and join each other openly or to forget about the whole thing.

<comment><USER10> I realize that HN is not the place to go if you're looking for people who can give good relationship advice, but an affair that comes about due to genuine mutual interest and is the seed for a strong, lasting relationship is pretty much non-existent outside of fiction. Either the original marriages were open to begin with, or there are other problems, either with the individuals involved or with the marriages themselves.

This is one of the more interesting things I've read inspired by the scandal: [http://www.outsidethebeltway.com/reconsidering-the- petraeus-...](http://www.outsidethebeltway.com/reconsidering-the-petraeus- hagiography/)

<comment><USER11> > an affair that comes about due to genuine mutual interest and is the seed > for a strong, lasting relationship is pretty much non-existent outside of > fiction

What are you basing this on?

<comment><USER10> Gut, mostly. I haven't seen an effective study on the matter, so there isn't much else to go on. The terms I'm using are also badly operationalized.

Substantially, all I'm saying is that affairs don't work like your standard relationship: they're always contextualized by the existing relationships.

</comment></comment></comment></comment><comment><USER12> They should have been careful to access the Gmail account _only_ via TOR. One slip, and the game would be up.

Actually, using Gmail was a reasonable COMSEC plan for what they were doing. Much safer than using his official email.

Problem was, she couldn't restrain herself from frightening Jill Kelley into calling the cops. Once the FBI got on the case, Gmail was worthless. They'd see the unencrypted texts and get to correlate and trace the IP accesses, leaving no hope for the communicants to remain anonymous.

Ultimately, this was a case of a general and a major letting their privates call the shots.

<comment><USER13> Accessing anything from TOR in his position would be silly, because that sort of activity is likely to be picked up on many computers he might use. Why have to explain why you're trying to hide your tracks?

I would buy a cheap tablet or an iPod touch or something, use it only at public wifi spots, only ever use it for this purpose, keep music or whatever on it as a cover, and wipe the device as frequently as possible.

Not foolproof, but reasonably practical.

<comment><USER12> I thought about that. But consider, if you are on a book tour or your job as DCI takes you from place to place. Investigators will map the location of each access point you used. Airports, coffee shops, hotels, whatever. Then they'll match their map up with the various possible suspects. They'll look at security cam video and inspect hotel guest lists and airline passenger manifests. Bingo!

Thus, you need to be logging on from IP addresses that have no connection to you or your whereabouts. Maybe if you could arrange to VPN into a Russian office that does not keep logs and laughs at FBI subpoenas. Even then, there's the danger they traffic-analyze the data in and out of the VPN server.

</comment></comment></comment><comment><USER14> If I were in this position, I'd have run an IRCd from static boot media. I don't think Petraeus should've been fired for being unable to keep his pants zipped, but I do think that a CIA employee who doesn't know how to cover his tracks properly when having an affair needs to be shown the door ASAP.

<comment><USER15> He was not fired. He resigned [1].

1\. [http://www.usatoday.com/story/news/nation/2012/11/09/david-p...](http://www.usatoday.com/story/news/nation/2012/11/09/david- petraeus-cia-resign-nbc/1695271/)

<comment><USER9> That's known as keeping the honour to yourself.

It means that you've become a liability to the organization that you serve so it is better to leave out of your own accord than to be thrown out. This is the quickest way to kill the storm. Imagine what it would have been like if this had been drawn out over many months, it's bad enough as it is.

Resigning looks so much better than being thrown out so if someone has messed up but on the whole has done an ok job (or in the case of Petraeus even a pretty good one) then giving them the option to bow out rather than to be tossed under the bus is a good thing. For everybody involved.

<comment><USER15> Resigning is honorable, but he might have kept his job if he made a public apology. The American people are generally quick to forgive public figures whose sins have come out and then confess. Lance Armstrong, on the other hand, is not likely to recover his image.

</comment></comment></comment></comment><comment><USER16> As far as I can tell both Hushmail and Tigertext are US based or have US operations. As such they can be subpoenaed (and this probably applies to other countries the US has law enforcement agreements with). Throw in a "matter of national security" to keep it secret and they would eventually have found out what is going on.

</comment><comment><USER17> The folks planning 911 used this same technique of using drafts to send messages back and forth. I'm certain that these are now always saved permanently for law enforcement use to comply with CALEA. [http://en.wikipedia.org/wiki/Communications_Assistance_for_L...](http://en.wikipedia.org/wiki/Communications_Assistance_for_Law_Enforcement_Act)

<comment><USER18> I don't think CALEA applies to email. Am I wrong?

</comment></comment><comment><USER19> Hushmail? really? <http://en.wikipedia.org/wiki/Hushmail>

</comment><comment><USER20> _Agents were able to use "digital forensic techniques to determine the person behind the keyboard at the time the emails in question were sent,"_

How does this work? Does this refer to tracking IP addresses accessing the account? And is this information provided by Google?

</comment><comment><USER21> They would share messages using un-flown pigeons in the same location of Washington d.c.

<comment><USER22> Finally a use for RFC 1149

<http://tools.ietf.org/html/rfc1149>

</comment></comment><comment><USER23> > What Petraeus and Broadwell could have used

Their brains?

<comment><USER24> PrivateSky - <http://privatesky.me>

</comment></comment><comment><USER25> No mention of <http://tormail.net/>? Would be hard to top its degree of anonymity.

</comment><comment><USER26> S/MIME and an email client like Thunderbird?

</comment><comment><USER27> I sort of wish the news reporter would do their own damn job and ask a security consultant.

</comment><comment><USER28> They could have just used GnuPG over pretty much any e-mail service?

<comment><USER16> That doesn't really help as GnuPG encrypts the message contents, but doesn't hide sender and recipients. In this case they could still have figured out who sent the messages even if they didn't figure out the contents. Then you apply traffic analysis to proceed to the next step <http://en.wikipedia.org/wiki/Traffic_analysis>

The only way they could have avoided detection using GnuPG is if they did something like always sending the same length message at the same time every day.

<comment><USER29> They left messages in the draft folder - no traffic analysis possible, at least from the messages themselves.

<comment><USER16> The message I responded to was proposing an alternative to what was used (draft folder) and instead using GnuPG.

However even if they only used encrypted text in the drafts and never sent them, you could still use traffic analysis techniques to help categorize the content (size, frequency etc) assuming the drafts could be read.

</comment></comment></comment></comment><comment><USER30> first the trampire, now betraeus lol</comment>


Show HN: Writt – Make and publish notes from Telegram - <AUTHOR> https://writt.glitch.me <comment><USER2> You're offering a product for 19/yr, but will this product exist for that long? Lack of privacy policy, hosted on glitch, and you're constantly throwing things up on there.

Things noticed: SSL is not enforced, therefore your payment page is being served plaintext. Telegram already has this service embedded for free. You're using a free host, with no custom domain, what if they decide to disappear next month without notice?

<comment><AUTHOR> Yes, it will.

Just enforced SSL - And if the free embedded service you're talking about is telegra.ph, that is different than writt because you can't write and publish within telegram, you have to use a web interface.

Glitch is made by FogCreek, the guys who co-founded Stack Overflow, and have built Trello and other stuff - so pretty sure they're reliable [https://glitch.com/faq#trust](https://glitch.com/faq#trust)

<comment><USER2> That doesn't change my view on this. You're paying Glitch _nothing_, yet are asking for 19/yr for this product. At the very least, a few users converting can get you a quality VM from Linode or Digital Ocean. Who's currently doing backups? You? Or Glitch.

<comment><USER3> Glitch wants apps like this to succeed, so will work with him to make sure everything stays running. And Glitch apps are just totally standard Node apps, so worst case he can run it anywhere else. Anything can happen, and there are no guarantees, but I wouldn't worry about the choice of platforms as a risk.

Source: I'm the CEO of Fog Creek, which makes Glitch, and we're excited about apps like this one!

<comment><USER2> I suppose. I'm just a bit cynical because Glitch is full of these "apps" that feel like a train wreck waiting to happen. jajoosam has a number of services on your platform now, and it's hard to tell if it'll get any updates or support 6-9 months down the line.

<comment><USER3> Sure, but that's true of apps on any platform. May be worth a try to state the same concerns about iOS or Android apps and see what the CEOs in charge of those platforms say, then make the call on which you trust.

</comment></comment></comment></comment></comment></comment><comment><USER4> This sounds potentially useful. I mostly used (for more than a year) Telegram for ad-hoc interactions at technical conferences. (E.g. EuroPython 2016 & some groups I interact with at FOSDEM 2018 used it, so I "had" to use it.). And right now, it's sitting dormant since February.

[Slightly off-topic & potentially opening a barrel of worms ahead; and this has been discussed to death here.]

Now ... I have been using (also for well over a year) the _Signal_ Messenger. Given Signal's funding model, their crypto protocol (which was peer-reviewed & got robust positive feedback -- corroborated through its adoption by WhatsApp & Facebook Messenger) and their (almost) zero metadata collection, I found it's better to invest my energies in Signal.

So, FWIW, I have been trying to "converge" on Signal and slowly persuading (without forcing, of course) friends to use it; I've had good success so far. A lot of my colleagues & a decent chunk of people I collaborate with in the open source community already use Signal. So that's not a problem there.

This is all just to make an attempt at maintaining sanity with the Chat Tool proliferation. Wonder how others are handling it.

<comment><USER5> My exact same route and experience, although I heartily detest Signal's _phone number required_ misfeature and the agressively forced contact sharing.

<comment><USER4> Yes, you raise a difficult point about the _phone number required_ feature. And you're at least the second person I hear (legitimately) complain about it; the other was from a technical friend, during an in-person conversation.

I realize that requiring phone number compromises the anonymity aspect (despite Signal's unparalleled privacy, given the alternatives). But I was giving (and still do) Signal the benefit of doubt and assume they're also going to implement username / password scheme – not sure if it's a good idea or not.

</comment></comment></comment><comment><USER6> Telegram has a similar service called telegra.ph - free of charge. It doesn't support written markdown, but has a built-in WYSIWYG editor.

And your credit card form is accessible over http.

<comment><AUTHOR> Writt is different from telegra.ph because you cannot write posts to telegraph from telegram, you have to use the web interface, which makes Writt's use cases very different, even though some of them might coincide

</comment></comment><comment><USER7> You don't even have a domain and are already charging $19 for "unlimited" notes? I like the app, it's not a bad idea, but the business model is not great.

Just storing notes isn't a great feature, there are thousands of note-taking apps out there. On Telegram itself you can just send messages to yourself and they will be stored, editable and deletable.

<comment><AUTHOR> Hmmm... What would you recommend instead?

<comment><USER8> FOSS with donations. I'm not sure if people would want to pay to use this, but if it looked nice and was FOSS then people would probably donate.

<comment><USER9> agree because paying 19$ for this product is not worthy. as telegram already provides telegra.ph which is free of cost. FOSS with donations is a perfect monetisation strategy.

<comment><USER10> > FOSS with donations is a perfect monetisation strategy.

Only if your monetization goal is less than a few bucks per month.

That's some serious Shit HN Says. Thanks for the chuckle.

<comment><USER8> > Only if your monetization goal is less than a few bucks per month.

It's a service that exists solely through a platform that already provides a competing service for free. A few bucks per month is as good as it's going to get.

</comment></comment></comment></comment></comment></comment><comment><USER11> Interesting. Tried it on some very simple example text and it seems well- executed. I'm intrigued by the concept of chatting to write. I wonder if the medium can stimulate thoughts in a way a traditional text editor can't.

Great work, and good luck!

<comment><AUTHOR> Thank you

Indeed, notes in the form of multiple texts and chatting could make things more interesting.

<comment><USER11> Reminds me of Tap by Wattpad. That could be a great market for you.

[https://taptaptap.co](https://taptaptap.co)

</comment></comment></comment><comment><USER12> Why is telegram suddenly so popular? AFAIK it doesn't provide proper decentralised cryptography and yet everyone in the crypto coin community is up in arms about it.

</comment><comment><USER13> Nice. Do I need it though that's the question....

</comment><comment><USER14> I like how this works. Need to use more extensively to understand cons.

</comment><comment><USER15> but I cannot edit the text?

<comment><AUTHOR> Editing with a web interface is in the pipeline, for now you can rewrite by using the $ sign. /help for details.

</comment></comment><comment><USER16> Isn't Telegram compromised by the Russian government..?

<comment><AUTHOR> No, they have refused to provide the encryption keys to Russian security agencies.</comment></comment>


Startup Mixer in Cambridge, MA on Friday 9/10/210 - <AUTHOR> http://anyvite.com/j1rbbpizi7 <comment><USER2> Looks great! Are you going to have more? I can't make it this week. You're the first Boston Properties building across from the MIT Press bookstore, right - just before Google (5 Cambridge Center).

BTW, I just checked out your site - Panjiva is a really cool idea, I want to try to use it for nothing in particular. It's so interesting that I feel I ought to try to find someone to fabricate some machine parts thousands of miles away _just because I can_! : )

<comment><AUTHOR> Yeah, we have something like this a few times a year. That's our building. Feel free to invite friends to this since you can't make it.</comment></comment>


Tesla's Summon-Your-Car Feature Spurs U.S. Safety Inquiry - <AUTHOR> https://www.bloomberg.com/news/articles/2019-10-02/tesla-s-summon-your-car-feature-spurs-u-s-to-ask-about-safety <comment><USER2> I worry that Tesla's self-driving ambitions put their entire mission (sustainable transit) and products at risk. Being conservative just isn't in their DNA—and you NEED that DNA to work on autonomy.

Advanced Summon is a good example of this problem. You might say it needs to be deployed to get the data needed to refine it. But they could have spent more time silently collecting data, comparing what the computer would do with what real drivers do. And they could have at least staged the rollout to happen much more slowly.

I just don't think they should have released it this widely in the current state. It's disingenuous to label it as beta and ask users to uncover the kinks with real risk to life and property.

<comment><USER3> Electric cars may be less polluting, but that doesn't make them "sustainable transit". It's a band-aid on the many problems caused by car-centric development, which does nothing to solve the fundamental issues:

[https://www.donkey.bike/wp- content/uploads/2018/02/autonomou...](https://www.donkey.bike/wp- content/uploads/2018/02/autonomous_vehicles_space_problem.jpg)

<comment><USER4> If the same car can do 20 trips on its own that graphic looks completely different.

<comment><USER5> Because now 20 cars will hold less than 20 people on the average. 20 cars on the road with many of them autonomous may only have 15 or maybe only 10 people.

<comment><USER6> It's a good point, but counting the taxi driver as increasing efficiency is disingenuous.

<comment><USER5> Taxis are an invalid alternative.

The truly "Green" alternatives are electric scooters, bikes, busses, and subways.

Electric scooters seem to be the winner in point-to-point movement within a city, although we're going to need to figure out proper safety regulations moving forward (scooter speed limits, road vs sidewalk, etc. etc.) Although still immature, the explosion of scooter-based inner city travel gives me a lot of optimism to that methodology.

20 people on 20 scooters will be more space efficient, cheaper, and environmentally friendly. They move slower than cars, but seem ideal for urban point-to-point travel under 2 miles. Apparently, Segway had the right idea (but the tech wasn't mature enough. Modern Li-Ion batteries + Scooter rentals + Scooter Ride Sharing apps didn't exist in the 00s, but today make it a reasonable technology to use).

\</comment></comment></comment></comment></comment></comment><comment></comment><comment>-

Given a good enough bus network, scooters should work for Point-to-bus-to- point travel, or Point-to-subway-to-point travel, since they're small enough to carry in a backpack.

<comment><USER6> I think this is just wildly out of touch. Yes, a family of 4 with two small children, getting around on buses and scooters carried in their backpacks.

<comment><USER5> Children in my area are largely carried around in school busses. I couldn't drive in High School and relied upon walking to the bus stop.

There are unorganized groups ("groups of friends") who travel together from bus-stop back into their neighborhoods. This is just walking. And I don't think I grew up in a very urban area.

\</comment></comment></comment><comment>---

Scooters are helpful when walking further distances for personal city travel. But... when it comes to school, busses are extremely efficient at the job. Probably the best at the job: a large group of people are getting to the same location at roughly the same time.

</comment><comment><USER7> The feature is not ready. The only reason Tesla pushed it out this early is that they want to recognize some of the Autopilot deferred revenue in Q3 to mask a YoY decline in revenue driven by lower ASPs, which damages their growth story.

<comment><USER8> Damaged growth story? Tesla delivered more cars this quarter than ever before: [https://www.cnbc.com/2019/10/02/tesla- tsla-3q-2019-productio...](https://www.cnbc.com/2019/10/02/tesla- tsla-3q-2019-production-and-delivery-numbers.html)

Lower ASP is to be expected as people are buying many more (and increasing) Model 3 than Model S/X.

<comment><USER9> Selling more cars is great, but YoY reduction in revenue will not justify the valuation of TESLA stock.

<comment><USER8> Depends how you define "reduction in revenue." YoY growth in units but reduction in per-unit profit is often a very good sign. See Apple when they introduced the iPod mini. Average sale price went down but number of units continued to increase.

<comment><USER9> Total revenue.

Apple did not make less total revenue when they introduced a lower-priced iPod. The claim above was that Tesla would have had a reduction in total revenue without being able to book pre-sales for this half-baked feature.

</comment></comment></comment></comment></comment><comment><USER10> Even at low speed, Tesla seems to still have trouble avoiding stationary obstacles.

</comment><comment><USER11> All of the videos I have seen of the summon feature have been pretty bad. A lot of time the vehicle can't seem to figure out what to do. I imagine that will get better with time, but if I had a Tesla I wouldn't be using it now.

I'm surprised they released the feature as-is.

<comment><USER12> If you use it with a little common sense, it works great. I just got the software update today and tested it out in a parking lot that had minimal foot traffic, worked perfectly.

</comment></comment><comment> mitasu-hachi Does the summon-your-car feature use the same technology that full self- driving robotaxi mode will use when it's released in under a year?

</comment><comment><USER13> Not sure why anyone is surprised anymore by Tesla releasing unfinished features and then polishing. That's literally what they do.

<comment><USER14> We're the 'training data'

</comment></comment><comment><USER15> [https://news.ycombinator.com/item?id=21141149](https://news.ycombinator.com/item?id=21141149)

[https://news.ycombinator.com/item?id=21105388](https://news.ycombinator.com/item?id=21105388)

</comment><comment><USER16> I could be wrong about this but it seems the way autonomous driving works is that it needs a ton of real-world failures to learn from. There seems to a mostly-lay-person that there is no way around getting it out there and having people test it out in very odd parking lots with lots of babies toddling around it.

This is the process of evolution, can it be artificially simulated in a way that makes this unnecessary? Can we evolve without people dying or having sex?

<comment><USER17> Do it in a controlled environment or simulation, there's no reason or excuse to risk real world babies.

<comment><USER16> But then how do you train it for real world situations where there's real world babies running around?

</comment></comment></comment><comment><USER18> [https://outline.com/V89rFR](https://outline.com/V89rFR)

</comment><comment><USER19> It's sad. They will force regulators to act and it will affect legitimate companies working on Self-Driving. It seems most of other companies working on self-driving understand there is a lot of responsibility in pushing half-baked product on the street.

This is why we cannot have nice things.

<comment><USER20> Do you remember the time when Uber run over a pedestrian?

It's pretty reasonable to regulate 2 metric tons steel bodies moving among fragile creatures.

Beta tests where bugs mean dead people aren't cool.

<comment><USER19> I would definitely not put Uber in the list of responsible ones. As far as I know Waymo and Cruise for example are taking security extremely seriously.</comment></comment></comment>


Flight rules for Git - <AUTHOR> https://github.com/k88hudson/git-flight-rules <comment><USER2> It never ceases to amaze me that:

1\. Git, a software product with more footguns per square inch of user-exposed surface than any I've ever encountered, ended up winning the distributed VCS sweepstakes; and

2\. Even now, more than five years after it was clear to everyone that Git had won, there is absolutely no momentum to go back and clean it up to _remove_ any of those footguns. All we get instead are increasingly shiny GUIs to hide them under, and well-meaning blog posts advising you on which is the best kind of tourniquet to apply when you trigger one of the aforementioned footguns.

Sigh.

<comment><USER3> It's basically a manifestation of programmer arrogance that they're just too damn smart to encounter a software learning curve again.

Ever sat through user training on microsoft office? How to use email? The weird one-off enterprise apps that nobody else has ever heard of?

They're unbearably tedious to users that can get by in a new tool with a 5 minute crash course, intuition and light googling.

Then git shows up and has the audacity to ask professionals to learn its mental model.

Once you do the interface isn't great, but not particularly esoteric. The problem isn't that users see dense-sounding things like "directed acyclic graph of revisions", it's that they assume they can competently manipulate one without knowing what that means.

I do think the interface could use cleanup, but can any ammount of cleanup prevent needing to know the mental model of git?

<comment><USER4> Bullshit. The "mental model" has never been the issue with git, it's always been that the UI is garbage and a giant abstraction leak, requiring knowing the implementation details of the commands and the storage model to work with and step over footguns. Because rather than an actual, designed, considered UI, the porcelain is crummy shortcuts on sets of operations on the underlying storage (not mental) model.

Mercurial has the same "DAG of revisions" model as git, and works just fine, because it has a good UI.

Darcs has a more unusual mental model than git, and works just fine, because it has a good UI.

<comment><USER5> The biggest leaking abstraction of git IMO, are commits: they equate a set of _changes_ with the state of the repo _after said changes_. So if there are multiple commits, reverting a commit is ambiguous. Does it revert the state of the repository to the state before the commit, or does it revert the changes in the commit?

There are a couple more flaws. The commandline is not very consistent, and the complete model is way too complicated IMHO. I can work with it now, but it took me a long time to grasp all of the concepts that I need - without feeling the need for distibuted version control. About that: It's kind of ironic that git is hailed for being distributed, but everyone just uses github, a centralized system.

<comment><USER6> Just to be pedantic, a commit is actually "the state of the repo after said changes" _plus_ "the commit(s) before said changes" (which allows one to recover the "set of changes" by comparing the before/after states).

<comment><USER5> I don't think that's true. At least, not if you revert something - it reverts only the changes introduced in that commit. (That there is confusion - or even discussion possible - is exactly my point: the abstraction is leaky at worst, and confusing at best).

Edit: You're right, but my point was that the revert command treats a commit like a set of changes, not like the state of of a repo (since you can't revert the state of a repo, but you can revert changes).

<comment><USER6> See for yourself: do a "git cat-file commit HEAD" on any repository. The "tree" line is the current state, the "parent" line(s) are the preceding commit(s). That command shows everything git has recorded for any commit (you can similarly use "git ls-tree <sha1>" to show everything git has recorded for any tree, or "git cat-file blob <sha1>" to show the contents of a file from the tree).

<comment><USER5> You're right from a technical perspective. What I meant that _as an abstraction_ , it serves a double purpose (it represents both a node and an edge in the DAG, and it depends on the command that you use which interpretation is right).

If you checkout a commit, the commit represents the state of the repository. If you revert a commit, the commit represents a set of changes.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> Everyone talks about how confusing Git is, and I think it's because they do'nt understand what the commands do. Don't get me wrong: you can royally screw up your repository with the wrong command, but 85% of your issues can be solved with checkout, pull, add, commit, push, merge, reset, status, stash, branch. When things get really heavy you may have to use rebase. And in spite of what people say, there is a nifty log that's perfect for revisiting a snapshot of your CLI's action: reflog.

When I first started using Git, I was told, "When you want to make you commits public, use `git push -u origin <branch>` where branch the branch you're pushing to." That advice is technically correct, but I had no idea what -u nor origin was. I took 5 minutes to decipher the command. I walked away understand that origin an alias for the remote server, and that -u associates a remote branch with your local.

I don't think Git is the greatest thing ever. I do think there's a bit of a learning curve. I also think that it's not as complicated as people make it seem. This post is a great reference for how to do X in Git, but it's a terrible source for beginners learning how to use Git. Cheat sheets are most effective when you have a basic understanding of the material.

<comment><USER2> _> Everyone talks about how confusing Git is, and I think it's because they do'nt understand what the commands do._

If nobody understands what the commands do, the problem is _with the commands,_ not with the people.

<comment><USER8> It's not this simple. Git's porcelain is very unintuitive, but the underlying model is not. Once you understand the model, memorizing the commands is a relatively minor annoyance. But even if you fixed the porcelain there is essential complexity in understanding and utilizing the underlying model. To further clarify would require simplifying the model or features exposed, reducing the power of git as a tool.

There are those who say you should never rebase as a form of simplifying git, but people who follow that workflow have much _much_ messier histories than those who know how to rebase properly. A well-curated history of well- explained atomic commits has _incredible_ value, far outstripping the cost of maintenance on a long-lived project.

Git is a professional tool for software engineers, it has severe short-comings for non-coders, but as a software engineer learning it well will magnify your value.

<comment><USER2> If the Git developers have made an explicit decision that it's impossible to simplify the interfaces without losing some important functionality, hey, it's their software, that's their prerogative. But if that's the case, then people who have trouble understanding the interface shouldn't be called dumb, or chided for "not understanding the model," because it means _the model is going to be too complicated for some people to understand._

Either Git is intended to be a tool anybody can pick up and use, in which case it's shameful how thoroughly ignored widespread complaints about the UI have been; or Git is intended to be a tool for a select priesthood of Enlightened Masters, in which case it shouldn't be promoted to the unwashed masses. You can't really have it both ways.

 _> Git is a professional tool for software engineers_

Lots of tools for professionals of all kinds take the time to protect their users from inadvertently harming themselves through misuse. Cars for commercial drivers aren't sold with a set of wheel chocks in place of brakes, and if you argued that they should be because they're meant for _professional drivers_ who _ought to know better_ than to walk away from their vehicle without setting the chocks first, you'd earn yourself a lot of funny looks.

</comment></comment></comment></comment><comment><USER9> I started version control with CVS. Just a bit better than copying.

I started distributed version control with monotone. That felt better but the central db felt a bit unnatural.

A bit later came mercurial and I was really happy with it. It all made sense. Easy interface and when you wanted to do something strange like rewrite history, you had to use a plugin.

Then came git (rhymes to better know your sh*t). I'm not surrounded by the smartest guys but most people I met only have a superficial understanding of git and looking at these flight rules kind of confirms the complexity of git.

I strongly believe that 80% of the current users of git would be better off with mercurial. But then everybody uses git for the remaining 20%.

<comment><USER10> Mercurial doesn't have branches, right? That drove me _bonkers_ when I was doing a bit of contract work after having used git for awhile.

edit: They must have been using a different SCM; ignore me!

<comment><USER11> Not only that it has branches -- it has four different ways to branch. [http://stevelosh.com/blog/2009/08/a-guide-to-branching-in- me...](http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/)

<comment><USER4> Of note: Git has 3 of those:

* git has clones

* git has bookmarks, that's "normal" git branches

* git has anonymous branches, it's called "detached head" and is completely broken

* git _does not_ have named branches (that's the branch name being part of the commit's immutable metadata)

<comment><USER12> I've seen plenty of git users put "[JIRA-1234]" at the beginning of a string of commit messages. That's basically poor-man's mercurial named branches.

<comment><USER13> Isn't that what a tag int Git is for?

<comment><USER11> A tag applies to an individual commit, and is unique (i.e. you can't have the same tag on multiple commits); a named branch is applied to all commits in a branch and persists after merging. In the example mentioned by @krupan, the users tend to (and some team even require it) prefix each commit with the ticket name or similar identifier.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> One of the reasons git is "hard" is because too many people read crap like this, which rapidly gets outdated and often contains more bad advice than good. Read the manual! It's not dry at all, and you'll learn a lot. Pick a random git command and read its man page right now.

[https://git-scm.com/docs](https://git-scm.com/docs)

Read the git book, too. It's free and serves as a good introduction and helps you learn which man pages you need to consult and when. You don't need to read the whole thing, but the first 3 chapters are indispensible.

[https://git-scm.com/book/en/v2](https://git-scm.com/book/en/v2)

<comment><USER15> Can you give some examples of how git has rapidly changed?

</comment></comment><comment><USER16> A simpler guide, perhaps, made by an incredibly talented former co-worker of mine: [https://ohshitgit.com/](https://ohshitgit.com/)

<comment><USER17> This looks great, thanks!

</comment></comment><comment><USER18> Reminds me of the git man page generator joke site. [https://git-man-page- generator.lokaltog.net/](https://git-man-page-generator.lokaltog.net/)

</comment><comment><USER19> This only briefly touches on revert commits, and instead shows a lot of (in my opinion) dangerous history-destroying commands.

Why aren't reverts more widely used? It seems a lot of devs I've worked with don't understand them, and to me they seem so simple and are so much safer than destroying/rewriting history.

<comment><USER20> I will explain a common problem with revert commits:

1\. Developer writes some code in a branch.

2\. Developer merges code into master before the code was actually ready to be deployed.

3\. Developer reverts the merge commit in master to roll back everything.

4\. Developer fixes the code in their branch.

5\. Developer merges their code into master.

6\. Their code doesn't show up in master, since it was already merged in and the revert is part of the official history of the branch in master.

7\. Developer has to make a revert of the revert and then untangle the mess.

<comment><USER19> I can understand that, but I still would prefer that over the alternative of changing history, losing context at why the revert happened (or that it happened at all!), and having to dig through reflog and hope that the machine that it was done on is available and hasn't dumped the code to be able to recover the commit needed.

In your scenario, it's a puzzle to unfuck, but it's unfuckable after a bit of work. In the alternative you run the risk of losing work, or blowing out other's commits and causing work for multiple other team members.

I guess my point was that I feel "revert commits" should be treated as the default, and the goto in most situations, and only using something like `git reset HEAD^ --hard` when you really need to, or a revert won't do. I'll be honest, I sometimes use a "reset and force push" when I've fucked up an ugly merge, or I want to undo the last few commits at once, but it's the exception, not the first tool I reach for.

</comment></comment></comment><comment><USER21> This is interesting - I like looking at corner cases of a tool to understand it better.

For people having difficulty with git, I recommend going through the official book [0] on a couple of topics with a toy repo and play with the tool a little bit. Another alternative is to find a simple workflow, and stick with it until you have time to explore git more.

Compared to other tools I've had to use in the past (AccuRev ughh), git is fantastic. Just being able to see what files have been modified without waiting for 10min+ or needing to be connected to a server changes your workflow, but we've probably become accustomed to those benefits, but still notice the arguably crusty command line.

[0] [https://git-scm.com/book/en/v2](https://git-scm.com/book/en/v2)

</comment><comment><USER22> FWIW here is a previous discussion from 4 years ago: [https://news.ycombinator.com/item?id=8102624](https://news.ycombinator.com/item?id=8102624) (29 comments)

</comment><comment><USER23> Any suggestions for quickly doing the following: When I clone a former repo on GitHub, I want to also add the original repo I formed from, as the upstream remote. This way I have the original repo which is mine and the upstream repo which is the original. So far I've been doing this manually.

<comment><USER24> go to [https://github.com/github/hub](https://github.com/github/hub) install hub and run: hub fork user/repo

</comment></comment><comment><USER25> Seeking advice from the git masters :

I joined a new small team several weeks ago. All of us are fairly new to git, coming from cvs and svn backgrounds. We use Intellij as our primary IDE, and bitbucket as our remote origin.

 _Do y'all recommend using the IntelliJ git front end GUI, or calling git from the command line?_

I thought I had a grasp of git from some side projects on GitHub, but now that it's a key dev tool in our daily process, the more I learn about it and need to extend beyond simple checkout/commit/push workflow, the deeper the rabbit hole goes.

<comment><USER26> Learn the git cmd prompt first.

Use a text editor eg. Code to fix merges.

Git is powerful. And can have a hard learning curve, but don't have to. Learn it in CMD.

I mostly use

status push pull commit -a add checkout checkout -b branch list merge

Gitk and git GUI usually comes with most installations and works for Mac Windows Linux. Then any other UI is just sugar.

</comment></comment><comment><USER27> IMO, by blindly follow these kind of cheatsheet, you will hard to get better with Git.

I used to remember command in certain situation and have a hard time understanding stuff like stash, rebase, cherry-pick, detach head etc...So whenever come up I have to sit google the heck out of it.

Then one day I decide to sit down and just read Git properly. They are on: [https://git-scm.com/](https://git-scm.com/)

Eventually I understand it better and now I know what I need to do in certain situation.

</comment><comment><AUTHOR> I also highly recommend Learn Git The Hard Way. Most developers who spend an afternoon with it will come away with a firm understanding of git's core abstractions.

Book: [https://leanpub.com/learngitthehardway](https://leanpub.com/learngitthehardway)

</comment><comment><USER28> I was surprised there is no mention of git bisect... an incredibly useful tool.

</comment><comment><USER29> Isn't the first rule, never talk about Flight rules?

</comment><comment><USER30> If git confuses

Read chapter 10 of pro git

Mourn enlightenment

</comment><comment><USER31> that remember this so much [https://xkcd.com/1597/](https://xkcd.com/1597/)</comment>


Red programming language switched from markdown to asciidoc for documentation - <AUTHOR> https://twitter.com/red_lang/status/817720495214727168 <comment><USER2> I wanted to switch from markdown to org-mode since it has great tool support ;) but maybe asiidoc is a better choice?

</comment><comment><USER3> Can someone explain the significance? I'm eagerly awaiting Red btw.</comment>


Scaling Continuous Integration: How We Cut Test Time by 77% - <AUTHOR> http://www.crashlytics.com/blog/scaling-continuous-integration-cutting-test-time-by-77/

<comment><USER2> Looks like they also need to work on scaling their blog.

Google cached version: [http://webcache.googleusercontent.com/search?q=cache:www.cra...](http://webcache.googleusercontent.com/search?q=cache:www.crashlytics.com/blog/scaling- continuous-integration-cutting-test-time-by-77/&hl=en&sa=G&prmd=imvns&strip=1)

<comment><USER3> Whoops - yes, you're right. Our blog server is completely separate and apart from any of our other servers. It originally only had 1GB of ram, and we're in the middle of resizing it to 4GB. Thanks for the Google cached link! :)

Update: And the blog server is now back up and on a much faster machine.

</comment></comment><comment><USER4> Looks like they are redirecting their blog to their root website (in beta?).

<comment><USER3> We redirected it to our main site while we resized our blog server. Resize completed and the blog server (and post) is now up! :-)

</comment></comment><comment><USER3> I remember when this problem was solved. Saved a ton of time.</comment>


Ask HN: Has anyone had success with "Canada Start-up visa"? - <AUTHOR>

I wanted to know if anyone here has success with "Canada Start-up visa" program.[1] and if so can you share your story and experience here?<p>[1] http://www.cic.gc.ca/english/immigrate/business/start-up/index.asp <comment><USER2> Comments moved to [https://news.ycombinator.com/item?id=9179838](https://news.ycombinator.com/item?id=9179838).</comment>


Learning Python for Social Scientists (2015) - <AUTHOR> https://nealcaren.github.io/python-tutorials/ <comment><USER2> One of the reasons R has been popular among social scientists is that you have libraries dedicated to statistical methods that are less common in Python. Examples include Mixed Effects Models, Hierarchical Linear Models, Psychometrics/Measurement Theory, Structural Equation Modeling, Survival Analysis, and a wide range of other techniques that shine when you have small amounts of data and lots of practical/physical barriers around data collection and experimentation.

Even for methods such as linear and generalized linear regression, there's just a range of libraries and diagnostic capabilities in R that aren't present in Python libraries.

Granted, a lot of these statistical methods can be expressed in Stan or some other probabilistic programming engines that have interfaces in Python. However, when you are a social scientists and not a statistician there's a big difference between having a few functions you can call and expressing a likelihood function.

<comment><USER3> In social science research, "getting the model right" is often the goal (i.e explaining _why_ something happens), and "making good predictions" is of secondary importance (i.e. good predictions only matter insofar as they indicate that your model is good). That's one reason you end up doing involved probability modeling, compared to just slapping it with machine learning like you'd do in an industry setting. The flipside is that, as you said, many social scientists don't have the programming chops to do it themselves.

The other big advantage IMO is that it straddles the space between "traditional stats package" (SAS, Stata, SPSS, Minitab, etc.) and "programming language". We here know that it's more or less the latter, but the "batteries included" standard library with first-class support for a "data-frame" type makes it feel much like the former, especially if that's what you're already familiar with. It's a gradual transition from a stats package to general- purpose programming, and you don't have to jump through hoops, install third- party packages for basic stats work, or deal with ugly/counterintuitive syntax.

<comment><USER4> A problem with Python is the lack of a "best practices" style for scientific computing.

I find it amazing that Linear Regression in scikit-learn doesn't return standard errors/t-scores for the coefficients. (These can be useful for model selection, for example). I know about statsmodels and pysal, but the workflow with these is already slightly different.

<comment><USER3> _I find it amazing that Linear Regression in scikit-learn doesn 't return standard errors/t-scores for the coefficients_

Case in point! You don't typically need those for machine learning, so why bother with them in a machine learning toolkit/library?

<comment><USER5> You don't need them if you don't need to diagnose your analysis. But the hard part is of course thinking about why your model doesn't work, which is the strength of R.

</comment></comment></comment></comment></comment><comment><USER6> > Social scientists don't really analyze images much, but that might be the > next big thing.

Satellite images have been used in research in economics to estimate poverty and deforestation.

<comment><USER7> That all depends - there are some geography departments that do a lot of data driven analaysis, as do some economists. There really is a lot of overlap - with various disciplines doing data driven work. Would also expect that criminologists might be doing similar things.

</comment></comment><comment><USER8> Link to very first notebook by Julia Evans is dead.</comment>


Render Pipelines design in C++ - llorens-marti http://www.marti.works/render-pipelines-desing-in-c/ <comment><AUTHOR> This is a somewhat simplified version of what you will see in a AAA graphics engine (pretty much all of them have abstractions around draw passes/stages and pipelines). The weakness in the approach here I think is that it won't handle temporally aliased things particularly well and also, I've found that actually render pipelines are more of a pipe"tree" with dependent objects becoming inputs for more than one pass for various things. Then you throw in multithreading into the mix and start to understand why a simplistic API like this gets complicated really fast in real world settings.

</comment><comment><USER2> There is nothing new in Abstraction of passes and pipelines. We need to think about practical complications also.

</comment><comment><USER3> I would be curious to see what anyone with more than my armchair experience in graphics thinks of this.

<comment><USER4> This post is an example of armchair programming, so your armchair experience is more than enough.

On a more serious note, it's software architecture done on paper. Any experienced programmer will tell you it's close to worthless. You don't have to be a graphics expert. Abstraction of passes and pipelines isn't anything new, it even bled into the modern APIs like Vulkan. This post is just some inexperienced programmer's vague description of how he or she would like things to work in theory. It's not worth a discussion.

<comment><USER5> I think the pipelines in this posts mean something different. As Vulkan/Metal pipelines refer to the GPU rendering pipeline with vertex, fragment shaders, tesselation etc not a sequence of passes.

</comment></comment></comment><comment><USER6> > This way we can describe that the Camera will have a relation of 1:1 with a > Render Pipeline.

If all you're making are smartphone games, this might work [1]. But for more complex scenarios, this sort of thing would be a non-starter. You ideally need to support N:1, Cameras to Pipelines. Not just for N=2 for VR, but N=3 for VR+spectator mode, and N=4+ for projection mapping to surfaces as well.

[1] I say "might" because smartphones are getting more powerful and will probably very soon (as in 2 to 3 years, i.e. about when such a new project would be usable) be decent VR platforms.

</comment><comment><USER7> Doesn't this approach incur a lot of cache misses?

<comment><USER8> Why? All these data structures are mostly just handles to the actual data, which lines in the GPU memory.

</comment></comment><comment><USER9> The concept in this blogpost anticipates a more general architecture. Graphics pipelines are a good fit for dataflow programming, such as FBP [0] or reactive [1]. When you use those styles, the pipeline architecture is more flexible, more straightforward to run concurrently, while the coding challenge turns into one of how to allocate and address different assets, address different _types_ of assets, create and address temporary data, and describe the node graph connecting these things together.

The practical complication in rendering can be stated like this: combine assets A and B and parameter P to make asset C, then (subsequently) combine asset C with asset D and parameter J to make output O. Some of the assets/parameters change with each frame rendered, others are entirely static, and oversight over GPU time and memory usage is needed, so you have to consider which things are loaded when. Then, while in production, a design change necessitates that a new parameter be added somewhere, and it turns out that you have to reorder which things are processed first and add a custom codepath because one of your target GPUs doesn't support the feature you need without a hack.

[0] [https://en.wikipedia.org/wiki/Flow- based_programming](https://en.wikipedia.org/wiki/Flow-based_programming) [1] [https://en.wikipedia.org/wiki/Reactive_programming](https://en.wikipedia.org/wiki/Reactive_programming)

<comment><USER10> Why not simply use a software cache (memoization, [1]) to prevent doing things more often than necessary?

[1] [https://en.wikipedia.org/wiki/Memoization](https://en.wikipedia.org/wiki/Memoization)

<comment><USER11> All kinds of caches are used in graphics programming. The devil is in the details: Does a generic memoization strategy help you decide when to re-render a shadowmap based on the updating of entities within range of said shadowmap? And when you decide technically incorrect, stale results, are still 'good enough' (based on the duration they've been stale, distance from the camera, size of the resulting shadows, etc.) how do you represent that? How do you decide when shadowmaps can been reused between multiple perspectives? How much can you calculate offline, before the game is even run? If you squint really hard you might be able to couch some of this in memoization terms - but how, exactly, is that helping?

Memory is tight - GPU memory might be a few gigs, and your artists _will_ fill the space. Time is tight - 16ms/frame _max_ if you're targeting a smooth 60fps, and even higher framerates (and tighter time budgets) are recommended to e.g. reduce nausea for VR. You know exactly how long you need some resources, they are large (over 100MB for a single set of 4k gbuffers), and spilling active data out of the cache can be disastrously slow, or lose necessary information you cannot recreate.

At some point you'll get rather hands on and start micromanaging a lot of this stuff - it's much simpler to write, debug, and optimize a lot of this if you implement it 'by hand', than to design, write, debug, and optimize an algorithm trying to handle a perfectly generic answer/solution to all these problems.</comment></comment></comment>


Nikon and Verily team up to fight diabetes-related eye disease - <AUTHOR> https://www.engadget.com/2016/12/30/nikon-and-verily-diabetes-eye-disease/ <comment><AUTHOR> Google Research recently published a paper in the JAMA on using Deep Learning to detect these same eye diseases: [http://jamanetwork.com/journals/jama/article- abstract/258876...](http://jamanetwork.com/journals/jama/article- abstract/2588763)</comment>


Mathematicians Encrypt Images Using Mathematics of Sudoku - <AUTHOR> http://www.technologyreview.com/view/428639/mathematicians-encrypt-images-using-mathematics/ <comment><USER2> This seems silly. They "use Sudoku to tackle a different problem--how to encrypt images before sending them".

You know what else works for encrypting images before sending them? Any other encryption algorithm.

Who cares if your "encrypted" image is still an image? Doesn't that kind of defeat the point of encryption? It's obviously just a scrambled image, leaving only the question of what the unscrambled image is. If someone was planning on sending illegal or sensitive images with this, you could probably still easily guess what the nature of the original image was.

It's not like steganography where you don't know there's a message there, or normal encryption where you have no guess as to what produced the cyphertext or what the payload is.

<comment><USER3> Yes - I couldn't believe it when I read the article. This is like - "Create any random permutation matrix. Use it to permute all the pixels. Done."

Or, use any encryption algorithm and set the key to be your Sudoku grid.

</comment></comment><comment><USER4> I'm not sure why the authors are concentrating on 'Sudoku' (which would perhaps better be called a Latin square in this context) because I can't see that they need the property that each 'number' appears once per row and column.

If they want in place image scrambling using grids, can't they just generate a random grid that mixes up the pixels in a square from any (x,y) to some other (x',y') in the same square? They can still arrange for this to be a bijection, it's trivial.

Perhaps they could Google 'permutation matrix'.

The interesting thing is how you generate the grids, and for that to be secure you'd need some cryptographic method. In which case you might just as well encrypt in the first place.

<comment><USER5> I'm certain that the Sudoku part is a hook, to bring in more viewers and then keep them for a bit. And although the site is called technology review, it is basically just a general audience site so concepts like permutation matrix would be lost on the audience and massacred by the time pressed journalists. This comes down to a trade off.

You want to draw people in and show them how cool modern tech is, but if you make it too much like a lecture you don't reach as many people. I think this trade off is okay, people who are attracted and care will unlearn and those who don't care will forget but still have a general idea of science as awesome. E.g. I used to read NewScientist although now I see much of it is garbage. That stuff can't be _much_ more damaging than non-hard scifi and comics IMO. The only site that makes me angry is ScienceDaily and not for their horrid site design.

</comment></comment><comment><USER6> Color me extremely skeptical until I see a rigorous security proof. Fun little schemes like this are a dime a dozen, and often are essentially useless for any real application.

<comment><USER7> FTA: "Wu and co make no claims about its potential security but there is clearly room for further exploration here. "

<comment><USER6> They need to be more careful about using the word 'cryptography' then. Sure, crypto people and informed laypersons know this is hopelessly insecure, but all it takes is one overzealous intern implementing this in a banking application to cause a huge problem.

<comment><USER7> The only word I see is "encrypt", but regardless, this IS cryptography - converting to a code. You can encrypt with a Ceaser cipher, though it's not terribly secure.

</comment></comment></comment></comment><comment><USER8> Since a sudoku puzzle implies it's solution, it is a highly compress(able) representation of the solution. I wonder if it might be possible to leverage this sort of rule based matrix to improve compression algorithms, or if compression algorithms are already more optimal than this.

<comment><USER9> That was my thought as well (see my comment elsewhere in the thread). I wonder if this could be used as some kind of fixed sparsity encoding?

<comment><USER10> I'm no expert but just an thought:

A sodoku puzzle(requiring only 17 digits for an unique solution) which has an single solution isn't exactly useful for compression.

How I am thinking about this? The article states:

"And since for a 256 x 256 image, there are at least 256!=2^1684 possible Sudoku matrices, it's not easy for an adversary to hit on the solution by accident or even by brute force. "

Compression would make sense if the number of puzzles was something like 1684, but it is here 2^1684!! Which is quite a lot and to be decoded would be even worse than the image itself.

The sodoku puzzle's compression would not be of any practical use not only because they give the data more problems to account for but also because the data would need to be interpreted.

Also anything that uses encryption generally does the complete opposite of compression.

<comment><USER8> For sure it increases the computational power required to decompress, but that does not preclude its usefulness. Sometimes computation is cheap and bandwidth is expensive, as perhaps when sending messages to distant stars.

</comment></comment></comment></comment><comment><USER11> This is really cool to me. But it makes sense. It just uses a unique numbering system. I suppose instead of adding a new tens or thousands placeholder to increase the number, you just add a new grid reference. Pretty cool really.

edit: I guess it's basically a base-9 numbering method, yes?

</comment><comment><USER12> If I'm understanding the technique correctly, it seems very likely that you could use statistical properties of natural images to guess at partial unscramblings, progressively rebuilding the image in far better than brute force time.

<comment><USER9> A bit of Googling also reveals that some Sudoku puzzles are invertible, so it sounds like in addition to this problem, you'd better pick your sudoku solution carefully, as it might be possible to recover the key if the number of candidate images is small.

</comment></comment><comment><USER13> One could exploit the continuity of the image regions. (The similarity of neighbors in intensity and colors). Also the output is an image with the same histogram ...

</comment><comment><USER14> Wow this is quite stupid for this application. When I read the title I thought along the lines of compression by 'solving' the sudoku.</comment>