A Year to The Day - <AUTHOR> http://blog.ghost.org/a-year-to-the-day/ <comment><USER2> Is there anything about Ghost that, intrinsically, makes it better than other blogging platforms?

Seems to me it's just reinventing a many times reinvented wheel (except this one is hip and popular).

<comment><USER3> From an end-user perspective:

1\. It's easier to use than WordPress, Drupal etc. That's because it does much less than those platforms on purpose. The WordPress admin area has become a scary place for new users, particularly once they start to add plugins that litter the admin menu with extra entries.

2\. The admin area is fast. Moving around feels slick – it is a desktop-like experience; hitting Command+S will save the draft you're working on, for example, rather than bringing up your browser's "Save Page As..." prompt. The Markdown preview is as-you-type, and they use the screen estate very well. It's a comfortable environment to write long posts in, and it hasn't been tacked on as an afterthought, like WordPress' full-screen editing mode.

3\. The organisation behind Ghost, the "Ghost Foundation" is a non-profit. It doesn't have investors or shareholders; it is answerable only to its users and funded by those who pay for official Ghost hosting.

4\. Pricing for official Ghost hosting is very reasonable ($5/month for 1 blog, $14/month for 5 blogs).

5\. Although it's still in development, the proposed Dashboard looks far more useful than competitors. WordPress' Dashboard often looks like a wasteland once a few plugins have been installed – a mish-mash of "Upgrade to the PRO version!" prompts and RSS updates from the WordPress feed.

And from a developer perspective:

1\. GitHub-based workflow.

2\. MIT licence instead of GPL.

3\. Less code debt (for now!).

4\. Handlebars-based theme templates that enforce separation of presentation and logic. (It will be possible to add custom handlebars tags via the Ghost Apps API in the future for more advanced stuff, but it's already pretty solid.)

5\. The official hosted version allows you to upload your own themes (unlike wordpress.com, for example).

As a developer and end-user, it's already a more enjoyable experience than competing platforms for simple blogs, and I think by the end of the year – once the app API has launched and developers are adding forum apps and more – it will be a real competitor for more complex sites too.

<comment><USER4> _disclosure: I 'm a former founder of a successful WordPress hosting platform and spent many years in the WP space._

I'll take the opportunity to counter, if I may...

 _1a. It 's easier to use than WordPress, Drupal etc. That's because it does much less than those platforms on purpose._

A key stumbling issue with Ghost is writing in Markdown - most non-tech people use WordPress's WYSIWYG editor because they don't understand markup as a concept. That's a BIG barrier to entry that Ghost needs to address or remain niche.

The other big barrier to entry with Ghost is that it's super difficult to install compared to WP - requires Node.JS which doesn't run on most shared accounts (where most people host blogs). The average non-technical person shouldn't be trying to manage a VPS or server.

Yes, there are hosted Ghost services but those cost money, hosted WordPress starts a $free on wordpress.com.

 _1b. The WordPress admin area has become a scary place for new users, particularly once they start to add plugins that litter the admin menu with extra entries._

Most plugins don't abuse the addition of menu items - most of the ones that create a new root-level menu item do so because they have sophisticated/deep enough configuration that requires it. I'm not sure that added menu items makes WordPress particularly "difficult to use" especially if the alternative is that functionality isn't available in the first place.

 _2\. The admin area is fast. Moving around feels slick – it is a desktop-like experience; hitting Command+S will save the draft you 're working on, for example, rather than bringing up your browser's "Save Page As..." prompt. The Markdown preview is as-you-type, and they use the screen estate very well. It's a comfortable environment to write long posts in, and it hasn't been tacked on as an afterthought, like WordPress' full-screen editing mode._

WordPress's /wp-admin/ can be super fast - that's down to where you are running it. Drafts are saved automatically and with browsable history so the need for Command-S is removed.

 _3\. The organisation behind Ghost, the "Ghost Foundation" is a non-profit. It doesn't have investors or shareholders; it is answerable only to its users and funded by those who pay for official Ghost hosting._

Materially, that's no different to the WordPress Foundation.

 _4\. Pricing for official Ghost hosting is very reasonable ($5 /month for 1 blog, $14/month for 5 blogs)._

Pricing for official WordPress hosting is even more reasonable - free. But also has a clear path to scalability via various Managed WordPress Providers if needs be.

 _5\. Although it 's still in development, the proposed Dashboard looks far more useful than competitors. WordPress' Dashboard often looks like a wasteland once a few plugins have been installed – a mish-mash of "Upgrade to the PRO version!" prompts and RSS updates from the WordPress feed._

The "Upgrade to Pro" part is a bit of a straw-man argument. I'm sure there must be some WP plugins that do that but not many, and any widget on the Dashboard can be easily moved or removed. Users can customize their dashbord with what they want, and a reasonable counter-perspective would be that a small RSS feed of WordPress news and articles isn't a negative.

\---

At the end of the day, debating WordPress vs Ghost is like debating Python vs Rails or Debian vs Centos. Differently philosophies - Ghost is certainly more focused on attractive features for developers, WordPress is focused on mainstream adoption. Makes sense given their relative positions in the market.

I think HN gives WordPress a hard time, applying niche values like code elegance and utilization of the latest technologies and ignoring in plain sight the fact that both of those are actually WordPress's advantage. Anyone can create a plugin, and while many of them are utter shite, there is an amazing breadth of quality ones out there. And PHP, while derided mostly on HN, actually is one of the most prevalent languages in terms of install base and suitability to low-resource shared-environment mainstream use.

A key strategic issue Ghost will need to deal with is that the use-cases for "blogging" have polarized over the years. People run a blog either as part of a wider site - which is where WordPress wins given it evolved into a site CMS. Or blogs are written to be part of a content network - which is where Medium and Subtle are playing where you get a "Ghost" like experience but the distribution network to go with it.

<comment><USER3> I agree with you – WordPress is often criticised within the tech community. That's because its philosophy has always been to optimise for end-user happiness first and developer happiness second. If you think something in WordPress is crazy – such as supporting PHP 5.2 – the reason is normally that it may be worse for developers, but it is much better for end-users. As soon as I understood and accepted that, I had much more fun with WordPress.

But I think the criticism in the tech community is also a cry for a strong alternative. And we have so few good ones today. That's why I'm happy to champion and celebrate anyone taking on an entrenched platform. If nothing else, Ghost and competing platforms will encourage WordPress to become something even better.

Some responses to your comments:

 _Right, so what happens when your needs grow and you need functionality that extends outside of the envelope of Ghost?_

You'll install Ghost apps, just as you install WordPress plugins like Advanced Custom Fields if you want to bend WordPress into a more feature-complete CMS. It's too early to say whether Ghost will succeed at creating a better add-on system, so perhaps this particular debate is premature. The challenge will be in creating an API that allows developers to bolt-on features without also enabling UI cruft and the sort of performance headaches and conflicts you see with WP plugins, which don't enforce sane practises like namespacing (thanks, PHP 5.2!).

 _Ghost may be "easier to use" but one thing that is difficult with it over WordPress is it's a lot harder to install - requires Node.JS which doesn't run on most shared accounts (where most people host blogs). The average non- technical person shouldn't be trying to manage a VPS or server._

I agree with you – at the moment the install process is too hard for non- technical users and a burden for technical ones. But the official Ghost hosting platform is inexpensive, easy to get started with, and happy to accept custom themes, so there is little reason for most users to look elsewhere. Those who want to host a blog themselves already have several alternatives that don't require sysadmin skills – WebFaction has a Ghost installer, for example, and they charge shared-hosting prices ($9.95/month). I suspect we'll see other suppliers who offer Ghost-only hosting to bloggers with lots of traffic, just as WP Engine and others have done with WordPress.

 _Most plugins don 't abuse the addition of menu items - most of the ones that create a new root-level menu item do so because they have sophisticated/deep enough configuration that requires it. I'm not sure that added menu items makes WordPress particularly "difficult to use" especially if the alternative is that functionality isn't available in the first place._

The problem is that these menu items can be created in at least three different areas – under Settings, under Tools, and at the top level. This makes things harder for users because they have to remember where the menu item of the configuration page for that plugin they installed four months ago is. I use WordPress every day and the menu system is a giant game of hunt-and- peck.

The menu setup also makes things harder for developers because they have to make the right choice of where their menu item will live. A great many get it wrong. (They will frequently place items at the top level that should have been under Settings or Tools.)

Additionally, the menus API invites games of Competitive Menu Trumping, where developers fight to get their menu item above those of other developers. Automattic are guilty of this – install their Jetpack plugin and you'll find a new "Jetpack" menu item right at the top of the menu bar, just below Dashboard: [http://d.pr/i/AZyt](http://d.pr/i/AZyt) This is either a desperate call for attention or an admission that the menu system is broken – that if it was placed under the Settings header instead, the Jetpack settings page might never be found.

Plugin author "WingerSpeed" has even created a plugin named "Slim Jetpack" that functions almost exactly like Jetpack, but removes the menu item and banners from the top of the page. It's been downloaded almost 50,000 times – nowhere near the 10 million or so downloads that Jetpack itself has, but still a sign that menu bar bloat, dashboard adverts, and enforced sign-ins are complications and annoyances that many users would prefer not to have.

 _WordPress 's /wp-admin/ can be super fast - that's down to where you are running it._

I don't think it's just down to hardware – WordPress is not yet a full MVC- style JavaScript app. Side-by-side, it does not feel as fast as Ghost on the front or back end. I just logged into an account I managed with one of the big WP hosts, and it's a huge improvement over how it feels with shared hosting, but if you compare it side-by-side with a Ghost blog it feels slower. Perhaps that's a little unscientific and anecdotal, but raw perception matters. To be fairer, Ghost's perceived speed is largely because it got a fresh start and doesn't carry 10 years of baggage, but also because it does less and makes fewer requests (11 vs 18 for the basic admin page). But the upshot is that it behaves more like a modern front-end JS app than WordPress currently does (even though that's changing) – the perceived performance mirrors the underlying architectural choices.

 _Drafts are saved automatically and with browsable history so the need for Command-S is removed._

Auto-save is great. I agree that this is better for users. My point was that, for writers and users with muscle-memory Command+S twitches like me, the fact that the shortcut triggers a save action instead of a browser menu shows that the app's been developed to feel more like a desktop one. It's a nice touch. (I believe the WP post editor disables Command+S when the box has focus, so it's not like they haven't thought of it – it's just a different approach.)

 _Materially, that 's no different to the WordPress Foundation._

Absolutely. Except that the Ghost Foundation is just the Ghost Foundation. There is no commercial arm like with Automattic – no for-profit company feeding from the work of open source contributors (and giving a lot back, to be fair). Most won't care or appreciate the difference, but it's an important distinction for some.

 _Pricing for official WordPress hosting is even more reasonable - free._

Sure, it's free if you want to use a modified version of WordPress. It's free if you don't need to add plugins or themes. It's free if you don't need to edit CSS. It's free if you don't want to use your own domain. Once you add the various upgrades or one of their (very decently priced) bundles, and once you pay for separate email hosting (which WP.com doesn't include), you'll soon be looking at shared hosting pricing or the headache of migration to a self- hosted setup. It's great that there's a free entry point, but perhaps sometimes that can turn into more of a burden than a benefit. Ghost Pro has a free full-featured trial, for what it's worth.

 _But also has a clear path to scalability via various Managed WordPress Providers if needs be._

Ghost Pro offers a clear upgrade path that scales based on traffic, just as managed providers like WP Engine do: [http://d.pr/i/SWHO](http://d.pr/i/SWHO) If you start with them you can grow with them too.

 _The "Upgrade to Pro" part is a bit of a straw-man argument. I'm sure there must be some WP plugins that do that but not many, and any widget on the Dashboard can be easily moved or removed. Users can customize their dashbord with what they want, and a reasonable counter-perspective would be that a small RSS feed of WordPress news and articles isn't a negative._

It's not a straw-man argument – it's a genuine problem. I started collecting screen shots of the Dashboards and various intrusive banners from Dashboards I've logged into over several years of web consulting and it paints an ugly picture. (I'll try to publish them in the future if I can strip identifying features.)

<comment><USER4> _I just logged into an account I managed with one of the big WP hosts, and it 's a huge improvement over how it feels with shared hosting_

I'm curious, which one were you using?

<comment><USER3> The site I logged into was with WP Engine (not a site I operate; just one I help maintain). It's fast, but there's still a perceivable lag when clicking around the admin area.

I'm not sure if you've played with Ghost much, but I recommend signing up for a free demo account at [http://ghost.org](http://ghost.org) if you're at all curious. Create a few posts, click between them on the contents tab, dive into the (currently very bare!) settings page, and try editing and saving a post – everything feels almost instantaneous. I don't get that feeling with WP, even on WP Engine or WP.com.

</comment></comment></comment></comment></comment></comment><comment><USER5> Just as an aside. If you're looking for something built in Node that's more of a CMS, my Kickstarter for Webhook recently hit it's goal. I really like Ghost as a blog service, but you might like our system if you need to build a more customized CMS experience for multiple editors.

[https://www.kickstarter.com/projects/1749618880/webhook](https://www.kickstarter.com/projects/1749618880/webhook)

<comment><USER6> Why Firebase + static pages instead of, say, varnish? (Honest question)

<comment><USER5> Because people who know HTML and CSS don't know how to set up or manage a database locally. Even worse, they'd have no idea how to port that data onto a live production server. Firebase makes the install pretty painless. Our only dependency is Node. Right now that means even windows users can get up in running in less than a minute.

As a designer I really like the ease of systems like Jekyll, but it was a dead stop as soon as I wanted those sites to be editable by other users. This way we get the best of both systems.

</comment></comment></comment><comment><USER7> > \- Ghost Dashboard

I'm really excited for that one, that was the reason I installed it in the first place only to realise it's not available yet.

<comment><USER8> -> "Very soon you'll be seeing a fresh, slick Ghost(Pro) dashboard"

Is "(Pro)" indicating that the dashboard component will be available only for paying/hosted customers and will not be open-sourced?

<comment><USER9> I doubt it, since the dashboard was a main selling feature of its Kickstarter campaign. A move like that would probably sacrifice any goodwill its garnered among early adopters.

<comment><USER10> I think it is, worryingly for early backers.

[http://blog.ghost.org/introducing- ghostpro/](http://blog.ghost.org/introducing-ghostpro/)

</comment></comment></comment></comment><comment><USER11> curious about what happened from current tagline and differentiating point from wordpress: 'Just a blogging platform.', to the statement here: 'Ghost has the potential to be far more than just a blogging platform' ?

</comment><comment><USER12> Please don't rewrite titles unless they're linkbait or misleading. ("Ghost blogging platform, future." wasn't even intelligible.) The post's title, when combined with the site name, is fine.

<comment><AUTHOR> Got it.

</comment></comment><comment><USER13> Ghost promised to be a simple blogging platform. It's simple if you use it as installed. It's not so straightforward to tweak and upgrade. I'm not very excited about it anymore.

<comment><USER14> They may improve the upgrade process for users. It's also open source, I'm sure they'd love to hear about your upgrading concerns.

On the other hand, there's also the paid service, which you don't have to manage at all.

</comment></comment><comment><USER15> Does anyone have any recommendations for comment systems on Ghost? It looks like they themselves are using Disqus and Jeff Atwood is using Discourse (obviously) are there any other contenders?

<comment><USER16> I saw this one tweeted out yesterday by one of the Ghost devs. Integrated commenting similar to what Medium does. [http://ouija.io/](http://ouija.io/)

</comment></comment><comment><USER17> I'm really glad that Ghost is progressing nicely — I've used it for a small project, and it was a really nice experience.

</comment><comment><USER18> I didn't think another blogging platform was really needed, but their stats and revenue make it sound successful. Congrats, they succeeded where I didn't think it could be done, or would have just hosted WordPress and tweaked it if I was doing it myself.</comment>


Tool Recreating the "Decrypting Text" Effect Seen in the Movie "Sneakers" - <AUTHOR> https://github.com/bartobri/no-more-secrets <comment><USER2> Seriously, this is the smartest, most interesting, and oddly the most accurate movie I've ever watched about computer and network security. And it was hugely entertaining, compelling and exciting!

This movie introduced a lot of people to cryptography, tiger teams and ethical cracking/hacking. Also, it had Sidney Poitier, Robert Redford, Dan Ackroyd and River Phoenix.

One of my all time favourite movies.

<comment><USER3> If you want to do a road trip, the Sneakers office was in the second floor of the Fox Theater in Oakland. No idea where Seatec Astronomy was. Maybe Fremont.

<comment><USER4> IMDb has a nice list of filming locations: [http://www.imdb.com/title/tt0105435/locations](http://www.imdb.com/title/tt0105435/locations)

Was Setec one of the locations? Is that where they did the handoff with the fake FBI folks?

I remember Playtronics was the one large building they broke into.

<comment><USER5> Setec Astronomy is just a code name of the project, not an actual place.

</comment></comment></comment></comment><comment><USER6> Oh, now this I like! Now I just have to 3D print Whistler's custom braille keyboard...

</comment><comment><USER7> He should have published it under a github organization called Setec Astronomy.

</comment><comment><USER8> For the people who want to try this on OS X run `locate ncurses.h` to see if you have the ncurses library. If you have Xcode installed, you should have it in a few places. Then update the Makefile, set `NCURSES_H = {dir}` and it should make and install just fine.

<comment><USER9> brew install no-more-secrets

Works.

<comment><USER8> Sweet, thanks! I'm not sure why I didn't think to try that. Worked for me!

</comment></comment></comment><comment><USER10> 'What did it sound like?'

I loved that film. Loved it.

<comment><USER11> I really like this, too... I have this movie (and others including Antitrust and, yes, Hackers) in my "night time playlist" for when my wife goes back home to visit family for a few weeks each summer and Christmas. It is a very comfortable movie for me.

</comment></comment><comment><USER12> Very cool! My favorite movie.

See also: [https://github.com/jtwaleson/decrypt](https://github.com/jtwaleson/decrypt) although yours much more faithfully recreated the Sneakers scene.

<comment><USER6> I loved that movie too, and I find it holds up far better than a lot of technologically-minded fare from the early 90's. It was one of those allstar casts that really got some mileage from the various personalities and styles, I think.

<comment><USER13> Given the current government war on encryption and the snowden revelations, we might say it's as timely today as it was in 92' if not more

<comment><USER6> It certainly proves the thesis of the film, which is that the "good guys" are not so good as to be trusted with everyone's secrets.

</comment></comment></comment></comment><comment><USER14> Kinda reminds me of what I did at school in about 1980 while dicking around on a Commodore PET 3016.

<comment><USER15> One of my earliest machine language programs was an experiment to see how fast a 1 MHz Apple II+ could count to a million. The code initialized some characters in the display buffer to ASCII '0' and then incremented them directly.

</comment></comment><comment><USER16> Sneakers was just as much about social engineering as it was its technical nature. I love this movie. I think I must listen to is as I work a few times a month.

One thing I took away from this movie is Whistler able to solve pieces while listening. It showed me that I should always listen first and interject my input after :-)

</comment><comment><USER17> Sneakers came out before I was even born but now I feel compelled to watch this old, and perhaps cringe-worthy hacker movie. Hope it does not disappoint.

<comment><USER18> "Cringe-worthy"? Hardly. You owe it to yourself to watch it. It's easily one of the best hacker movies.

<comment><USER19> it's the best one, by far.

<comment><USER20> I'm partial to operation takedown which is a pretty solid representation of Kevin mitnick.

<comment><USER21> No way.

In that movie Mitnick was outright violent towards Shimomura (he wasn't in real life) , along with a few totally ficticious real-life encounters between the two.

Also, this movie depicted Mitnick destroying files and evidence (didn't happen).

The script was leaked during his defense. No good.

I understand movie magic and drama, but what a kicker to be trying to defend yourself legally while a Hollywood production glorifies you as a criminal while simultaneously injecting fiction into the true story.

</comment></comment></comment></comment></comment><comment><USER22> Anyone feel like creating an xscreensaver hack using this?

<comment><USER23> Nah, The Matrix code was the last screensaver we ever needed. Even nostalgia for Sneakers doesn't make the effect cooler. Probably just that the Matrix makes better background noise, tricking your mind into spotting patterns.

</comment></comment><comment><USER24> And of course, the anagram software they used (as mentioned in the commentary) was most likely NAMEGRAM, an old DOS program.

</comment><comment><USER25> Dang I thought it would be the one that changed the characters into a line drawing of a map.

</comment><comment><USER26> Your C API seems to take a null-terminated string without a specified length.

</comment><comment><USER27> I feel suddenly motivated to work on my telnet bbs to put this to use... :-)

</comment><comment><USER28> 'Tis very nicely done.

I wish I had the time or talent to turn this into an actual shell.

</comment><comment><USER29> Doesn't work with screenfetch and neofetch :'(

but... [https://gfycat.com/WanSecretGyrfalcon](https://gfycat.com/WanSecretGyrfalcon)

</comment><comment><USER30> Why are we adding (2016) to current year submissions now?

<comment><USER31> So time travelers feel more comfortable. You know how offended everyone gets these days.

<comment><USER32> Goddam fateless tickers. Why can't they just stay put?

</comment></comment></comment><comment><USER33> I love this. I love the movie. It's geeky and I went on a date to see this movie with a female friend of mine at the time (we weren't girl/boy friend at that time). Now she is my wife of almost 20 years (2017). IT was playing at the Midway Theater in Forest Hills NY.</comment>


ChromeOS: How VM and Containers will seem to run - <AUTHOR> https://www.reddit.com/r/chromeos/comments/742f8j/how_chromeos_vm_and_containers_will_seem_to_run/ <comment><USER2> For context, ChromiumOS is already available on Google Cloud as their recommended Kubernetes host OS. Launching a new GCE instance gives me "Container-Optimized OS 63-10032.71.0 beta (ChromiumOS-4.4.86 Kubernetes: 1.7.8 Docker: 17.03.2)" among the other base images.

<comment><USER3> Oh, interesting. I see "Kernel: ChromiumOS-4.4.86" in the list and some of their docs on it have other cros-isms (/mnt/stateful_partition), and that version 63 lines up with Chrome. cros is sort of a surprising choice of upstream for a GCE thing, but I guess it is a hardened, public Linux kernel that Google already maintains.

Makes everything confusing, though, since there are some commits specifically mentioning VMs and the codename for the Pixelbook (Eve), so something is happening there, but there's also an intersection with a whole other container-related project :/

I wouldn't mind if ARC, the Android subsystem, could live in a VM. Android apps can poke at a bunch of surface area that webpages can't.

</comment></comment><comment><USER4> Is any of this enabled yet (even if only in the dev channels)? Being able to run arbitrary containers locally would make a Pixelbook a lot more powerful.

<comment><USER5> Technically, termux is a container although of a very different kind...

But then again, you don't need a pixelbook to run that :(

<comment><USER6> Same with GNUroot which I prefer over Termux. They both run in a container using a fake Chroot.

</comment></comment></comment><comment><USER7> I always assumed this was why the Pixelbook was so gratuitously over-spec'ed.

</comment><comment><USER8> Being able to run Android Studio on a container would be really nice.

<comment><USER9> You mean like this: [https://github.com/menny/docker_android/tree/master/android_...](https://github.com/menny/docker_android/tree/master/android_studio)

I really hope that ChromeOS's way would be better.

</comment></comment><comment><USER10> Considering Android apps run in containers in ChromeOS,I noticed that apps (media player) doesn't have HW acceleration support. Does the VM and Containers mentioned in OP, offer HW acceleration ?

</comment><comment> X-Istence I'd love to use my HP G1 for something other than browsing Imgur while I am watching the news at night :P</comment>


H-1B Secrecy Veil Torn Off as Trump Steps Up Targeting of Visas - <AUTHOR> https://news.bloomberglaw.com/daily-labor-report/h-1b-end-users-exposed-in-latest-trump-targeting-of-visa-program <comment><AUTHOR> Data is available here as an Excel Spreadsheet (sort by SECONDARY_ENTITY_BUSINESS_NAME): [https://www.foreignlaborcert.doleta.gov/pdf/PerformanceData/...](https://www.foreignlaborcert.doleta.gov/pdf/PerformanceData/2019/H-1B_Disclosure_Data_FY2019.xlsx)</comment>


iMac Pro: Apple releases its most expensive computer - <AUTHOR> https://www.theguardian.com/technology/2017/dec/14/imac-pro-apple-expensive-computer-creative-professionals <comment><USER2> I wonder if they are going to release a Mac Pro too, or if this is it.

<comment><USER3> Yup, was mentioned in the iMac pro presser:

>in addition to the new iMac Pro, Apple is working on a completely redesigned, next-generation Mac Pro architected for pro customers who need the highest performance, high-throughput system in a modular, upgradeable design, as well as a new high-end pro display.</comment></comment>


Must-have Django packages - <AUTHOR> https://devcharm.com/pages/79-must-have-django-packages

<comment><USER3> Interesting, my fairly large django project uses none of those. Instead, the addons we consider to be critical are these these:

        django-debreach
        django-redis
        jinja2
        pytz

Redis for queues and caching, debreach to secure the CSRF tokens and vary your content length to avoid the BREACH vulnerability. jinja2 for doing templates outside of returning data, and pytz so you can use timezones in your data models.

I also recommend installing `flup` and running via fastcgi, but that's open to interpretation.

<comment><USER4> Does django-redis block? I'm not sure I'd make a connection from django to anything other than a low latency queue.

<comment><USER5> I think Redis fits your definition of low latency queue. It does in my book, and I run a few thousand concurrent connections to Redis from Django. I also endorse django-redis and use it for caching, pub sub, and various data types.

See [https://github.com/samuelclay/NewsBlur/blob/master/settings....](https://github.com/samuelclay/NewsBlur/blob/master/settings.py#L584-L597) for my redis configuration. I also use django-redis for caching (which is higher up in that file).

</comment></comment></comment><comment><USER6> Amusingly, my must-have Django packages are mostly things that replace large chunks of Django (Jinja2 for templates, a DB connector/ORM for whatever database the project needs, which even for PostgreSQL is often plain SQLAlchemy).

I guess I'll just use flask.py or something for my next project. This is not meant as criticism, just my personal experience.

<comment><USER7> Yep, having everything built-in is a double-edge sword.

I myself love Django, though I find some parts of it a bit over-engineered (in the Java style). That is changing for the better in recent releases though. I'm thinking the render_to_response stuff and management commands, etc.

</comment></comment><comment><USER4> Django guardian's performance is ridiculously bad. Better roll your own. We're using it and stuck with it at this point, but some of our page loads have thousands of queries because of all the permission lookups. Permissions probably need to be cached in memory.

</comment><comment><USER8> Notable omission of South ( [http://south.aeracode.org/](http://south.aeracode.org/) )

<comment><USER9> I think that's because South has been rolled into Django now. Is that correct?

<comment><USER10> Close, but not exactly, Andrew Godwin, the creator of South, has added support for migrations into the dev branch of Django, to be part of the Django 1.7 release.

[https://docs.djangoproject.com/en/dev/topics/migrations/](https://docs.djangoproject.com/en/dev/topics/migrations/)

Andrew has significantly rethought migrations and applied the lessons he's learned from years of maintaining and developing South.

He's kept a developer diary that has been pretty fascinating to read (YMMV):

[http://www.aeracode.org/2013/11/27/new-apps- migrations/](http://www.aeracode.org/2013/11/27/new-apps-migrations/)

and others at:

[http://www.aeracode.org/category/django- diaries/](http://www.aeracode.org/category/django-diaries/)

[update: route3 is faster than me, but still it's worth checking out the developer diaries]

<comment><USER11> For a greenfield app, would you recommend starting with 1.7, or do you think it will be simple enough to convert from South that it's not worth it? (I have no idea how stable 1.7 is)

<comment><USER10> Andrew has said in the past that his intent is to provide a migration path from classic South to "South 2.0" which will backport migrations to earlier versions of Django (versions 1.4-1.6, I believe) and also build on top of the migration infrastructure in Django 1.7.

You should be fine sticking with South for now.

<comment><USER11> Cool, thanks very much to you and the other commenter.

</comment></comment></comment></comment></comment></comment><comment><USER12> "Must-have" anything always feels like clickbait to me.

<comment><USER13> Eh, Internet is based on clickbaits and lolcats. $ad but true.

</comment></comment><comment><USER14> Cripsy forms, because simple things like adding the necessary markup for bootstrap to forms is super-boring otherwise ; in general, it helps with custom form layout, which is usually a template maintenance hell

[https://github.com/maraujop/django-crispy- forms](https://github.com/maraujop/django-crispy-forms)

</comment><comment><USER15> I've found django-allauth a bit friendlier than django-social-auth, and it covers registration of local accounts in one package.

<comment><USER16> Yep, an interesting comparison is found here: [https://speakerdeck.com/tedtieken/signing-up-and-signing- in-...](https://speakerdeck.com/tedtieken/signing-up-and-signing-in-users-in- django-with-django-allauth)

</comment></comment><comment><USER7> Everyone recommends the django-toolbar for spotting slowdowns in your SQL... ok, installed it. I almost never see any tips on how to actually do that though. What now?

(I suppose the SQL part of my app is not the problem though, rather the number crunching that happens on the data returned. I do lots of caching on the results instead to make the site seem responsive.)

<comment><USER17> If you're doing a lot of # crunching, you might try using Cython for those bits. You can get a 10x speedup without changing semantics and just annotating types.

<comment><USER7> Thanks for the tip. Most of it is done by python-dateutil though so I think it in C already. I'm basically checking ~500 recurring events to see if they occur today with django-schedule.

<comment><USER18> Oooow. Interesting one. So the events can be on different schedules / frequencies? It sounds like something you should be able to do with a single sql query. Could you provide a bit more info?

<comment><USER7> Hmm, currently it uses python-dateutil to find recurring dates. I chose django-schedule at the time as it seemed the best choice, which is built upon it.

I'm not certain postgres for example has that kind of functionality around dates. Googling, I found this though:

        http://justatheory.com/computers/databases/postgresql/recurring_events.html

Looks promising. Rewriting the core of the app is a pretty big project though, it's just a side-project so I'm not sure I'll get that amount of time.

<comment><USER18> Just thinking that maybe you had a single core lookup that you needed to optimise. A slow running "find all events that repeat on today." If it's something that's more deeply scattered through the system then it might not be worth attacking.

If you just needed the events for today you should be able to do something simpler. Depends on the complexity of the frequency controls. But imagine weekly recurring - you should be able to do something like _" (today - <USER2> day) mod 7 == 0"_ to find all weekly events that hit today. You could then construct a similar rule for each different frequency:

        -- not real code!
        select * from event where
        (frequency = 'daily')
        or (frequency = 'weekly' and (today - starts) % 7 == 0)
        or (frequency = 'monthly' and day(today) == day(starts))
        or (frequency = ...)

Just a thought.

</comment></comment></comment></comment></comment></comment><comment><USER19> I've been away from the Django ecosystem for the last few years.. how is Pinax doing these days?

<comment><USER7> Seems dead, a shame because I wouldn't mind a more complete starting point when creating a project.

</comment></comment><comment><USER20> What are people using these days for compiling their static assets (js minification + concatenation, etc)? We have been using django-pipeline but it has been a nightmare.

<comment><USER21> We're currently using Django Compressor [0], and it works pretty well. It was slightly annoying to get it to work well with staticfiles [1], as we host all of our content on S3 and serve through cloudfront in production. But, it's nice because now we can do this in our template:

        <script src="{% static js/mylib.js %}"></script>

and with compress like this:

        {% compress js %}
          <script src="{% static js/mylib.js %}"></script>
        {% endcompress %}

so that referencing static files is consistent.

[0]: [https://github.com/jezdez/django_compressor](https://github.com/jezdez/django_compressor)

[1]: [https://docs.djangoproject.com/en/dev/ref/contrib/staticfile...](https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#module- django.contrib.staticfiles)

<comment><USER12> I used to use this, and it is convenient for small sites. But it quickly gets hard to manage. It also makes it extremely difficult to share compiled assets between pages with different templates.

</comment></comment></comment><comment><USER22> What about South?

<comment><USER23> I agree South should probably make the list although database migrations have finally made their way into the Django core so perhaps that's why the author didn't mention South.

</comment></comment><comment><USER24> A few I work with on most projects are...

Backend: one of the most popular API Frameworks is Tastypie. Authentication and authorization: Django Userena. Debugging: Django Extensions. Django CMS Django Haystack

These are a must for your tool kit.

<comment><USER25> I would recommend DjangoRestFramework any day over Tastypie after having used Tastypie for two years.

DjangoRestFramework code base is better, it's better thought out, it should be part of Django core.

<comment><USER24> I've used both in projects I honestly don't like the auto documentation Django Rest Framework assumes you want. I enjoyed using both but I lead towards TastyPie, yum!

</comment></comment></comment><comment><USER26> I currently use Tasypie.

Would I gain anything switching to Django REST framework?

<comment><USER27> Bonus points that Daniel Lindsay (tastypie author) got a job with Amazon, works on boto, and does very little with tastypie anymore. D-R-F is actively maintained and is a bit nicer to work with. Also, the autogenerated apiview "self documenting api" is super duper nice.

<comment><USER28> Also, the DRF author, Tom, actively attends meetup and does talks here in Brighton.

</comment></comment></comment><comment><USER29> Wow, man. That's a lot of boilerplate.

</comment><comment><USER30> "Django REST framework is an insane framework"

I'm fed up with this kind of oppressive language. This is heteronormative and is discriminatory to the mentally ill. It's stuff like this that leads to the imbalance and under-representation in the industry. Let's boycott this author.

(I would love not to have to write "</satire>" but having seen some of the comments here, I think I need to. I wonder how many white-knights-for-women use language like this without thinking. We need some flexibility in the way people are allowed to express themselves.)

<comment><USER31> How is this heteronormative? I didn't see any reference to gender in the entire article.

Also, you're really stretching the "discrimination to the mentally ill" thing to the point of silliness.

<comment><USER32> should have used a bigger </satire>...

<comment><USER31> Or perhaps waterlion should have just not made the joke. It does not relate to the article in any fashion I can find relevant nor does it add anything to the existing discussion.

<comment><USER30> I think it adds to the broader discussion that goes on on HN, specifically over the last month or so.

Could you remove that joke about the dog from your profile page please? I went there looking for serious information.

<comment><USER31> If you could tell me what you're looking for instead of what you're not looking for, I'd be happy to provide it.

<comment><USER30> I'm looking for interesting discussion with intelligent people on a range of subjects guided by user-submitted stories. I'm looking for explorations of ideas, learning new things, synthesis as well as derivative ideas. Open- mindedness and acceptance. Otherwise there's no point.

I'm not looking for pedantry and closed-mindedness (I'm not accusing you of those things).

I'm probably in the wrong place for that on HN, but the articles are interesting and often so is the discussion.

<comment><USER31> That's an odd thing to look for on my profile page.

<comment><USER30> I'm going to give you the benefit of the doubt that I would want others to extend to me and not accuse you of pedantry. But I don't think there's much steam left in this conversation.</comment></comment></comment></comment></comment></comment></comment></comment></comment>


How an internet mapping glitch turned a random Kansas farm into a digital hell - <AUTHOR> http://fusion.net/story/287592/internet-mapping-glitch-kansas-farm/ <comment><USER2> I'm pleasantly surprised to see that the MaxMind folks are doing the "right thing" and relocating the default coordinates to the middle of bodies of water.

It'd also be nice if they returned (-inf, -inf) or something so people know it's not a "real" location, lest folks start looking for the house closest to the lake.

<comment><USER3> I would say the right thing would be to explicitly return a region rather than default coordinates at all. In the article it claims accuracy to about the ZIP code level, but when it's a default for the whole US, it's clearly not.

If the uncertainty can vary, you have to explicitly indicate it. If returning an irregularly shaped region is too hard, you can at least return a center and a radius.

...or is that what they were already doing, and the problem comes from people ignoring the radius? In that case, yeah, I guess they were doing what they could, and good on them I guess for moving the center to somewhere that won't cause problems when people ignore the radius. But it doesn't sound like that's what they _were_ doing in the first place?

<comment><USER4> According to their API documentation[1] this is what they are doing: There is an "accuracy_radius" element being returned right next to latitude and longitude.

And according to the Internet Archive, this element existed already in 2015, so they do not seem to have added it in response to this article.

What they did seem to have added in response to this article, however (as it is new), is a more explicit warning for their customers:

"Latitude and Longitude are often near the center of population. These values are not precise and should not be used to identify a particular address or household."

To be honest, I think it is very hard for an API provider to ensure that their customers will use all the right disclaimers when providing their data onwards to their customers... I think the new approach with pointing the general coordinates into uninhabitated spaces and adding that warning seems probably the best they can do without completely changing their API design (which would make many customers very angry, I would guess).

[1] [http://dev.maxmind.com/geoip/geoip2/web- services/](http://dev.maxmind.com/geoip/geoip2/web-services/)

</comment></comment></comment><comment><USER5> i feel really sorry for these people. the idea that law enforcement and service providers are using ip based location for anything serious is absolutely incredible.

the location from ip address is a fallback for when you don't have anything better, and it is a fallback you take with the caveat of "could be completely wrong".

this was my opinion of it when i first discovered the concept over 10 years ago.

the victims should bring legal action imo and law enforcement should get its shit together and act competently at the very most basic of levels.

ip based geolocation is and never was reliable. the idea that you can get a search warrant based off of it is laughable and indicates gross incompetence, not just in one place, but at every stage that such a thing is knowingly used.

using things you don't understand to draw concrete conclusions is stupid. literally. its the very definition of the word 'misguided'.

</comment><comment><USER6> Clearly the right thing to do is to have them map every single "default" IP to the nearest law enforcement facility rather than an individual address.

<comment><USER7> I can think of a few issues which make that solution more murky.

Why put more work onto the local law enforcement, who now have to deal with people from well outside their jurisdiction?

There are currently people who go to point X to complain or threaten. In your model, some will arrive at point X to find out it's a law enforcement facility. Many fewer will try to go to X if it's in the middle of a body of water, because it's more obviously not a place where people live. Placing X in the water means less time is wasted on a wild goose chase.

There are corrupt law enforcement officers. Isn't there the chance that, say, the state police will come to investigate small-town Mayberry with a single sheriff and deputy?

Law enforcement facilities do move. Who is in charge of updating the coordinates when the sheriff's office moves a few blocks away and another business moves into the old facility?

</comment></comment><comment><USER8> > At its least precise, it can be mapped only to a country.

Even that isn't true. Plenty of corporations have all their IP addresses registered to their main office. In general the way these services work is that they ask the owner of the IP where they are and have no way to verify it at all. And of course there are always VPN services and proxy servers that will let you choose which country your IP address will read as being in. To say nothing of actually malicious parties that can break into a computer anywhere in the world and then make all of their traffic appear to have come from its IP address.

I've also seen the IP address entries for residential ISPs be off by more than a thousand miles simply because the ISP has customers in both places and assigned customers in one place IP addresses listed in the other place.

Geo IP can be useful for showing ads as long as you don't care that it's wrong some significant percentage of the time. Thinking it can be used to locate a bad actor is dangerous and absurd.

</comment><comment><USER9> Fun lesson on why judging others doesn't scale.

</comment><comment><USER10> For Russian IP addresses default location is Moscow Kremlin. Why not make it The White House for the USA?

<comment><USER11> Just what we need: _More_ conspiracy fuel for those everyday Americans who put complete confidence in default coordinates.

</comment></comment><comment><USER12> > 39°50′N 98°35′W. In digital maps, that number is an ugly one: > 39.8333333,-98.585522. So back in 2002, when MaxMind was first choosing the > default point on its digital map for the center of the U.S., it decided to > clean up the measurements and go with a simpler, nearby latitude and > longitude: 38°N 97°W or 38.0000,-97.0000.

If 39°50′N = 39.8333333, then how does 38°N = 38.0000?

And why wouldn't they round to 40 and -99 (or -98) ?

<comment><USER13> The 50' (read as 50 minutes) means 50 60ths of a degree, so that part makes sense.

And not sure about the rounding, but 38N, 97W just "looks" more central looking at the map, so I'm guessing they just played around with the numbers for a few minutes to get it to look nice.

I've definitely done this, picking arbitrary points to place a marker on a map for, say, a big city, county or national park.

</comment></comment><comment><USER14> I think a lot of these reverse ip solutions return the center point of the zip code they have linked the ip to. For an estimation of location for marketing type purposes I think that works well. But when people assume that these databases really map to a physical address, this fails miserably as the article describes.

</comment><comment><USER15> This shows the folly of using a meaningful value when there actually isn't one. Put a null in your database, or (better) structure your database so that if there isn't a value, there isn't a corresponding entry in your database, and a query for it returns the empty set.

</comment><comment><USER16> When an API "default" can ruin the life of people, it's really sad that such "IP intelligence" was not thought right from the beginning to answer "can not be found".

Similar story [0] to a lesser degree is still horrible to the people it affects.

[0] [https://gimletmedia.com/episode/53-in-the- desert/](https://gimletmedia.com/episode/53-in-the-desert/)

</comment><comment><USER17> This story takes place in the USA, one of the world's most litigous countries. Surely someone can be sued for this.

<comment><USER18> You hear this refrain so often it must be true. Comparing the "litigiousness" of countries is not easy due to different legal systems. One of the best metrics in my opinion is cases per capita. Germany and Sweden are at the top of the list. I think the U.S. comes in at 5th or 6th.

</comment></comment><comment><USER18>
      1. 600M IPs attached to the [Kansas] home were 10 orders of magnitude
         higher than at any other location
      X. 17M IPs located in Pav's home in Ashburn.

Am I missing something obvious? 600M is 10 orders of magnitude higher than 17M?

<comment><USER19>
        2. location here has probably 60M associated IPs

<comment><USER18> That does not play well with my concept of orders of magnitude.

<comment><USER19> What is your concept?

<comment><USER18> [http://public.wsu.edu/~brians/errors/orders.html](http://public.wsu.edu/~brians/errors/orders.html)

[http://www2.pvc.maricopa.edu/tutor/chem/chem151/metric/magni...](http://www2.pvc.maricopa.edu/tutor/chem/chem151/metric/magnitude.html)

[https://en.m.wikipedia.org/wiki/Order_of_magnitude](https://en.m.wikipedia.org/wiki/Order_of_magnitude)

<comment><USER19> Oh I just misread the sentence. Of course it should be one order of magnitude.</comment></comment></comment></comment></comment></comment>


The Snowden Legacy, part one: What's changed, really? - <AUTHOR> https://arstechnica.com/tech-policy/2018/11/the-snowden-legacy-part-one-whats-changed-really/ <comment><USER2> Everyone is very aware of private mass surveillance now. Everyone knows that they are being tracked in their daily lives by Google and Facebook and they know in which way the are being tracked too. So maybe, not much changed in terms of laws but people are significantly more aware now. What used to be conspiracy theory is now accepted fact. Stallman is no longer the butt of the joke. Technologists that used to dismiss privacy concerns can no longer do so.

<comment><USER3> Yeah, my brother commented to me that in retrospect, my consistent objection to handing over my drivers license/SSN and other personal details needlessly wasn't crazy (which at the time he considered odd behaviour).

After working with the credit bureaus, I don't llike giving out enough info for anyone to mess up my personal life. For example, even the front of a drivers license without the license number is enough info for fraudulent creditors to wreck your credit.

<comment><USER4> Forget government agencies - anyone not afraid of giving them away is a fucking moron for identity theft reasons alone. Which I believe demonstrates a fundemtnal problem with how the system works - that the burden is even on the individual instead of the crediter is itself a problem.

If a schoolboy orders 3000 pizzas in your name that isn't your problem but the pizza place's problem. So why the /hell/ do we consider it different with big banks except for the general stupid human tendency to hold people in power to lesser standards?

<comment><USER5> Banks employ lobbyists and hand out more than pizzas to politicians?

</comment></comment></comment></comment><comment><USER6> It doesn't really help the issue at large but in my personal life it was nice to say, "I told you so." and not get called a crazy person anymore.

<comment><USER7> Worked in security for over decade before Snowden, kept finding things I couldn't talk about, knowing I'd found the tip of an iceberg in many places without being able to piece it together. When it all came out, was relieved not to be nuts, and was more interested in what regular people thought of the whole thing.

Normal people know what "parallel construction," is and how elected governments will use these systems for political ends.

What I don't understand is why it's not a bigger public issue with all the controversy around the current U.S. president. The scenario where a radical takes power and has control of these systems was the precise nightmare scenario everyone was concerned about, and yet mass surveillance just isn't a part of the popular discourse in the culture war.

Is it because the other team wants to use it too, or is it because the current perceived radical tyrant isn't dangerous enough?

<comment><USER8> Fortunately Trump's supporters aren't fans of the intelligence community either, maybe partly thanks to Snowden.

<comment><USER9> Oh, they just want the intelligence community to spy on and murder who the president points at. To shut up and do what they are told.

<comment><USER10> It's more accurate to say "the king". People still want a king. I hope Trump critics can start identifying this soon...

<comment><USER9> It reminds of the biblical story of Israel, when the people were fed up with the corruption of the judges, who ruled the land. The judges were a sort of a "checks and balances" system. The people were not happy with the sorry state of the system. They said, other countries have kings and order, we want one too. So the most prominent prophet listened to the will of the people and said, "ok, you will have a king, I will anoint one for you, you might not like what you get though".

To me the current situation rings like an echo of that, people want strong leaders to fix their problems.

"History never repeats, but it rhymes" \-- probably not Mark Twain

</comment></comment></comment></comment></comment></comment><comment><USER11> He (and Greenwald etc) moved the reality of mass surveillance in the West from somewhere between "it happens in other countries, but not here where we are Free" past "tinfoil hat derision" and into to a reality that we have to accept and deal with.

For one person, it's quite an achievement.

As for what has changed, Only in the movies would there be a neat resolution, and we go back to being safe.

We can never go back. But at least we know where we are. Fixing it is now our problem.

<comment><USER12> ^This

The people of the west can no longer look at China/Iran/Russia and say they're police states without looking at what they've allowed in their own countries. Once China's "Social Credit" system is in full force and in the governments eyes working well (undoubtedly crime/social tension will decrease, at the cost of freedom) will the west start copying these technologies just like their spy agencies did after 9/11?

<comment><USER13> Social credit is already a thing for the people of the west. Where people have to pass various checks for everyday things, so that ideologically wrong people could get punished and removed from ideologically right circles by not allowing them to get decent jobs and decent places to live.

<comment><USER11> Social Credit is merely a virtual representation of something that is already very real. Our reputations precede us, a decade of hard drinking, abuse and fighting in pubs will earn you a social score amoungst your local area, as will a decade of volunteer work with a local charity.

Putting those online is not inherently a stupid or bad thing. Putting them online without a means to view them, without controls on who can view them, without ability to redress wrongs or corrections, without institutions that support it, yes that's crazy.

We are never going back to a world where only your neighbours knew what you were like, but we do not either have to go to a world where only the Stasi or ad-tech firms do too.

<comment><USER14> Nobody seriously thinks that the Chinese system is your "local reputation," it's an East-German style reporting system where everyone informs on everyone else and only the state's interests are served.

<comment><USER11> Yes, I know. But the point is that a cctv camera that is accessible only to the police is the same camera that is open to the web. It's the use not the technology.

</comment></comment></comment></comment></comment></comment><comment><USER15> Nothing, and no one cares. It's like the aged grandfather wheeled out at special occasions to reassure everyone they care, and promptly wheeled back out of sight. Life goes on.

Snowden is an embarrassment for those who use human rights, freedom and democracy to further other interests. Hence the deafening silence. The entire ecosystem that supports dissenters with asylum, grand freedom narratives and wall to wall coverage about evil regimes and heroic protest closed shop for Snowden, Assange, Manning and others.

Imagine the hysteria of the 'free world' in unison against the totalitarian chinese if there was no Snowden and a chinese dissenter leaked something like this, and his or her subsequent global fame as a defender of freedom. Now look as Snowden stranded in Russia and Assange in the Ecuador embassy. That is the pathetic state of pretension.

And instead of demonizing others and filling pages upon pages scaremongering about totalitarian surveillance vs democracy, its surveillance that is being demoted to a lesser transgression, even something 'acceptable' if not 'necessary', and that's what Snowden changed.

<comment><USER16> It's amazing and almost completely unremarked how pwned the NGOs are by western intel agencies. I mean some of them have always been known to be spook outposts (Freedom house), but ... it seems virtually all of them are.

</comment></comment><comment><USER17> this, and pretty much only this:

"Suddenly, everybody knows, and nothing's changed," security technologist and author Bruce Schneier told Ars. "It was never a campaign issue. We tried to make it one. We failed... the subsequent changes are very small."

<comment><USER18> > _nothing 's changed_

2FA, E2E communications, TLS everywhere, tightening security on our phones and computers, Congress rolling back some of the NSA's dragnet powers, allied countries switching to open source and re-evaluating intelligence sharing agreements, and a sharp rise in VPN use is nothing?

<comment><USER19> > E2E communications

What popular by masses E2E encrypted comms are there?

<comment><USER20> I'm surprised by the amount of my non techie friends who are using Signal.

<comment><USER21> I try to get all my friends to use it. Since a lot like to use WhatsApp anyway it usually isn't too hard. But Signal doesn't have a push from a big company like WA does. Generally it is my friends from India that I can't move, because all of their communication is done with WA anyway. But it doesn't seem unreasonable to me that such a stronghold could happen for Signal too. But it needs to be spread by word of mouth.

</comment></comment></comment></comment></comment><comment><USER22> Snowden revealed that big companies voluntarily cooperated with NSA+CIA (with the exception of Twitter) and provided api's for snooping.

It would be nice to see if things have changed behind the scenes. My hope is that more of them are just complying with court orders and refuse cooperating with mass surveillance. Mass surveillance without cooperation is very expensive and don't always work.

<comment><USER23> No, Snowden "revealed" the big tech companies voluntarily cooperated with the FBI. In quotes because we already knew that. PRISM simply takes data that the FBI requested via court order and ingests it into NSA data processing system.

<comment><USER24> No, we didn't. We believed it. Media portrayed it.

Now, we know it.

<comment><USER23> No, we knew it. The FBI has always had the ability to get data from communications companies for specific subscribers via a court order. This is as true for email and chat as it is for phones.

<comment><USER24> Certain cases via court order is different than firehose.

<comment><USER23> Certain cases via court order is what was and is happening. The FBI has never had a firehose, and Snowden's documents didn't say otherwise.

</comment></comment></comment></comment></comment></comment><comment><USER25> People who spy on their own countrymen should be regarded as traitors, in my opinion. And they should be prosecuted as traitors, and punished as traitors. However, the theory of the "Deep State" and its influences makes it difficult to accomplish this. How can we fix this?

<comment><USER26> There is a very specific constitutional definition of treason. Another class of crime should likely be used.

Section 3. Treason against the United States, shall consist only in levying war against them, or in adhering to their enemies, giving them aid and comfort. No person shall be convicted of treason unless on the testimony of two witnesses to the same overt act, or on confession in open court.

The Congress shall have power to declare the punishment of treason, but no attainder of treason shall work corruption of blood, or forfeiture except during the life of the person attainted.

<comment><USER24> Regarded as traitor doesn't mean we can indict as a traitor.

</comment></comment></comment><comment><USER27> I'm slightly surprised that neither the article nor commenters here have mentioned Safe Harbor (or Privacy Shield).

Well, maybe not _that_ surprised, this is a very American-centric community.

Snowden's revelations had a direct effect on transatlantic ties, with the Court of Justice of the European Union holding that the EU-US Safe Harbor system violated the essence of the right to a private life. That's a big deal, and it's a very important piece of case law here now. I'm not sure if Schrems would have been able to make his case without those details being made public.

</comment><comment><USER28> Just a comment to ask general HN members, do you think the Snowden legacy has resulted in a more stressful world? As they say, ignorance is bliss, but Snowden's allegations have forced the public to realize the extent of the surveillance being conducted. Humans (and crows) become stressed when being observed by others, and this could mean citizens now have a higher stress rate. Could it be that Snowden's legacy has resulted in a decrease in net productivity from stressed Americans?

<comment><USER19> Not an American :) But it pushed me to un-google myself and switch all my data to Apple. They do security properly. I have not a single byte of Google code on all my desk/laptops and phones. No Chrome, no Android, no Gmail, no Google Maps.

I do feel less stressed now, because i _was_ aware of what he made public. As for general public, I believe it _added_ stress, and it is right thing, it might lead to some changes, and slow down surveillance apparatus.

</comment></comment><comment><USER10> Whistleblowing always makes it more difficult for the human rights violators. Some of these people will finish their workday in government or tech companies and be with families and friends who really don't care about their actions, but even then, they still have to wonder and worry.

"Do they think I'm spying on them, after Snowden reported that NSA men passed around private images of girls they found? Will I be named in the history books as a monster?"

It's always worth it. It's karma.

</comment><comment><USER29> Now we have letsencrypt.org

<comment><USER3> And more and more normal people are on Signal & Riot. Hopefully Briar and Mastodon continue to improve, Briar for metadata free chats would be a great thing to normalize.

<comment><USER30> Do both signal and riot work on matrix.org?

<comment><USER31> Riot is a Matrix client. Signal is not.

</comment></comment></comment></comment><comment><USER19> Snowden's stuff is about how _Gov_ sources your data. That is a good start, but is only one part of the problem.

We need another Snowden to leak how _corporations_ source, aggregate and store your data. A whistleblower from inside Google/Facebook/AdTech/CDNs.

</comment><comment><USER32> Semi off topic, but I used to work with his girlfriend 10 years ago or so. It was really strange seeing pictures of her and Snowden pop up in my Facebook feed, and even stranger seeing her on the stage of the Oscars for Citizen Four.

</comment><comment><USER33> Snowden and Trump both showed how brittle the societies are that we are living in

</comment><comment><USER34> By analogy...

Our relationship with shops has changed over the last 30 years thanks to CCTV (affordable webcams) and point of sale systems. We went from an era where we were not expected to be 'filmed as we shopped' to taking it for granted.

In the pre-CCTV days different strategies were needed for preventing slippage. Helpful shop assistants would ask suspicious customers if they 'needed any help'. Having stock locked away in glass fronted cabinets helped too, having all the goodies behind the counter or in hard to get to window displays helped too. There was a lot more going on than physical security though.

If your prices were actually reasonable and if everyone in town loved your store then you wouldn't get robbed. Being locally owned rather than part of a chain helped as theft from a big chain could be imagined to be only costing some notional insurance company (in a thief's mind wanting to justify stealing). Community also mattered in that nobody wants to steal from a shop they rely on, so getting barred from the local newsagents was not a desirable outcome. Nobody would steal a packet of sweets from a shopkeeper who they knew the name of and depended on for their daily newspaper/milk/fags/top-shelf magazines. Kids could be watched or only allowed in two at a time, adults could be trusted due to soft levers of trust.

Nowadays though you just wouldn't have a fortune in stock laid out ready for people to slip into their pockets, you would have CCTV, on every aisle, from both sides and from both directions. You would have some outsourced security contractor monitoring the CCTV and telling staff if they needed to apprehend anyone. There would be no 'he said vs shopkeeper said' discussion with the police, CCTV does the evidence providing bit.

The thing is that the CCTV works without anyone looking at the monitor screens. The 'smile you are on CCTV' signs are a huge part of it. They instil fear in the souls of the shoplifter. Coupled with this there is no need for patrons to actually know the staff or for them to know neighbours that could also be shopping. With increased mobility (people shop far and wide these days) the nature of shopping has changed.

The difference that the Snowden leaks have made is that we now know that the CCTV is 'everywhere', in our email and phone calls too. This ubiquitous spying works in a similar way to CCTV in retail - behaviour is controlled. We accept CCTV in retail in part because we want our pint of milk (or whatever it is) and there is no option to buy what we need from places that don't have CCTV. We are not going to buy a pasture and get a herd of cows going just to have that spot of milk in our tea. CCTV can't be objected to. Similarly in post- Snowden world we still have needs to communicate and we just have to accept the spying. That is what has changed, an acceptance of it.</comment>


China Threatens U.S. Airlines Over Taiwan References - <AUTHOR> http://foreignpolicy.com/2018/04/27/china-threatens-u-s-airlines-over-taiwan-references-united-american-flight-beijing/ <comment><USER2> What is China so scared about it needs to resort to this kind of shenanigans, over and over again?

Use of threats, bullying and excessive force definitely gives an impression that China is very weak in some way(s). Does China's leadership think the regime is about to implode to turn to this desperate measures?

Truly strong nations do not care about petty details. Strong can afford to be soft.</comment>


Broccoli: Syncing Faster by Syncing Less - <AUTHOR> https://dropbox.tech/infrastructure/-broccoli--syncing-faster-by-syncing-less <comment><AUTHOR> Hi folks, I'm Daniel from Dropbox, and I am happy to answer any questions about this tech.

<comment><USER2> When are you going to offer a cheaper plan with less storage for people that only need <50GB?

I lucked out and have 2 free plans that have bonus storage from various promotions. I get about 25 GB per account. I haven't maxed either one.

I absolutely love the product. My wife scans a file, I can grab it right away. I'm at work and need some document (e.g., my driver's license photo), I hop on the website and download it.

I pay $5 for backblaze to backup 5TB. I don't want to spend $10 a month for storage I'll never use (I couldn't even keep that much synced on most of my devices) but I'd gladly pay $3-5 a month for 50-100GB.

For now, I'll keep mooching with my free plan.

<comment><AUTHOR> There's the family plan which offers up to 6 members an account for a great monthly price.

[https://help.dropbox.com/accounts-billing/plans- upgrades/dro...](https://help.dropbox.com/accounts-billing/plans- upgrades/dropbox-family-plan)

With Dropbox Family, each member of the plan has their own Dropbox account. A single person, the Family manager, will manage the billing and memberships for the entire Family plan.

</comment></comment></comment><comment><USER3> This is why I continue to use Dropbox for daily work and constantly changing files. The syncing is unmatched. It's surprising how bad the others like OneDrive and google drive are in comparison.

<comment><USER4> OneDrive completed its rollout of differential sync in April 2020[1], after beginning in Sep 2019. This should improve OneDrive's sync speed substantially.

[1] [https://techcommunity.microsoft.com/t5/office-365/onedrive-c...](https://techcommunity.microsoft.com/t5/office-365/onedrive- completes-roll-out-of-differential-sync/td-p/1343279)

<comment><USER3> They already had this for Office files, it's just finally extended to all file types after several years. It's still nowhere near as fast as Dropbox, especially for complex directories, and the fact that it took until 2020 to finish this feature shows how far behind they are.

</comment></comment></comment><comment><USER5> I'm more of a security-focused engineer so I'm most interested in the "specially crafted low-privilege jail". What protocol gets data in and out, not shared memory I'm sure? Do the jail processes also have to implement an RPC server (protobuf/gRPC/HTTP?) or is there another mechanism for giving them work and receiving results?

<comment><AUTHOR> Dropbox uses a similar toolbox as [https://chromium.googlesource.com/chromiumos/docs/+/master/s...](https://chromium.googlesource.com/chromiumos/docs/+/master/sandboxing.md)

And yes, much of the overhead stems from the RPC server that needs to be implemented. For lepton we used a raw TCP server (a simple fork/exec server) to answer compression requests. For Lepton we would establish a connection and send a raw file on the socket and await the compressed file on the same socket. A strict SECCOMP filter was used for lepton. It was nice to avoid this for broccoli since it was implemented in the safe subset of rust.

<comment><USER5> Thank you for the technical answer!

</comment></comment></comment><comment><USER6> In my opinion broccoli does not go so well with bread (brötli = bread roll in swiss german), so some more matching name suggestions are: gipfeli (Croissant), weggli, pfünderli (500g bread), bürli, zöpfli

:-)

<comment><AUTHOR> Savory with a touch of sweetness, Broccoli Bread cooks up like cornbread but offers fiber and calcium. The original name was Brot-cat-li (since files could be concatenated and compressed in parallel), but when we said it fast it sounded like "Broccoli" and the name stuck.

</comment></comment><comment><USER7> Surprised they didn't look more at zstd.

IME it's faster than brotli and often has a better compression ratio.

<comment><USER8> It looks like they did, but having an implementation in a memory-safe language was one of their requirements. Learning _that_ was for me the most fascinating part of the article.

<comment><USER9> Surely Dropbox would have the engineering power to re-implement zstd in a memory safe language if it was sufficiently beneficial.

<comment><USER10> I'm sure they could implement it technically speaking, but if a compression protocol is not widespread enough to have others doing such a thing, they can probably consider that a sign of how supported it is.

</comment></comment></comment></comment><comment><USER11> The header on the page keeps hiding and reappearing as I scroll making it incredibly difficult to read.

</comment><comment><USER12> > Maintaining a static list of the most common incompressible types within > Dropbox and doing constant time checks against it in order to decide if we > want to compress blocks

There is also a format-agnostic and adaptable heuristic to stop compression if the initial part (say, first 1MB) of the file seems incompressible. I'm not sure whether this is widespread, but I've seen at least one software doing that and it worked well. This can be combined with other kinds of heuristics like entropy estimation.

</comment><comment><USER13> This is a really interesting write up of their use of Brotli! Makes me wonder if there might be a novel way I could leverage it beyond HTTP Responses.

I never realized the advantages of brotli over zlib could be so extensive, in particular, it appears they're getting a huge speed boost (I think also in part that its written in Rust)

>we were able to compress a file at 3x the rate of vanilla Google Brotli using multiple cores to compress the file and then concatenating each chunk.

Side note: I admit, at first I thought they were talking the Broccoli build system[0]

[0][https://github.com/broccolijs/broccoli](https://github.com/broccolijs/broccoli)

</comment><comment><USER14> The tradeoff between client CPU time and upload speed is interesting. If they need to be able to output compressed text at 100mbps, that gives a budget of ~100ns/byte, or pretty much what they would have been spending with zlib in the first place. But on my fiber connection I only have a budget of 10ns/byte. Does that mean you'd use the equivalent of `brotli -q 1` for me? If so, doesn't the march of progress continually erode the advantages of compression in this use case?

</comment><comment><USER15> Is it possible to use this as rsync replacement ?

<comment><USER16> They aren't on the same level of abstraction. Rsync currently uses zlib for block compression on the wire. Brotli/broccoli would be an alternative option.

<comment><USER17> New compression options were added in rsync 3.2. From [https://download.samba.org/pub/rsync/NEWS#3.2.0](https://download.samba.org/pub/rsync/NEWS#3.2.0)

Various compression enhancements, including the addition of zstd and lz4 compression algorithms and a negotiation heuristic that picks the best compression option supported by both sides.

</comment></comment></comment><comment><USER18> Is there a pun between Broccoli and Brotli I'm not aware of? There's another Brotli compression tool called Broccoli (written in Go), just a coincidence?

<comment><USER19> _We codenamed the Brotli compressor in Rust "Broccoli" because of the capability to make Brotli files concatenate with one another (brot-cat-li)._

</comment></comment><comment><USER20> Curious if there's enough of any one type of file that a specialty compression for it would be worth the added complexity.

<comment><AUTHOR> Great question! We developed and deployed Lepton to losslessly encode JPEG image files. Lepton continues to deliver substantial storage and cost savings every year. You can read more about it here [https://dropbox.tech/infrastructure/lepton-image- compression...](https://dropbox.tech/infrastructure/lepton-image-compression- saving-22-losslessly-from-images-at-15mbs)

</comment></comment><comment><USER21> I wonder whether syncthing can use it.

</comment><comment><USER22> None of the images are loading. :(

<comment><USER23> Should be fixed now :)

</comment></comment><comment><USER24> Good supporting data

<comment><USER25> Yeah, I really like how well the performance is quantified

</comment></comment><comment><USER26> Middle out compression has shown considerable performance over the investigated options listed in the article. I wonder why it was not mentioned?

Just kidding :) great article. As others have said, supporting data was very informative.</comment>