What's a good replacement for Google Apps in a NSA-poisoned post-cloud era? - <AUTHOR> These days, anyone who values their privacy are probably considering migrating away from US-based cloud-services... In an age of NSA masss-surveilance and service-providers getting hit with secret requests for information, it seems the only thing you can actually trust is open-source software you host yourself.
Does anyone have a good guide or experience moving from cloud-services like Google apps to a completely self-hosted open-source solution for email, contact-management and calendering? What about productivity tools?
Can you make all of this (more or less) seamlessly integrate with mobile-platforms? <comment><USER2> This does not exist yet, but I've been thinking about it for a long time. I think we generally need to move away from a centralized cloud to a more distributed cloud, where you can host your own applications on your own server, encrypt and protect them as you like, and share them with whomever you need via a secure API. You can imagine having a photo app, a social networking app, video sharing apps, document apps, whatever people can build and deploy.

You'd need standards. Standard web service APIs, standard authentication, standard privacy controls, standard database layer, standard encryption between nodes. And that would take most of the effort off the individual apps for handling all this. They'd just say "I have this chunk of data, and it's accessible to these people" and if the everything checks out, the app can grab it from whatever node it's on.

There needs to be this sort of operating-system-for-the-web on which people can build distributed apps. Something like Diaspora is trying to be, but without the limited social networking overtones. A platform where security, privacy, distributed social networking, and sharing are the main concerns, and the apps built on top of it more general.

It might be interesting. I've had this idea since about 2001 but never got around to building something. Maybe it's the right time...

<comment><USER3> Your idea sounds a lot like "the internet".

<comment><USER2> You'd think that, but then, why doesn't it work this way?

It needs a bit more structure on top, a protocol that makes it easy to do certain things in a standard way. Otherwise it's just clay and we're making things that don't fit together real well. We need Legos instead.

</comment></comment></comment><comment><USER4> 1\. Use an email client with end-to-end encryption.

2\. Buy a notebook. They sell ones that have a calendar in the front and an address book in the back.

3\. Use a stamp and an envelope. The USPO does not keep a record of all sent or received packages. Public drop boxes are also widely available, PO boxes which mask your identity from the sender, and third party mailboxes to mask your identity from The Man. I mean, really, if you're going to send messages to your mistress, girlfriend, lover, or wife, at least have the class to send paper letters she can keep in a lockbox for her great-grandchildren to find after she's dead.

<comment><USER5> _The USPO does not keep a record of all sent or received packages._

I'm not sure that's true.

 _JUNE 7--A high-tech computer system that captures images of "every mail piece that is processed" by the United State Postal Service was critical in helping federal agents track the Texas woman arrested today for allegedly sending ricin-tainted letters to President Barack Obama and New York City Mayor Michael Bloomberg._

[http://www.thesmokinggun.com/documents/woman-arrested-for- ob...](http://www.thesmokinggun.com/documents/woman-arrested-for-obama- bloomberg-ricin-letters-687435)

<comment><USER4> Then I stand corrected! One of the memes floating around was that snail mail was less tracked. I wonder if I assumed this applied to the metadata and people were only ever referring to the contents.

<comment><USER2> Less tracked, sure: I don't think they have the ability to see inside your mail just yet.

<comment><USER6> Even as far back as WW2 (and probably earlier), mail was being routinely opened, inspected, resealed, and passed on its way.

The technology for doing this in a virtually undetectable fashion has been around for a long time, and I'm sure it's far more advanced now. I very much doubt that ordinary letters have to be opened to see their contents anymore. It can probably be easily done using any number of scanning technologies.

The main problem for them is that it takes more time, effort, and money to do this for snail mail than to just ask a company for a hard drive containing all the emails and tweets of a few hundred million people.

On the other hand, a lot fewer people are using snail mail to communicate these days. So total surveillance of the relatively small amount of snail mail might just be done as a matter of course anyway.

Of course, there are some pesky laws presumably protecting the sanctity of physical mail. But if you think spies have ever let the law stand in their way, you live in a much rosier and prettier world than I do.

</comment></comment></comment></comment></comment><comment><USER7> I'm in the process of moving more stuff to a self hosted server right now. Sadly, this is almost impossible if you're not a seasoned sysadmin or willing to learn a lot.

My setup looks like this:

Mail: Postfix/Dovecot/Roundcube

Setting up a Mail server is the hardest part i think. It's essential to implement measures against spam and abuse. In my setup, this includes: \- Get a valid certificate for your mail host and enable opportunistic TLS for SMTP and SMTPD. \- Implement DKIM and SPF (and DMARC, [http://dmarc.org/](http://dmarc.org/)) \- Use Blacklists: policyd-weight, but look out for changes to public RBL services \- Reject Spam in the SMTP session using Milters, never bounce SPAM. \- Use spamass-milter and clamav-milter to Filter unwanted stuff. \- Use [http://sanesecurity.com/](http://sanesecurity.com/) ClamAV signatures, in Debian use the clamav-unofficial-sigs package

Instant Messaging:

[http://prosody.im/](http://prosody.im/) is a breeze to configure and it just works.

Telephony / Video calls:

Repro is a secure and simple SIP proxy: [http://www.resiprocate.org/About_Repro](http://www.resiprocate.org/About_Repro)

But: Resist the urge to install a VoIP PBX like Asterisk and connect it to your phone line & the internet. It _will_ most likely be hacked and abused.

Security:

I've set up a self-signed CA using XCA ([http://xca.sourceforge.net/](http://xca.sourceforge.net/)). With a client certificate on a smart card which i embedded into a USB token ([http://www.gemalto.com/products/usb_shell_token_v2/](http://www.gemalto.com/products/usb_shell_token_v2/)), i can run around and plug this USB token in random machines and instantly have my client certificate available in Chrome. This is especially nice if you (like me) don't trust PHP software like the Roundcube Webmailer.

<comment><USER8> Can you point to some good tutorials for starting with XCA?

The home ([http://xca.sourceforge.net/](http://xca.sourceforge.net/)) is quite overwhelming.

<comment><USER7> I don't get most of the stuff myself. PKIs _are_ overwhelming. There are thousands of flags you can set when creating a certificate it's not funny anymore. I did a lot of trial and error until i had a VPN using my certificates running successfully (that's what i needed the CA for in the first place).

Maybe look for CA stuff in general or some TinyCA tutorial. I've used TinyCA, it's exactly the same complexity as XCA, although I like XCA's UI better.

</comment></comment></comment><comment><USER9> I've been running my own web services for the last three years.

For email I use exim/dovecot/roundcube and am also experimentally using Bitmessage

For social networking I use Friendica, and eventually I'll be on the Red Matrix.

For dropbox type functionality I either just use plain old ssh or Owncloud. Owncloud also includes a lot of other stuff such as photos and calendar.

For blogging I use Flatpress. It just works.

From a security point of view conventional email remains a problem. It's not yet possible to function in the modern world without email, and trying to persuade even quite technical people to use PGP/GPG routinely has been very unsuccessful, since it's just too inconvenient for most people.

I've never used Google apps to any extent since I don't have much need to collaboratively edit documents. But if I did then there probably are open source systems out there which would do the job. Certainly if I was running a business then I wouldn't even contemplate using Google apps, especially after the recent NSA scandal.

I expect that readers here are more than capable of doing all the above, but sadly the average internet user is still going to remain stuck in the surveillance state with few other options to turn to.

<comment><USER10> I've been thinking about this in light of recent events. I was going to blog about it but I'll jot down my thoughts here.

At the moment we have companies making all sorts of set-top boxes, games consoles and media boxes which come in all sorts of shapes and sizes. It seems to me that we can rely on corporations to build user friendly devices for us to consume media.

What we (as citizens) rather than as consumers need is a comms box or a social box or whatever it gets called. Some mentioned "personal data box" on here or words to that effect.

We want (I'm going out on a limb to speak for other around here) secure, federated and private communications: chat communication (instant messaging), mail communication (email), audio communication (voice over ip), video communication (video over ip). Also we'd ideally like our digital social interactions kept secure, federated and private. Finally secure and private online storage of documents would be nice if only for convenience. And to top it all off what would be super-delicious would be a workable single sign-on thingy for the internet age. Please, pretty please? We only have to solve this once. We're at the threshold of the digital age, future generations will thank us for solving these problems when the issues cam e to a head (as they are now).

What is needed for all this is _open federated protocols_. And the corporations that flog proprietary offerings have to embrace these protocols. As far as I can see we have those protocols for email (smtp+tls+...?), for voice over ip (sip), chat (jabber?), social (open soical?), online document storage (?), single sign on (openid?)

We need all this in a box that people can just connect to the net via their router using wifi or whatever. We need it to be peer to peer so that large portions of it cannot be bought out and centralized after the fact. We need to shout loudly when companies roll back on support of open protocols.

We need this in a box. My personal social comms box. I dunno. It seems like a huge but vital need. But we only need to get it right once. We can't trust those in power not to abuse this tech, there will always be an erosion of checks and balances so we need to build a technical solution while we struggle for better social and legal norms.

Who's with me?

<comment><USER11> Totally agree. The bad thing about blowing the cover of NSA is that now they don't need to pussy around to keep the secret. They'll be much more brazen.

The good thing is that now we have momentum to change how we communicate. If we don't, we'll lose this train. We have had research in anonymity going on since 2000 but people haven't been motivated enough to adopt them en masse.

<comment><USER10> Yes indeed. We must use the energy created by this debacle to push forward with change. Of course, those who do will be said to be in league with the traitors and terrorists.

I should given an honorable mention Freedom Box in my previous post: [http://freedomboxfoundation.org/](http://freedomboxfoundation.org/)

</comment></comment></comment></comment><comment><USER12> Part of the problem is that there aren't enough people building such things. We've all had the luxury of a 'free' offering, which can understandably stifle progression of alternatives.

There are other challenges too. Decentralised services are probably the way forward but that creates a new set of problems around identity, communication and connectivity (things that are easier when everything just 'dials home'). User experience and design are also something that would have to be considered for such products before most people would/could switch. Keychain management will also be important to ensure security. Think of it like vaccines, where you need to have a certain proportion of the population inoculated before you can benefit from herd immunity [1].

If you're interested in a google-alternative that would deal with mail, contacts and calendars then get in touch with me (email in profile). There are a few of us who'll be building something and we'll need feedback.

Despite all of the above, I'm pretty sure that Hotmail is still the biggest hosted email provider, followed by Yahoo and then Gmail. I can't remember where/when I heard this though.

[1] [http://en.wikipedia.org/wiki/Herd_immunity](http://en.wikipedia.org/wiki/Herd_immunity)

</comment><comment><USER13> If you're concerned that the NSA will catch you in their dragnet...why are you using cloud-services at all?

To sync data with your mobile device? But doesn't the NSA already record all mobile transmissions? What difference to the NSA intercept program does it make if the data comes from Google Apps or your own open-source platform?

Would you still rely on cloud services for collaborative apps? Well, besides the issue of whether or not the NSA is catching all data packets sent through the fiber...you still have the weak point of any one of your collaborators receiving and transmitting information insecurely.

So if the NSA is a main concern for you, why are you storing portable data in anything else but a large flash memory key that you can use for any of your devices on the road?

<comment><AUTHOR> The nsa only collects information I send them over the internet via us-based services. I live outside the US.

<comment><USER6> I don't know what gave you this impression.

The NSA are interested in any and all sources of information, and probably collect just as much or more of it from foreign sources than domestic ones.

Yes, the recent media hoopla has been over the NSA collecting information on Americans in America. But that does not mean that the NSA is limited to collecting information in America. Quite the contrary. Historically, they have been more about collecting foreign intelligence abroad (ie. spying on the rest of the world).

This reminds me of a quote: "You have a choice: you can live under US domestic policy or US foreign policy". If you think you're safe or exempt under either of them, you might have some surprises in store for you.

<comment><USER14> But they can only get massive pipes of data via legally compelling a company, which means they need to either have a footprint in the USA or US employees willing to betray their company for their government.

I'd imagine using zero days to crack into foreign companies would only be for the most specific of information, making them theoretically safer from general abuse.

<comment><USER6> _" they can only get massive pipes of data via legally compelling a company"_

No. They could simply ask for the data, without any compulsion. Most companies are only too happy to help, and will bend right over -- especially after whoever the boogieman of the day is has appeared on the news lately. They wouldn't want to be seen as not doing thier utmost to help fight the good fight.

Compulsion is the second easiest and second most legal way. This way is also politically, legally and psychologically desirable for them, because they would like to push the envelope of what is considered legitimate surveillance.

Apart from asking and compelling companies, they could just tap the fiber, or use other direct and indirect methods of surveillance. This is really what they do best, and what their original and primary mission has always been: mostly to spy on foreign governments and their agents. After all, the NSA isn't going to get very far asking or trying to compel China to give them a tap in to their data centers. So they'll have to find some other way to do it. And if you think they're going to have a hard time doing it in their own back yard as compared to a hostile country, you might want to reconsider.

</comment></comment></comment></comment></comment><comment><USER15> If you're e-mailing people on the public internet, you're leaking your contacts and patterns of correspondence anyway, even if you're using end-to- end encryption. Even if you're outside the US, it's near impossible to ensure your traffic won't cross through US controlled infrastructure at some point.

That said, Zimbra does all of what you're after, it's "freemium" open source (meaning that it's a commercial product with a free community edition) and it was fairly easy to deal with last time I set it up 4-5 years ago, presumably it's better now.

</comment><comment><USER16> Ignoring the dubious baseline assumption that your data really is at risk of exposure, why do you think what you're proposing would make you safer rather than put you at more risk? You still need to actually home your servers somewhere, which means a colo or VPS that is subject to the exact same laws. Do you expect the providers of those services to have the capability to fight such requests better than Google, given that Google has a full time legal team dedicated to doing so? What about the basic security of the provider itself? Do you really expect that to be anywhere near the level of what Google provides on its own systems? To top it all off, there's the maintenance and securing of whatever you're actually running, which requires significant expertise and takes quite a bit of time if you don't want your box popped and turned into a zombie host (which is what is most likely to happen).

I've actually done what you're proposing for several years, and I wouldn't recommend it to anyone. And I'm a well respected security professional with the right background who knows all the tricks I'm supposed to do. It was still a huge pain and time sink, which meant I often let patches and updates slide more than I should. The real cherry on top is that all the three different colo/VPS providers I've used were hacked at one point or another, causing disclosure of my personal data and significant outages of my ability to communicate.

My point here is that you're not being honest with yourself if you really think you can manage this as a side project better than the full-time teams at Google do. If you want to pick another provider of email, etc, then investigate carefully and find one you're happy with. But I'd strongly discouraging running your own services.

<comment><USER17> > You still need to actually home your servers somewhere, which means a colo > or VPS that is subject to the exact same laws.

Nobody is going to dispute that a colo box is still vulnerable to targeted surveillance. But targeted surveillance is not really the issue of the day.

From the NSA's perspective, the expected value of obtaining your data is very small, but they'll bother doing it anyway when the cost is even smaller. Big centralized providers are very cost-effective targets. A sea of tiny servers operated by hundreds of different data centers, hosting heterogeneous applications, monitored by hundreds of different individuals and organizations are really not cost effective to surveil in bulk.

(I agree with the rest of your post -- doing it yourself is a major pain, and no side-project is going to do a great job.)

<comment><USER16> I don't know why you think this. Requests for the contents of communications are warrants (or effectively such). They're narrow enough that they target individuals, and it doesn't matter where the data is homed. There's certainly an argument to pick a provider outside the US if you're specifically concerned about US policy (although I'm dubious of that argument), but then you still have to investigate the provider's policies and the applicable laws in whatever jurisdiction you choose.

</comment></comment></comment><comment><USER14> It doesn't quite answer the question but I've been thinking something like Open Stack could be the future. You write your cloud apps for open stack, then the user can select where they deploy it, be it in a family/community set of open stack compatible servers, corporate servers or some trusted local open stack service.

</comment><comment><USER6> This is a great question, but you should bear in mind that there's almost always going to be a tradeoff between security/privacy/anonymity and convenience.

Encypting your email and having to deal with key managment issues, and hosting your own email server is going to be more of a hassle than just using someone else's "free" cloud-based service.

Setting up and managing your own services is going to be a burden and a time sink, even if you have the skills, knowledge, and copious free time to do it all.

The question becomes: how much is your privacy and security worth to you? Unfortunately, for most people the answer is "not that much". Nevermind that most of them don't have the knowledge or skills to set up and manage their own services even if they valued their security and privacy enough to sacrifice so much time and effort to it.

The problems are magnified when you consider that most applications are not built with security, privacy, or anonymity in mind. And when you have to use these applications to interact with other people (most of which probably won't be nearly as security or privacy conscious as you are), you're going to have an uphill battle to protect your own privacy. For instance, good luck getting your computer illiterate parents, friends, colleagues, and bosses to properly use GPG to encrypt their emails to you.

This doesn't even begin to address the issues of privacy and security in the "real world" as opposed to the digital one, nor of the digital footprints you leave as you consume online information and media, and interact with others on various forums.

The end result is that online privacy and security are largely becoming the mainstay of the super-paranoid technological elite and the ultra-rich, who can afford to hire security teams to take care of their privacy for them, and of the technological outcast, who (willingly or unwillingly) manages to avoid using the net at all.

<comment><USER13> I think you've approached the question in the right way. There is no such thing as security without a tradeoff...everyone who's ever traveled through an airport knows that. The cloud is useful because it's so easy to share things -- and that "leakiness" makes it inherently unsafe.

The NSA had its own easy-access problem lately...Snowden reportedly snuck data out using USB drives, something that was purportedly banned after a 2008 virus incident...and yet, some sysop apparently thought it'd be a good idea to let a few USB ports survive, because the things are so damned useful. And now look at what that wrought the NSA.

It's not just inconvenience that you have to put up when going for security, it's the risk that the inconvenience will cause you to enact blind spots to ease your day...and easing inconvenience was exactly the reason why people use cloud services (and USB keys) in the first place.

I'm not trying to espouse fatalism (i.e. everyone just give up their data to the G-men). Just pointing out that what the OP wants is basically a unicorn.

<comment><USER18> > _There is no such thing as security without a tradeoff_

The original implementation of Skype was convenient and secure.

<comment><USER6> Skype was "secure", for some limited definition of the term.

It was secure if you trust a closed-source program made by a company that could be decrypting your communication because they've got your keys, or that could have put any number of backdoors in the program.

And it's secure if you don't consider the possibility of doing traffic analysis, or the fact that most people are identifiable by their IP addresses, etc.

If you lower your standards enough, I guess anything could be considered "secure".

<comment><USER18> Starting with closed-source clients, yeah, Skype had limitations w.r.t security and trust. OTOH, I don't recall a successful attack on Skype.

My comment was more directed toward the fact that ephemeral keys and P2P can be secure _and_ convenient for real-time communication.

<comment><USER13> However, the trade off is that whoever you want to communicate with has to have that same client (Skype or anything)...no big deal, you say? just have everyone use it?...well, once a service gets that much penetration, a company will own the popular implementation, and then you're back to square one.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> As for mobile platforms, iOS is obviously not a trust-worthy solution. Android is OK, but has issues by being run by Google and core services provided through them.

I'd love for a third viable option to pop up, but for the same reasons as iOS, Windows Phone is out. FirefoxOS seems not quite ready at this point, and until proven otherwise I will consider Tizen vaporware.

That pretty much leaves us with Android. While _owned_ by Google is at least open-source. If you go super-paranoid you _can_ always make/get builds not integrating with Google's services in any way.

Does anyone have any experience with making opens-source solutions for things like email, calendering and contact-management integrate nicely with Android?

<comment><USER19> Best would be to use a CyanogenMod ROM. I hear in the next version they plan to introduce some extra privacy options:

[http://www.androidcentral.com/easy-privacy-may-be-coming- soo...](http://www.androidcentral.com/easy-privacy-may-be-coming-soon- cyanogenmod-incognito-mode)

Then TextSecure for SMS/chat, RedPhone for calling, and some other alternatives you may find for other apps.

At least for now it might even help if you use services from small, unknown companies, as opposed to services from the big ones, which are a given to be the focus of NSA. But it may be a matter of time before they collect the data (even if encrypted) from those, too, if they aren't already. It would also help a bit to user services hosted in other countries.

<comment><USER20> _It would also help a bit to user services hosted in other countries._

How so? At least in theory the constitution applies to US citizens and US companies. With a foreign service you have no such claim. Historically the NSA has always been concerned with communication into and out of the US, so if you live the US and use foreign hosting you're essentially stepping right into their spotlight.

Also you can guess that most other governments snoop as much if not more so than the NSA does. It's just the nature of the beast.

</comment></comment></comment><comment><USER21> It's possible for a company like Google or Yahoo to offer services without 1) requiring that we identify ourselves; 2) profiling us; 3) keeping detailed logs and data in a way that it can be mined and forklifted.

They just all choose not to.

</comment><comment><USER22> One solution is to host your own NAS. Synology e.g. provide for their NAS multiple services that might replace some Google Apps: Mail server (Gmail), CalDAV support (Google Calendar), Cloud Station (Google Drive/Dropbox), Photo Station (Picasa), Audio Station (Google Play),… all these with VPN connections. And you can easily install some productivity tools like CRM, wiki, Wordpress, Zafara…

<comment><USER23> It doesn't come prepackaged with apache/php/mysql does it? I take it one would have to install those. Tonido does most of these things, but there are some limitations/complications when trying to install your own stuff I believe.

</comment></comment><comment><USER24> I considered switching to Fast Mail a while ago, but never got around to it. It's owned by Opera Software, and hosted in Australia.

I'd prefer if it was hosted in Norway (Opera is a Norwegian software company), but Australia feels like a better choice than the US (too me anyway, perhaps others can chime in).

[https://www.fastmail.fm/](https://www.fastmail.fm/)

<comment><USER25> Australia is a member of the UKUSA intelligence treaty, so you can assume your mail there is under broadly the same scrutiny as in the USA, if not more so.

Frankly I wouldn't trust any commercially hosted mail service to be immune from government monitoring no matter where it's hosted.

</comment></comment><comment><USER26> How about email in general? These days, one cannot simply host an SMTP server out of their home, without it being automatically blocked by the big guys (i.e. Gmail's servers block any non-approved sources, for spam reasons).

Of course, email in general isn't secure, and there's no way of knowing what the recipient does with the message anyway, so this is a bit of a tricky problem.

<comment><USER27> I know outlook.com wont block or more accurately auto assign mail to spam if your domain has spf enabled.

Try this wizard for TXT record to assign to your DNS [http://www.microsoft.com/mscorp/safety/content/technologies/...](http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/)

</comment></comment><comment><USER28> You're assuming they are getting the data from Google. They are getting the information from the internet backbone, not the service providers themselves. Assume EVERYTHING on the internet is being recorded, regardless of service provider. Setting up your own email solution does not even begin to solve the problem.

</comment><comment><USER29> Related topic: [http://prism-break.org/](http://prism-break.org/)

Replace your workflow with open-source tools that aren't attached to Microsoft, Yahoo, Google, Facebook, PalTalk, AOL, Skype, YouTube, and Apple.

</comment><comment><USER30> People seem very concerned about backdoors in Huawei equipment. Isn't it just as likely that there are backdoors in US manufactured equipment. If there is then there are two privacy solutions: end to end strong encryption, or become a hermit.

</comment><comment><USER31> I provide a service to manage a personal mail server. See [http://appxecute.com](http://appxecute.com) for more details.

</comment><comment><USER32> A quick Google search turned up this: [http://cipherdocs.com/](http://cipherdocs.com/)

</comment><comment><USER33> Between 8 and 14 sysadmins.</comment>


Google doesn't think I'm a person - <AUTHOR> https://email@example.com/is-machine-learning-at-google-falling-apart-googles-system-doesn-t-believe-i-m-a-person-88c92159c979 <comment><USER2> _I'm sure you've heard people rant about being banned from a service when they don't think they should have been. This is not one of those rants._

Well, that's exactly what it is, though. Nothing wrong with that.

<comment><AUTHOR> I suppose it turned into that after all. As layer after layer of the impact kept unfolding I was a bit in shock - not that such a thing is possible, but that it all hinges on Google's neglected social media platform. Ban me on G+ if you must, but stripping my identity on Gmail, Hangouts, Google Photos, OAuth2 for other sites? That's what hurts.

<comment><USER2> It sounds nightmarish. You thought Google was your friend... (I'd never even heard of most of those Google things you had in your life.) Especially when you're paying the &$#*s.

</comment></comment></comment><comment><AUTHOR> Update: my Google+ profile has been restored, with no explanation so far. My concern remains that there is such a coupling between their social media service and every other product. I have added an update to this effect at the top of the post.

</comment><comment><USER3> > ... all of the hard work I put into labeling people I've met in New York, > Pennsylvania, Massachusetts

Are they all Ok with it, you have their permission? because I wouldn't be personally.

<comment><AUTHOR> Yes, they're all people I've met at conferences, meetups, partner companies. I only label people that I've met and that knew I was taking their picture. It helps me to keep a personal connection with a much larger group of acquaintances than I could keep in my head alone.

<comment><USER4> > I only label people that I've met and that knew I was taking their picture.

Since you don't say it, one infers that you _didn 't_ have their permission to tell Google who and where they were, and that you met them.

<comment><AUTHOR> Interesting. Ok, I've added clarity there in case that is misleading.</comment></comment></comment></comment>


Fortran Standard Library - <AUTHOR> https://github.com/fortran-lang/stdlib <comment><USER2> Interesting. I'd also like to see compiled a curated list of standard Fortran libraries, that is commonly used and hopefully academically peer reviewed and part of prominent published work.

Scientific programming in fields that have significant Fortran legacy is often a matter of writing new code to build upon or adapt existing experimentally verified code for new studies.

An annotated library of notable code and details about it with references to publications could be very valuable.

Unfamiliar to most developers, scientific programming has the quirk that the more code you write, the more your experiment becomes about the code instead of your actual thesis. This is one of the reasons Fortran stays relevant, decades of code and experiment built up as a foundation for something new.

<comment><USER3> NAG Numerical Algorithms group has been around for decades

<comment><USER2> Is it only a paid product? That would have some resistance in academia, both for openness and funding reasons.

<comment><USER4> I'm not sure if that holds. Every university I've ever done research in has no issues paying for software like MATLAB, ANSYS, Mathematica, Intel MKL, Intel Compiler Suite, and the like. The people I know that still use Fortran (including myself) are all heavily focused on computational numerics and the fact that ifortran is closed source vs gfortran being open just does not even factor into the equation. If it's available to me, and I can show that a program compiled by ifortran and MKL is faster than one compiled by gfortran and BLAS then I will use the faster one.

<comment><USER5> You must have done research at well funded universities :)

My experience is quite different. Although after some hassles I was able to get license but it's online only.

Overall I'm trying to move away from commercial software (I don't teach any, only use foss) but yeah Mathematica, MKL + ifort are unbeatable in some respects (Matlab is slowly going to be replaced except some specialised toolboxes, don't use ansys)

Anyway, proprietary and closed source software is in my opinion harmful for academia. Sometimes a lot, sometimes a little but in general it's no good, although I understand that it funds some development that wouldn't be done otherwise.

<comment><USER6> Not sure how I could proprietary/closed software being 'harmful'. If the product is superior and allows researchers/students to do a better or more efficient job then why not (absent severe budget constraints)? It is highly unlikely that those end users would ever modify the code or even need to view the source.

<comment><USER5> Simply via inability to reproduce or check research code without proprietary software. Science shouldn't be built on something like that.

<comment><USER3> back checking / testing is not just run the same "codes"

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> If Fortran has been around for this long without a standard library, then what do people use now? Does each compiler have its own? Are they very different? Is there much of a need for this since it hasn't existed for 60+ years?

<comment><USER8> > then what do people use now?

Language has basic string type (character array, really) and basic file access. People would use NetCDF or HDF libraries for reading and writing big numeric datasets. Algorithms, people would write their own when needed. For linear algebra, BLAS, LAPACK and (sometimes Intel MKL) is de facto standard, it's just not called a standard library.

<comment> fortran-guy I also tend to use `iso_c_binding` a fair amount these days if I need something in `stdlib`.

</comment></comment></comment><comment> non-entity How many people are writing new Fortran code? I know it has its niches, and is used in some popular numerical Python packages, but what about outside that? I've seen it mentioned in a few job postings for government contractors or "scientific programmer" (more of a PhD scientist who happens to program) roles, but typically listed in a way that makes it look like an HR person took a list of the tech a company used in the last few decades and make them all listed as requirements.

<comment><USER9> I have not kept up with recent developments, but Fortran has historically been noticeably faster than even C/C++ for numerical computations. This difference used to be quite stark due to limitations in C itself: [http://www.ibiblio.org/pub/languages/fortran/ch1-2.html](http://www.ibiblio.org/pub/languages/fortran/ch1-2.html) (see " b) The design of FORTRAN allows maximal speed of execution:") I think things have gotten better in C, e.g. with C99's "restrict" keyword, along with general improvements in C compilers, but for a long while Fortran had a clear upper hand when it came to allocating and manipulating large arrays of floats. Outside of the language specification, I would wager that many supercomputers have better-optimized Fortran compilers than C compilers for this reason. And even if in principle C and Fortran can execute equally performant numeric code, Fortran's historic superiority has created a great deal of high-quality legacy Fortran code that nobody feels an urgent need to port into C (which would be tedious, expensive, and risky work).

In the modern day: R uses a standard Fortran BLAS implementation, as do many other libraries and platforms (NumPy, Numerics.NET, etc). LAPACK is also widely used for low-level numerical linear algebra: [https://github.com/Reference-LAPACK/lapack](https://github.com/Reference- LAPACK/lapack) Intel also maintains a BLAS implementation. So there's still a healthy need for a (small) community of Fortran programmers, it's not all about maintaining 70s legacy code. Very different than the situation with COBOL.

<comment><USER10> > Fortran has historically been noticeably faster than even C/C++ for > numerical computations

Nowadays: faster - maybe, noticeably - I doubt it.

> supercomputers have better-optimized Fortran compilers than C compilers

I don't know about supercomputer compilers but mainstream compilers usually have the same backend for FORTRAN and C (as well as other implemented languages).

> created a great deal of high-quality legacy Fortran code that nobody feels > an urgent need to port into C

Optimized and tested FORTRAN code - maybe, but not high-quality. I've seen some of it, FORTRAN makes it difficult to write readable, maintainable code. For this reason even scientist are rewriting their tools and libraries (that also require good performance) in C++: for example see Pythia, GEANT, cern root.

<comment><USER11> The thing is, you can write a perfectly normal fortran code, and instantly gain speedup (CUDA, distributed computing with OpenMP, etc) just by enabling some compiler flags. You can't do this in C/C++ as you have to deliberately write your program to use those tech. Also, vector/matrix operations are first class in fortran and you don't need to rely on 3rd party libs.

<comment><USER10> > The thing is, you can write a perfectly normal fortran code, and instantly > gain speedup (CUDA, distributed computing with OpenMP, etc) just by enabling > some compiler flags.

I'm not sure I understand you correctly. Can you give examples of such flags?

> Also, vector/matrix operations are first class in fortran and you don't need > to rely on 3rd party libs.

It may be useful as long as you're hell-bent on not using libraries (which is somewhat contrary to one of the pro-FORTRAN arguments that FORTRAN has lots of libraries that are tested and ready to use).

This is a weak consolation though, since anything complex enough deals with custom matrix/vector types for sparse matrices or data types used in parallel computations.

<comment><USER11> Not sure about gfortran, but commercial fortran compilers supports automatic parallelization (e.g. intel fortran compiler -parallel flag [1]). You can even go as far as parallelizing you program across a cluster of machines via OpenMP by simply sprinkling some directive in your program to mark the code that must be parallelized via OpenMP. I remembered incorrectly about cuda. PGI fortran compiler supports CUDA but you still need to deliberately use it in your code, though there are projects that attempt to make this automatic (not sure if they're really took off).

> It may be useful as long as you're hell-bent on not using libraries (which > is somewhat contrary to one of the pro-FORTRAN arguments that FORTRAN has > lots of libraries that are tested and ready to use).

Yes, library is still used but it's typically only for data input/output. For example NetCDF is a popular data format and many fortran projects support the format via 3rd party library. But for complex matrix computation, this is essentially what fortran was made for so it's not typical to use 3rd party library for this. Most big fortran projects in the area I was involved with (meteorology and air pollution) uses minimal amount of 3rd party library and mostly rely on built-in fortran functionality, with optimization being left to the compiler (typically intel or pgi fortran). There is definitely code reuse, but it's in the form of the scientist collecting snippets of useful algorithm over the years and copy it to the project when they needed.

[1] [https://software.intel.com/content/www/us/en/develop/documen...](https://software.intel.com/content/www/us/en/develop/documentation/fortran- compiler-developer-guide-and-reference/top/optimization-and-programming- guide/automatic-parallelization.html)

<comment><USER10> That's very cool but it's in no way specific to the FORTRAN language:

[https://software.intel.com/content/www/us/en/develop/documen...](https://software.intel.com/content/www/us/en/develop/documentation/cpp- compiler-developer-guide-and-reference/top/optimization-and-programming- guide/automatic-parallelization.html)

contrary to the earlier comment:

> You can't do this in C/C++

On a side note: having (semi)automatic parallelization with code generation for GPGPU would be very nice.

> There is definitely code reuse, but it's in the form of the scientist > collecting snippets of useful algorithm over the years and copy it to the > project when they needed.

I think that's just bad programming practices.

<comment><USER11> Well, doing complex matrix calculation yourself in C/C++ without 3rd party library is hard. Unless you write everything yourself or specifically use intel MKL library, the benefit of enabling automatic parallelization on C/C++ won't be as impactful as in fortran where it's common do all calculation without using any 3rd party math library.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> no real skin in the game, but basing this on a preprocessor language that requires python seems curious (if still pragmatic)..

realize that fortran has a long history of being used with preprocessing, but usually these have been standalone tools AFAIK

anyone have any insight here?

<comment><USER13> I am one of the authors. We will pre-generate the files into a tarball, so that end users do not have to worry about generating it or have Python installed. Also we'll probably rewrite the preprocessing tool to C++ for robustness and speed. The reason we use it is that we need some easy way to generate the subroutines to work with any array dimension. There are not many other good options.</comment></comment>


Response to HackerNews comments on my trading - <AUTHOR> http://ptotrading.blogspot.com/2014/11/a-message-to-all-non-daytraders-and.html <comment><USER2>
      Needless to say, a lot of people, based off one story that I haven't yet finished,
      now think I'm just another halfwit gamble-my-savings-away day trader.

The closest thing to a thesis of this post is that he is not a halfwit gambler as critics on Hacker News (including practicing market makers and quantitative traders) believe him to be.

      On some days, I feel the same way about myself [that he is in fact a
      halfwit gambler].

_Not looking good._

      You have no idea how much I've read up on financial history, probability, and
      behavioral finance. You have no idea the number of profitable
      traders (whose track records I get to see or witness in real time) I've spoken
      to whom use a variety or combination of different methodologies.

Because you don't know what he knows, you can't be certain that he doesn't know anything. True. But, he doesn't record any specific knowledge here, so we still don't know whether he knows anything.

 _Looking worse now._

He then sets up several straw man arguments against him. But, amazingly, does not refute them.

 _Okay, I give up._

</comment><comment><USER3> You seem like a nice and rational guy, but it looks to me as if your post ended just after the introduction. Apart from the info that there are many profitable trades which success that cannot be explained by luck, I'd love to see counterarguments to other things.

For example, if you are a successful trader, it seems a kind of activity that scales very nicely. If you can be sure you are profitable in say a month (or a year), you are multiplying money, so using leverage or loans you should be able to basically have any amount of money you want (and what about trades in that case?). At least that is my understanding as a non-trader. I'd love to read some explanations for this, especially that I like your writing style.

<comment><USER4> > If you can be sure you are profitable in say a month (or a year), you are > multiplying money, so using leverage or loans you should be able to > basically have any amount of money you want (and what about trades in that > case?).

Its not that simple, sadly.

You can't buy $InfinteMoney$ worth of stock X in a single transaction, you end up moving the stock price as you buy/sell. This allows others to respond to your activity. At small scales, a system that appears to work will break down at large scales.

<comment><USER3> > You can't buy $InfinteMoney$ worth of stock X in a single transaction, you > end up moving the stock price as you buy/sell. This allows others to respond > to your activity. At small scales, a system that appears to work will break > down at large scales.

OK, true it doesn't work up to infinity, but given how much money there is on the Wall Street, I would imagine that when your activities are big enough to move the market, you are way past the point of wondering what you can and what you cannot afford (say $1B+ not to be too vague). Especially given how many instruments there are out there.

<comment><USER4> He is trading penny stocks with a market cap is less than $50 million, generally.

[http://www.investopedia.com/terms/n/nanocap.asp](http://www.investopedia.com/terms/n/nanocap.asp)

The amount of money needed to move the needle on these is very small, its why they are often used in pump & dumps.

He doesn't have anything to do with "Wall Street" which works with stocks that follow SEC rules, commodities, etc. in the way you mean.

<comment><USER5> Deleted my response when I saw this better one. I'll only add these links as they're good for background:

[http://en.wikipedia.org/wiki/Slippage_(finance)](http://en.wikipedia.org/wiki/Slippage_\(finance\)) [http://en.wikipedia.org/wiki/Market_impact](http://en.wikipedia.org/wiki/Market_impact)

</comment></comment></comment></comment></comment><comment><USER6> Refuting crappy internet comments about your blog isn't really a recipe for success, unless your criteria for success is more pageviews. There were tens of people in that thread who were spouting off armchair economics like they were financial experts. That type of ignorant rambling should really be reserved for /r/economics or, better, /r/bitcoin.

It doesn't really deserve to be responded to, much less refuted.

<comment><USER4> Yes, but he is talking about trading penny stocks/pink sheets. He is on par with the people the "armchair economists" unless he is super wealthy and can prove it.

You sure you want to act like he is a financial guru?

"If you've been following penny stock trading circles on twitter for awhile, you'll know certain tales of legend were born on that day. One particular trader profited $1.2 million in a day. When that many traders produce their largest gains on one particular trade, you know it doesn't require some crazy genius, It's the stock that does all the work and makes life easy.

My day was just peanuts in comparison but it was still my personal best, as I made $33,000 (with no profit split! which felt amazing). The trader across from me, who I will affectionately nickname El Chango, had seen enough. El Chango was branch B's head trader. It killed him to see easy money slip away due to inaccessibility. He pulled CCG's principle owner aside and pleaded for him to enable access to the OTC."

His method?

"Simple. Look for the extreme volatile moves on massive volume. If the direction was down, find the first held bid. Buy. If the direction was up, find the first major seller. Short."

If it was really how that worked in the real world, it'd have been eaten up by HFT.

<comment><USER7> HFT on fucking pink sheets? LULZ.

What about the HFT risk from getting goods for free on CL and reselling them back to CL?

<comment><USER4> Like I said, if it really was how it worked someone would find a way to automate it. :P

I suppose I could have been more verbose. I tend to use acronyms sometimes and not think through that it could be read differently than I intended.

<comment><USER7> Yeah, I was once considering building an OTCBB trader that would trade all the pennystock promo bullshit and see which (if any) lists actually worked and just trade the spam. The amount of degenerate gamblers in that industry is insane.

Almost built a claim jumping / claim expiry bot for mineral titles. Claim expires tomorrow? Grab it at 9 AM? Claims made today? Stake the blocks around them.

I think anyone with half a brain would clean up on OTCBB with a little automation.

<comment><USER8> You'd think so but the problem is volume. When it's time to buy you can't get enough shares at the price you want and when it's time to sell, there's no buyers.

I used to trade pennies for fun, because maybe I was one of those degenerate gamblers. Options trading can be just as much fun.

</comment></comment></comment></comment></comment></comment><comment><USER9> I think the big complaint was that your post was titled "How I lost a shit-ton of money in one day" but the content was "How I started making some money on OTC trades and got on a leaderboard at work" \-- felt click-baited.

<comment><USER10> Full of himself, b.s. trader can't even make a blog title correctly. No wonder he lost thousands of dollars by accident, then couldn't even finish the last 3/4 of his story. Not to mention leaving out the most interesting part.

</comment></comment><comment><USER11> Am I the only one who sees this entire story as decidedly "un-hackerly?"

On one hand, OP chides commenters for describing trades as "coin flips" (this is apparently some sort of composite as he quotes no individual commenter in saying this). Then expresses the following transparent cognitive dissonance:

> Trades are not binary outcomes like coinflips. If you think someone with a > consistent 70% win rate...

Well, if trades are not binary, what is a "win rate?"

OP makes no mention of the 'hack' here - namely, the underlying systemic outcome of this conduct. Are the companies whose stock is at issue ethically sound? Are they promoting open source and free thought? If they aren't, and if their value is the metric, how can any gains be regarded as a "win?"

Perhaps in a self-enriching sense, but self-enrichment isn't per se a 'hack.'

</comment><comment><USER12> Do TA proponents ever talk about the suggestions that the reason TA works is because of a feedback cycle?

That is, if you get enough people with enough influence (cash) all believing they can read tea leaves, and make similar movements based on that, it kinda works.

<comment><USER3> > kinda works

I think it does influence the market, but it cannot make them all profitable. Unless you think that group of TA believers act as a big pump and dump scheme (but I guess most markets are too big for that?)

<comment><USER4> Penny stocks [as the OP trades] wouldn't be.

He tries to obscure it a little with things like "pink sheets" and acronyms but I don't think that is intentional. I'm guessing people just don't read the original article he is defending.

<comment><USER3> I read it, but wasn't aware of the difference. Thanks for the link and explanation in another comment.

</comment></comment></comment></comment><comment><USER13> I'm glad this was posted; I thought it was well-written and humble (both the original and follow-up), and interesting to me personally anyway.

My impression from the limited research I've done into technical trading is that - just like a lot of things in life - it's a lot of work. People who go into it hoping to make it big on one lucky score seem to get slaughtered.

A lot of technical analysis indicators seem to be not about the companies themselves (which is fundamental analysis) but about determining the trend of what _other people_ think about those companies/stocks (the "Kenesian beauty contest" [http://en.wikipedia.org/wiki/Keynesian_beauty_contest](http://en.wikipedia.org/wiki/Keynesian_beauty_contest))

So I think it's a little more than sheer chance/luck, but it also definitely seems to require a lot of work and staying glued to the market/indicators/news.

It's appealing intellectually because of the "game" aspect, but I think the reality is that like anything else it requires a ton of work to succeed at it, especially in the beginning when you're still cutting your teeth and learning hard lessons. I appreciate that the author wrote candidly about his experiences learning the game/trade.

</comment><comment><USER14> There is a pretty solid hypothesis that when traders talk about "reading a chart", they are not in fact using easily explainable patterns, but rather more hardcore estimations of market psychology, derived via the brain's rather robust pattern-finding machinery. The chart patterns are more of a mnemonic than a decision rule in and of themselves.

<comment><USER15> Pattern matching has its own risks.

My favorite story is a study that involved trying to predict the outcome of a 1:2 random distribution. Pull a random card from an infinite deck, 1/3rd of the time it's black, 2/3rds of the time it's red. The optimum strategy is to guess red all of the time; you will be right 2/3rds of the time. Humans have a tendency to try to match the random number generator, guessing red 2/3rds of the time and black 1/3rd of the time, which is sub-optimal, and you're only correct 5/9ths of the time.

We're powerful pattern matchers, yes: we can observe a 1:2 random distribution and mimic it pretty accurately. But we're also a little stupid: we think it's a good idea to do so.

<comment><USER16> Maybe traders have learned on an intuitive level to "fix" this pattern matching - clearly the processing circuitry / power / ability is there, so a trained difference in outlook could allow us to harness it properly.

</comment></comment></comment><comment><USER17> Here's something I've witnessed: Those that understand trading do it. Those that don't, swear that nobody else is making money doing it either. They are _convinced_ of that and will argue their point endlessly.

I find it strange.

(Caveat: of course I don't mean absolutely everyone, but it's a phenomena you can witness whenever it gets discussed here.)

<comment><USER18> [http://longbets.org/362/](http://longbets.org/362/)

[http://fortune.com/2014/02/05/buffett-widens-lead- in-1-milli...](http://fortune.com/2014/02/05/buffett-widens-lead-in-1-million- hedge-fund-bet/)

</comment></comment><comment><USER19> I don't usually bookmark blogs, but you've got me hooked.</comment>


Facebook Phone is a Bad Idea - mck- http://www.businessinsider.com/facebook-phone-is-a-bad-idea-2012-5

<comment><AUTHOR> A duplicate of this link was posted 15 minutes before yours, plenty of time to check to see if the link had already been posted.

I'm sure you just logged on and contributed, but please check next time.

Upvote for effort either way.</comment>


Infrequently Asked Questions in comp.lang.c - <AUTHOR> http://www.seebs.net/faqs/c-iaq.html <comment><USER2> Hm, I don't agree with some answers.

For 1.1: Which integer type should one use? This depends. Do you need space? Use the smallest one possible. (char, short, ...) . Do you need speed? Use the natural integer type for the machine (int). Do you need a large value range? Use the largest one (long). Do you know no clear answer? Use the simplest one (int).

For 1.9: check out forward declarations instead of brittle structs where you need to check if the sizes fit.

For 2.1: This is a very, very weak reason. No mention of type safety? No mention of mistype-safety for the enumeration indexes? No mention of increased maintainability?

For 2.9: What? C data has to be byte aligned, thus those "holes" are required by the standard. Unions and structs just cannot be interchanged. Am I confused on this one?

Too bad I don't have time to go through all of this, but this appears to be pretty weak. Read it with care, and don't read it at all if you cannot care due to a lack of C-fu.</comment>


Days between Apple's innovation and Microsoft's imitation  - <AUTHOR> http://timkeller.me/2011/09/16/innovation-to-imitation/

<comment><USER2> Some of the things that the author claims Apple "innovated" on here are ludicrous. The original Macintosh OS was not an innovation so much as it was an imitation of Xerox research into GUIs, for example. The original iPod was a fairly modest set of improvements to prior digital media players, such as the Rio Diamond (which was much more original). Perhaps most strikingly, the iPhone was a fairly incremental step from many preexisting ideas like the Treo and Tungsten lines from Palm, or even the old Windows Mobile line.

Mind you, I don't see it as a problem that Apple and Microsoft have both by and large profited from imitation, as that's what drives the industry forward. Someone has a good idea, and builds it on top of an established base of other good ideas, imitated from others. Standing on the shoulders of giants, so to speak. What I mind are the contortions of historical revisionism one has to go through to paint this picture of "Apple innovates, Microsoft imitates."

</comment><comment><USER3> The claim that Microsoft does nothing except imitate others is blatantly false; it would have been much more interesting if you had delved into the cases where they _didn't_ imitate, or tried to identify the reasons why they're content to merely follow the market leader in these cases. The post's tone also overshadows the actual content to an unnecessary degree.

It's fair to call out Thurrott, but if you want to do so, you should try to at least rise above his level.

<comment><USER4> >>The claim that Microsoft does nothing except imitate others is blatantly false

Sure, Microsoft tried to build lots of tablets without copying anyone. How did that go?

Edit: To be fair, I'm just reviewing Microsoft's old OLAP invention MDX, since I might want to suggest using it as part of a solution (in a different problem domain). So far, it seems quite neat (I have only read, yet).</comment></comment>


Saving Data: Reducing the Size of App Updates by 65% - jor-el http://android-developers.blogspot.com/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html <comment><USER2> It's a bit humbling to think that my "quick hack" is yielding bandwidth savings measured in _petabytes per day_.

<comment><USER3> I just tried bsdiff, and can I complain a little? I tried to diff two 1 GB files, and within a few seconds bsdiff was using enough memory to lock up my Ubuntu desktop. 10 minutes later I managed to get a text console and kill it. Maybe I should blame Ubuntu for letting a single process hose the system, but it was still annoying.

Edit: bsdiff is documented to use memory 17x the file size.

<comment><USER4> > Maybe I should blame Ubuntu for letting a single process hose the system

Speaking of: is there a good way to ask an OS to "reserve memory and CPU-time" for emergency interactive maintenance (e.g. spawning a console on a thrashing box), in a similar way to how ext2/3/4 "reserves space" for the root user for emergency maintenance?

The only real way I can think of to do this (that doesn't involve already knowing what processes could thrash and cgroup'ing them) is to run a hypervisor and treat the dom0 as the emergency console. That still doesn't let you get _inside_ the thrashing instance to fix it, though.

<comment><USER5> If you use systemd you can assign services to slices which can be assigned cgroup resource reservations / limits. I don't think any distro does too much with this out of the box yet (IIRC by default all services run under a single "system" slice, and user sessions under a user slice), but it should be possible to run an "emergency environment" slice in which the processes are always guaranteed some small amount of resources necessary to recover.

<comment><USER6> The LinkedIn people report that it's not that simple.

* [https://engineering.linkedin.com/blog/2016/08/don_t-let-linu...](https://engineering.linkedin.com/blog/2016/08/don_t-let-linux-control-groups-uncontrolled)

</comment></comment></comment></comment></comment><comment><USER7> I'm reminded of Courgette: [https://www.chromium.org/developers/design- documents/softwar...](https://www.chromium.org/developers/design- documents/software-updates-courgette)

It's used to deliver unbelievably small Chromium OS updates.

To quote the link above:

      Full update:      10,385,920
      bsdiff update:    704,512
      Courgette update: 78,848

The reason it's not being used here is because Courgette's disassembler/symbol resolver only works on x86/x86_64/ARM assembly language.

But it got me thinking - how much bulk does binary code take up in an Android APK? If it's noteworthy, a JVM disassembler could produce really interesting results.

If it's mostly images and sounds then things are pretty much as optimized as they already are. (I have no idea myself.)

<comment><USER8> > a JVM disassembler

Note that Dalvik bytecode is very very different from the Java bytecode read by JVMs. For one, Dalvik is a register machine (like LuaJIT's internal bytecode) and the JVM is a stack machine (like Python bytecode).

<comment><USER7> Oooh, that might be one of the reasons why LuaJIT is so fast.

I did some reading about Forth some time ago, and learned that modern CPUs have minor struggles with stack-based interpreters. I understand it's almost negligible but has a small impact. (Forth is stack-based.)

<comment><USER9> There are many reasons why LuaJIT is false, Mike Pall has talked pretty extensively about it (e.g. [http://lambda-the- ultimate.org/node/3851](http://lambda-the-ultimate.org/node/3851)).

I don't recall being register-based coming up as a first-order reason, though it may have allowed optimisations which a stack-based machine would not allow (e.g. the JIT doing register hinting and renaming).

<comment><USER7> What do you mean by "LuaJIT is false"?

Thanks heaps for that link though, reading through that thread has been really interesting.

<comment><USER9> > What do you mean by "LuaJIT is false"?

Meant to write "LuaJIT is fast", but was probably thinking about something else and the fingers went haywire, sorry about that.

> Thanks heaps for that link though, reading through that thread has been > really interesting.

Pall has written in other venues (mostly LuaJIT mailing lists but I think I remember comments of his on reddit or HN, though I might be mistaken), comments of his are always interesting reads.

<comment><USER7> Ah, I've had the same kinds of finger-"autocorrect" issues myself. Thanks!

I've been trying to find more info about Mike, he seems to be a really private/mysterious type. What he writes about is definitely interesting though.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Disclaimer: I used to work at Google exactly on this project.

I'm so happy this finally made it to daylight! =) I was the original one researching for the feasibility of this, but at the time (3.5 years ago), the re-compression burden made it a no-go even for the most modern phones so we decided to table it.

<comment><USER11> Possibly dumb question: why doesn't the hash verification take place on the _uncompressed_ contents, moving recompression from the updating-critical-path to the nice-to-have-later-to-save-disk-space category?

My dream updater would just be a bunch of stream processors, so the downloading / decompression / disassembly (a la courgette), patching / recompression happens all at once, as the data packets come in (thus utilizing both CPU and IO). If done right an update shouldn't take much longer to complete than the download step, no?

<comment><USER12> Many (most?) Android phones are chronically short of flash space, even in the status quo storage shortage often prevents applying app updates. There is really no room to make storage use more lax.

</comment></comment></comment><comment><USER13> Instead of faffing about re-compressing all the App data in order to compare signatures, wouldn't it would be simpler to store two signatures alongside each App in the first place: one for the compressed version & one for the uncompressed one?

Then computing the signature after using the compressed diffs to upgrade an existing App in place would just require walking over the upgraded files & comparing the hash to the previously computed one. No CPU intensive re- compression required.

(Clearly you'd have to sign the diffs in some way to prevent bad actors injecting data into the system, but that's a separate problem to the 'does this data match what the App developer sent us in the first place' question which Google is currently solving by re-compressing everything at great CPU expense.)

</comment><comment><USER14> I really wish Google would let us download pre-compiled native blobs on fast WiFi connections. Downloading is by far the fastest part of app installation / update for me, the time spent recompiling the bytecode can measure in the minutes for some apps.

Even worse is the forced-recompilation of every single app for the tiniest system patches, which can take upwards of an hour.

<comment><USER15> That's no longer the case with Nougat. Apps are now interpreted with a JIT compiler and a profile is created for performing optimized AOT compilation when the device is charging and unused for subsequent executions.

<comment><USER14> Does this have to be turned on somewhere? I updated to Nougat last night and it still recompiled all my apps on the first boot. I didn't notice any improvements while updating apps from the play store either.

<comment><USER16> The upgrade from Marshmallow to Nougat should be the last time you see "Optimizing Apps".

</comment></comment></comment></comment><comment><USER17> So they are recompressing the files after applying the patches and hope that it's the same? Wouldn't it make more sense to work on having the signature work on uncompressed files?

Also it's kind of shocking that almost everyone is using deflate in with compression level 6 or 9. Why is 6 even used? Why does almost nobody use zopfli which achieves 3–8% higher compression rates. Why aren't other compression formats supported like lzma, brotli, etc.?

<comment><USER18> _> Wouldn't it make more sense to work on having the signature work on uncompressed files?_

No! That would create an additional security risk, in addition to potential performance issues.

Ideally, unchecked data should _only ever_ hit the signature check and nothing else. Otherwise, it could also attack the decompression, not just through security holes, but also through denial of service (by injecting something that explodes in size on decompression).

This is a general principle! You should neither decompress, nor decrypt, nor unpack/untar, nor parse (XML/JSON/...), nor anything else before the signature check. See also:

"Preauthenticated decryption considered harmful"

[http://www.tedunangst.com/flak/post/preauthenticated- decrypt...](http://www.tedunangst.com/flak/post/preauthenticated-decryption- considered-harmful)

However, I agree that it makes sense to have an _additional_ check after decompression. However, that check usually happens in most decompression tools anyway, and usually through fast, lightweight checksums. Because, again, the point here is to detect bugs in the decompressor, not to protect from malicious tampering, which should always happen before complex stuff like decompression.

<comment><USER19> _> Otherwise, it could also attack the decompression, not just through security holes, but also through denial of service (by injecting something that explodes in size on decompression)._

 _> This is a general principle! You should neither decompress, nor decrypt, nor unpack/untar, nor parse (XML/JSON/...), nor anything else before the signature check._

Google have stated that they're going to be decompressing, patching _and_ recompressing data in an attempt to recreate the original data stream - <AUTHOR> to perform a signature check.

So, whilst minimising the attack surface is a great goal in general, your advice is not relevant here. They're already vulnerable to having their decompression, patching and compression software attacked.

<comment><USER20> Your old apk is already signed. I guess it would make sense to sign the patch too (if they don't do it already).

</comment></comment></comment></comment><comment><USER21> Now if someone could figure out how to reduce the size of installed apps. And preferably explain why most Android apps seem to be about 60MB even for trivial things.

<comment><USER22> I have worked on a few mobile games where there was a big developer push for reducing the app size but it was shot down by management to add more new features instead. A lot of features had expected performance targets to hit before they got the go ahead and it was hard for reducing the app size to have a sizeable difference on KPIs until it got to critical levels.

I think there is also a certain amount of developer apathy towards application size.I was recently talking to a colleague who had worked on a new banking app and they saw no problem with developing, what was essentially a web front end, with the Unity game engine. When I asked if he was concerned the effect on the app size including a game engine would have he said it was what they where used to using and that was more important.

<comment><USER23> A banking app in Unity??!?

Come to think of it, my banking app takes forever to start up, I wonder why...! I've never bothered to check how big it is, though, which possibly proves your colleague's point. :(

<comment><USER24> Well, if they are a C# shop, that might be a really easy way to get their app on phones while being able to reuse code that might already exist, while also not having to deal with phone specific toolkits.

</comment></comment></comment></comment><comment><USER25> I actually did exactly the same thing over a year ago. I needed to download Cynogen nightlies quite frequently which was not feasible at that time as I was on limited data. So I used to download the nigthly on my digitalocean droplet and calculate patch over their. Patch sizes were just around 12MBs. I would then download the patch file and patch the old nightly on my local system. Voila! phone upgraded by downloading just 12MBs! I did thought why Google can't do the same thing with Android applications.:)

</comment><comment><USER26> Slightly offtopic: What algorithm does git use for binary diffs in packfiles? Is it bsdiff, or the improved algorithm form Colin's thesis, or something else?

<comment><USER27> As often is the case, SO has an answer: [http://stackoverflow.com/questions/9478023/is-the-git- binary...](http://stackoverflow.com/questions/9478023/is-the-git-binary-diff- algorithm-delta-storage-standardized)

</comment></comment><comment><USER28> > _Disclaimer: if you see different patch sizes when you press "update" > manually, that is because we are not currently using File-by-file for > interactive updates, only those done in the background._

Out of curiosity, does anybody know why that would be the case? I had assumed automatic app updates used exactly the same code paths / APIs as manually pressing "update" in the Google Play app did, just triggered automatically. But it sounds like they actually go via a different route?

<comment><USER29> The article implies they prioritise installation _time_ for manual installs (which sounds sensible to me, since waiting time is the biggest source of frustration for users, and decompression is the bottleneck there, especially on older devices.

</comment></comment><comment><USER30> The code is open sourced at: [https://github.com/andrewhayden/archive- patcher#table-of-con...](https://github.com/andrewhayden/archive- patcher#table-of-contents)

</comment><comment><USER31> This kind of problems could be made public.

Small or big company, government or organisation could define problem, give test input, expected output and performance points (delta size, cpu usage etc); submissions open for all.

Similar to programming competitions but instead of made-up problems, there is a real world problem to be solved.

This could be good for everybody - companies, because they'd get (I hope) the best solutions (there could be some price for top solutions, but still fraction of in-house r&d cost), for people, because they could be hired or at least would have a chance to test their skills on real problems; it could be used as part of hiring process to skip early nonsense etc.

The biggest win, IMHO, would be to have government organisations getting involved in this kind of "Open R&D" \- open for contributions and also open for new, not yet articulated ideas.

<comment><USER32> It already is public. That's why Google is using bsdiff, an algorithm developed for FreeBSD, and which they are legally able to use due to the way in which it is licensed. We don't need some sort of central clearinghouse of problem instances.

bsdiff predates the entire concept of Android by many years. To the extent that Google wasn't using it from day 1, it isn't because the solution didn't exist somewhere. (It sounds like it was due to problems that would have existed for any clever solution and they chose deliberately to ship down un- diffed-in-any-way updates.)

There's actually a profound lesson about how open source really works here. And I'm not really saying this as an open source evangelist... it's more an observation that our instinctive (and probably genetic) instinct in favor of centralization is not always an accurate model of how the world either _does_ or _should_ work. There's not much some sort of centralized contest could do in the present that wasn't already done in the real world over a decade ago without it.

</comment></comment><comment><USER33> On a side note: I "love" when I have to download 100+ MB update of the mobile app with only changelog description: "minor fixes".

<comment><USER34> Or worse, opening up the app on the train just to be greeted "This version is outdated. Please update to continue using."

<comment><USER35> It's great when it happens with Uber and you're outside, with a crappy 3G connection.

</comment></comment></comment><comment><USER36> Why recompress at all?

Rather than using .zip, an improved APK format could be more like a tarball. You'd tar up the files, sign it, and compress the result.

To verify, you decompress (streamily, without saving the result anywhere) in a tight sandbox and verify the signature on the output. Then you decompress again to install. Call it APK v3.

This adds a second pass at decompression, but decompression is generally quite fast. In exchange, you avoid a much slower recompression step.

<comment><USER23> Yes, it definitely seems like fixing the flaws in the original APK design would be a win! Just signing the uncompressed data, as many others here have pointed out, would allow most of your suggestion to be implemented.

I wonder if it's a sign of communication problems between the Android and Google Play teams? The Play team seems to have spent years bending over backwards to build this really awkward solution, which could have been done a lot better with some low-level Android changes (i.e. a new APK format).

<comment><USER36> I bet there's a valid historical reason for the current design: APK looks a lot like JAR and Android is kind-of-sort-of Java. Java random-accesses its JARs, so the awkward format makes sense in the context. Android may, too, to a limited extent, but Android has no concept of running an APK in place, so it doesn't need this capability.

<comment><USER23> Well, exactly, both APKs and JARs are just zipfiles with some extra conventions over format and contents.

Why not update the format? Android has been around long enough to make the advantages and disadvantages pretty clear.

</comment></comment></comment></comment><comment><USER37> Enabled only for automatic updates because it's CPU intensive.

<comment><USER38> Auto update is the default and I'm sure it's what most people is using. I disabled it many years ago to prevent app abusively add permissions and to prevent crashes. It was pretty common for new versions not to work properly on many devices. I remember checking the reviews before updating. This hasn't happened anymore for at least a couple of years and Android 6 solved the issue with the permissions but I didn't lose the habit. I'll download the full update. Furthermore I keep the phone in airplane mode at night. I want to sleep.

<comment><USER39> Before Marshmallow it would refuse to auto-update apps if they added a permission. It would just show you a notification about the new permission and whether to accept/decline the update.

<comment><USER38> So I did it because of the crashes. I forged the other memory. Thanks.

</comment></comment></comment></comment><comment><USER40> This is great for games like Hearthstone that currently requires a re-download of the whole app which takes 2 GB whenever there's an update.

<comment><USER9> > This is great for games like Hearthstone

Nope.

Hearthstone is a Unity game, and the issue (for mobile packaging at least) is Unity creates a giant bundle of all the assets, so it's a single binary file which tends to greatly change from one release to the next, even with per-file diffing you're not getting any improvement[0].

Other developers get around that by downloading assets separately from the main application bundle à la Supercell. I don't know that Supercell uses Unity, but they do follow that pattern, they have an application bundle of code & trivial assets, but most of the assets are downloaded at the loading screen.

Checking on my phone, Hearthstone is ~2GB with 10.5MB of "documents and data" (= a ~2GB base app bundle), clash royale is ~450MB with 336MB of data (= a ~100MB base app bundle), jetpack joyride is similar with ~330MB split between a ~90MB base bundle and ~250MB "documents and data".

The default unity approach makes sense for rarely updated applications (Monument Valley or République), it's an absolute bear for dynamic frequently updated on-line games like Hearthstone.

[0] IIRC per-file diffing is what apple does on the appstore, it does not work at all for HS

<comment><USER41> Unity has asset bundles precisely for use cases like streaming and downloading of assets:

[https://docs.unity3d.com/Manual/AssetBundlesIntro.html](https://docs.unity3d.com/Manual/AssetBundlesIntro.html)

Blizzard just doesn't seem to be using them...

</comment></comment></comment><comment><USER42> Even before bsdiff and this new file-by-file patching mechanism I find Android updates much faster than iOS. I don't know if iOS updates are just bigger becuase application are bigger, or is just that apple has a worst CDN (I live in Argentina).

</comment><comment><USER43> TL:DR? Compress diffs, don't diff compressed things?

</comment><comment><USER44> Could rsync algorithm be used to prepare binary diffs? How would it compare to bsdiff?

EDIT:

Thanks for the downvote.

There is a tool using rsync algorithm for binary diffs. It's called rdiff [1]. I found Master's thesis Binary Differencing for Media Files by Vladimir Komsiyski [2]. What I take from there is that rdiff is fast, but bsdiff produces smaller patches.

Excerpts:

> _bsdiff_ is marginally better than xdelta as far as compression is > concerned. _rdiff_ and especially edelta generate much larger patches. The > upload gains from the viewpoints of the server are exactly the same as the > compression rates and the saved disk space is proportional to it. For > example, an expected number of versions per file of 3.7 (which is the case > in Dataset A) and differencing with xdelta result in 70% less required disk > space.

> As expected, _bsdiff_ 's slow differencing time results in 3 times slower > upload from the viewpoint of the client when compared to the server's > experienced time. Apart from that, the added time due to the differencing is > smaller compared to the transfer of the produced patch and lead to up to 21 > times faster transfer than when no differencing is done. Even the worst > performing tool edelta reaches a speed-up with a factor of 3. xdelta, > showing the best results out of the four tools, achieves a 21x speedup when > transferring a file to the server. The second best tool, _bsdiff_ , is only > 9 times faster. However, its coefficient may dramatically increase with > increasing the file size (Adobe Photoshop files) because of the non-linear > runtime of this tool. edelta is severely outperformed by all other tools. > The fastest tool _rdiff_ loses its lead due to the bigger patches it > produces.

> For Adobe Documents (and most likely other image editing software files) the > binary differencing tool xdelta shows the best run-time for the compression > it achieves. _rdiff_ is faster, but its patches are bigger. _bsdiff_ shows > better compression, but its non-linear runtime makes it unusable for files > of this size. The benefits of xdelta are considerable - decrease of the file > size with more than 96% for Adobe InDesign files, causing up to 30 times > faster file transfer. Adobe Photoshop documents also show substantial gain > despite their much larger size.

[1] [https://en.wikipedia.org/wiki/Rsync#Variations](https://en.wikipedia.org/wiki/Rsync#Variations) [https://linux.die.net/man/1/rdiff](https://linux.die.net/man/1/rdiff)

[2] [http://scriptiesonline.uba.uva.nl/document/490827](http://scriptiesonline.uba.uva.nl/document/490827)

</comment><comment><USER45> I wonder how this deals with renamed files.

If it doesn't, one should avoid renames of bigs assets when working on updates of an already released app.

</comment><comment><USER46> The question, why took this so long ? 8 years for figuring out how to make differential updates...

<comment><USER47> Read the thread.

[https://news.ycombinator.com/item?id=13121903](https://news.ycombinator.com/item?id=13121903)

</comment></comment><comment><USER48> Is this basically something like git? Isn't it crazy that something like this hasn't been implemented until today?

<comment><USER49> I am not sure what you mean by git. This is binary data diffing using bsdiff, applied at the individual file level instead of the whole package (single file).

</comment></comment><comment><USER50> Nice work there but who really talks like this?

"in order to provide users with great content, improve security, and enhance the overall user experience"

<comment><USER51> The author is listed right above that: Posted by Andrew Hayden, Software Engineer on Google Play.

</comment></comment><comment><USER52> The author of this post now works at Amazon!

[https://www.linkedin.com/in/andrewhayden](https://www.linkedin.com/in/andrewhayden)

<comment><USER23> They deserve serious kudos for managing to finish and ship their product at Google first. (Another commenter here said the project started 3.5 years ago, wow)

</comment></comment><comment><USER53> If only they spent this much effort in getting Android OS updates to consumers.

</comment><comment><USER54> How does this differ to the delta updates that came out over four years ago?

[http://www.androidpolice.com/2012/08/16/google-flips-the- swi...](http://www.androidpolice.com/2012/08/16/google-flips-the-switch-on- smart-app-updates-in-the-play-store-video/)

<comment><USER55> Read the article before commenting?

<comment><USER54> Which part of the article covered the 2012 announcement? There was the piece near the top: `earlier this year, we announced that we started using the bsdiff algorithm (by Colin Percival).`

But earlier this year would be 2016, not 2012 so I assumed that this was something else.

<comment><USER55> And clicking on the link in your quoted text would give you this:

> Google Play has used delta algorithms since 2012, and we recently rolled out > an additional delta algorithm, bsdiff</comment></comment></comment></comment>


Ask HN: Are web security seals worth the money? - <AUTHOR>

Hi All,
Our company is a leading provider of financial services. From our website we gather a lot of client information. We are currently looking into security verification seals to give clients some reassurance that their information is safe.
At the moment we are looking at Trust-Guard & Truste. Does anyone have experience with either of these providers? And if so what has your experience been?
Thanks, TST <comment> tomh- Having any logo which looks like a "security seal" boosts conversions by a percentage, I recommend to put something up, even if its a logo linking to your own page with information about the security using an A/B test to see if it works for you.</comment>


Want to make free calls? Dial this number - <AUTHOR> http://timesofindia.indiatimes.com/tech/tech-news/internet/Want-to-make-free-calls-Dial-this-number/articleshow/31456232.cms

<comment><USER2> I remember using an ad-supported phone calling service when visiting a cousin in Sweden. It sounds similar to this (ads that both people hear every few minutes) but it didn't use call-back.

Can any Swedes remember this service, or comment on whether it still exists?

</comment><comment><USER3> "nearly four lakh FreeKalls have been made so far"? A lakh or lac (/ˈlæk/ or /ˈlɑːk/; abbreviated L) is a unit in the South Asian numbering system equal to one hundred thousand [Wiki]</comment>


Augmented Reality & Motion Capture Shopping App - <AUTHOR> http://www.youtube.com/watch?v=NxQZuo6pFUw <comment><USER2> This is a start in the right direction. If this was combined with markerless motion capture and cloth physics simulator, then this might be more viable, but that's obviously a good ways off. Until something like that is created, I'm not sure how useful I would find this.</comment>


Serial young entrepreneur and previous 2x Y-Combinator applicant launches new venture revolutionizing real estate industry. - <AUTHOR> http://www.atomicbrokers.com <comment><USER2> Well all right then. I'll consider it revolutionized.</comment>


Ask HN: What are some exemplary React apps to study? - <AUTHOR> <comment><USER2> Take a look at [1] React Redux App by Richard Park. It has very well-formed app structure.

[1] [https://github.com/r-park/todo-react- redux](https://github.com/r-park/todo-react-redux)

</comment><comment><USER3> Not a full fledged app, but React Boilerplate ([http://reactboilerplate.com/](http://reactboilerplate.com/)) is pretty nice in terms of the structure and pattern it follows.

Check out this video which explains it pretty nicely: [https://vimeo.com/168648012](https://vimeo.com/168648012)

</comment><comment><USER4> [https://github.com/bigardone/phoenix- trello](https://github.com/bigardone/phoenix-trello)</comment>


How to Right a Good Internal Company Newsletter - <AUTHOR> http://www.inc.com/janine-popick/2010/11/how-to-write-a-company-newsletter.html

<comment><USER2> Given the nature of the post, it seems ironic you confused "Right"/"Write" in your title.</comment>


Lightning Strike Kills More Than 300 Reindeer in Norway - <AUTHOR> http://www.nytimes.com/2016/08/30/world/europe/hardangervidda-norway-lightning-reindeer.html <comment><USER2> Detecting lightning automatically isn't that hard-- it's a sudden burst of powerful, wideband static. There are several sites where users collaboratively trilaterate lighting strikes by sharing precise time-of-arrival data-- including live maps! [http://en.blitzortung.org/live_dynamic_maps.php?map=11](http://en.blitzortung.org/live_dynamic_maps.php?map=11)

The process is fast enough that in a thunderstorm, you'll probably get a packet specifying the lightning's coordinates before you hear the thunder!

Here's what the storm looked like: [http://i.imgur.com/mkIqnlD.gif](http://i.imgur.com/mkIqnlD.gif)

<comment><USER3> You do, and on lightningmaps.org you can even watch on the map the thunder front traveling at the speed of sound. Disappointingly, though, it appears that the lightnings positioning isn't reliable enough to actually predict when you'll be reached by the thunder, let alone go out there and look for a nice piece of fulgurite!

<comment><USER4> Do what the guy does in Sweet Home Alabama: go off into the wilderness, put up some big metal rods, and after a thunderstorm check the rods for lightning strikes.

<comment><USER5> Or go build your own lightning field like Walter de Maria

[https://artappreciation101.files.wordpress.com/2011/06/demar...](https://artappreciation101.files.wordpress.com/2011/06/demaria_the- lightning-field-19771.jpg)

[https://www.theguardian.com/artanddesign/jonathanjonesblog/2...](https://www.theguardian.com/artanddesign/jonathanjonesblog/2013/jul/29/walter- de-maria-art-lightning)

Of which there is a great description if Geoff Dyer's latest book White Sands.

</comment></comment></comment></comment><comment><USER6> I can remember a similar event in the Alps in France. What was interesting (it was with cows) is that only the cows showing or turning their back to the lightning strike were killed.

It was because of the potential difference between the front and back legs (you have more than 1m), the ground resistance was higher than the wet cow and the current just went through the cow. When the cows were not facing (or back) the strike, the ground resistance was lower, the current did not went through the cows.

So, if you are wet, in the mountains and lightning strikes are firing, make small steps :)

<comment> saiya-jin this is a basic advice on any electrical course I had to go through. if things get really hairy, you should only jump around with both feet together, and have arms close to the body (not so easy somewhere on the wilderness)

<comment><USER7> Actually you hop on one foot, that way the current path does not go fully across your heart.

<comment><USER8> Not quite. It is the voltage potential caused by having your feet on two different pieces of ground, which have a voltage gradient between them, that causes the current to flow. If you keep your feet together you are less likely to be harmed.

The farther apart your feet are the greater the voltage between them. This is called "step potential" in electrician and lineman safety training.

<comment><USER7> Yes, but if are hopping on one foot, you have no opportunity to complete the circuit with both feet.

<comment><USER9> but if your feet are touching it's the same thing, because you've shorted the circuit. Also, any chance of arcing to the ungrounded leg?

</comment></comment></comment></comment></comment></comment><comment><USER10> I'm probably a terrible person for this, but my first thought was: [http://media- dominaria.cursecdn.com/attachments/104/60/63503...](http://media- dominaria.cursecdn.com/attachments/104/60/635032499431831568.jpg)

</comment><comment><USER11> "They were standing on a hill, moving up that hill."

Bad idea in a thunderstorm. It is recommended to stay in low-lying areas instead: [http://www.wildbackpacker.com/wilderness- survival/articles/s...](http://www.wildbackpacker.com/wilderness- survival/articles/surviving-a-lightning-storm/)

<comment><USER12> Technically true but teaching that to reindeer may be something of a challenge :)

<comment><USER13> They just learned by natural selection :)

<comment><USER14> IANAB, but lately i have been thinking a lot about evolution and how it changes a species. The context i am considering is social evolution, but i think the rules of biological evolution apply. From what i understand, Reindeer have learned nothing from the event in general. In the localized area, a bunch of Reindeer are out of the gene pool and any of their kin that are close enough to encounter their carcasses but not the the cause are going to freely associate the deaths with whatever the perception sensory suite allows. My assertion: Evolutionary "learning" occurs over a long period of time through strings of events with similar inputs and outputs.

Not an argument or rebuttal; your comment made me think about it. :]

<comment><USER13> Nah, I was just kidding. Unless there are similar events occurring during a lot of time, making those reindeers that prefer staying more separated and in lower areas have a significant advantage, and these preferences are hereditary...

</comment></comment></comment></comment></comment><comment><USER15> Wait a second. Anyone else see the 'The Tick' pilot? Wasn't this the opening sequence, Superion falling to earth and killing a bunch of reindeer?

That aside, this must have been one heck of a lightning bolt. I've been within few meters of one with no ill effect. It must have been nothing compared to this monster. Or could it have been several bolts?

</comment><comment><USER16> An recently, in July: 1500 reindeer dead, 40 humans hospitalized amid anthrax outbreak in Siberia - [https://news.vice.com/article/reindeer-dead-humans- hospitali...](https://news.vice.com/article/reindeer-dead-humans-hospitalized- anthrax-outbreak-russia-siberia)

</comment><comment><USER17> Not to sound to heartless but I wonder if the animals will be taken for meat, buried, or nothing done at all (ie left for nature). What normally happens in these situations?

<comment><USER18> Literally the last sentence in the article:

> "The normal routine in such cases is to let nature take its course."

<comment><USER17> How embarrassing! I must have coalesced that last paragraph with the advertisement and missed it.

</comment></comment></comment><comment><USER19> I wonder if this ever happened to ancient armies

</comment><comment><USER20> Music in the video really ties it together, but they really should've dropped the bass when switching to the aerial footage.

</comment><comment><USER21> I can't shake the feel that the thunderstorms have gotten stronger in recent years.</comment>


Neoliberalism has had its day. So what happens next? - <AUTHOR> https://www.theguardian.com/commentisfree/2016/aug/21/death-of-neoliberalism-crisis-in-western-politics <comment><USER2> > The banks were bailed out; hardly any bankers on either side of the Atlantic > were prosecuted for their crimes; and the price of their behaviour was duly > paid by the taxpayer.

Banks paid back everything borrowed to them:

> "Barack Obama says banks paid back all the federal bailout money". Not all > of the banks who received TARP money have repaid the loans. In fact, > 400-plus banks are still on the government's red list -- more than have > completed the payments -- and, according to the GAO, some may never pay the > money back. But, the fact remains, due to interest, dividends and other > revenue streams, the government has received more money back ($266.7 > billion, according to the Treasury) than it handed out to banks under the > bailout law ($245.2 billion). We rate this claim Mostly True. [0]

Furthermore:

> U.S. banks have crossed a significant post-financial crisis milestone, > tallying more $200 billion in fines paid out regarding questionable > behavior. [1]

How did neoliberalism survive post-financial crisis? Politicians discovered that they can just hit up banks for tens of billions of dollars in fees and use the proceeds in any way they want:

> Yet some of the biggest chunks of money stayed with the entity that levied > the fines in the first place. Of $109.96 billion of federal fines related to > the housing crisis since 2010, roughly $50 billion ended up with the U.S. > government with little disclosure of what happened next, according to a Wall > Street Journal analysis.

Yes, bailouts are bad, but let's not pretend like politicians are not complicit in promoting this type of crony capitalism.

[0] [http://www.politifact.com/new- hampshire/statements/2012/oct/...](http://www.politifact.com/new- hampshire/statements/2012/oct/25/barack-obama/barack-obama-says-banks-paid- back-all-federal-bail/)

[1] [http://www.cnbc.com/2015/10/30/misbehaving-banks-have-now- pa...](http://www.cnbc.com/2015/10/30/misbehaving-banks-have-now-paid-204b-in- fines.html)

[2] [http://www.wsj.com/articles/big-banks-paid-110-billionin- mor...](http://www.wsj.com/articles/big-banks-paid-110-billionin-mortgage- related-fines-where-did-the-money-go-1457557442)</comment>


Code2040 Summer 2017 Application - <AUTHOR> https://fellows.code2040.org <comment><AUTHOR> Calling all Black & Latinx students interested in interning in the Bay Area and joining a large and supportive network of URM and allies.</comment>