Hoodie: very fast web app development - <AUTHOR> http://hood.ie <comment><USER3> Unlike a lot of comments here, I don't need to be sold on the concept. As luck would have it, I am actively building an app that runs mostly in the browser using CouchDB, Kanso, and PouchDB. It's a pretty amazing tech stack; I get a local DB in the browser, which I can sync with the main DB in the cloud the client is connected. Plus I get some nice APIs for stuff like sessions, registering users, security, validation, etc.

I see you've already mentioned switching to PouchDB on your backend, and you're already using CouchDB for the cloud, so it seems like you're basically competing straight up against Kanso. So.... How is what you're doing different from what Kanso does? Why should I use Hoodie over Kanso?

I see this code in your docs:

        hoodie.account.signUp('firstname.lastname@example.com', 'secret')
        hoodie.account.signIn('email@example.com', 'secret')

Very nice. But Kanso has:

        users.create('testuser', 'testing', {roles: ['example']}, function (err){});
        session.login('testuser', 'testing', function (err, response){}});

Seems like six of one, half a dozen of the other. What's your killer feature? Kanso has shaky documentation and is currently more aimed at couchapps (they strongly assume that you'll serve your frontend code from CouchDB), but I'm not sure how that really differs from Hoodie. :)

If we take the overall architecture as a given (browser based app, PouchDB or equivalent for local storage, syncing with CouchDB in the cloud, and node workers handling the backend work), what's the focus of Hoodie that sets it apart from Kanso?

(In any case, nice stuff. I think the concept is great, and there's definitely room for several competing frameworks in this space.)

<comment><USER4> hi

I am trying very hard not to be dumb, but this is beating me

seemingly Kanso? and Hoodie both have JS stubs at the client and server end, these synch with each other, then synch with whatever you are using at client (say backbone) and server (node? CouchDB? My Postgres server behind my python app?)

If thats roughly right, why do I do this? I control my server apps no? Why wrap them in another layer? Json->backbone is hardly a difficult synch operation - I mean its called synch().

I really am interested in understanding the philosophy and driving ideas here - everyone has a moment of clarity where they can see how to build an app or a framework, and how it will come together - I am not attacking you (yet !) - please describe your moment of clarity.

<comment><USER3> Okay, first off, let's discuss a standard three tier webapp architecture:

        Browser   <--->   Server   <--->   DB

You get some HTML/CSS/JS from the server, you display stuff to the user. As the user interacts with your app, HTTP requests hit your server which processes them. As needed, the server will make requests to the DB, which will reply with data, which then gets passed back to the browser. Everyone knows how this stack works, I hope. :)

It's simple and it justs works. Almost every website you use works this way. But for webapps, it's not always ideal. Let's discuss some of the issues:

First off, this architecture is "always online". If the browser loses connection to the server, then when your user clicks on a button nothing will happen. Not too bad in a desktop environment, but with laptops or - <USER2> - mobile clients this isn't ideal.

Worse, what if you're trying to keep the state in sync with multiple clients (like Trello does)? You're probably using websockets or long polling or similar techniques so that the server can tell clients about the actions of other clients. A network hiccup might cause you to miss one of these updates; detecting and recovering from this sort of sync error is problematic. Generally the solution is to throw away the last few bits of work the user did, and force a full reload and re-fetch the current state. Which is fine, in most cases. Unless the work the user is difficult to replicate, or if the application state is quite large. Trello takes a second or two to grab the current cards; if your app takes 30s to grab its data, you may want to avoid this though. :)

In fact, the more you want to replicate the feel and functionality of a classic desktop app, while allowing multiple users and clients, the more you start to struggle against this architecture.

What other architectures are there? Well, there's one obvious one:

        Browser   <--->   CouchDB   <--->   Workers

CouchDB is kind of unique among databases because it speaks HTTP natively. It can serve your app to the client, and then turn around and have the clients connect directly to it. It can't, eg, send emails, but what you can do is use CouchDB as a queue. Add a document that says "send an email to X", then you spin up some worker processes that monitor it and process any documents like that as they appear.

CouchDB also has a _changes feed that makes a lot of sync tasks very easy. It also does master/master replication, and is generally a really nice database to work with from a devops point of view. Still, we're missing a crucial bit of magic to really improve over a classic three tier webapp. And that is...

...local DBs, in the form of PouchDB for browsers or TouchDB (apparently called Couchbase Mobile these days, although confusingly it's unrelated to Couchbase, which in turn is unrelated to CouchDB. Don't ask.) for Android/iOS. CouchDB is really _really_ good at syncing and replicating, so now we have a new architecture:

        Browser/Mobile App  <-->  PouchDB/TouchDB  <-->  CouchDB  <-->  Workers

This isn't appropriate for every task. For one thing, it's quite a lot more work to implement (partly due to inferior tooling which projects like Hoodie are, thankfully, addressing). But if you really want the best possible user experience, you want it to work even if your on an airplane or in a tunnel, you want a really snappy UI, you want each client to see the changes made by other clients in as close to real time as practical, you want a consistent experience for native mobile apps and webapps running in a browser...then this is the architecture for you.

So to directly answer your questions:

First, Kanso (by default) wants to do something like:

       Browser  <-->  CouchDB

Which is a really slick, really easy way to do single page applications that don't need local persistence or server processes. But it's not hard to add them on, in which case you have the architecture above. You seem to be imagining something like this:

       Hoodie Client  <-->  Hoodie Server <-->  CouchDB

But that's not the idea at all. You're also asking why not do this:

      Backbone  <--->  Python   <--->  Postgres

And that's a perfectly fine architecture. It works, it's rock solid, and it's easy to code. For many apps (maybe most) it's the best possible architecture (well, I'd suggest Knockout over Backbone, but that's personal preference).

But for the project I'm working on right now, we decided that wasn't _quite_ good enough. :)

<comment><USER5> If my phone has no signal I'd rather see an error message. I don't want the app to pretend everything is fine when that's not the case. If I press "Delete" in the app, don't pretend the thing is deleted if it's not really deleted yet, that's deceptive. What if I take off on my boat and I press "Send" and your app says "Sent!" but it's really not sent because now I've left the coast--your app just lied to me.

<comment><USER3> With all due respect, I think you're wrong.

If I'm writing an email in the Gmail app on Android, what I want when I hit send is for the app to save the email locally, and then do its best to try and deliver it. Ditto for, basically, every other app on the phone that talks to a remote server. If I try and send a tweet, I want it to go "sure", and then be able to go on about my business while it works in the background. (And, again, that's how the actual twitter app works.)

And if I'm using some app that lets me take notes about a meeting or interview and syncs them with the cloud, I sure as HELL want to be able to take those notes _and_ save them locally, even if the cloud sync had to be delayed. "Sorry Mr. CEO, but I can't seem to get wifi in this meeting room. Can we go out on the balcony and you just run back over those last few points?"

Now, yeah, I need some way of checking to see if it has actually sent the email or whatever. And if it fails, a notification that it has done so is not amiss. Mobile apps, typically, already do this sort of thing. (For gmail for android, you can check the outbox to look for unsent emails, and I believe twitter for android will notify you if it fails to send a tweet.)

To the extent that you're just saying that the app needs to have a UI which doesn't lie or surprise the user, then I agree wholeheartedly. If the user cares whether or not the cloud has been updated, or the message has actually been sent, don't lie and claim it has when it hasn't!

But if you're saying that as a general rule we need to _not_ handle network sync stuff in the background, I think that's nuts. Nobody wants to sit there with an email draft open hitting send every so often to see if the 3G signal is good enough to talk to the mail server yet. Nor do we want to save a bunch of emails as drafts when offline, then have to go open each one when we get net access again and send them. (Come to that, in your model, could we even save an email as a draft in gmail without net access? Remember, the drafts folder is synced. Do we want the app to claim that a draft has been saved when it's not accessible from your desktop gmail client? Isn't that another case of "pretending we've saved a draft when it's not really saved"? If not, why not?)

TL;DR: Good UI that doesn't lie to the user is important. That's not a good reason to cripple your apps functionality so it doesn't do anything you can lie about.

<comment><USER5> You think I'm wrong but you wholeheartedly agree? Funny. Every situation is unique but bottom line: you can't receive content without a connection. Most apps NEED a connection to function. So an error message "No Connection, Retry?" is both common and expected behavior.

</comment></comment></comment></comment></comment></comment><comment><USER6> Looks lovely, but I wish people would focus less on fast development and more on something that better maintainability 2 years down the road.

When it comes to the lifecycle of most applications, the speed of developing something new has a marginal impact on the overall costs, and most tools and frameworks than enable rapid initial development tend to suck once you have a big app on top of them. This is why we fawn over a tool when it's shiny and new, and bitch about how much it sucks 3 to 5 years later.

<comment><USER7> > ... the speed of developing something new has a marginal impact on the > overall costs

But it has a large impact on the overall probability of getting off the ground at all - and this is what most projects (and people) never do.

<comment><USER8> > [dev speed has] a large impact on the overall probability of getting off the > ground

I don't know. This seems common sense but it could be a cliché too.

If we look at other domains, it is not always true. For instance, if you write novels that are not tied to some evanescent trend like a pop star's private life, then the chance of getting off the ground if not directly linked to the timing of publication. It would be the same if you are creating a newly designed chair.

Even on the web, let's take two success stories: when Wikipedia or Twitter were created and made public, I don't remember seeing five other contenders running behind them, and being only two-features-months behind. Same with, say, Minecraft.

I am all for quick development, because it is more fun, but not because it would magically increase to probability of success, which is too low to compute and unpredictable anyway.

I am also all for choosing carefully which path you take, especially when you cross virgin territories. You don't want to find yourself spending a night half naked with a toothpick for your sole defense in Borneo's jungle.

<comment><USER9> Let's say today 1 of 1000 people is able to create an app to solve a problem, let's say time time tracking. Now imagine we change that drastically, to 100 of 1000 or even more. People will start to build their own little apps to solve their own little problems. And if these break, nobody cares, but them.

For example, friends of mine like to go climbing. They've build an app in an afternoon, just for their group, to keep track of what tracks they finished and how they liked it. I think this is fascinating. I wonder what would happen if students would learn to build simple apps like that in school, instead of Excel?

<comment><USER10> "Now imagine we change that drastically, to 100 of 1000 or even more. People will start to build their own little apps to solve their own little problems. And if these break, nobody cares, but them."

This was the old dream of Hypercard, and it worked very well for a long time. <http://en.wikipedia.org/wiki/HyperCard>

Both Danny Goodman and Bill Atkinson have stated publicly that they got programming questions from every kind of person: small business owners, school teachers, ski bums, taxi drivers, etc.

Hypercard let non-programmers build simple apps, and so a lot of apps were created that would never have been created if app-creation was left in the hands of professional programmers. And many of those little apps were tailored to what the creator needed at that moment, in a way that could never happen if a professional programmer drew up some spec.

<comment><USER11> It's ironic that the tool that has replaced Hypercard is in fact Excel. Excel makes it amazingly easy for non-programmers to develop simple applications

</comment></comment></comment></comment></comment></comment><comment><USER12> I haven't had time to look at try out the walkthrough, so here's a kind of dumb question that wasn't answered explicitly in the docs: how does it work with existing front-end frameworks, such as Backbone, Angular, Ember, etc?

I ask this because of the description found on the Github page: <https://github.com/hoodiehq/hoodie.js>

       hoodie is a JavaScript library that runs in your browser.
       It gives you

          user authentication
          data storage and sync
          sharing
          emails
          and so much more

None of these things are really handled by the client-side frameworks mentioned above, so I'm wondering if Hoodie is basically drop-in easy for existing projects.

<comment><USER9> at first, Hoodie is just a Store for data. So instead of Backbone, Angular etc storing their data via AJAX on a server, or in LocalStorage, we can very simply build adapters for all of the MV* frameworks to use Hoodie as their store.

This way, you get accounts and data synchronization for free. And offline ;-)

Emails and other future modules like payments are just extensions of the available JavaScript API in the browser.

<comment><USER4> I have not read any more than the intro linked, but I am kind of "concerned" by the big claims

      Behind the scenes, Hoodie takes care of account creation,
      email validation and all the boring backend tasks you have
      to think of.

yeah, but _how_. You still need to _do_ account creation, even if you are storing the hoodie-account-object locally in some fashion.

Am I being dense, or is this like a nice client framework that still needs to talk AJAX to the backend, so still needs to allow oAuth, so still needs to change code if the API changes, and so on.

"We just want to make apps, and add billing by Monday" is a bit worrying as a goal to me...

edit: oh, so this stubs out the server side at the client ? It seems like the same amount of work to keep backbone models in synch with the server as it does to keep hoodie? What do I gain?

<comment><USER13> > yeah, but how.

Hoodie comes with a backend that does all that stuff that can only be sensibly done server-side for you.

> edit: oh, so this stubs out the server side at the client ? It seems like > the same amount of work to keep backbone models in synch with the server as > it does to keep hoodie? What do I gain?

@gr2m can you take this one?

<comment><USER4>
      Hoodie comes with a backend that does all that stuff that
      can only be sensibly done server-side for you.

wibble? I run your client and your server and talk to the stubs both ways? Why? Sounds like Microsoft.

Edit: I take back the MS jab - unnecessary when simply trying to understand something. My apologies

<comment><USER13> Yeah, we are basically copying Microsoft.

Edit: No harm done :)

</comment></comment></comment></comment></comment></comment><comment> old-gregg I have a question for experts in Hoodie and/or other similar projects (like Meteor):

How are database connections managed? If every client (browser) basically talks nearly directly to the database, don't you end up with of thousands of database connections?

Large number of connections is not a problem by itself, and servers like nginx handle them well, but I am not aware of a database which will feel comfortable being exposed this way. MongoDB, for example, really starts struggling with 2K+ connections. PostgreSQL can barely handle a hundred.

Is this why CouchDB is used? Is connection pooling used on the server? Or maybe the connections are short-lived and never persistent? Basically, how does this work with large number (say, 10K) of concurrent visitors?

Thanks!

<comment><USER13> As long as the underlying OS can provide file descriptors, CouchDB can handle the concurrent connections through the magic of Erlang, both persistent and shorter lived.

There are obvious limits to this, but the Hoodie architecture allows easy scale out (more DB servers and manual sharding, or a dynamo-like BigCouch, more workers etc.) that we'll get to making use of once Hoodie apps become that big.

<comment><USER14> Magic of Erlang? I hate to be cynical, but have you got benchmarks to back that up?

<comment><USER13> Yeah.

</comment></comment></comment></comment><comment><USER15> Neat, I really like the idea of being able to sync local storage with a Couch backend.

> Hoodie currently only runs on OS X

Oh, a hipster framework.

<comment><USER13> > Oh, a hipster framework.

Never attribute to hipsterism what could adequately be explained with picking priorities :)

The hoodie backend works anywhere node and Couch work (anywhere, really), it's just that the local dev setup with the fancy domains and everything is tailored to Mac OS X. We have Linux support in the works.

<comment><USER16> Yeah, a hipster framework.

<comment><USER17> I'd say OSX is pretty mainstream for web development in my experience. If Hoodie was only available for the Atari ST then perhaps it would be a hipster framework.

</comment></comment></comment></comment><comment><USER13> Hey there, other @hoodiehq dev here. Here is a bit longer intro to Hoodie: <http://hood.ie/intro.html>

And a presentation from earlier at @berlinjs including a live demo: <https://www.youtube.com/watch?v=X3Ttb0BD8pg>

<comment><USER18> Nice presentation. In the description of back end modules, you mention payments as a future capability. What kind of payment service would you be able to work with? (I don't see how you could handle card details in the Hoodie architecture). Thanks.

<comment><USER9> we definitely won't send any credit card information to a hoodie server. Instead we use a service like stripe, but then receive the payment notifications or errors, which lets us notify the user if payment worked or not.

</comment></comment></comment><comment><USER19> Self hosted Firebase, nice!

How do you handle permissions by the way? is it the same concept as Firebase's? how do you handle data validation? (e.g. preventing users gaming the data, e.g. saving all their stolen ebooks in your system by using a console, or just cheating in a game by doing illegal moves?)

<comment><USER9> permissions is handled directly by CouchDB. Every user has an own database that only he/she has access to. Synchronization between accounts can be done via shares.

We don't have validation at the moment. You can store whatever you like, it's just JSON. We could add validation at some point, but it's currently not on our agenda.

We try to keep it really simple at the moment, getting the core right, then empowering others to build modules on top of it.

</comment></comment><comment><USER20> I'm a Unix developer and would love to have a play, a little bit disheartening to see that the installation instructions are OS X exclusive.

<comment><USER13> I'm a Unix dev too, check out my beard in the linked video :D

Like mentioned elsewhere in this thread, other *nixen support is in the works. Hoodie core is platform agnostic, we just chose Mac as the dev environment because we figured that was what most people use. We are definitely not exclusive and would love to see any help towards getting it run on other systems :)

</comment></comment><comment><USER21> This looks pretty optimal for a little app I'm building. I've been bolting bits to Sinatra and using it as a half-API-half-template engine for a frontend I built, this looks far cleaner. I'm curious about security though, what's stopping a user from plugging in random data via the console?

<comment><USER13> Nothing, all they can do is fuck up their own data.

<comment><USER22> Which sounds kind of bad, if that is the case.

1\. Even if the users are idiots you shouldn't let them ruin their own application experience. 2\. I would never put anything remotely connected to security and user privileges in the same storage accessible by users, so I would have to set up a separate service.

<comment><USER13> 1\. what if that idiot messes with gmail's web console?

2\. Security handling is purely server-side. You can't fake your way without using a proper auth-channel.

<comment><USER22> The distinction here is that Hoodie is supposed to free you from dealing with servers, but that is currently limited to scenarios where you have users with uniform access privileges and no concerns about users messing around with their database information. So until they add modules most projects will have to get down and dirty in the end if they want to attach any kind of privileges to users. In both points 1 and 2 you need to have some server side logic beyond Hoodie.

<comment><USER13> Hoodie can only promise to free you from worrying about the backend by providing one that you can just use.

The sharing module e.g. makes heavy use of server side logic and database security and access control features. The Hoodie frontend just makes it accessible to frontend devs.

</comment></comment></comment></comment></comment></comment><comment><USER23> It's odd that no one's mentioned it yet, but this reminds me most of Meteor.

How does this compare to, say, Meteor?

<comment><USER9> hood.ie has a different approach. Meteor brings backend logic to the frontend. hood.ie tries to hide the backend entirely and provide an API that feels natural for the frontend environment.

The JavaScript API is what we care about most. We do currently have the hood.ie backend implemented in Node/CouchDB, but it could be everything, really. The frontend developer doesn't care.

<comment><USER13> We love Meteor though :)

</comment></comment></comment><comment><USER24> Berlin and Zurich seem to be doing great lately. I'm also planning on building a startup in Zurich in the coming months. How about a Hacker News Zurich meetup?

<comment><USER25> Would be cool. We had one over a year ago, and the attendance was good, but no one organized a follow-up.

See <http://techup.ch> and meetup.com for what's going on in the tech meetup space in Zurich. I'm often at zhgeeks, jszurich, web tuesday.

</comment></comment><comment><USER26> It's awesome when you read posts on HN about interesting things and realize your friends were involved. Congrats gr2m! (P.S... +1 for an Angular adapter =))

<comment><USER9> 10 cary-minutes is all it takes ;-) Thanks Dylan

</comment></comment><comment><USER27> I think promising "we want to enable you to build complete web apps in days" might be promising too much. Programming is difficult, there is no silver bullet, and a different data store certainly won't change that.

Interesting choice of couchdb. Any reasons for that compared to the alternatives?

<comment><USER28> If I had to guess, the fact that HTTP and JSON are CouchDB's native tongue (removing the need for a middleman in front of the DB) was likely a factor.

<comment><USER13> That, and that CouchDB has Sync already figured out. Hoodie didn't have to invent this. Sync is hard, leave it to the experts.

</comment></comment></comment><comment><USER29> @hoodiehq : are you using something similar to PouchDB ?

<comment><USER13> We have our own little lightweight version of PouchDB that isn't really a version of PouchDB. It works well for now, but we might migrate to Pouch later.

<comment><USER30> Yeh we should definitely work on this when you are in town :) the storage part of pouch is nearing being complete and stable, the server infrastructure was next on the list (after docs etc)

<comment><USER9> <3

</comment></comment></comment></comment><comment><USER31> I know everyone hates that question.. (I hate it too), but what about non-js users or, maybe more importantly, SEO or crawlers? Honestly, this is what is holding me back to go pure client-side for most of my projects.. I know there's the phantomJS server-side hack but how does that work in practice?

Also, something else that really trouble me with javascript client-side is that, often, when something bugs, _everything_ just stop working. I.e. links don't work anymore, button obviously don't.. Is there a better solution than a hard-refresh? Would a first-level try/catch solve this problem?

Thanks!

<comment><USER32> > non-js users

the web is no longer just text. if you need to consume text then you need a text-api client, which is what web browsers used to be.

<comment><USER33> I run noscript. If I encounter a site that's broken or useless for seemingly no reason (e.g. not an interactive thing like Google Maps), I often don't even whitelist; I just leave. Relatedly, I've stopped clicking on Photobucket or Blogspot links.

I kinda resent being expected to run your pile of arbitrary code just to render static text and images on my screen—something that worked just fine without JS twenty years ago.

<comment><USER34> While I admire your principled stand - you are in a dwindling minority so there isn't much reason to factor in people such as yourself when making technical decisions.

The SEO issue is a stronger argument but Googlebot now seems to be executing javascript in some cases so even that might cease to be an issue.

<comment><USER33> It's not just people like myself; it's people writing one-off scrapers, people writing new search engines or browsers (Google is not the entire universe), _everyone_ when you forget a brace and break all of your JS, etc. The Web is not and has never been merely human beings sitting at a keyboard and using one of three known GUI browsers.

</comment></comment></comment></comment></comment><comment><USER35> so bots can sign up easily:

while (true) { hoodie.account.signUp(_.uniqueId() + '@gmail.com', _.uniqueId()) }

great effort though, but I still don't think you've got client/server separation right.

we've built something very similar over a year ago: an offline client db backed by memory/websql/indexed-db that replicates with dynamo-db so the app works offline. Looked at Couch/Pouch solution but decided against Couch for performance reasons.

Might open source it later once we figure out the user management/security bit. I still find the existing solutions (including Firebase) not ideal.

<comment><USER36> I was thinking there might be some sort of throttling/banning mechanism within the signUp function itself, but then being client based that could be easily circumvented unless there's some server side logic aswell. I wonder whether there's a sane way to handle this client side short of minify and/or obfuscate the code and hope for the best.

<comment><USER37> Let me refer to a great answer Lazare gave further up (more generally about sync, but very useful nonetheless): <https://news.ycombinator.com/item?id=5517632>

</comment></comment></comment><comment><USER38> Apologies for an off-topic question - how did you go about registering a .ie domain name? I've been wanting to register one for a while, but the last time I looked there was a requirement to show proof of residency (or a business address) in Ireland.

Edit: The reason I ask is that it seems like the developers are living in Zurich and Berlin, not Ireland. I live in SF, so can't fulfil the residency requirement.

<comment><USER37> No problem :) There are actually more than three of us working on Hoodie, one (actually two by now) of them an Irish national. The about section just lists the core team, if you will.

<comment><USER38> That makes sense. I guess I will just have to go without until they (eventually?) open up registration to foreigners! Thanks for taking the time to reply.

</comment></comment></comment><comment><USER13> Alright, Team Hoodie is calling it a day. Thanks for the lively discussion and nice feedback, you've been great! :)

See you again tomorrow CEST. <3

</comment><comment><USER39> I'm very interested in using this together with emberjs, but honestly I'm having trouble understanding what Hoodie actually does. Why do I need Hoodie to go with my CouchDB? Is it only for the local storage?

</comment><comment><USER40> Very cool :).

A few questions. What is the strategy is for concurrent modification of data? What are the consistency guarantees for the sync operation?

<comment><USER9> In the current implementation, CouchDB takes care of it. In case of a conflict, both versions are kept, with one winner. The other version can be recovered and the conflict resolved, both automatically with a worker process or by the user.

</comment></comment><comment><USER41> As a new web developer I'm very interested in this. Looks like a not so complex next step after building basic static websites.

<comment><USER9> Welcome to Hoodie world :-)

</comment></comment><comment><USER42> definitely going to check this out in depth soon, and build some "dreamcode" with it to test it out!

<comment><USER13> That's the Hoodie Spirit <3 :D

</comment></comment><comment><USER43> How is hoodie solving / handling conflicts when its syncing local storage to the remote database ?

</comment><comment><USER44> Test comment for <http://hackerfollow.com>

</comment><comment><USER45> Hoodie devs: Just out of curiosity, what frontend framework (if any) is used on minutes.io?

<comment><USER9> backbone

</comment></comment><comment><USER46> How does this compare with Firebase, besides offering a way to host everything locally?

</comment><comment><USER47> Does "hoodie" mean "juvenile delinquent" in the US as well as the UK?

<comment><USER48> No.

</comment></comment><comment><USER49> All art requires courage. ~Anne Tucker</comment>


Azure cracks, AWS strong: Covid-19 stress-tests the cloud - <AUTHOR> https://interconnected.blog/covid-19-stress-tests-cloud-azure-aws/ <comment><USER2> > Since Microsoft's other businesses are either hardware or office software, > with the exception of maybe XBox's gaming, it generally lacks operational > experience in running "always on" services and preparing for unanticipated > traffic surges.

Uh, as a former Microsoft equivalent of a SRE this is a utterly ridiculous statement.

I was in the Windows division on a team handling all the incoming telemetry - we definitely knew how to prepare for traffic surges and keep our services "always on".

Has the author never seen Office 365? Bing? The entirety of Azure in general that has dozens if not hundreds of publicly facing always on services?

<comment><USER3> >> we definitely knew how to prepare for traffic surges and keep our services "always on"

Okay, but as a heavy customer of AWS for 10 years and Azure & GCP for 6 years... there's some very valid criticism of the capacity planning and elasticity.

I hit capacity, provisioning and scaling issues all the time in Azure regions. And many times the cause is not obvious because most services seem to be unaware that the resources are exhausted, and fail with seemingly unrelated errors and partial failure.

edit: and to be fair, I'm not saying it's easy to build something like Azure or that I would do better. And I have a huge list of cases where Amazon's product engineering is terrible compared to Azure's. But... the problems I've seen all week in the UK aren't imaginary. I've had no such problems on Amazon.

<comment><USER2> I don't disagree with anything you are saying here, but I was heavily taken back by the statement the author made as it felt right out of the early 2000's, ignoring the last 20 years of Microsoft's shift to services and the cloud.

Azure clearly prioritized massively increasing their geographic footprint without the availability zones that AWS almost always deploys when they enter a new region, and I have no doubt they run leaner with less compute available then AWS.

When I was on the telemetry team about 3 years ago, we were in the early days of transitioning from the internal only "physical cloud" that the Bing team had created years before into Azure, and it was full of pain for a service our size at that time.

</comment></comment></comment><comment><USER4> > One misleading headline that's been pushed out by Microsoft and > unfortunately picked up by various tech media outlets, is the "775% > increase" in Azure cloud service usage in geographical areas that are most > committed to some form of social distancing or shelter-in-place policy.

It's way worse than that. The original statement:

> We have seen a 775 percent increase of our cloud services in regions that > have enforced social distancing or shelter in place orders

The updated statement:

> We have seen a 775 percent increase in Teams' calling and meeting monthly > users in a one month period in Italy, where social distancing or shelter in > place orders have been enforced.

Somehow an increase in usage of a single feature of a single product in a single country was miscommunicated as an increase in demand for all of Azure's cloud services. Microsoft's PR team royally screwed up on this one.

[https://azure.microsoft.com/en-us/blog/update-2-on- microsoft...](https://azure.microsoft.com/en-us/blog/update-2-on-microsoft- cloud-services-continuity/)

<comment><USER5> This "single feature" is video-conference and is used by almost every Office 365 business customer. In chat propositions, it's Slacks biggest competitor.

There is no feature bigger in bandwidth than this one.

I believe the numbers are similar in Europe.

By reference, I think it's similar as having a Netflix business appear from nowhere.

<comment><USER6> >By reference, I think it's similar as having a Netflix business appear from nowhere.

Only without all the edge caching devices Netflix has had a decade to deploy.

</comment></comment></comment><comment><USER7> > limited to places where shelter-in-place is in full force, which at least in > the United States, is only a small part of the country with densely > populated urban areas: New York, California, New Jersey, Michigan, etc.

Ummm, it is NOT a "small part" of the country that is under "shelter-in- place". By geographic area, maybe, but California, New York and New Jersey alone include some of the largest cities by population and population density. And there are now several other states with large populations (Illinois, Florida, Washington, for example) not listed here that have shelter-in-place as well.

UPDATE: I found an interactive map that shows which states have statewide shelter-in-place orders in the United States; it looks like all the states with the largest populations (and more than 50% of the country's total population) are covered, which is actually significantly larger than I had actually thought before I looked this up: [https://www.nytimes.com/interactive/2020/us/coronavirus- stay...](https://www.nytimes.com/interactive/2020/us/coronavirus-stay-at-home- order.html)

<comment><USER8> By population, New York + California + New Jersey is 22% of the USA.

<comment><USER9> and have 6% representation in the Senate...

<comment><USER10> That's what the house is for.

The senate is there so the larger states don't trample over, say, Idaho. Majority rules does NOT work for the USA. You have to think of us as the EU with member votes for each country (country being state). This nation is not supposed to be about the federal government, it all went to shit around 9/11 and has continued to do so.

<comment><USER11> > it all went to shit around 9/11

Not really. You could argue that it's been like this for a while. Depending on what you want to judge, there's

\- Johnson's Great Society (Medicaid and Medicare, urban renewal, War on Poverty)

\- Eisenhower (Interstates, the first nationwide public transportation network)

\- FDR (The New Deal, Social Security)

<comment><USER12> I would go even earlier

\- Fugitive Slave Act (Federal government uses military force to kidnap people who were, according to the laws of the state they were in, free citizens and force them back into captivity)

<comment><USER8> I'd go even further:

\- The Constitution, which was a coup that displaced the old confederation of states.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> A better comparison would be between Azure and GCP, both of them provide office suite (SAAS) to customers.

Google recently released its side of the numbers ([https://cloud.google.com/blog/topics/inside-google- cloud/how...](https://cloud.google.com/blog/topics/inside-google-cloud/how- google-cloud-is-helping-during-covid-19))

<comment><USER14> AWS also provides office suite to customers: WorkMail, WorkDocs, Chime messaging, etc.

<comment><USER15> Are they even in the same league as O365 and G-suite, usage wise?

Only asking because I'd never heard of these services before

<comment><USER16> Are G-suite even in the same league as O365? I have no numbers and might be incredibly biased by location but I feel that all the "boring" office companies in Europe has moved towards Office 365 as a natural step.

That is of course not true and I'm sure there are plenty of G-suite companies but for the large ones it seems very common that they have just followed Microsoft into the cloud.

</comment></comment></comment></comment><comment><USER17> Doesn't seem like a particularly balanced article. Plus some downright bizarre statements. GCP is "not big enough to be stress tested on the same level."??? Even smaller operations are subject to stress proportionately and I'm using a hell of a lot of google hangouts conferences...

<comment><USER18> Do major Google services even run on GCP? My understanding was that they don't...

<comment><USER19> No, this is why GCP is fairly crap when comparing to AWS. Google doesn't actually use GCP for building anything so they have less incentive to actually make it better, that or it means they don't understand the issues with it.

At least with AWS you know that Amazon dog-foods the hell out of it and have a major incentive to improve it.

<comment><USER20> In what way is GCP crap?

</comment></comment></comment></comment><comment><USER21> Where exactly are the "cracks" of which the author speaks? There are some links in the article pointing to generic MSDN help articles but I'm not seeing this claim substantiated by data (reports of downtime for example).

No one is saying online services and cloud providers aren't affected by the surge in traffic but I'm not seeing where the cracks are, so I'm calling the title BS (or clickbait at best).

Hate on M$ all you want, the worlds' enterprises run on their software (and on their cloud).

</comment><comment><USER22> "It's safe to say that AWS runs a much bigger, if not more critical, chunk of the digital world than Azure" noooo... it isn't. It's just that the people we know use AWS. Most people use office, sharepoint, outlook and exchange - especially exchange.

<comment><USER23> AWS even admits that only 5% of enterprise IT spend is in the cloud.

<comment><USER24> I think its safe to say that more consumer services are using AWS than Azure, things like Spotify, Netflix, Snapchat, etc. But when it comes to large enterprises, I'd bet that Azure is first place by a significant margin.

Given that these enterprises power a significant chunk of the worlds supply chains, manufacturing processes and so on, I'd say that the piece of the digital world that Azure offers is way more critical.

<comment><USER23> Amazon always brags that they have twice as many Windows VMs running as Azure.

I'm by no means a Windows hater. I've never developed on anything professionally besides a Windows computer - these days .Net Core, Python and Node. But once I started working with AWS, I avoided deploying to Windows servers like the plague. Anytime you add Windows to the mix you basically more than double your cost between Windows licenses and the increased hardware required over running Linux.

All that to say, no one chooses to use Windows on AWS. So if there are that many Windows instances on AWS, they are more than likely old enterprise workloads.

</comment></comment></comment></comment><comment><USER25> I have had a Service Down Ticket open with Azure now for four days with out as much as a call back from them. Just today I finally got the automated email saying they acknowledge my issue. My system is totally dead in the water on Azure.

On the other hand amazon just called me to make sure everything was ok. Their system has been a rock.

</comment><comment><USER26> The entirety of French universities are using Microsoft Teams for their online classes, that's a very serious workload. Previously almost no one used that tool.

</comment><comment><USER27> Very subjective article trying to pass off as an objective report.

</comment><comment><USER28> One thing that probably helps AWS is that they have experience preparing for peak usage given Amazon's peak traffic (and others) on some events during the year, such as black Friday, Prime Day, major holidays, etc.

<comment><USER23> Please don't start down the false road that AWS came out of excess capacity that Amazon had.

You didn't state that explicitly, just making sure that no one rose thinks that.

</comment></comment><comment> purple-again Eh. A legion of non remote, non technology, companies were forced to quickly thrust themselves into the web. Want to bet how many of those companies were already running as a Microsoft shop? Nearly 100%.

I would love to see details on the total requests placed on AWS and Azure during the time frame of COVID. My instinct tells me that Azure got hammered driven by those microsoft shops scrambling to get into the cloud while AWS had only a small increase, driven by technology companies (a small portion of companies) where the devs have final say and prefer AWS.

<comment><USER6> 100% echo this statement. I have dozens of customers who went to VDI in Azure as their immediate work from home solution. I can't think of a single one that went to AWS. The article makes a bunch of baseless claims to get to an unproven conclusion.

>It's safe to say that AWS runs a much bigger, if not more critical, chunk of the digital world than Azure.

Based on?? If o365 goes down for any extended period of time, most of the fortune 100 can't do business.

> it generally lacks operational experience in running "always on" services > and preparing for unanticipated traffic surges.

Ya... Microsoft lacks experience running always-on services... they'd only been managing hotmail for a decade before AWS was even in beta. They have no experience running always-on services _AT ALL_.

<comment><USER29> I have seen the same. One of our clients stood up 25,000 Azure VMs (Win 10) in a single day. No problems so far.

<comment><USER6> > No problems so far.

Wait until they get the bill :) I will say, it's amazing how quickly they find budget for that back-burnered VDI project after the first month of billing.

</comment></comment></comment></comment><comment><USER30> Seems to me like microsoft probably: underestimated their customer's potential demand and/or oversold their cloud capacity.

</comment><comment><USER31> MS has more user facing SaaS offerings than Amazon, so both are not impacted the same way due to the lock down. What is the AWS equivalent of Teams & Office 365? Chime and work docs? The comparison is apples to oranges.

</comment><comment><USER32> Except for EMR

<comment> contol-m Could you explain further?

<comment><USER32> When we started using EMR, we'd never exhaust the number of available instances (in both spot and on-demand). Now, we frequently do. And, for the last 3 weeks, it's been close to impossible to launch a huge cluster.</comment></comment></comment>


PCLinuxOS gets number 1 spot on Distrowatch; Ubuntu is now number 2 - <AUTHOR> http://distrowatch.com/weekly.php?issue=20070917#editorial

<comment><USER2> And that's why Distrowatch is invalid as a measurement tool. Nobody uses PCLinuxOS. It's like five guys for whom Mandriva (the Linux distro seemingly designed by 14 year old boys) is a little too grown up and serious and Gentoo (the Linux distro designed for people with way too much of that pesky "time" thing) is too hard to understand.</comment>


Understanding the process of finding serious vulnerabilities - <AUTHOR> http://lcamtuf.blogspot.com/2015/08/understanding-process-of-finding.html

<comment><USER2> It may not be that comprehensive yet, but I thought this was interesting:

> Only a small minority of serious vulnerabilities appear to be disclosed > anywhere outside a vendor advisory, making it extremely dangerous to rely on > press coverage (or any other casual source) for evaluating personal risk.

I've always had a feeling that the cycle of reading an article on publicized security flaw X and then jumping in response to update or replace something immediately was kinda pointless. Seems better to have a regular update schedule and practice good defense in depth than to watch the tech media and obsess over their reports.

</comment><comment><USER3> It's just a teaser with no results yet. I'd be really interested to read the next article when he'll have compiled the answers.

<comment><USER4> I must admit, he totally _got me_ with watch this space. Bookmarked for later. I wonder how honest the security researchers will be? I'd also love to see the same survey replicated with those who exploit the vulnerabilities they find.... Those may not be as honest

<comment><USER5> I also added their ATOM feed to my RSS reader, and I'm curious if some interesting articles will come.</comment></comment></comment>


The Error Model in Midori - <AUTHOR> http://joeduffyblog.com/2016/02/07/the-error-model/ <comment><USER2> As is usual the Rust crowd is excited about the shoutouts and the applicability of the consideration to Rust's analogue.

In particular, I enjoy that Rust just decided to change their "try" syntax for calling a function that could possibly fail to a simpler "?" suffix, and now we get Duffy praising how well "try" worked out for them. :P

<comment><USER3> Yeah, we struggled with how to do errors right in Rust for a long time, and it's really interesting to see that we almost ended up in the same place as Midori. The main differences are:

\- Midori (and Swift) uses effects instead of monadic types (probably good, since we dodge the interactions with generics described at the end, but the "throws" syntax is so sweet);

\- Rust uses a return codes implementation instead of an unwinding implementation (for simplicity's sake; some Rust users really don't like unwinding, for legitimate reasons in many cases, but Midori's arguments in favor of unwinding are compelling);

\- Rust lets you catch panics at thread boundaries (one of the most controversial decisions in Rust; I think it was the right one, but I definitely acknowledge that it's a big tradeoff).

<comment><USER4> > \- Rust lets you catch panics at thread boundaries (one of the most > controversial decisions in Rust; I think it was the right one, but I > definitely acknowledge that it's a big tradeoff).

(Unstable) Rust lets you catch panics _anywhere_. You have to be able to do that for sane FFI - as you know (but perhaps other readers don't), a panic that unwinds across the Rust/C FFI boundary causes undefined behavior, so AIUI extern functions in Rust should all look like:

        pub extern "C" fn foo_the_quuxes(x: i32) -> i32 {
            std::catch_panic(|| do_stuff_in_rust_land_that_may_panic(x))
              .unwrap_or(-1)
        }

(Actually, checking several libraries off the top of my head that expose C APIs, most don't do this -- probably because std::catch_panic is relatively new.)

Let's hope developers don't abuse std::catch_panic for regular Rust code. Maybe a lint that warns about uses outside of an extern fn is worth having.

<comment><USER5> > (Unstable) Rust lets you catch panics anywhere.

Note that you're replying to the original creator of Rust, he's most likely aware of that discussion.

<comment><USER6> pcwalton has been heavily involved with Rust since long before anyone remembers, but you're confusing him with Graydon Hoare, Rust's true original author.

<comment><USER5> I am for some reason…

</comment></comment></comment></comment></comment></comment><comment><USER7> They got as far as porting the Microsoft Speech Server, for multiple languages, yet the project was canceled. That's really sad.

<comment><USER8> They accomplished a lot!

[http://joeduffyblog.com/2015/11/03/blogging-about- midori/](http://joeduffyblog.com/2015/11/03/blogging-about-midori/)

</comment></comment><comment><USER9> Kind if surprised with all mentions if reliability and lightweight processes that there was no mention of Erlang.

<comment><USER10> Did you read the following paragraph and the section on Abandonment? Erlang isn't mentioned by named, but the problem domain is.

"I expect more of the world to adopt this philosophy as the shift to more distributed computing happens. In a cluster of microservices, for example, the failure of a single container is often handled seamlessly by the enclosing cluster management software (Kubernetes, Amazon EC2 Container Service, Docker Swarm, etc). As a result, what I describe in this post is possibly helpful for writing more reliable Java, Node.js/JavaScript, Python, and even Ruby services. The unfortunate news is you're likely going to be fighting your languages to get there. A lot of code in your process is going to work real damn hard to keep limping along when something goes awry."

<comment><USER9> Yup read that part and that's exactly my point.

The author hammers home how important abandonment is in a light threaded system without any mention of all the work that Erlang has been doing for years in that space. Given how much reference there is to Rust, Go and a host of other languages I would at least expect Erlang to be mentioned.

<comment><USER11> The frequency of the mentions of Go made me prick up my ears so I've done a frequency analysis of language mentions:

       Midori: 31
       Go:     27
       C++:    27
       C:      18
       Rust:   10
       Swift:  03
       D:      02
       Scala:  02

None for Erlang! Could it be said that Joe Duffy is focussing on statically typed languages or systems programming languages? Interesting that Go is most referenced, not C++ :)

</comment></comment></comment></comment><comment><USER12> > Bugs Aren't Recoverable Errors!

This is true in the context of Midori but not in general. In a dynamic environment, you recompile your buggy function and continue working.

</comment><comment><USER13> > Maybe statistical correctness is okay for scripting languages, but for the > lowest levels of an operating system, or any mission critical application or > service, this isn't an appropriate solution. I hope this isn't > controversial.

I would say that in any case, statistical correctness is all you can have. It's a bit of a nitpick, though, as it certainly doesn't mean that it can't be some orders of magnitude more likely.

(Which point the author seems to make, himself, later in the essay.)

</comment><comment><USER14> > "This distinction [between bugs and recoverable errors] is paramount. > Surprisingly, most systems don't make one, at least not in a principled way! > As we saw above, Java, C#, and dynamic languages just use exceptions for > everything; and C and Go use return codes."

Not a Java programmer, but isn't that what RuntimeException is for? Yeah, Java uses exceptions for both errors and recoverable errors, but it can use _different_ exceptions.

<comment><USER15> You'd like to think that, but they can be caught and used just like other exceptions. The one caveat is that you can't force code to account for potential RuntimeException's. With things like an IOException, the code has to account for the possibility that the exception will be thrown.

So, they are a little different, but the tooling for them is pretty much the same, greatly diminishing the effect.

</comment></comment><comment><USER16> A huge post. He just keeps piling on one thing on top of another and another and another. Many topics should be discussed separately. There is a lot of information. And yet, I feel that critical topics are misunderstood.

</comment><comment><USER17> Does anyone know if they plan to release any of this stuff as open source?

</comment><comment><USER18> This is kind of confusing when you start reading the article thinking it is about the Midori browser [0] while it is actually: "Midori was a research/incubation project to explore ways of innovating throughout Microsoft's software stack." [1]

[0][http://midori-browser.org/](http://midori-browser.org/)

[1] [http://joeduffyblog.com/2015/11/03/blogging-about- midori/](http://joeduffyblog.com/2015/11/03/blogging-about-midori/)

<comment><USER19> Why do the Midori Browser screenshots look like they're a Mac app and yet the download page says it's not available for OS X?

<comment><USER18> Yeah, a bit stupid, they even showcase the browser on macs only (guess they like the clean look). It's a GTK app, Gnome3 apps do look similar to osX apps. The full screen button though, it is very out of place. It is problably the result of using it on Elementary OS (where it is de default browser), a Linux distro that aims to emulate osX. [0]

[0] [https://elementary.io/](https://elementary.io/)

</comment></comment></comment><comment><USER20> His earliest blog post on this is from November. Midori the web browser, which I figured this would be about, was first released in 2007.

I don't understand how this type of name overlap happens. It's got to be of the, "Yeah, I know this obscure name is already used for an open source project and I just don't care" variety.

<comment><USER21> > "Yeah, I know this obscure name is already used for an open source project > and I just don't care"

Curious how you jump to this conclusion. Not to burst your bubble, but the overwhelming majority of programmers even have likely never heard of the Midori browser. Additionally, the Midori browser did not invent this word, it is a Japanese word...and also the name of a band, the name of an olympic figure skater, the name of a liquor, all predating the browser.

Why on earth would people working on an internal Microsoft project go searching for obscure open source projects that might be using the same name before agreeing to move forward with the name?

<comment><USER20> I have nothing to say to that. You're 100% right.

<comment><USER22> FWIW, I recall there was a terminal emulator called Power Shell back in the day. I don't know if we bought the name or...500 lb. Gorilla...</comment></comment></comment></comment>