Spotify moves its back end to Google Cloud - <AUTHOR> https://news.spotify.com/us/2016/02/23/announcing-spotify-infrastructures-googley-future/ <comment><USER3> What Google Cloud needs is a better sales story. AWS consistently beats out Google in this respect. This story is welcomed and there should be more success stories published like this.

My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform.

We (myself and another guy) started at zero and shipped our product in 3 months on what is effectively an infinitely auto-scalable platform where we don't have to do any devops or carry a pager. I'd much rather work on features than chores. =) 4 months later and we've had profitable growth and zero downtime (knock on wood) and we're hiring.

Thanks Google!

<comment><USER4> I've been relying on multiple Cloud offerings (including PaaS, IaaS etc) outside Google for many years, but for some unclear reason, I cannot manage to consider Google a reliable partner for a type of hosting or another, at this point.

I would love to be proven wrong really, because I often look at their GC offering, and I /really/ want to dive in, yet I feel that I would risk having to migrate quickly in 2 weeks at some point because they are changing things significantly (either shutting down or other major change).

So to my question: for how long have these services been around in a fashion that is stable (feature-wise, quality-wise etc)? Is there anyone here with a longer history of using Google Cloud offering, able to comment on this?

EDIT: thanks to everyone who replied below. I think I'll give it a try :-)

<comment><USER5> For Google Cloud products that have made it to General Availability (GA), we have a one-year deprecation policy ([https://cloud.google.com/terms/](https://cloud.google.com/terms/), Section 7.2). That is, if we're going to make some big change, we give you a 1 year heads up.

Disclaimer: I work on Compute Engine, but I'm not a lawyer ;).

<comment><USER6> Promises to do something for a year which you can change or withdraw after 30 days are really promises for 30 days:

"Google may make changes to this Agreement, including pricing (and any linked documents) from time to time. Unless otherwise noted by Google, material changes to the Agreement will become effective 30 days after they are posted, except if the changes apply to new functionality in which case they will be effective immediately. If Customer does not agree to the revised Agreement, please stop using the Services." [https://cloud.google.com/terms/](https://cloud.google.com/terms/), Section 1.7b

Also, Sections 13.1-13.3 means even though Google promises nt to do something, you really don't have any recourse if they do it anyway and cause you damage.

Although in the grand scheme of things, not the worst terms of service document I have read. The mere fact the drafters addressed the issue is worth ... something.

Disclaimer: IAMALBIANYL.

<comment><USER7> > Disclaimer: IAMALBIANYL.

I like that one. :)

I have a question though, is the prevalence of taglines in any way related to legal requirements? Do you have to stipulate that you aren't a lawyer when stating things because otherwise people to construe it as legal advice, or is it just a fun little identifier of how valid we should consider information to be (like it is in other acronyms of the same type)?

<comment><USER6> I think it may stem from laws and rules related to practicing law, but there is certainly not any "requirement" to stipulate you are not a lawyer when speaking about legal things. However there are a few things to be mindful of: In the (all of the) United States (and many other countries) the profession of lawyer or attorney is regulated and requires a license to practice as such. In many places to practice law without a license is a criminal offense. Obviously this doesn't contemplate criminal charges against a group of people spouting off about something legal related on the Internet (otherwise we would have to jail perhaps 80% of reddit), but providing legal advice and interpreting the legal effect of contracts, statutes, or regulations is part of the practice of law.

The reason we heavily regulate the practice of law is similar to the reason we heavily regulate the practice of medicine. If you screw something up, the consequences can be bad. If someone takes your advice thinking you know what you are doing, and you are wrong, that person can end up a lot poorer or even in prison. (Though some will argue we regulate it so lawyers can be the only ones charging high fees for legal advice!) Just like one probably shouldn't go around telling folks not to worry about their incredible chest pain and shooting arm pain unless they are a trained medical doctor, the same could be said for those that go around offering home spun legal advice.. ("Yeah man, if you ask the undercover cop if he is law enforcement, he HAS to tell you! Otherwise it is entrapment man...")

Also the reason you always see lawyers preface everything by saying "I'm not your lawyer.. This isn't legal advice... Yadda yaddda yadda" is because once there is an Attorney-Client relationship a lot of stuff happens. The lawyer has a host of duties to the client, from confidentiality, to competence, and many others. That obligation isn't taken lightly and many times if there is any doubt whatsoever whether or not there is an attorney-client relationship, the law will find in favor of one because --- well -- the lawyer should know better. Therefore lawyers will make it annoying clear they aren't your lawyer before spouting off some deep legal thoughts about whatever the topic of discussion is.

<comment><USER8> Great answer. Has the acronym "IANAL" ever been tested in court in such a way, to your knowledge? Now you have me curious, and that'd honestly be quite amusing.

<comment><USER6> Ha! Interesting question. I just ran an all state/federal Westlaw search for "IANAL" and didn't hit any case law. Two or three briefs, a couple of scholarly articles, but no court opinions. I also searched for "I am not a lawyer", but that brought up about 130 caselaw hits which seemed to primarily be references to various filings by pro se parties actually saying in a court filing "I am not a lawyer".

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I want to commend some of the Google Compute Platform employees that commented in this thread.

I was at AWS for 6 years (left 2 years ago... today!), and I've always been a proponent of being more open and communicative with developers, but it rarely happened - I guess that AWS' PR policy and such are a big showstopper for these kind of discussions. Although, some individuals did their best (e.g. Jeff Barr) to try to share as much as possible.

It seems that the Google guys know how to do it.

Keep doing it. It will help your business a lot.

<comment><USER10> AWS employees may not speak out very often, but they hardly need to. AWS is a marketing machine, from Jeff's regular blog posts, to the volume of free information they provide and to the conferences that they are running everywhere. I think Microsoft is starting to catch on with Azure promotions & conferences, but it's like Google didn't even know that the there was a race .... and they left their running gear at home.

<comment><USER11> This might be true but I think the issue is that many AWS employees actually want to speak out or respond to issues personally.

I used to work at AWS and the things we could say were very controlled. It could just be my team/managers though.

<comment><USER9> No, I agree with your last statement - at AWS you always feel the invisible shadow of PR :)

</comment></comment></comment></comment><comment><USER12> I tried Google Container Engine (GKE) and really liked it - it's the best cloud solution for deploying Docker to production in my opinion, mainly due to its use of Kubernetes. Unfortunately in my Web apps I make heavy use of Postgres-specific features, and since Cloud SQL only supports MySQL, Google Cloud is a total non-starter for me.

So for now I'm on AWS, using Postgres on RDS and deploying containers with ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple (a half dozen task definitions), it's not a big deal. I really hope Google adds Postgres to Cloud SQL at some point.

<comment><USER13> There are vendors running managed Postgres services on Google Cloud Platform, like ElephantSQL [1] and Aiven [2]. And you can of course run your own on GCE, even to the extent of 24/7 commercial support - you can run EnterpriseDB from Cloud Launcher [3].

And, you can also run Kubernetes on AWS - we have a group focused on making sure it's an excellent experience.

I work for Google Cloud Platform; ping me if you'd like more help with either option.

[1] [https://www.elephantsql.com/blog/2014-11-17-google- compute-e...](https://www.elephantsql.com/blog/2014-11-17-google-compute- engine.html) [2] [https://aiven.io/](https://aiven.io/) [3] [https://cloud.google.com/launcher/solution/public-edb- ppas/e...](https://cloud.google.com/launcher/solution/public-edb-ppas/edb- postgres-enterprise?q=enter)

<comment><USER12> > There are vendors running managed Postgres services on Google Cloud > Platform, like ElephantSQL

I did check out ElephantSQL but my pricing needs are somewhere between their $100 and $20 plans and there seems to be a lack of configurability compared to RDS's parameter groups (e.g. enabling extensions).

> you can also run Kubernetes on AWS

I've had success turning up Kubernetes clusters on AWS for demo purposes, but I really don't want to manage a k8s cluster myself (anecdotes I've read about etcd failures / partitions especially scare me). Also I use Terraform for provisioning, and kube-up.sh is not something that fits into that paradigm. I've also made the mistake running kube-up.sh with the wrong arguments after a previous invocation that had created a cluster, which caused it to try and create a new cluster, which wiped the local cache of the previous cluster I had made, making kube-down.sh unable to automatically clean up the old cluster (so I had to do it manually in the AWS console).

The other thing I tried was the kube-aws CoreOS tool, which is nice, but it comes baked with a 90-day expiration due to TLS certificate expiration, so I'd have to set up some sort of PKI process to make that production-ready. All in all just too much work for a single person trying to deploy a small number of containers for small to medium sized projects; if I was a medium-sized company with hundreds of containers and some dedicated DevOps resources maybe it would be worth it, but for myself I'd prefer a turn-key solution like ECS or GKE.

<comment><USER14> We also just had a user check in a cloud formation template for Kubernetes on AWS - would that meet your needs? (No kube-up.sh required)

Disclosure: I work on GKE and Kubernetes at Google

</comment></comment></comment></comment><comment><USER15> > Google has long been a thought-leader in this space, and this shows in the > sophistication and quality of its data offerings. From traditional batch > processing with Dataproc, to rock-solid event delivery with Pub/Sub to the > nearly magical abilities of BigQuery, building on Google's data > infrastructure provides us with a significant advantage where it matters the > most.

Big Data is the core strength of Google Cloud. Good to see this move by Spotify!

> What really tipped the scales towards Google for us, however, has been our > experience with Google's data platform and tools. Good infrastructure isn't > just about keeping things up and running, it's about making all of our teams > more efficient and more effective, and Google's data stack does that for us > in spades.

What I really really liked about Google Cloud is the ease of use. Spin up a VM, start Cloud shell, SSH into your instance, install a bunch of software and you will know what I mean. It's "Quality" Cloud.

</comment><comment><USER16> At work we moved to GCE at the beginning of this year, from Linode after they were having stability issues over the christmas break. No complaints from us. So far have been very happy with it. We were considering moving to AWS, but to realize the same pricing as GCE we'd have to purchase reserved instances - <USER2> sustained usage discounts have been huge for us.

Additional things we liked: \- gcs is way more responsive than S3. also, was fairly painless to migrate our S3 buckets to GCS via the web console. \- peering with cloudflare lets us save on bandwidth costs! \- network load balancer has shown itself to be very reliable and solid for holding open A LOT sustained websocket connections. \- http load balancer has shown itself to be very capable of ssl termination & routing (love that we can route /api to our API servers, and the rest to our static servers). additionally, no pre-warming was required. when we did the datacenter switch, it was just 5 mins of downtime. didn't have to worry about pre-warming for our production load like we would have on ELB.

Things we didn't like: \- salt-cloud's driver for GCE is still lacking. Can't specify disk size for provisioned storage. Can't specify local SSD storage either. Also parallel provisioning didn't work. Something about PyCrypto not liking the way it forked. Not GCE's fault \- billing support non-existent unless you purchase a support plan. \- documentation still needs some work.

<comment><USER17> Billing support is provided for free to all customers. If you had trouble with a particular ticket, please feel free to ping me.

Disclaimer: I work in Cloud Support.

<comment><USER16> Thanks for the reply! I don't remember the specific issue, but we had a billing inquiry and were told we had to upgrade to a paid support plan to get an answer.

<comment><USER17> Please feel free to forward the ticket to me at jani at google, and I'll look into it.

</comment></comment></comment></comment><comment><USER18> The big point of this move is the hit it will have on on-premises Hadoop offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence in their offerings. Spotify are effectively ditching a 2000 server, 90 PB Hadoop cluster to go GCE. As Spotify say themselves, that's big news....

<comment><USER19> Spotifier here. This is an important point. I have nothing bad to say about Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good experience), but I don't really see a compelling reason at this stage to manage your own cluster(s) (HIPAA/regulatory constraints, maybe?)

Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.

<comment><USER18> In fact, I don't think large orgs like you (Spotify) really want independently operated clusters. That prevents easy sharing of data, causing data silos to appear. You really want to have true multi-tenancy, which isn't in Hadoop yet. Hadoop has worked more on Kerberos support at the cost of features like easy- to-use access control - Apache Ranger or Sentry anybody!?!?

<comment><USER20> Yahoo's Hadoop clusters operate in a multi tenant fashion for precisely this reason: to ease sharing of data between groups.

</comment></comment></comment></comment><comment><USER21> When did Spotify abandon their original peer-to-peer architecture?[0]

[0][http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.p...](http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.pdf)

<comment><USER22> Quite a while ago. Not sure on the exact time, but judging from this article, I'd guess sometime in 2014 [https://torrentfreak.com/spotify-starts-shutting- down-its-ma...](https://torrentfreak.com/spotify-starts-shutting-down-its- massive-p2p-network-140416/)

<comment><USER23> I'd be curious if anyone knows the details about why: complexity, customer complaints, reliability?

<comment><USER24> Too much complexity for a world that's moving to mobile clients that can't be active peers of the network.

<comment><USER23> Seems to be a pattern: wimpy mobile endpoint devices drive everything to the cloud.

If so this is perhaps a dated phenomenon. Moore's Law is still a thing and today's generation of mobile devices are getting fatter and fatter.

<comment><USER25> Moore's Law has not been improving batteries, which is the real problem. Mobile P2P nodes need to be awake most/all the time, which is a real battery killer. Modern devices have improved battery life by being asleep almost all the time.

<comment><USER23> Depends on what those p2p nodes are doing. We do p2p on mobile but it's to allow the mobile device to act as a client to access an IoT device, not to force the mobile device to act as a CDN node. Spotify was doing the latter, which is more problematic.

Moore's Law eventually _will_ matter, since faster and more efficient chips equal the ability to do more with lower power consumption. Batteries aren't getting better fast enough, but power consumption per fixed unit of compute is falling like a rock. As a result "effective battery life" for a given fixed workload is subject to Moore's Law.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> We host a lot of our applications on Google Compute Engine but have had many issues with even simple features available from other cloud providers. For example, Google's HTTP(S) Load Balancer offering does not support SNI [1], HTTP to HTTPS forwarding [2], or sticky sessions [3], making it unusable for us. The support we pay for has been pretty helpful, but all we hear is "we have a feature request for this but I cannot provide an ETA on when and if this will be implemented". Many of the issues in the Compute Engine public issue tracker [4] haven't been updated in many months or over a year in some cases. Additionally, it can be very confusing finding out which beta services can only be accessed via the gcloud tool and not the web console. Without the ability to even schedule a window for maintenance on our VMs, we often feel like we do not have the control we need over our hosting environment. Please increase your transparency around addressing Google Cloud issues, all the Docker updates have been great, but many of these issues are over a year old and still show-stoppers for some users. Thanks again for churning out new features/services constantly, but please help close out some of these older feature requests!

[1] [https://code.google.com/p/google-compute- engine/issues/detai...](https://code.google.com/p/google-compute- engine/issues/detail?id=289) [2] [https://code.google.com/p/google-compute- engine/issues/detai...](https://code.google.com/p/google-compute- engine/issues/detail?id=255) [3] [https://code.google.com/p/google-compute- engine/issues/detai...](https://code.google.com/p/google-compute- engine/issues/detail?id=240) [4] [https://code.google.com/p/google-compute- engine/issues/list](https://code.google.com/p/google-compute- engine/issues/list)

</comment><comment><USER27> We (Sojern) moved to 100% GCP as well this month, admittedly not nearly at the same scale as Spotify, but we did. Other than some growing pains - that GCP team is already aware of - this has been a very smooth migration for us.

We use Google Cloud {Storage, BigTable, Container Engine, BigQuery, Monitoring, DNS, Compute} and may be some services I'm missing. I'd be happy to answer any questions.

</comment><comment><USER5> This was mentioned somewhere deep in a thread, but Spotify (and others!) will be speaking at our upcoming customer conference (GCP Next) in a few weeks:

[https://cloudplatformonline.com/NEXT2016-schedule.html](https://cloudplatformonline.com/NEXT2016-schedule.html)

You'll be able to livestream the sessions here:

[https://cloudplatformonline.com/NEXT2016-Live.html](https://cloudplatformonline.com/NEXT2016-Live.html)

Disclosure: I work on Compute Engine (and I'll be at Next).

</comment><comment><USER28> The GCE model is the future. Google knows this because they've already discovered it internally by running their own services. They know containers are he future and they built a lot of the LXC kernel updates that makes things like docker work.

Most people don't need the future yet. Really people are just trying to get out of the business of managing and scaling their hardware with as little work as possible. Containerization is much farther down the line.

So I posit the future GCE customers are not people "moving to the cloud" like AWS, but rather people "moving to the small, contained, service" \-- indeed, it's current AWS customers.

</comment><comment><USER29> Also on this: [http://www.wired.co.uk/news/archive/2016-02/23/spotify- move-...](http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-to- google-cloud) and [http://googlecloudplatform.blogspot.com/2016/02/Spotify- choo...](http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses- Google-Cloud-Platform-to-power-data-infrastructure.html)

</comment><comment><USER30> I notice App Engine isn't mentioned in their stack. I'm a bit worried that Google abandons it once Snapchat moves off it. It has already been in maintenance mode for some years now (save for Managed VMs). The writing was on the wall when Guido left the team. Apparently Google still uses it internally (as the recent switch to Monorail made evident), so perhaps it's safe for a few more years.

<comment><USER31> Hi pdknsk,

I'm a Product Manager on App Engine and wanted to reach out and say that 100% we are invested in app engine and are working to make it your favorite platform. For example, we heard that GAE documentation needs enhancements and we've kicked off a large project to do just this. It's an ongoing effort as you can imagine, but please take a look at the new landing page and managed VM content as well as the left-hand nav reorg we did. Here is the link. More to come of course.

[https://cloud.google.com/appengine/docs](https://cloud.google.com/appengine/docs)

Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a feedback tool on the top right hand side of every page.

Thank you! Amir

<comment><USER32> Do you have any plans to launch a data center in India? AWS is launching one this year, and I imagine this will be a huge boon to the startup industry here.

As of 2016, AWS is the better bet.

</comment></comment></comment><comment><USER33> >"...feature testing and more intelligent user-facing features," the Google staffer wrote.

I have to wonder if Spotify is doing this so that they can (eventually) use Google's deep learning technology in improving their recommendation engine. They purchased Echo Nest in 2014 to achieve this.

</comment><comment><USER34> I have some questions based on previous experience many moons ago experience (3 or 4 yrs).

When I tried to port some of our platform to AppEngine I had some issues with various Java libraries (writing tmp files, opening sockets, etc). Once I sort of got past that and used some of Googles own APIs (which is sort of annoying that I had to couple my stuff) I had timeout issues. See we have to integrate with all these enterprise third parties (SOAP/REST) and these guys can be really slow. We also couldn't use our own pub/sub (AMQP/RabbitMQ) so that was going to have to be ported as well.

Things must have changed because its hard for me to imagine Spotify being able to get all their needs met on a rather coupling platform.

* How does one write code for Google Cloud while being agnostic of Google Cloud?

* Maybe appengine supports AMQP?

* Or maybe the container engine fixes these problems? (ie need something more custom run it in a docker image).

Our current platform runs on Rackspace and Digital Ocean. These guys are IaaS so your basically doing DevOps which is a pain... but Rackspace does have kick ass customer service.

<comment><USER35> AppEngine is Google Cloud Platform's PaaS offering. Its the oldest product on Google Cloud Platform, but its not one of the ones mentioned that Spotify is using (the Spotify post doesn't talk much about the products, but Google's [0] does.)

The Google Cloud Platform services they are identified as using are: Cloud Storage, Compute Engine, Cloud Datastore, Cloud Bigtable, Direct Peering, Cloud VPN, Cloud Router, Cloud Pub/Sub, Cloud Dataflow, BigQuery, and Cloud Dataproc.

So, its not surprising that they might be doing something that Google AppEngine doesn't support particularly well.

[0] [http://googlecloudplatform.blogspot.com/2016/02/Spotify- choo...](http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses- Google-Cloud-Platform-to-power-data-infrastructure.html)

<comment><USER34> I guess they are not entirely on Google Cloud but rather their data warehousing / big data stuff is now?/going? to be on GCP.

I don't really call big data processing "backend" but rather the microservices the backend. Its not clear to me if those are going to be ported over or if they are already.

Does spotify plan on porting everything over or just the heavy computing? A detailed techy case study I hope will follow soon as I am interested (the DevOps pain is very real).

<comment><USER35> > I guess they are not entirely on Google Cloud but rather their data > warehousing / big data stuff is now?/going? to be on GCP.

I don't know why you would guess that: the Google announcement indicates that the migration has both a "services" track and a "data" track. And Spotify's announcement certainly seems to indicate that they are moving off of their own datacenters, onto Google Cloud Platform, and not in a limited way.

<comment><USER34> So... basically nothing is live yet? Are you from Google? Is anything live. I admit I got confused by the OP title: "Spotify moves its back end to Google Cloud".

</comment></comment></comment></comment></comment><comment><USER36> This reads like they've been acquired by Google and are integrating. Now Google just has to write the check and own music streaming.

<comment><USER37> Hmmm...interesting point but Google already has an awful lot of music infrastructure in the form of Google Music and YouTube does it not? Would it really make sense for them to purchase Spotify too?

<comment><USER38> I've only really looked at Spotify and Tidal, but Spotify is really good. There's more to streaming than content.

Makes me curious about how good Google's recommendation engine is.

<comment><USER39> As a long time Spotify user who switched to Google a few months ago: I really like the playlists and I've found a lot of new music that I like. The recommendations have worked really well. I'm not looking back.

<comment><USER40> I've also found that Google's radio stations are more diverse and less repetitive.

<comment><USER38> Just checked out Google Music on Wikipedia. No native desktop clients. Not even for Windows, and I use Linux. They can't be serious. Total dealbreaker.

<comment><USER41> I tend to always have a browser open so a desktop client doesn't matter to me. I keep Play Music open in a tab. Then I use an extension[0] to add the other controls I normally use.

[0] [https://chrome.google.com/webstore/detail/prime-player- for-g...](https://chrome.google.com/webstore/detail/prime-player-for- google-p/npngaakpdgeaajbnidkkginekmnaejbi)

</comment></comment></comment></comment></comment></comment></comment><comment><USER42> Congrats to the sales team at Google Cloud. I wonder how much Google's new AI services played a role in convincing Spotify to use them.

<comment><USER43> Hey guys it's Tino. When will you contribute to our Big Data blog with your excellent ideas? :)

[https://cloud.google.com/blog/big-data/](https://cloud.google.com/blog/big- data/)

<comment><USER42> I am more than happy to contribute but I have a feeling you've mistaken me with someone else? :)

<comment><USER43> yup sorry posted in wrong thread.. but please do :)

</comment></comment></comment></comment><comment><USER44> I would be interested to get some more technical info. OK so you use GC, but how? What products do you use, how do you build your scalable infrastructure so anywhere in the world I can have all the music in the world? I love reading those type of posts from Netflix and was hoping to find something similar here

<comment><USER5> As I mentioned in a top-level comment ([https://news.ycombinator.com/item?id=11159840](https://news.ycombinator.com/item?id=11159840)), they will be speaking at GCP Next (which you can livestream) and should have some more detail to share.

</comment></comment><comment><USER45> Today: Highly publicized move onto Google Cloud

Later: Silent, gradual move away from Google Cloud after experiencing a number of issues and bugs, constant connectivity issues with services like Cloud Storage, silent failures that do not return error messages on a random basis, detailed billing info only available through arcane APIs and not the administrative UI, broken functionality and rigid inflexibility of BigQuery, and other non-obvious negative behaviors

</comment><comment><USER46> I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?

<comment><USER13> Spotify are not moving to BigQuery alone. They are also using Dataproc, Google's managed Hadoop service (which co-incidentally went GA yesterday), which lets them get value out of all the orchestration tools they already use, like Luigi. (They are also adopting Cloud Dataflow, Cloud Pub/Sub and other parts of our Big Data stack.)

Luigi has had BigQuery support for some time: [https://github.com/spotify/luigi/pull/1002](https://github.com/spotify/luigi/pull/1002)

Spotify will be talking more about the specifics of their data pipelines at GCP Next in SF in March, - I would expect to see a lot more on the new Google Cloud Big Data blog at [https://cloud.google.com/blog/big- data/](https://cloud.google.com/blog/big-data/) too.

[Disclosure, GCP guy, etc]

</comment></comment><comment><USER47> Om Malik sees it as a first step toward acquisition by Google (or rather, Alphabet).

[https://twitter.com/om/status/702194246489546752](https://twitter.com/om/status/702194246489546752)

<comment><USER48> This has been rumored before. E.g here's a headline from 2014: [http://www.wsj.com/articles/google-considers-buying- spotify-...](http://www.wsj.com/articles/google-considers-buying-spotify-but- finds-the-price-too-high-1406061732)

</comment></comment><comment><USER49> Great now they can free up some engineers to bring their Linux client out of beta, which has been in beta since dinosaurs.

<comment><USER50> Or work on a successor to libspotify. Ha! Who am I kidding?

</comment></comment><comment><USER51> I've run a few small side projects on GAE (Google App Engine). I use it as a very simple and fast way to get an MVP off. My major issue is that their documentation tends to lag (sometimes way behind) when they change stuff, especially things that are integrated to other services they provide. For example after dropping support for OpenID 2.0, the documentation for 'Authentication Type' on Google App Engine was not updated to reflect this, neither was the documentation for securing your urls. This cost me considerable time and effort.

<comment><USER31> Hi NearAP,

I'm a Product Manager on App Engine and I truly apologize for the trouble you went through when you tried the platform. We have kicked off a large initiative to update and enhance our docs, and we just recently launched our new App Engine docs, an ongoing project of course. You can check them out here:

[https://cloud.google.com/appengine/docs](https://cloud.google.com/appengine/docs)

Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a tool on the top right hand side of every page that explains this.

Thank you! Amir

</comment></comment><comment><USER52> Now how about Spotify fixing their front end so that Premium customers on Android can just hit "play" to play the album they want to play, in the right order? And how about an app in the Windows Store so that you can use it on a touchscreen? And how about some improved metadata that work for classical music? And a way to report corrupt or mistagged tracks and get them fixed?

/rant

<comment><USER53> I'm on mobile so can't directly link you but there's a support form for getting songs fixed.

<comment><USER52> I have never found a form, but I have found forums where people complain about persistent issues and nothing gets done. They seem to go out of their way to make it difficult for users to report issues. In addition they seem to have contempt for user experience, or they're incompetent at it, or they are under weird nonsensical licensing pressures that require them to make things difficult for their users. Their front ends need an overhaul and they could easily introduce a "flag this track as having a problem" feature.

</comment></comment></comment><comment><USER54> I had no clue spotify operated their own data centers.

Does anyone have a link to that compares google cloud vs aws on a per price + per service basis ?

<comment><USER13> Last year, we commissioned an independent 3rd-party vendor to prepare a whitepaper comparing GCP pricing with AWS: [https://cloud.google.com/files/esg- whitepaper.pdf](https://cloud.google.com/files/esg-whitepaper.pdf)

We also have a handy guide for AWS professionals who are looking at Google's cloud: [https://cloud.google.com/docs/google-cloud-platform-for- aws-...](https://cloud.google.com/docs/google-cloud-platform-for-aws- professionals)

[Disclosure, I work for GCP, but I love all TLAs equally]

<comment><USER55> There are some half-truths in that whitepaper, like "reserved instances cannot be resized" (They can, within the same "class" of instances)

</comment></comment></comment><comment><USER56> I wonder if this has to do with Amazon's streaming music competitor being more of a threat to Spotify than Google's?

<comment><USER57> Amazon's streaming music competitor is awful. The apps are bad, the web UI is bad, the selection isn't as big, and the audio quality is middle-of-the-pack. In general, it's a "oh right, I have that Amazon service because I have Prime"; a similar comparison would be Prime Video to Netflix.

Spotify is better off in most ways (their desktop app still sucks, but the web UI is better).

<comment><USER56> I wouldn't judge a competitive threat solely on a V1 implementation, given it's from a company with engineering chops like Amazon.

</comment></comment></comment><comment><USER58> The short and sweet of my user story as a hobbyist fullstack developer and wannabe data analyst is AWS has standalone services for your apps that either don't exist on GCP or are better than what GCP offers.

I'm talking about pay-as-you-go compute functions with Lambda (I'm aware that GC Functions are on the way, Lambda has been out for more than a year already), managed Postgres with RDS, and managed task queues with SQS. But GCP's console UX is killer compared to AWS, and using BigQuery feels too good to be true compared to how difficult it is to accomplish the same thing with Hadoop/Spark.

</comment><comment><USER59> A little birdy told me that if you find a bug in GCE's live migration they'll buy you a bottle of scotch. Apparently the team gets blamed a lot for problems that turn out not to something else. :)

</comment><comment><USER60> I'm curious regarding the economics of this but could anyone approximate the cost associated with this move or expenses going forward. It sounds like this would be a pretty big account for Google.

<comment><USER18> You can bet your bottom dollar Spotify are getting heavy, heavy discounts. Google are ramping up here in Stockholm, partly to get this account. This has been in the works since before last summer, so they've taken time before announcing it.

The precedent for Spotify was switching from Cloudera to Hortonworks a couple of years ago. Rumours are that they got their licenses for close to nothing from Hortonworks, they were so desparate to close them. Same rumours going around vis-a-vi GCE...

<comment><USER61> Just with respect to some details: Spotify changed from being a self- supporting (free) user of Cloudera's open source platform to being a paid support customer of Hortonworks. So, it was never actually a Cloudera customer (despite what was widely "reported").

Disclaimer: Cloudera emp here.

<comment><USER18> Nice, reinforces the point that Spotify likes and Spotify gets discounts on infrastructure :)

</comment></comment></comment></comment><comment><USER62> Still no IPv6 on Google's cloud... why?

</comment><comment><USER63> Has anyone seen stats published on how big of an infrastructure Spotify runs?

Or how it compares to Netflix, which runs on AWS?

</comment><comment><USER64> Why google over aws? Why isn't this man answering the real question?

</comment><comment><USER65> Sorry but spotify's infrastructure is not a shiny beacon of correctness.

They've compounded mistake upon "Oooshiny lets use that"

here is a post where someone dissects spotify's infrastructure bit by bit: [http://www.secretbatcave.co.uk/software/docker-and- you/](http://www.secretbatcave.co.uk/software/docker-and-you/)

This isn't an anti google compute, its got some really great features. But, spotify's backend infrastructure ain't one

Yes, I appreciate that that post is from a while ago, and they might have completely paid back that massive technical debt/innovation token debt.

<comment><USER19> Spotifier here. For the record, I hate puppet with the fiery intensity of a thousand suns. It's also pretty hard to magically make go away. I could probably do a whole talk on why puppet is difficult to kill, it's the kudzu of config management. _I think it is the worst._

We've got our warts and a pile of tech debt, and I wouldn't want anyone to think otherwise. Containers are a part of a long-term strategy to get away from puppet and onto more idempotent units of deployment, and move a lot of what is considered to be "configuration" back into the build process where it belongs.

<comment><USER66> Is there any public available information where we can read about deployment process in the company?

</comment></comment></comment><comment><USER67> Does Google Cloud offer an equivalent to AWS Direct Connect?

<comment><USER5> Yes. We have both "Carrier Interconnect" ([https://cloud.google.com/interconnect/](https://cloud.google.com/interconnect/)) and "Direct Peering" ([https://cloud.google.com/interconnect/direct- peering](https://cloud.google.com/interconnect/direct-peering)).

</comment></comment><comment><USER68> Isn't it risky to move to competition's infrastructure?

<comment><USER69> Not really - many people rely on Google Cloud and if Google made it harder for Spotify to compete (via, say, raising prices for Google Cloud for them), they'd have to be pretty specific to Spotify to not cause other customers to move to AWS or Azure.

Spotify has much bigger risks in their licensing deals.

<comment><USER68> cost it not a problem and google wouldn't do it. I'm talking about stealing recommendation algorithms and models. Have a peek new features before release etc.

<comment><USER70> Recommendation algorithms and models don't do you any good if you don't have users.

<comment><USER68> Google Music has a lot of users?

</comment></comment></comment></comment></comment><comment><USER71> (Serious question) Isn't it risky to rely on _any_ Google product as a foundation for your business?

They've shut down more popular services that people depend on than Google Cloud. They tend to enter a market, undercut all the competitors, and then once they are dominant in the niche they terminate or change the service in a way that makes it not fulfill the original promise.

<comment><USER72> Well Amazon killed the Fire Phone though ;-) [https://fortune.com/2015/09/09/amazon-killing-fire- phone/](https://fortune.com/2015/09/09/amazon-killing-fire-phone/)

Jokes aside, probably you are thinking about free products like Google Reader, because this:

> once they are dominant in the niche they terminate or change the service

would not make any sense if the product in question makes $$$

</comment></comment><comment><USER73> I think I could try moving to GAE if they had Postgresql offering for database storage.

</comment><comment><USER74> I wonder if they are still using Cassandra or moved that over as well.

</comment><comment><USER75> I always used to think Facebook would make a move for Spotify..

<comment><USER75> -1 :(

Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook would one day like to move to the subscription streaming market. A barrier to entry for Spotify to stream movies I always suspected would be cash.

Moving to google cloud won't stop a Spotify sale but it may discourage some?

</comment></comment><comment><USER76> Smells like a puff-piece

<comment><USER77> A puff piece would be if some tech reporter wrote about the wonders of GCP.

This is the Spotify team, and they've clearly put their money where their mouth is by actually making the move.

<comment><USER78> Marketing blog spam is the new puff piece.

<comment><USER43> Would you prefer testaments like these from Spotify engineers?

[https://twitter.com/plamere/status/702168809445134336](https://twitter.com/plamere/status/702168809445134336)

[https://twitter.com/plamere/status/702169733454495745](https://twitter.com/plamere/status/702169733454495745)

[https://twitter.com/mpjme/status/702167231623516160](https://twitter.com/mpjme/status/702167231623516160)

<comment><USER76> Kind of. We all know that companies cut deals with providers in exchange for blog posts, tweets, and other sorts of promotion. I'm always wary of posts like these, especially when they lack detail or awareness of competing products.

<comment><USER43> I think if Spotify dressed down Google Cloud Platform competition, no matter the merit, there would be many more complaints (ex. see comments on this technical post [https://news.ycombinator.com/item?id=11029032](https://news.ycombinator.com/item?id=11029032)).

Spotify did say that they really enjoy specific aspects of Google Cloud Platform, through their own engineering blog. Spotify engineers have also been independently vocal on Twitter.

Netflix has been very similarly vocal of their support of AWS.

I'd love to hear your thoughts on a better mindshare model, perhaps we can all learn to do things a bit better.

<comment><USER76> Right, I don't think they have to directly attack anyone. It's just a matter of tradeoffs - here are the things the Google Cloud gets us, here's a specific example that hones it in, and here's why that's critical to our business.

At the moment, the blog post just says, "Yay we moved to the cloud" and threw out some marketing buzzwords. It should be beyond a shadow of doubt that they got a lower price as a result. That's ok - now give us the details! On a product level, what made Google Cloud so compelling that you were willing to move your entire infrastructure to it? That question really wasn't answered, and is really the biggest thing, as a customer, I'd want.

Right now, I'd have to call Spotify as a reference in order to do to get that answer, which again allows them bring the price lower. Not necessarily democratize the information.

<comment><USER19> Spotifier here. Frankly, price is not the biggest factor in a decision like this. If we were going for the _lowest cost_ cloud option, it probably wouldn't be either AWS or Google - there are other providers who are hungrier for business that would be willing to do deep cuts at our scale.

The way we think about this is that there are basically two classes of cloud services: commodities and differentiated services. Commodities are storage/network/compute, and the big players are going to compete on price and quality on these for the foreseeable future (as with most commodities).

The differentiated services stuff is a bit more interesting. Different players have different strengths and weaknesses here - AWS has way, way better capabilities when it comes to administration and access control and identity management, for example (which is actually pretty important when trying to do this in a large org). The places were Google is strong (data platform) are the places that are most important for us as a business.

Compelling: dataproc+gcs, bigquery, pubsub, dataflow Made it safe: high-enough quality, cheap enough.

What more would you like to know?

<comment><USER76> Hey nrh,

Nice! Did you have the data tooling built out before you went to Google Cloud? If you did, I could imagine the migration was pretty hard as well.

Also, all of those seem relatively possible with AWS Redshift, Kinesis, and Data Pipelines. I'm interested what Google Cloud had to offer, spec-wise.

<comment><USER19> I would categorize data tooling as a moving target - we have some, it's never enough, it probably won't ever be enough. It's a moving target (p.s. obligatory we're hiring!).

I think this Quora post does a good job of redshift vs. bigquery: [https://www.quora.com/How-good-is-Googles-BigQuery-as- compar...](https://www.quora.com/How-good-is-Googles-BigQuery-as-compared-to- Amazons-Redshift)

More generally speaking, (excuse me for being a little hand-wavey here) many of the AWS offerings feel like polished, managed versions of familiar tools. Redshift, for example, feels a bit like "hey we figured out how to abstract away a bunch of mysql instances to feel like a big processing cluster". That's not a bad thing, necessarily. The google stuff feels much more intentional - "we need to solve the problem of doing these sorts of queries at scale" vs. "we need to solve the problem of scaling mysql to solve these types of queries"

Maybe they're just better at abstraction, but whatever - that works for me!

<comment><USER76> So it sounds like BigQuery was the deciding factor in this case?

<comment><USER19> I'd say the data platform overall, bigquery is certainly great . So are some other bits.

<comment><USER76> Awesome, good to know. Thanks!

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER79> In the same way that facebook shut down parse, I am afraid that google is going to shut down google cloud. And reluctantly have become okay with relying on AWS.

<comment><USER80> Google Cloud interacts with the core of Google's business (web services that needs scalable data infrastructure) in a way Parse never did for Facebook.

<comment><USER79> Google's core business is ads. Parse makes a lot of sense strategically for facebook, a big moat that they have built is around being your identity on the internet. Parse's platform encourages app developers to use facebook login as a core component of their apps. So even despite being ancillary to their business, they shut it down. Building a technology company around an ecosystem is potentially a lot riskier than people take it as. Especially one with a lot of proprietary bells and whistles..

<comment><USER80> Google's core business is providing web services and putting ads on them. That core business means Google focuses a lot on running those web services effectively, which is why they are now focusing hard on the business angle of running other people's web services.

Google spends $2+B per quarter on capital expenditures; where do you think that money is going if not datacenters?

Using Facebook as a default login was never core to Parse (e.g. PFUser: [https://parse.com/docs/osx/api/Classes/PFUser.html](https://parse.com/docs/osx/api/Classes/PFUser.html)) nor is Parse nearly on the same scale as Google's investment in its datacenters / web software efficiency.

<comment><USER79> And at some point your company might decide that it's already its biggest customer for its cloud products and close it to the public.. or it might not.. but it's up for me to decide where I host :)</comment></comment></comment></comment></comment>


This is how Meltdown works - <AUTHOR> https://dev.to/isaacandsuch/how-meltdown-works-28j2 <comment><USER2> The Raspberry Pi explanation is better in my personal view, although to the knowledgable who do not need explanations of CPU design by analogy the actual papers by Intel and Google are of course better still. (-:

* [https://news.ycombinator.com/item?id=16080002](https://news.ycombinator.com/item?id=16080002)</comment>


President will look into Twitter "SHADOW BANNING" prominent Republicans - <AUTHOR> https://twitter.com/realDonaldTrump/status/1022447980408983552 <comment><USER2> This story is backwards. Twitter is disabling shadow banning of prominent Republicans. The algorithm shadow banned them, and they manually re-enabled them. So the favoritism is towards Republicans, not against them.

</comment><comment><USER3> Yesterday:

> Some clarification from Twitter on the whole 'some conservative accounts not > showing up like normal in search' thing: some accounts are not automatically > populating in the search box and the company is apparently shipping a change > to address this.

[https://twitter.com/cwarzel/status/1022213455149817856?s=21](https://twitter.com/cwarzel/status/1022213455149817856?s=21)

</comment><comment><USER4> What will he do when he finds out that they were shadow banned because they were using armies of bots to increase their reach? Will he admit that it's a totally appropriate way to reduce the spread of fake news?

</comment><comment><USER5> This is political theater, not a credible threat against Twitter's right to censor its own platform. It isn't even censoring based on political affiliation, but rather the content of tweets.

<comment><USER6> Let's just see after Trump and his team look into it.

/s for anyone wondering. This will go nowhere.

</comment></comment><comment><USER7> republicans are not a protected class, twitter can ban them if they like. It's not remotely illegal.

</comment><comment><USER8> Sad to see that the first 3 comments here are justifying and defending twitter shadow banning republican politicians. That's quite something. I would really love if there was an investigation into why the internet and social media all of a sudden has so many comments advocating for censorship. Almost everyone I talk to is against censorship and for open discourse and against social media picking sides. And yet, on the internet, it's nonstop advocacy for censorship. I don't think it's russian trolls either.

It's not just censorship on social media. There has been a surprising attack against net neutrality as well. And even defense of europe's "meme law". Maybe I'm just in a bubble where people are for free speech, against throttling and against criminalizing memes. But I don't think so. It just feels like there is an organized campaign for censorship being pushed all over social media and the internet.

<comment><USER9> It's quite simple: their server, their rules.

Much like you can ask anyone to leave your property, but can't censor them on public property. The Internet, as a network, should uphold free speech, that's why Net Neutrality is important. Particular servers, even if the POTUS decides to use one as his soapbox, have no such requirement.

<comment><USER10> This is a fine sentiment at small scale, but once a service becomes significant, close to mandatory, in daily life, the other aspects become overwhelming.

For sake of example, as much as I detest various extremists [†], I would strongly oppose blocking[‡] the extremists from local power company's electric grid (which is a publicly traded stock company in here), from the local telephone exchange (which in my city is a public-private partnership, IIRC), or yes, from the most popular microbloging platform out there. In 2018, Twitter access is about as necessary as phone exchange access in 1990s.

200 years ago cutting off either would have been fine, a non-issue really, as there was infrastructure, services, and knowledge & skills in place to lead normal life without electricity, running water, comms, etc.. But right now one doesn't simply go about the necessities of daily life - work, shopping, health care, etc. - without electricity and fast communications.

There's two competing approaches to the problem of extremism, hate, and disinformation: either "giving them voice empowers them", or "sunlight is the best disinfectant / let them speak so everybody can see their idiocy", and I happen to firmly belong in the later camp. From my POV, we aren't experiencing increase in extremism right now - quite the opposite; thanks to the modern tools we've noticed how bad it used to be in the hiding, and are successfully fixing the problems now that they surfaced.

I shed no tears for all the Twitters, Facebooks, Googles, etc. fall victims to their own success - becoming so successful they became widely considered necessities. They are still quite profitable with all the problems that causes, and all the constraints it puts on them. And that's not necessarily permanent state either; going by historical examples, access to MS Windows, or ability to listen to radio, or access to horse pastures used to be such necessities back in their heyday, but aren't terribly important anymore.

[†] that is not to imply republicans are extremists.

[‡] outside of criminal activity, which is mostly accepted as fair ground for bans and content removal - but is neither specific, nor unique, to trolls, haters, or extremists.

<comment><USER5> I could not disagree more that Twitter is as necessary as phones ever were in the last 50 years.

I don't ever visit it and am a happy, productive tech worker. I never miss it, and no one I know talks about it in person. I don't personally know any active users.

Its main influence on daily life seems to be when real media re-broadcasts the Twitter speech of famous people.

Facebook could be argued to be essential much more easily and has an order of magnitude more MAUs, but I'd disagree even with that. Lots of normal people I know (including many reading this) have quit FB and not looked back.

Someone who quit landline phones in the 90s would've been considered a nut or a recluse. The only alternative for communicating with them was the postal service, and there was no instantaneous alternative.</comment></comment></comment></comment>


In Nigeria, Solar-Powered Fridges at Outdoor Markets Save Food from Spoiling - <AUTHOR> http://gizmodo.com/in-nigeria-solar-powered-fridges-at-outdoor-markets-sa-1737306287 <comment><USER2> The text of the article mentions cold-rooms but the picture shows freestanding refrigerators. For small-scale storage they should should be using front- opening/chest-style refrigerators instead of freestanding ones.

Since cold air is denser than warm air, it falls. That means that every time someone opens a freestanding fridge a significant amount of your cold air falls right out. Top-opening fridges provide a well that retains the cold, significantly reducing energy usage especially on fridges that are frequently opened.

You see it a lot on freezers because the air is much colder and the cold tends to fall out even faster as a result. I think it would produce improvements here as well, since the energy produced by a solar panel is rather limited.

Assuming you could protect the cooling system and provide sufficient airflow - could you operate a fridge flipped on its back? Or does the compressor depend on proper orientation to have gravity return coolant and so on?

<comment><USER3> I've seen an article online where an "off-grid" home powered by solar re- purposes a chest freezer into a fridge for the very reason you state.

[http://newlifeonahomestead.com/convert-chest-freezer-to- frid...](http://newlifeonahomestead.com/convert-chest-freezer-to-fridge- solar/)

</comment></comment><comment><USER4> I wonder what the novel feature is here. Surely somebody would have thought of renting space in a solar powered fridge before? Maybe it's only recently become viable because of improvements in the economy?

<comment><USER3> The cost of solar panels (and batteries) has been dropping at a reasonable pace and is set to continue to do so for a while, so there's probably a whole bunch of "solar powered X" businesses that will become feasible as time goes on.

There's some nice synergies here, you need the refrigeration most when the sun is shining brightest, a well insulated fridge acts as a kind of battery, the solar panels shade the fridge from the sun, and so on that are strangely pleasing to some aesthetic sense of mine.

</comment></comment><comment><USER5> Interesting that the article specifically mentions that the solar panels feed and inverter first, and then power the refrigerator. There are refrigerators designed to work directly off of 24V DC, which would seem to be a more logical choice.

[http://m.alibaba.com/showroom/24v-refrigerator- compressor.ht...](http://m.alibaba.com/showroom/24v-refrigerator- compressor.html)

These are less common, though. Maybe cheaper commodity prices for AC-powered compressors offsets the additional inverter cost and efficiency losses?

<comment><USER6> You can cannibalize an AC compressor from a car. Alibaba has AC-compressors for as little as $1/pc. They are much cheaper, and much more common.

<comment><USER7> Most car A/C compressors are belt driven from the engine, eg no electric motor so not much use for Solar operation.

<comment><USER6> That's a good point - I didn't realize that. Still easy to cannibalize if you have an AC motor :)

</comment></comment></comment></comment><comment><USER8> Are batteries involved? Or is the insulation simply thick enough the keep the food cold enough during nighttime?

<comment><USER9> Bear in mind that in many "hot" places the temperature during the night falls down to nearly 0C, so it's probably enough to stop the food from spoiling.</comment></comment>


Ask HN: Bought a new laptop i7 7th gen. Should I return it? - <AUTHOR> Spent $850 on an ASUS development laptop. Specs are pretty good with 16gb ram, 256gb ssd + 1tb hdd, GTX 1050. But having seen the recent intel bug news, I feel that I overpaid for a machine that will only perform worse over time. Should I just return it and wait for the next gen CPUs that will have this fixed? <comment><USER2> New chips without these bugs aren't going to be ready for at least 2 years if not longer. While one bug affects intel, the other bug affects intel amd and arm chips. So it's not like you can get a better computer. I suppose you might join a class action suit though. Make sure your signing up for asus doesn't require binding arbitration.

Ultimately your chip will be plenty fast for dev work. and $850 is damn cheap compared to most macbook pros.

<comment><USER3> >compared to most macbook pros.

What you wrote is flamebait. It's an apples and oranges comparison, as has been detailed ad nauseum here on hn.

<comment><USER2> I mean I use (and post from) a macbook pro myself, so I'm hardly trying to flamebait anyone.

</comment></comment></comment><comment><USER4> As always, if you need the computer to get your work done now then you should buy it now. The next generation will always be better and faster.

<comment><USER5> He's specifically asking in the context of the recent Intel CPU bugs.</comment></comment>


The three I's of startup blogging - <AUTHOR> http://blog.chuwe.com/the-three-is-of-startup-bloggi

<comment><USER2> _I was struck by the relevancy to writing startup and business blogs. Immediacy is everything - a hindsight view offers a lot of well-selected, edited, abstracted lessons, but it's boring._

Geez, really? I better stop writing blog posts then.

 _I suffer from always trying to appear rational in my actions, so I spend too much time trying to analyze the underlying motivations about why I did something when I should simply write about what I did and the short, simple thoughts that brought me there._

Good point. Why analyze where we went right or wrong? What's the point of that? It's not like other people, or even ourselves, can learn from our mistakes.

 _... Amazon is amazing about this ..._

Amazon's blog is a start-up blog? Ok, then obviously we have a different definition of what a "start-up blog" is. I thought it was a blog about start- ups, written by an entrepreneur. Apparently tech blogs written by industry behemoths count as start-up blogs, according to this article.

I'm not quite sure what this article was talking about. Perhaps the author should have defined what he meant by "start-up blogs" before writing the article. Not at all sure where the "Indecipherable" part came from either.. who reads indecipherable blogs?? Does anyone intentionally write indecipherable blogs?

<comment><AUTHOR> Hey swombat, I definitely respect your opinion, so let me see if I can explain myself a bit more - and hopefully improve myself in the process.

By hindsight view, I meant retroactively analyzing everything after the adrenalin of the moment has worn off. It's pretty much what we always read. It's a bit cooler, more sterile, and methodical. And you're right, writing that down, careful and well-thought analysis is crucial. But for interesting writing (perhaps entertaining is a better word), immediacy offers a lot of emotional nuances that are lost with time.

And I simply chose Amazon because the way they openly run their AWS is a great example for startups. I was comparing their style to an article I read from box.net (an actual startup), and the difference was striking.

Finally, the indecipherable bit was simply from the original podcast on how to write a successful political diary. I don't think it's useful or really relevant for startups, and that's why I crossed it out. Perhaps I should have simply omitted it.

I'll make notes of all of this for the next time I write. Thanks!

<comment><USER2> I see what you mean. However, there's more than one purpose for a blog, and more than one approach to getting visitors - even from a start-up perspective.

To give you a very immediate example, my latest article on danieltenner.com is exactly as you describe - hindsight, cool, methodical (I like to think it's not overly sterile, but that's just me). Yet it gathered about 45k visitors so far (with more to come, no doubt.. even today it still got about 2000 visitors) and sent about 700 click-throughs to woobius.com through the front page on the day of publication.

I'd call it a very successful blog post in terms of promoting my business, even though it was not about my business and it was not even on my business blog (<http://www.woobius.com/scribbles>).

Other articles on Scribbles itself got thousands of visits and also sent hundreds of hits to Woobius.

Moreover, because those posts provide lasting value (precisely because they are not immediate), they tend to get linked by others, gain pagerank, and appear in google searches (an article I wrote about document control is getting a handful of hits from people looking for document control tools every day).

So, while I don't disagree that the "immediate" style can work very well, because it can be quite entertaining to many people and so they'll follow it. However, I don't think it's right to discard the "slow hindsight" style, because it can also work extremely well.

I think that would be my main disagreement with your article.

Btw, I agree that you should have left out the bit about political diaries.. it was more a distraction than anything else.

Keep it up, though, looking forward to more articles :-)

<comment><AUTHOR> Thanks swombat. I think we agree - it was simply my implication that this was "The one way". I'll be careful about that.

</comment></comment></comment></comment><comment><USER3> I believe you mean indiscreet, not indiscrete.

<comment><AUTHOR> Fixed. Thanks.

</comment></comment><comment><USER4> Ok, sometimes I feel authors think of a title first, then write a story around it. That being said, I don't have anything against the article.

<comment><AUTHOR> Sorry if it felt a bit vapid. I actually thought of writing it when I heard the interview on political diaries, and felt it matched very well with what I found was interesting for startup and business blogs as well.

I had been procrastinating writing about our "launch experience" on hn from Sunday because of coding to account for all the feedback we received. I tend to write in a very sanitary, evasive, and long-winded manner. The interview just jumped out at me with the "four i's". Out of those, I thought three were really relevant to startups/business blogs.

Thanks for your feedback.</comment></comment>


[Seriously OT] Led Zeppelin re-forming for concert - <AUTHOR> http://www.cnn.com/2007/SHOWBIZ/Music/09/12/ledzeppelin.comeback/index.html

<comment><AUTHOR> I'll understand if this gets whacked, but I've known few hackers who weren't big Zeppelin fans.</comment>


When We Eat, or Don't Eat, May Be Critical for Health - <AUTHOR> https://www.nytimes.com/2018/07/24/well/when-we-eat-or-dont-eat-may-be-critical-for-health.html <comment><USER2> Anecdotally I've noticed some benefits from skipping breakfast and eating only within an eight hour window. My blood sugar and energy levels are more even throughout the day, I can skip a meal if necessary when traveling or something without feeling too hungry, and I've lost a few pounds I was having a hard time getting rid of any other way.

In the morning I just drink coffee black and then have a somewhat early lunch and dinner. Maybe it would be more effective to skip dinner instead but this seems to be working.

<comment> saiya-jin Had a friend in university, he did military service for 1 year and the only thing he had for breakfast during that time was a strong black coffee without any sugar nor milk.

He screwed up his digestive tract for good - there is apparently something in coffee that is quite aggressive on stomach and gets neutralized when mixed with any sugar/milk/food. He will never be able to eat vast array of common foods - things like marmelades and other basic food. Sorry I don't have any further info, has been few years ago.

He might have some unique predisposition for this, don't know, but better be careful and put at least some milk in it. It makes it much more filling anyway

<comment><USER3> For another anecdote, I have been drinking black coffee, and only black coffee, for the first four to six hours of the day. Sometimes water as well.

I have done this for about six years without damaging my digestion in any detectable way. I would bet money your friend has or had a Helicobacter pylori infection; treatment is antibiotics.

<comment><USER4> I tried the same for for about 6 months but had to stop because of stomach ache. I'm not infected with helicobacter pylori.

<comment><USER3> Sure, people differ greatly in acid sensitivity and other factors that make no one diet right for everyone.

The stomach ache went away when you quit? If not... are you sure you don't have helicobacter pylori?

<comment><USER4> Yes it went away and yes I was tested multiple times for helicobacter pylori.

</comment></comment></comment></comment></comment></comment><comment><USER5> I just tried intermittent fasting (16hr fast, 8hr feeding window). It was very challenging at first and I had to catch myself multiple times opening the cupboard at night by habit, but it did help me realize how much snacking I really do.

My fat % is way down and muscle mass up, so it's been very positive so far and much easier after week 2 when my body fully adapted to the new eating style.

<comment><USER6> N.B. fasting in general works best if you are eating a high fat diet, instead of eating a lot a carbohydrates. You might also need more salt in your diet, including during your fasting ;-)

Carbohydrates leads to raised insulin levels, which inhibits the consumption of fat from your fat store. Our body is pretty smart, so when insulin goes down, it begins to consume your fat for energy, a metabolic state called _ketosis_.

But until that insulin goes down, what you're experiencing is the dreaded _keto flu_. So you can experience fatigue, dizziness, even head aches ... which if serious are not normal and you should stop your fast ;-)

Anyway, for successful fasting what I found is this:

1\. eat high fat, low carbs meals; if you can't go full keto, at least do that just before starting your fast

2\. you might get an electrolytes imbalance, which is THE reason for dizziness and headaches: the solution is to take salt supplements — e.g. right now I'm in my second day of water fasting, when I woke up I felt a little dizzy, but took 4 grams of salt (about one teaspoon) and now I feel great; there are also "salt tablets" on the market btw, which are easier to swallow

NOTE: there's science behind the keto diet, studies and high profile medics backing it up, but there's also a lot of denial about it. One good book describing what the deal is with _insulin resistance_ , why it makes us fat and why the keto diet is healthy would be the "The Obesity Code: Unlocking the Secrets of Weight Loss" by Dr. Jason Fung.

Here's the podcast dedicated to it, talking of keto and fasting; it's a great podcast: [http://obesitycodepodcast.com](http://obesitycodepodcast.com)

<comment><USER7> What is N.B fasting?

I do 16/8 I.F fasting on a HCLF diet and have never experienced any of those issues.

There is plenty of evidence showing the benefits of ketosis on the body, and benefits from people switching from SAD to Keto reversing diabetes and the likes - but that's more down to calorie restriction. With regular keto you're raising your risk long-term for diabetes, heart-disease, cancer and stroke due to increased meat/dairy/egg consumption.

[https://nutritionfacts.org/video/what-causes-insulin- resista...](https://nutritionfacts.org/video/what-causes-insulin-resistance/)

You're far better just eating a varied diet with as little animal produce as possible and doing I.F which releases 5x the amount of ketones than the keto diet.

<comment><USER6> > _I do 16 /8 I.F fasting_

That's not enough to trigger those symptoms for many people, as it depends on how much salt you have in your diet and your own metabolism of course. But do a 36 to a 72 hours fasting, which is otherwise a very healthy thing to do from time to time and you'll experience it head on.

>> " _that 's more down to calorie restriction_"

Why did the plane crash? Due to gravity. What should the plane have done to not go down? It should've had more lift.

Now tell me, how does that help in establishing why the plane actually crashed and what to do to solve it? That's how " _calorie restriction_ " sounds as an explanation, being a _prima facie_ observation. Of course you're reducing calories. But _HOW_ you're reducing those calories makes all the difference.

Another one: Why do people get rich? Because their spending are less than their earnings. What should people do to get rich? They should cut their spending or raise their earnings. Well duh! But it's a totally useless observation, isn't it?

>> " _With regular keto you 're raising your risk long-term for diabetes, heart-disease, cancer and stroke due to increased meat/dairy/egg consumption._"

That's absolute bullshit.

Here's one of the largest studies on dietary fat, done on 50,000 women, spanning for 8 years, showing that going low fat did absolutely nothing to prevent heart disease: [https://www.ncbi.nlm.nih.gov/pubmed/28515068](https://www.ncbi.nlm.nih.gov/pubmed/28515068)

I also wish people would stop posting YouTube videos btw. I don't watch YouTube bullshit. Post links to studies that I can verify.

<comment><USER7> >Now tell me, how does that help in establishing why the plane actually crashed and what to do to solve it? That's how "calorie restriction" sounds as an explanation, being a prima facie observation. Of course you're reducing calories. But HOW you're reducing those calories makes all the difference.

A ketogenic diet can also reduce hyperglycemia in diabetes, but it is not the wisest choice for the long term, diabetes or not. Addressing the cause of the disease, excess unhealthy fats in the case of diabetes, is better than merely treating the symptoms (hyperglycemia), especially if you can do both with the same intervention ([http://www.pcrm.org/health/diabetes- resources](http://www.pcrm.org/health/diabetes-resources)).

>Here's one of the largest studies on dietary fat, done on 50,000 women, spanning for 8 years, showing that going low fat did absolutely nothing to prevent heart disease: [https://www.ncbi.nlm.nih.gov/pubmed/28515068](https://www.ncbi.nlm.nih.gov/pubmed/28515068)

Whole-food plant-based diets are the only diets shown to reverse heart disease in a clinical setting:

Dr. Richard Fleming showing progression with low-carb diet and reversal with WFPB [https://www.ncbi.nlm.nih.gov/pubmed/1110832](https://www.ncbi.nlm.nih.gov/pubmed/1110832)

Dr. Dean Ornish reversing heart disease with a WFPB diet [https://www.ncbi.nlm.nih.gov/pubmed/9863851](https://www.ncbi.nlm.nih.gov/pubmed/9863851)

Dr. Caldwell Esselstyn Jr reversing heart disease with a WFPB diet [https://www.ncbi.nlm.nih.gov/pubmed/25198208](https://www.ncbi.nlm.nih.gov/pubmed/25198208)

[http://www.ncbi.nlm.nih.gov/pubmed/17518696](http://www.ncbi.nlm.nih.gov/pubmed/17518696) CONCLUSIONS: Long-term consumption of a low-calorie low-protein vegan diet or regular endurance exercise training is associated with low cardiometabolic risk. Moreover, our data suggest that specific components of a low-calorie low-protein vegan diet provide additional beneficial effects on blood pressure.

[http://www.ncbi.nlm.nih.gov/pubmed/7019459](http://www.ncbi.nlm.nih.gov/pubmed/7019459) The study suggests an adverse effect of consumption of beef on plasma lipid and BP levels.

[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1312295/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1312295/)

[http://www.ncbi.nlm.nih.gov/pubmed/15172426](http://www.ncbi.nlm.nih.gov/pubmed/15172426) Randomized trial data suggest atherosclerosis progression and coronary heart disease events are minimized when LDL is lowered to <70 mg/dl

Dr. Kim Williams - the President of American College of Cardiology said, "There are two types of cardiologists: vegans and those who haven't read the data."

[https://nutritionfacts.org/video/low-carb-diets-and- coronary...](https://nutritionfacts.org/video/low-carb-diets-and-coronary- blood-flow/#sources)

[https://nutritionfacts.org/video/how-not-to-die-from- heart-d...](https://nutritionfacts.org/video/how-not-to-die-from-heart- disease/#sources)

[https://nutritionfacts.org/video/the-problem-with-the- paleo-...](https://nutritionfacts.org/video/the-problem-with-the-paleo-diet- argument/#sources)

>I also wish people would stop posting YouTube videos btw. I don't watch YouTube bullshit. Post links to studies that I can verify.

Wasn't linking a youtube video. He puts a video on the page to make it easy to digest but all the sources are cited.

[https://nutritionfacts.org/video/what-causes-insulin- resista...](https://nutritionfacts.org/video/what-causes-insulin- resistance/#sources)

\--

You've picked the most obvious one. Heart disease and animal produce link is well established and accepted.

Fat 100% causes diabetes. So does low-fibre or high-sugar though ([https://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0072693/](https://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0072693/)). There is also a lot of extremely shady shit going on in diabetes research like this >>

"This study was supported by grants from the American Diabetes Association, the Minnesota Beef Council, and the Colorado and Nebraska Beef Councils".

Cancer is established but the risk isn't that great (still causes 21% of bowel cancers though)

</comment></comment></comment></comment></comment><comment><USER8> Related: the jetlag fast. Fasting for 12-16 hours and then eating your next meal in the morning in your new destination is supposed to help adjust to jetlag faster.

[https://harpers.org/blog/2012/03/the-empty-stomach- fasting-t...](https://harpers.org/blog/2012/03/the-empty-stomach-fasting-to- beat-jet-lag/)

</comment><comment><USER9> Well, that's a shame. I hate eating in the morning, and love eating in the evening. I find it surprising that earlier humans in history didn't follow this pattern, allowing the body to adapt to digesting food later at night.

<comment><USER10> Me too; I find it easy to fast when I skip eating in the morning / afternoon — its like my body realizes its not getting any food and turns off the hunger.

"On the time-restricted regimen, the men had lower insulin, reduced levels of oxidative stress, less nighttime hunger and significantly lower blood pressure."

If I eat in the morning I'm ravenous until bed time :(

<comment><USER11> Likewise, I normally don't eat until late afternoon or evening, and I'm usually not hungry until then. But if I eat breakfast, I then want to keep eating all day!

</comment></comment></comment><comment><USER12> I think they're on to something with the intermittent fasting but starting early in the day is not a convincing argument for a night owl like me. I'd like to see them compare the early starting six or eight hour intermittent fasting group to a later one to see if there is a difference and include morning larks in the first one and night owls in the second. My hypothesis is that the groups would fare similarly but without the study we won't know.

</comment><comment><USER13> I'd be curious to know why the many online advocates of intermittent fasting (and other time restricted diets) think it's any different than the many other fad diets that have come, gone, and come again.

Diets in general seem to bring out the obsessive optimization that many (especially on this website) are prone too and the "one quick trick" mentality.

I'll be surprised if there is ever any better advice than "eat food. not too much. mostly plants".

<comment><USER14> Because unlike other fad diets I've actually experienced weight loss, less hunger, better focus, more stable energy levels, lower blood pressure, better sleep, not to mention cost savings on food, since starting a 16/8 intermittent fast. YMMV, but for many people the benefits are quite stark and palpable beyond possible placebo effect.

Sure it's annecdotal, but there's serious science behind some of the benefits (see this article among other studies). To flip your question around, why are you skeptics so certain that it's just another fad diet that will come and go? IF can actually be a useful enforcement mechanism for the first two parts of "eat food, not too much, mostly vegetables"

<comment><USER13> > why are you skeptics so certain that it's just another fad diet that will > come and go?

Because, like most skeptics, we have seen this pattern before. Supposed scientific support and loads of anecdotal evidence are common to all popular fad diets. The scientific support of IF is also much less solid than it's advocates would have you believe.

> IF can actually be a useful enforcement mechanism for the first two parts of > "eat food, not too much, mostly vegetables"

I agree with this - so can basically any other fad diet. People that recognize IF as a method of enforcing a basic CICO diet, I agree with. There are plenty of people that, like any other fad diet, think that IF is somehow special in ways unrelated to it's restriction of calories. _That 's_ what makes it the same as any other fad diet to me.

> I've actually experienced weight loss, less hunger, better focus, more > stable energy levels, lower blood pressure, better sleep

People report this from atkins, vegan, paleo, keto, juice cleanse, weight watchers, south beach, etc. I am happy that you've found a diet that works for you. I hope that everyone finds a healthy diet that works for them.

I just want people to realize that CICO will _always_ be the key to weight loss regardless of how it is achieved, and "eat food. not too much. mostly plants" will (probably) _always_ be the key to a health diet.

</comment></comment></comment><comment><USER15> > "If you're constantly eating at a time of day when you're not getting bright > light exposure, then the different clock systems become out of sync,"

So if you live in a part of the world when the sun sets at 2 or 3PM in the winter, you will likely not have a very social eating life following these recommendations.

<comment><USER16> From what I heard, people don't usually eat out in Scandinavian countries.

<comment><USER15> There's some truth to that I suppose. But, TBH, this comment was mind bending to me. May I remind you that it is possible to eat with friends even though there's no business preparing your food? :)

</comment></comment></comment><comment><USER17> I didn't eat for 60 days. It was, without a doubt, the best decision I've ever made for my physical health.

<comment><USER18> Congratulations on a great decision. I'm at the most 10 kg overweight (86 Kg/ 20% body fat right now) and I regularly go a week without ingesting anything but water and electrolytes. Shooting for 20 days of prolonged fasting in August.

I've found it to be the easiest to stick to diet I've ever tried, and I've done everything. If you struggle with moderation, fasting for prolonged periods has been much more productive for me. Not only do you burn fat 3-5x faster than any other diet, but your cravings go away incredibly quickly. I used to compulsively eat 3 chocolate croissants a day and after a couple of long fasts, I don't really care for them.

The emotional benefits are almost bigger. After 3 days without food my mind stops racing and my anxiety melts away. It feels like zen no-mind, and much stronger than I've been able to achieve by meditating.

Needless to say I'm a fan of going without food now, to the point where I hope to do it regularly for the rest of my life (unless evidence comes out pointing to it being bad for your health). Happy to answer any questions that pop up about fasting.

<comment><USER17> > Shooting for 20 days of prolonged fasting in August.

Awesome. You should be able to do that no problem. Just make sure you're supplementing plenty of sodium, potassium and magnesium.

> The emotional benefits are almost bigger. After 3 days without food my mind > stops racing and my anxiety melts away. It feels like zen no-mind, and much > stronger than I've been able to achieve by meditating.

This was my experience as well. Somewhere between day 7 and day 10 everything just got easier.

<comment><USER18> That's really good to hear. How was your libido through the 60 days? Mine pretty much goes away by day 3. That's been the thing preventing me from going longer, girlfriends don't take kindly to the lack of action between the sheets.

On electrolytes, last time I fasted for 5 days I only drank Snake Juice (salts dissolved in water taken through the day) and it did wonders for me. The guy promoting it is more than a little on the wacko side, but he hit the nail on the head with that recipe.

</comment></comment></comment></comment><comment><USER19> I find it very surprising that the first meal should be after getting up. It seems unlikely to me that our hunter and gatherer foragers had immediate access to food after getting up. Also the article mentions multiple times that food consumption during times of little or no daylight is messing with our internal clocks, but I wonder if that is actually true, because in Winter we only have daylight from about 8am to 4pm, often less. I eat when my body tells me it is hungry, which literally can happen at any time of the day. Sometimes I am physically active for most of the day (sports) and then I am normally pretty hungry at even 9 or 10pm and I think it would be pretty unhealthy not to listen to my own body. This article is pretty much bollocks, the only rule for a healthy lifestyle is: Listen to your body. Don't follow any dogmatic approach!

<comment><USER20> > Listen to your body. Don't follow any dogmatic approach!

When my body tells me to inject more drugs into my veins because it feels good I should really listen to my body! Your comment is pretty much bollocks to be honest. My example is reductio ad absurdum of course but the point still stands. If we were to listen to our body 24/7 we'd all be mindless instagram/facebook/dopamine zombies.

<comment><USER21> The slogan should probably be "Learn about your body." Also, there may be some truth to the second part which resonates with what Arnold says: "Shock your body!"

</comment></comment></comment><comment><USER22> Does intermittent fasting make its health benefit claims based on any of this? Because this sounds like fasting windows.

<comment><USER14> Depends on they length of the fast. Intermittent fasting focuses on the benefits of not eating for extended periods, and fast lengths can vary from 12 hours to entire days. Time-restricted eating focuses on the benefits of eating daily within a specific time window.

The "standard" 16/8 intermittent fast would get all the benefits of time restricted eating as well.</comment></comment>


The search for a thinking machine - <AUTHOR> http://www.bbc.co.uk/news/technology-32334573 <comment><USER2> Sometimes I wonder if we're having problems due to trying to make a successful AI. If you really think about it human thought is a culmination of failures from both an evolutionary standpoint and a life-experience standpoint.

Aiming to have AI successfully identify, say, dog breeds better than humans can could be a mistake that we are making.

</comment><comment><USER3> Relevant: A Search Engine that Thinks, Almost [http://newscenter.lbl.gov/2005/03/31/a-search-engine-that- th...](http://newscenter.lbl.gov/2005/03/31/a-search-engine-that-thinks- almost/)

</comment><comment><USER4> > Thanks, in part, to a new era of machine learning, computer are already > starting to assimilate information from raw data in the same way as the > human infant learns from the world around her.

Computer are also able to spot basic grammatical errors that that BBC should not be making..</comment>


All fish in this Banff lake will be removed and killed due to whirling disease - <AUTHOR> http://www.cbc.ca/news/canada/calgary/whirling-disease-johnson-lake-banff-cutthroat-westslope-parks-canada-two-jack-1.4119302 <comment><USER2> So no mention of whether or not the disease was introduced naturally?

What metrics do the gubments use do decide between good meddling and bad meddling?</comment>