Stross on: why my Android tablet experience sucks - <AUTHOR> http://www.antipope.org/charlie/blog-static/2012/06/gadget-patrol-why-android-tabl.html

<comment><USER2> Interestingly titled 'why _my_...', as it heads off the trolls at the pass, decrying "yeah but it's great for MY needs".

But Stross makes some really good points, proprietary connectors are a relic of a long time ago and part of the reason why Apple is still hawking them is that they don't want to do something silly like Samsung have, and break compatibility. Which isn't a good idea on Apples part as the dock connector is still crap, but I'd wager they'd have a lot of upset customers if they killed it.

I only just recently bought an iPad, it took time to convince me I needed one. I did look at Android for a while though, mostly for the price point. My experience was much the same, I found at the name brand manufacturers quality was excellent but the software was terrible, and the off brand was vice-versa. Apple has the market stitched up at the moment by having a) excellent hardware and b) excellent software (which even if you don't _like_ iOS you can't argue it's very good at what it does).

<comment><USER3> > part of the reason why Apple is still hawking them

I've always assumed a large part of the reason Apple uses proprietary connectors (other than any potential circuit real estate savings) is that it significantly reduces their testing matrix. If you have to have a license to plug it into their system, they don't have to test it unless you paid the fee.

If you saw the terrible, horrible things that hardware vendors do to the USB specification just because "it doesn't crash when attached to a Windows <N> box" you would cry bloody tears.

<comment><USER2> You know what, that's a super smart idea that I hadn't even considered. I just assumed it was to not upset consumers and/or because they've already got an entire production chain geared for producing them for absolutely nothing, but forcing it down that route does ensure Apple has a slightly easier life.

</comment></comment></comment><comment><USER4> Reasons why Android sucks: OEM's installing crapware, no unified hardware strategy, lots of crappy apps

Reasons why iOS sucks: App store monopoly, crappy memory management, no hardware freedom, read-only devices

That's really it. If you're a power user, like most of the HNers, then Android tends to be much more attractive. My ASUS Transformer Prime with keyboard dock might be the the best piece of hardware I've ever owned. I think ICS multitasking is the best I've ever seen. I've owned no less than 3 iOS devices, and I can see how they're great for the layperson who is maybe just learning to use computers for the first time, but I can't use one for 10 minutes without being frustrated with a thought like "Is that app still running in the background?" or "Why can't I customize the way this works?"

But then, that just proves I'm not 90% of the Tablet market. I don't fault Apple for going after the lowest common denominator, but doing so means there devices are of no use to me.

<comment><USER5> _"Is that app still running in the background?"_

My Android experience has been on par with this, except with "how can I make that app stop running in the background?"

If there's a way to easily just close an app and have it be done with, I don't know of it. Task managers I've found to be cumbersome and people tend to frown on them saying the system runs more efficiently on its own. So why is this background app listed above Bluetooth in the power manager?

<comment><USER6> Yeah, this has been a paid. It's fixed in ICS. Hold home button, see list of running apps, swipe left any you want to close.

<comment><USER7> AFAIK, this doesn't actually kill the task. I believe that it just removes it from the displayed list.

<comment><USER8> You are correct, it doesn't kill the task.

</comment></comment></comment></comment></comment><comment><USER9> Should be titled: "Why my experience with one Android tablet sucks".

While his reasons are sound, I don't think these reasons are entirely encompassing of all Android tablet experiences. I for one have been using Asus Transformer for a while now and I am entirely satisfied with the experience. While I am entirely capable of rooting the device, the thought has never even crossed my mind. Asus has done a great job of actually bundling apps that I use (Polaris Office and MyCloud) and the few other applications that they've bundled are not even noticable.

Perhaps we could talk about standardizaton of connectors, but I think this problem transcends the Android Tablet experiece and even the "Android Experience". It would be nice to have one cable fit all my devices (tablet, phone etc), but alas I don't ever see this happening. I think the problem is less about conformity and more about proprietary hardware design and that's a whole bunch of issues I know nothing about.

I could easily write "Why my ASUS Android Tablet experience is awesome". With freedom comes choices, and Google has left it up to the hardware manufacturer to make the right software choices for their hardware. Asus has made good choices while Samsung has chosen poorly. With freedom comes responsibility..and all that. Shouldn't we be talking about this instead?

<comment><USER8> I will second that. I _love_ that my Transformer Prime has an (almost?) stock ICS experience, I think ICS is beautifully designed and very usable.

I'm trying to decide which phone should replace my Desire HD, and I'm leaning towards the SGSIII, but the custom UI is really putting me off. Too bad there's no stock-ish phone in the top contenders, I might have to go with the One X if it's less customized.

</comment></comment><comment><USER10> This is sort of an odd article. He is calling out Samsung for their walled garden approach yet yearns for a 7" iPad. Secondly, he is critical of the dock connector but, if rumors are to be believed and it looks increasingly likely, that Apple will adopt a new, smaller dock connector for iOS devices.

I can't blame Samsung in wanting to reduce their dependency on Google when they control over half of the Android phone market and sell as many, if not more, smartphones than Apple.

<comment><AUTHOR> You missed the key angle: ecosystem size.

Apple has first-mover advantage: an 8 year legacy of dock-connector-compatible third party peripherals.

Samsung does not, and if they keep changing proprietary connectors like underpants nobody in their right mind will even _think_ about producing compatible hardware peripherals for their devices.

Similarly, Apple has a pre-existing, pre-iOS market of loyal OSX customers who can be coaxed into embracing iOS as a portable extension of their desktop, with Calendar and Mail and Address book syncing. Samsung doesn't have such a captive audience, and won't build one by going about it this way.

You're right about Samsung having reason to reduce their dependency on Google, but my point is they're going about it the wrong way from a UX point of view.

Per Drucker, the job of any successful company is to acquire customers. Preferably loyal repeat customers. This is not how to do it.

<comment><USER11> And mail doesn't restrict you from emailing non iOS users (indeed, it supports Exchange and gmail), etc. etc.

<comment><USER12> Did Samsung really restrict an E-Mail application to one maildomain? Yes I read the article, but I still can't believe it... why would anyone want to use such crapware? Are they even asking themselves this question?

<comment><USER11> "chat to other Galaxy Tab users, mail via walled garden servers, share photos with other Samsung owners, and so on" so this is all incorrect? (I genuinely don't know.)

Consider the way iOS Messages works — it bypasses SMS when possible to save you money, but transparently supports SMS. The point stands even if the particulars are slightly incorrect.

</comment></comment></comment></comment></comment><comment><USER13> Much more balanced look than the article floating around yesterday. Although, the 7" form factor seems even less practical too me. When I was purchasing a tablet, I had my mind set on the Blackberry Playbook, until I went and played with it. So small! Not enough differential from my phone screen to warrant a new device.

> _"Why does Samsung insist on trying to steer users towards Samsung apps that > duplicate their functionality but miss out key features that make them > useful, like, oh, being able to share stuff with folks who don't own a > Samsung device"_

I don't understand this either. In fact, when I'm phone shopping, I refuse to buy a product that has applications installed, either by the service provider or manufacturer, that I can't remove. How naive are these companies to think that, of all the ways to mimic Apple, creating a "wall-garden" on an inferior eco-system is the choice?

First build an awesome tablet. If it takes off, invest in having some good apps developed. Or rely on the Market to sort it out. Applications should _add_ to the experience, not give you coronary every time you use your device.

</comment><comment><USER14> And this is why I will only buy Nexus phones. I cannot be arsed rooting my phone (and potentially bricking it), and I don't want to have to deal with crapware being forced on me.

(I understand that even then you're not entirely safe - with the Verizon version of the Galaxy Nexus _still_ having Verizon software on top. FFS.)

</comment><comment><USER15> He doesn't even yet get to experience what will happen when Google releases a new version of Android and you expect to see it on your tablet in a reasonable time frame.

I gave up on my Galaxy Tab 10.1 when Android 4 came out and Samsung chose to first announce and release a brand new line of tablet products running Android 4, and then slack on updating the ones that weren't even a year old at the time. The message they're sending was "oh, you want a tablet running a current OS? Why not ditch the one you just bought, and buy one of these?"

As of December, maybe three weeks after the source was released to Android 4, there were custom firmware builds for the Tab 10.1 that were 80% functional - no camera support and power management could be flaky at times. If some hobbyists coding at home at night can get a tracer bullet in a couple weeks, Samsung has no excuse for the fact that there still has been no official update released.

I prefer Android to iOS in general, but only on a Nexus device with less of a gap between Google as the software vendor and me as the end user.

<comment><USER16> Are the iPads updated with new OS releases? Do they autoupdate?

<comment><AUTHOR> Yes and yes. As of iOS 5.x, iPads can update their own OS over wifi without requiring a complete nuke-and-reinstall-from-backup (as was needed in previous iterations of the OS).

</comment></comment></comment><comment><USER17> He should have said: "why my Samsung Android table experience sucks". What he says does not apply to my Xoom. Yes, they are a few apps loaded that you can't uninstall, but most of them I was using anyway. The rest is all google stuff, or whatever you wish to use.

</comment><comment><USER18> While not a fan of the OEM crapware, I have found it completely harmless on my Tab2. Yes, it takes up some space on the internal flash, but I would be surprised if it was much more than 200mb.

</comment><comment><USER7> Also, keep in mind that ICS offers the ability to disable unwanted builtin apps: [http://blogs.computerworld.com/19435/disable_system_apps_and...](http://blogs.computerworld.com/19435/disable_system_apps_android_ice_cream_sandwich)

This removes them from the applications menus and prevents them from ever running in the background.

</comment><comment><USER19> Didn't see any real argument about the supposed bad user experience. Just another blog review that says "this sucks because isn't done the apple way"

</comment><comment><USER20> While the things he talked about can be a bit annoying, that's far from the biggest problem with Android tablets. The biggest problems are that Google needs to:

1) improve the tablet experience _dramatically_. It shouldn't just be similar or slightly better than iPad. It should be _much_ better, while also easy to get by most people. I suggest going the direction of the Chameleon UI, where your "widgets" can be a lot more useful and functional than Microsoft's "tiles" and of course than Apple's "icons" on the homescreen. Also make the OS a lot better at multi-tasking, so you can use at least 2 apps at once for example.

2) get more tablet apps. A lot more. They've wasted 1.5 years not giving a _crap_ about Android tablet apps. For crying out loud Google - take this issue seriously before you lose the tablet ecosystem forever. The _least_ you can do is get top 100 apps to have a tablet app. But you really need to give an incentive for all devs to do it.

I suggest lowering the fee you get from devs by 10% out of the 30% or whatever you're charging them now, for those who are willing to adopt the Holo design style. Apple had a style guide and resources from the beginning, so the vast majority of devs are already using the iOS style for apps, while Microsoft is forcing Metro on all developers.

So you go the "incentive" route, and have a "special" category of apps, or give them a certain badge on the Play Store, for those that use the Holo style, and also reduce the fee from 30% to 20% for those who are willing to do it. Also put them through a more rigorous approval process. And give them higher visibility in the Play Store. This way they have plenty of incentive to follow your style guidelines, _and_ to make a tablet version (ask for that, too).

<comment><USER21> > It shouldn't just be similar or slightly better than iPad. It should be much > better

That's easy for you to say. Google has a lot of the world's smartest people working for them, but beating iOS in design and usability is not something they are geared to do as an organization.</comment></comment>


Advice for Apple: Grow the iPod Touch - <AUTHOR> http://www.pcworld.com/article/172296/advice_for_apple_grow_the_ipod_touch.html

<comment><USER2> Most people are buying a Touch because they don't want to get involved in a long cellular contract. Otherwise you might as well just get an iPhone.

<comment><USER3> not really, I don't want an iPhone, I like the Blackberry much better, but I bought a touch for other uses.

<comment><USER2> In your case would you be willing to pay for a 3G data plan for the Touch (presumably along with a separate plan for your BB?) Probably a really limited market for that type of expense for mobile data. Lots of people have a hard time justifying one $30/month data plan. I think it would make more sense for Apple to offer the iPhone on multiple carriers and carrier competition would take care of the rest. Someone would offer an iPhone with a data-only contract probably with some tie-in on Pay-As-You voice minutes.

</comment></comment></comment><comment><USER4> The potential of the iPhone/iPod Touch platform is something I think everyone underestimates. What these two devices do is to give developers programmatic access to a bunch of things that have never been available before. Find me the other voice recorders with APIs or the other cameras with APIs. It really is the first truly portable computer.

The iPod touch makes that all available (camera coming soon I'm sure) at a minuscule price point. Meaning they can be widely deployed. Apple needs to put more focus on the security of the devices but once they do I could see iPod touches being deployed in hospitals, law offices, construction sites and just about anywhere that values portability.

</comment><comment><AUTHOR> I have be waiting and have been excited for the day to have an unrestricted Internet mobile device that does everything my laptop/desktops do, but in my pocket! I don't think Apple would ever do what the author is suggesting, but it does leave the door open for competitors to create something like this. I'd buy a Zune HD if it did this, but alas that device is more focused on music and does not have built in 3G/4G.

<comment><USER5> Try a Zii Egg instead: two cameras (video & still), wifi, a multi-touch screen, music & video playback, HD output...

</comment></comment><comment><USER6> Ah yes, because Apple is crying out for _advice_ \- they're really not sure what to do next.</comment>


Ask HN: Machine Learning for Non-Programners? - <AUTHOR> I am noob to coding, an learning Python for web dev. I am interested in machine learning too. Can a non-programmer learn machine learning? How to get started? <comment><USER2> Start with the mathematical aspects of machine learning. "Pattern Recognition and Machine Learning" by Christopher Bishop provides a general idea of the field. Pick up with learning related python frameworks on the way. It's better to know what you're really doing before going into the messy programming work.</comment>


Droplo - drag&drop subversion on the cloud - <AUTHOR>

tl;dr: We're looking to get some feedback on a new web app we're building called Droplo, a subversion on the cloud with drag&drop capabilities for committing files and keeping track of revisions.
Link: http://droplo.com/
It's different from Github or Beanstalk in that it's not targeted towards the developer who already has git installed and configured on their system and knows how to use it, but rather a designer or creative professional who wants an intuitive, private online solution for project hosting and revising accessible everywhere.
It's in early development stages and we're trying to get a lot of traction around the idea. We've set up an incentive to send invites to use the app for free prelaunch if you can get 5 people to sign up through your referral link.
We have an initial non-functional, gui-only mockup here: http://droplo.com/demo/main
Please feel free to give any feedback and tweet to your followers if you like it. <comment><AUTHOR> Clickable: <http://droplo.com/></comment>


A first look at Mojang's next game Scrolls - <AUTHOR> http://gaming.icrontic.com/news/first-look-at-scrolls-during-minecon-2011/

<comment><USER2> > Mojang won the rights to use the name Scrolls.

Didn't they just not get an injunction on using the name?

</comment><comment><USER3> I just hope it doesn't use Java like Minecraft.

<comment><USER4> Its use of Java is likely the only reason it has worked equally well on OS X and Linux since the beginning.

Also, many of its inefficiencies are not related to Java.</comment></comment>


Show HN: Go + Martini + Facebook - <AUTHOR> https://github.com/fredkelly/go-martini-facebook

<comment><USER3> I get the feeling there are bots/zealots upvoting anything with the word go in the title... and I say this as a fan of the language.

<comment><USER4> Probably just zealots.

It was the exact same thing on HN when Node.js was just starting to spike in popularity. Same with frontend JS frameworks.

<comment><USER3> Possibly. Like I said, I like Go. And I don't like to insult anyone's work. But this doesn't seem front page worthy. It's incomplete, short, and not idiomatic. Then I remember seeing posts about Go the game too, which made me suspect bots. I don't care quite enough to test, but I hate to think this quality source of news is being manipulated. Ah well.

</comment></comment></comment><comment><AUTHOR> My first dabble in to the world of Go. By no means perfect so I'd love to get some feedback/revisions from more experienced Gophers :)

</comment><comment><USER5> I don't understand this, Martini has an awesome middleware that supports Google, Github, Facebook and LinkedIn out of the box:

[https://github.com/martini-contrib/oauth2](https://github.com/martini- contrib/oauth2)

with easy ways to check to see if a user is authenticated.

Gorm is an awesome choice, but this code just looks messy to me

<comment><AUTHOR> Hey, thanks for your critique - I really appreciate you taking the time to check out the code. I wanted to be able to validate requests from a mobile application (or similar), rather than building an entire OAuth flow. I'm hoping this will offer a useful starting point to newcomers like myself who want to get something simple off the ground.

</comment></comment><comment><USER6> Why the code is so full of undocumented TODOs? Is this a finished product or work in progress? If the former then you need good documentation. Badly. If the latter then I'd only present it to HN after the work is finished.

<comment><AUTHOR> Thanks for checking it out. Yes I totally agree on all fronts. This is indeed WIP, my hope in posting to HN is to get some assistance with said work - <USER2> you're right I should make that more clear.

</comment></comment><comment><USER7> Okay, cool, but... MySQL?!

<comment><USER8> What about MySQL?

<comment><AUTHOR> Damn, you spotted my glaring omission! I guess my thinking is that with Gorp the database choice is up to you ..maybe go-martini-facebook-gorp then ;)

<comment><USER8> Oh, I was just asking what pshc's comment was about. I'm still unsure what: "Okay, cool, but... MySQL?!" is supposed to mean.

<comment><AUTHOR> Ah, in that case I'm not sure, perhaps an aversion to MySQL? Either way Gorp gives some level of choice.</comment></comment></comment></comment></comment>


Production-Oriented Development - <AUTHOR> https://medium.com/@paulosman/production-oriented-development-8ae05f8cc7ea <comment><USER2> Right. Commenting on a very specific part of this with an anecdote. I did some integration with a 'neo bank' a few years ago where the CTO said testing and staging envs are a lie. I vehemently disagree(d) but they were paying to only test on production. I guess you can guess what happened (and not because of us as we spent 10k of our own money to build a simulator of their env; I have some kind of pride); testing was extremely expensive (it is a bank so production testing and having bugs is actually losing money... also they could not really test millions of transactions because of that so there were bugs, many bugs, in their system...), violated rules and the company died and got bought for scraps.

I understand the sentiment but I agree with 2, 3 and 6 of this article, the rest is, imho, actually dangerous in many non startup cases.

Example; Simple is always better IF you can apply it; a lot of companies and people you work with do not do so simple. A lot of companies still have SOAP, CORBA or in house protocols and you will have to work with it. So you can shout from the rafters that simple wins; you will not get the project. That can be a decision but I do not see many people who finally got into a bank/insurer/manufacturer/... go 'well, your tech is not simple in my definition so I will look elsewhere'.

It is a nice utopia and maybe it will happen when all legacy gets phased out in 50-100 years.

<comment><USER3> Thanks for your comment! But I don't think all legacy will ever get phased out.

Today's code will become tomorrow's legacy.

<comment><USER2> I agree, I meant 50-100 yrs with a bit of wink as no-one can reason about that period of time in software obviously.

</comment></comment></comment><comment><USER4> This article makes a lot of assumptions that only hold true in a very specific set of circumstances:

\- that it's possible for the team developing the product to deploy or monitor it (example cases where it isn't: most things that aren't web based such as desktop, most things embedded into hardware that might not yet exist etc.)

\- that _if_ you can deliver continuously, customers actually _accept_ that you do. Customers may want Big Bang releases every two years and reject the idea of the software changing the slightest in between.

\- not validating a deployment for a long time before it meets customers is also only ok if the impact of mistakes is that you deploy a fix. If the next release window is a year away and/or if people are harmed by a faulty product then you probably want it painstakingly manually tested.

My point is: if you are a team developing and operating a product that is as web site/app/service and you are free to choose if and when to deploy, then most of the article is indeed good advice. But then you are also facing the simple edge case among software deployment scenarios.

<comment><USER5> > This article makes a lot of assumptions that only hold true in a very > specific set of circumstances:

Yes. The assumption that you are working on a web based service is so core to this piece that it doesn't seem any more necessary to say "this doesn't work for desktop" than it would be to say "this doesn't work without internet".

 _given_ that you are delivering software on the web, your customers are going to get changes to it and like it, because their other option is to run systems on the internet with known exploits. Customers who don't want changes host their own instance.

And if your next release is a year away and you have no way to roll back the release, but you have no manual validation - then you aren't following this advice to begin with, and you have an appallingly broken process.

<comment><USER4> Absolutely agree, my complaint wasn't that the advice was bad but that it lacks the specifier "what follows is good avice for this small subset of scenarios". I really dislike the phenomenon that "software development" without qualifier has begun to imply web app/servic development.

</comment></comment></comment><comment><USER6> Cool article, enjoy the summary of relevant knowledge that's been passed around various circles.

I do disagree with:

> Environments like staging or pre-prod are a fucking lie.

You need an environment that runs on production settings but isn't production. Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME.

There's just so many classes of bugs that are easily caught by some sort of pre-prod environment, including stupid things like "I marked this dependency as development-only but actually it's needed in production as well".

Development environments are frequently so incredibly far removed from production environments that some sort of intermediary between production is almost always so helpful to me that the extra work involved in maintaining that staging environment is well worth it.

It's not the same as production obviously, but it's a LOT closer than development.

<comment><USER7> > You need an environment that runs on production settings but isn't > production.

Why?

> Setting up an environment that ideally has read-only access to production > data has saved a huge number of bugs from reaching customers, at least IME.

That's an anecdote, not a reason. Also, just because you've done it that way doesn't mean it has to be done that way, like you asserted.

> There's just so many classes of bugs that are easily caught by some sort of > pre-prod environment

Also does not support the claim that you need a pre-prod env.

> Development environments

Whoa, there! You're sneaking yet another kind of environment into the conversation? Maybe not. This is unclear, given the many different ways that people do work.

> not the same as production obviously, but it's a LOT closer

You seem to want something like production. There is nothing more like production than production.

If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.

<comment><USER8> > If you're set up to do A/B tests or deploys with canaries or give potential > customers test accounts you're probably able to start testing in production > in a sane, contained way.

You seem to be assuming 1. some sort of large-horizontal-scale production system with multiple customers, where the impact of a failure can be minimized by minimizing the number of users exposed to new features, and where 2. there's no type of bug in the code that would potentially take down production as a whole.

What if your production system is, say, a bank's ACH reconciliation logic? A medical device? A car? The live server for a popular MMORPG? A telephone backbone switch? A television or radio broadcast station?

In these cases, your software isn't a _service_ with multiple distinct _customers_ that each make requests to it, where you can test your new code on one customer in a thousand; your software is just _running_ and _doing_ something— _one, unified_ something per instance of the system (though that process may _track_ multiple customers)—and if the code is wrong, then the whole system the software operates will fail.

How do you test software for such systems?

Usually by having a "production simulation" whose failure won't kill people or cost a million dollars in lost revenue.

<comment><USER9> Thank you for contrasting life building the latest social media platform from what many of the rest of us do.

Currently I work on systems to prepare and validate birth and death certificates for the state, counties, hospitals, et al, and this whole "throw it against the wall and see what sticks" methodology doesn't fly. Nor would it have worked when preparing and presenting investment account information 5 years ago, nor the job 10 years ago processing lawsuit and insurance claim cases and legal bills. Nor any place that I at least have ever worked in the last 30 years.

<comment><USER10> Agree, and I work in the latest 'social media platform' type end. We have many customers. I can assure the author of the post, when those customers pay for enterprise licensing and their system is broken with an obvious bug. The 'we didn't do any testing before hand because staging is a lie' doesn't actually fair well as a valid excuse for anything. In fact, you just look like an unprepared and immature muppet.

</comment></comment></comment></comment></comment><comment><USER11> Some good points but some controversial ones.

I think a manual QA team is very valuable. Sure the tests pass but what if the UI is confusing or disorienting. QA can be user advocates in a way a unit test can't be. I work in games so maybe it's just a squishier design philosophy but you can't unit test fun.

I also don't understand the worry about other environments. If you're automating deployments how is another environment added work? Shouldn't it be just as easy to deploy to?

<comment><USER12> I think the valuable purpose you are describing of QA is better achieved by having a UX team earlier in the pipeline.

<comment><USER11> I do support constant deployments to the QA environment (also a no-no apparently). That can keep the QA team involved at all times. I wouldn't suggest waiting on large changes before having QA do a pass.

<comment><USER13> Literally constant? As in whilst attempting to replicate a bug the software could change out from underneath the tester? Would that complicate things or am I misunderstanding something about the process?

<comment><USER11> Hmm you're right. We CI to a dev environment that QA pulls in at their discretion.

<comment><USER14> Do you mean you CD to a dev environment?

<comment><USER11> yes

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> I think I disagree with this about 100%. Sure, production is what it is all about in the end. But how do you know the letters you just typed are going to be any good in production? They might just crash and burn there. That is why we need all those quality gates. The sooner and the farther removed from production that you discover a problem the easier it is to fix.

Regarding the 'buy vs build' I think buying software is one of the most risky things that you can do. Since it cost money you cannot then say 'o well, i guess it just did not pan out, let us just not use it'. Now you are kind of married to the software. And some of the worst software out there is paid for. E.g., jira vs. redmine. This is actually a bit ironic considering the fact that I actually am writing software in my job that is sold.... O well, it actually is sold as a part of a piece of hardware, so it is not really sofware as such.....

Regarding the last point, failure can be made uncommon if a relatively safe route to production is available, starting with a language that verifies the correct use of types, automated tests that verify the correctness of code, a testing environment that one attempts to keep close to what production is like and so on. Getting a call that production is not working is the event that I am trying to prevent by all means possible, and I think research would be able to show that people who get fewer calls, not just because production is failing, but in general, fewer calls regarding whatever subject, will live longer and happier.

<comment><USER16> > Regarding the 'buy vs build' I think buying software is one of the most > risky things that you can do. Since it cost money you cannot then say 'o > well, i guess it just did not pan out, let us just not use it'. Now you are > kind of married to the software.

It is usually _way_ more costly and risky to develop your own. It's many hours spent on what is a separate product to your actual product, and you're way more married to it: you've just spent money, time and energy developping a custom homegrown solution. What are the chances you'll go "o well, i guess it just did not pan out, let us just not use it"? Very, very low

So you end up spending more money and a significant amount of time/energy for a product that's probably subpar because there's no reason you'd do better than companies that are focused on this product.

I think buying software is one of the _least_ risky things you can do, you know exactly how much money you have at risk and you usually know pretty well what you're buying. You don't know how much money/time/energy it will take to make your own solution, and you don't know what result you'll get.

<comment><USER10> Regarding your last point. You weigh buying software over building it when you know how much it costs to buy and maintain, and have a strong grasp on how much time money and energy it costs to build it yourself. That is how you make an informed judgement. Sure there is risk, but if your burning 15K a year on a build server and you can build it yourself for 5k and run it for 1k a year then math doesn't lie about what choice you _should_ make.

</comment></comment></comment><comment><USER17> He lost me at "non production environments are bullshit".

In dev you can break almost anything, no biggie. In stage if you break something, great just don't deploy it to prod. If you break something in prod, well ... you may end up going below SLA and may legit lose money and your customers trust.

Don't YOLO into prod. Build reliable shit.

</comment><comment><USER18> Non-production environments are useful for more than testing application code. Changing underlying infrastructure (Upgrading a database, networking shenanigans, messing around with ELB or Nginx settings) requires testing too. Having the same traffic / data shape in pre-prod is not as important.

</comment><comment><USER19> Radical! Some counter-points, though.

\- Infrastructure as code and schemas as code make it easier to keep environmental parity, because everything can be rolled back/forwards/reset with easy source control and CD operations. Visual environment diffing and drift detection can make this even easier.

\- Make your stage and prod into a blue-green situation, where if stage is ready to go, you flip users onto it. I can guarantee your stage and prod will both be respected as prod then. Failing that, just add load/stress tests to stage to make it more prod-like.

\- Non-prod environments and attention are not necessarily debt, but they are expensive insurance premiums. You should only pay those premiums if you need the insurance. It's about risk management.

\- As time passes, the people who wrote a specific part of a system don't know it anymore, so having them babysit 'their' code in production has diminishing returns. On the other hand, having a systems quality team who have a broad mandate to bugfix, put in preventative measure, reduce technical debt, improve observibility and establish good patterns for developers to do these things, can enabled these things to actually happen, when just telling devs who are busy making features that they should happen often doesn't make them happen. Also there are devs who enjoy creating new things, and others who love trouble-shooting and metrics.

</comment><comment><USER20> What exactly is would be the disadvantage of running something in stating environment before running it "for real" in production? I'm assuming the staging environment is an exact clone of production (except reduced size: fewer app servers + smaller DB instance)?

I understand the deploy-often-and-rollback-if-there-is-a-problem strategy, but certain things like DB migrations and config changes are difficult to rollback, so doing a dry run in a staging environment seems like a good thing...

</comment><comment><USER21> It's funny in my career I've observed similar development styles. But I always just thought of this as great/good developers verses average/mediocre developers. the A+ coders would always make their code very easy to access, deploy from a user standpoint, debug, read etc. The mediocre guys would wait for someone else to hit a landmine beefore fixing something that was obviously wrong.

</comment><comment><USER22> While I do not agree with everything presented in this article (especially item #2), I definitely share the overall sentiment.

For some of our customers, we operate 2 environments which are both effectively production. The only real difference between these is the users who have access. Normal production allows all expected users. "Pre" production allows only 2-3 specific users who understand the intent of this environment and the potential damage they might cause. In these ideal cases, we go: local development -> internal QA -> pre production -> production actual. These customers do not actually have a dedicated testing or staging environment. Everyone loves this process who has seen it in action. The level of confidence in an update going from pre production to production is pretty much absolute at this point.

The amount of frustration this has eliminated is staggering. At least in cases where we were allowed by our customers the ability to practice it. For many there is still that ancient fear that if we haven't tested for a few hours in staging that the world will end. For others, weeks of bullshit ceremony can be summarily dismissed in favor of actually meeting the business needs directly and with courage. Hiding in staging is ultimately cowardice. You don't want to deal with the bugs you know will be found in production, so you keep it there as long as possible. And then, when it does finally go to production, it's inevitably a complete shitshow because you've been making months worth of changes built upon layers of assumptions that have never been validated against reality.

This all said, there are definitely specific ecosystems in which the traditional model of test/staging/prod works out well, but I find these to be rare in practice. Most of the time, production is hooked up to real-world consequences that can never be fully replicated in a staging or test environment. We've built some incredibly elaborate simulators and still cannot 100% prove that code passing on these will succeed in production against the real deal.

<comment><USER23> I've worked with a customer who also had a post-production environment. They used it for the sole purpose of being able to replicate problems and do root- cause analysis in case things went horribly wrong. Then they took a snapshot of prod, synced it to post-prod, hotfixed prod as fast as possible, and then did their detailed analysis in post-prod.

This wasn't cheap; they payed Oracle somewhere between 50k€ and 200k€ a year just for the database for this environment, but they considered it worth it. (They were also in a pretty tightly regulated vertical).

My main takeaway is that I don't think there is a one-size-fits-all answer to the question of how many and what environments you need. IME having at least one "buffer" between dev and prod is a good thing, but I'm not sure to which extend my experience generalizes.

</comment></comment><comment><USER24> I agree with most of the points, but I have serious caveats on the first two.

1\. No, the engineers should not by default be on call; the owners of the product are the first call line. If they're not engineers or if they're engineers but don't have enough time to deal with all incidents–in short, if they need to delegate–they better be willing to pay _very_ generously for the extra hours of on call duty.

2\. No, hosted is not better than open source, both for philosophical and operational reasons: mostly, you become subject to the whims of the provider. A good compromise is _hosted open source_ solutions, which at least takes you half way to a migration, if the need for one comes up.

That aside, I very much agree on everything else.

</comment><comment><USER25> I agree with most of this but the point about QA gating deploys should be amended. A 5 minute integration test on a pre-flight box in the production environment by the deploying engineer is a form of QA, and can catch a lot of issues. It shouldn't be considered anti pattern. Manually verifying critical paths in production before putting them live is about the best thing you can do to ensure no push results in catastrophic breakage.

Without such a preflight box, or automated incremental rollouts, you are kind of doing a Hail Mary, since you are exposing all users immediately to a system that has not been verified in production before going live.

</comment><comment><USER26> I agree with most everything said in the article but with a big condition. If I as an engineer am responsible for everything the author says I should be responsible for then I want total control of the tech stack and runtime environment.

</comment><comment><USER27> Why would you ever share the medium version with all the added crud when the author has a version of the post on his personal blog?

</comment><comment><USER28> I'm sorry but I'm tired of Medium paywall to the point I don't want to read anything there.

<comment><USER29> Also published at: [https://paulosman.me/2019/12/30/production-oriented- developm...](https://paulosman.me/2019/12/30/production-oriented- development.html)

</comment></comment><comment><USER30> this is another Agile or DevOps

</comment><comment><USER31> > 2\. Buy Almost Always Beats Build

Strongly disagree with that, well maybe it is a good idea when you are over founded by VC where cost of money is equal to zero and you don't want to master what you are working on but in all other cases this is wrong, you shouldn't rebuild everything from scratch but creating a company is not the same as playing with LEGO

And this is the same argument as saying you should have everything in AWS because if you self host you will have to hire devops engineer

<comment><USER32> Could you expand more on why you disagree with this? Do you believe the opposite - that "Build Almost Always Beats Buy"?

I've made the build-vs-buy decision many times in my career. I don't necessarily regret /all/ of those times, but the general lesson I've learned time and time again is that you're going to end up investing WAY too much time maintaining your special version of X when you should have spent that time solving problems unique to your business model.</comment></comment>


"The Zen Programmer" is available - <AUTHOR> http://www.grobmeier.de/zen-programmer-published-06112013.html#.Unzs1fkqHnF

<comment><USER2> From the site:

 _DRM?

We trust our readers and therefore do not haven any copyright protection mechanisms installed. Read the book where you want. If you have reasons to give the book to a friend, do that. You would do this with a printed book too. Do the right thing._

I'm not a rabid anti-DRM guy, but this is so...sweet it makes me want to buy the book just to support someone with this attitude.

<comment><USER3> Actually LeanPub's terms are more or less similar (I guess this is why he also has published it there.) Thinking about purchasing it, too

</comment></comment><comment><USER4> I thoroughly enjoyed reading this book!

I followed my intuition and bought this book a few days ago when Christian Grobmeier first released it. I had no knowledge of his history, but there was "something" about how he presented it that influenced me to buy it immediately.

This book has a lot of great little stories to think about. The author weaves the concepts of Zen and Buddhism into practical thoughts for those who lead a life that involves professional computer programming -- work that is very mind-driven and which can consume your life if you're not careful and balanced.

The very last section of the last non-summary chapter ("Zenify Your Project") in particular made me smile. I won't spoil the surprise for anyone here though. :)

</comment><comment><USER5> I've never heard of this but the 10 rules of a Zen program are very poorly written. [http://www.grobmeier.de/the-10-rules-of-a-zen- programmer-030...](http://www.grobmeier.de/the-10-rules-of-a-zen- programmer-03022012.html#.Un0HmvlJPVU)

I found two glaring mistakes very quickly, which makes me wonder how many mistakes are in this book.

1\. Focus " You'll not become quicker, just you work multithreaded." 3\. Beginners Mind "Remember the days were you were a beginner."

<comment><USER6> The book supposedly had an editor, but there's more to editing than checking for typos. Random sampling: (about a co-worker who felt he was underpaid) "He did not need the money anyways. After taxes, he couldn't even fuel his car." Does that make sense to anyone else?

I haven't read the whole thing yet, and frankly don't know if I'll make it through. The book rambles along, and its cohesion seems to be lacking. I hesitate to ding anyone's hard work, but in retrospect I'd rather to have paid about half of the $20USD I spent on it.

<comment><USER4> The way I interpreted this part was that the extra 100€ the unhappy guy in the story wanted wouldn't have been able to fuel his car after taxes were accounted for.

</comment></comment></comment><comment><USER7> Congratulations - I know the feeling of finally, finally, finishing. It's great.

</comment><comment><USER8> I would love to get this as print version. Is there a way?

<comment><USER9> I think (please check) that LuLu will convert a .pdf file into dead trees for you.</comment></comment>


Russia: Google removes Putin critic's ads from YouTube - <AUTHOR> https://www.bbc.com/news/technology-45471519 <comment> Spartan-S63 This seems to run counter to the idea that Google wants to promote democracy. Hell or high water, if a country has stifling civil liberties, a company with _real_ principles in cultivating discourse and democracy would stick to their guns regardless of local laws.

In this case, it looks like a simple case of complying with local laws to continue to growing their bottom line. It's why we can't believe Silicon Valley when they say they want to cultivate discourse and democracy. They don't. They want to make money. When the two coincide with one another, that's great. When they don't, profit is more important than freedom. That's why we can't trust private enterprises to act in the best interests of society.

<comment><USER2> You're advocating for breaking a country's laws. What makes Russia's laws worth less than any other country's? Whether you agree or not with them, you have to respect them. They are a sovereign country and have the right to establish whatever laws they want within their territory.

<comment><USER3> > Whether you agree or not with them, you have to respect them.

A trivial comparison: would you respect laws of Nazi Germany in 1930s if Internet existed then?

<comment><USER2> I would choose not to do business with Nazi Germany. If I'm running a business and don't agree with their laws, I can choose not to do business there. If I'm choosing to do business there, I'm choosing to respect their laws. It's that simple - it's just like an individual immigrating to a country.

Doing business there, and breaking their laws, isn't right no matter what you think of them.

<comment><USER3> > Doing business there

Out of curiosity, how'd you define "doing business there" in a borderless Internet? Surely Google servers remain in the USA? Would you rely on imperfect client location determination?

<comment><USER2> The article is about Google pulling down Russian ads, paid for by Russians. It is clear that Google is doing business with people in Russia. This isn't about visitors to a website, which I agree is an entirely different situation.

</comment></comment></comment></comment></comment></comment><comment><USER4> Don't accuse me of being a Russian shill, I'm the opposite. But isn't this a local law in many countries?

"The laws prohibit political campaigning within 24 hours of an official poll."

We have a similar law in most of Canada, I think. No advertising just before Election Day. In some provinces, the blackout includes the day before an election (Ontario, for example).

Interesting that for national elections, Canada doesn't allow advertising on Election Day itself, but specifically exempts internet ads that were placed before Election Day. So if you uploaded a video to youtube before the blackout, you don't have to remove it.

The question here is whether Google also removed other Russian party ads from YouTube as well.

</comment><comment><USER5> It seems that the _Citizens United_ ruling doesn't apply in Russia.</comment>


Giving meaning to 100B analytics events a day with Kafka, Dataflow and BigQuery - <AUTHOR> https://medium.com/teads-engineering/give-meaning-to-100-billion-analytics-events-a-day-d6ba09aa8f44 <comment><USER2> I've been working with all these tools for a while now (and a bunch more along the way).

It's not the orthodox cloud-thinking, but you're often best off processing at the point of creation (or ingestion). Normalize the data as much as you can, (probably) compress[1], and send as close to the target as possible.

If you grab the data, send elsewhere, transform <repeat>... it all gets slow and expensive pretty quickly. Also a headache to manage failures.

This is especially true if you're using BigQuery. Stage your near-raw data into BigQuery and then use it's muscle as much as you can. A classic example here might be de-duplicating data. A painful prospect for many distributed systems, but pretty easy on the BigQuery side.

This is all especially true for time-series data. With BigQuery time partitions you can keep the queries fast (and the costs reasonable).

Also limits the range of technologies and languages you need to wrangle too.

1: Choosing your data format and compression approach can make a _huge_ difference.

</comment><comment><USER3> I built a similar pipeline using Kafka and ClickHouse - it's amazing how easy it is nowadays to ingest and analyze billions of events a day using standard tools.

ClickHouse can even ingest directly from Kafka (courtesy of Cloudflare - [http://github.com/vavrusa](http://github.com/vavrusa) contributed it).

<comment><USER4> Do you learn anything useful about humanity tho?

<comment><USER5> Fair comment. The title would imply there's a 'so what?' in the article, not just a 'how'.

</comment></comment></comment><comment><USER6> For anyone who wants to move this much data out of AWS and into another cloud provider, Kinesis Streams does not charge for bandwidth out. If you do the math, it works out to a very large savings over doing public internet data transfers if you are moving terabytes per day. The compute costs to copy data from Kafka to Kinesis should be minimal since you're essentially just operating a pipe and not doing much actual compute, and this can operate easily on spot instances.

<comment><USER7> Can you elaborate on that? I haven't been able to conclude that from what it says on the pricing page [1]:

> Data transfer is free. AWS does not charge for data transfer from your data > producers to Amazon Kinesis Data Streams, or from Amazon Kinesis Data > Streams to your Amazon Kinesis Applications.

That sounds like it's sonly free to Amazon Kinesis Applications (== inside AWS).

And on [2] it says:

> If you use Amazon EC2 for running your Amazon Kinesis Applications, you will > be charged for Amazon EC2 resources in addition to Amazon Kinesis Data > Streams costs.

So that sounds like you will eventually pay the normal egress cost of EC2.

[1]: [https://aws.amazon.com/kinesis/data- streams/pricing/](https://aws.amazon.com/kinesis/data-streams/pricing/) [2]: [https://aws.amazon.com/kinesis/data- streams/faqs/](https://aws.amazon.com/kinesis/data-streams/faqs/)

<comment><USER6> "If you use Amazon EC2 for running your Amazon Kinesis Applications" is the key there. You don't have to use Amazon EC2.

Amazon Kinesis Application does not imply EC2. They use on-prem examples in a few Re:Invent presentations. It is just an HTTP API like most everything else within Amazon. You will see other services spell out their bandwidth out charges much more clearly.

</comment></comment></comment><comment><USER8> "In digital advertising, ..." Stopped reading right there. Maybe a great article though. I started working on that thing called BigData not that long time ago but now realized that 50% of the job is advertisement, not fan of it at all, I want to like the end product or at least be neutral about it.

<comment><USER9> Could you elaborate? There's plenty of big data tasks and roles that aren't even closely related to marketing.

<comment><USER10> Not my experince. I'd say the majority of big data is unethical both in capture and use trying to sell you something or sell someone else something on you, a small amount is for general good.

I wasn't involved on the project but a major Telecoms company i was at was prototyping scanning for mobile devices on their provided home routers. If they didn't recognise a mibile as being on their network they would then target you with mobile deals. Devs saw the issues with this, essentially scanning your home network from their routers and sending the data back to be processed. Managers found Apple's mac address rotation annoying and lost the point why Apple do it.

Unfortunately besides quitting the job devs on the project don't have a say as they don't have the power or say in the final decisions at these companies. It's easy to say you would quit but it's harder when you have a mortgage, wife and kids for example.

<comment><USER11> The vast majority of big data is used today to provide everything you depend on, like email, healthcare, banking, transportation, government services, etc.

</comment></comment></comment></comment><comment><USER12> Very interesting read, but as more of an analyst, I kept waiting for the 'meaning.' Maybe I missed it, but for anyone else who may be considering reading this, it is more about 'giving business structure' to analytic events.

Good article none the less.

</comment><comment><USER13> You 'd be surprised how good modern hardware is. We are being hit with between 1.5 and 2 million requests per minute (steady traffic no spikes except increased usage in the weekends), and our analytics solution runs on just 2 main servers and costs $3k per month (total associated costs except human labour which is 1 engineer working on it part time now).

We talked with a famous analytics company and they gave us a quote of 1 million yearly to work with us. So how did we get it down from 1m to 35k?

We pretty much do the thing the article suggests (roll-ups section). We compute data hourly. Alongside with the hourly computation we also dump some extra data that can be used to compute numbers for the day (eg unique user ids, from unique user ids per hour), then from the day we get per week and then per month. We also have less moving pieces (our stack is way more traditional), and we manage our own hardware (key for keeping costs down)

When you get data back from the system you only hit the pre-computed cache, no query touches the main system from the dashboards. We only allow queries running in a 30 minute window to run on the live system - to ensure that no crazy load is going to be built on top of it and we use that to mostly catch anomalies on the real time data. (our parsing time is good too, between 10 seconds and 1 minute compared to the 2-30 minutes the article gives).

However this is the "You are alive but you 're not living" angst of analytics. All the data is there, but you cannot freely prod it for answers and patterns. If you want to get an answers about past data, you need to go through an overly complex process of raising a new cluster and ingesting old backups, multiple times, then waiting for a few days. It get's relatively expensive, slow and at times will demoralize you and make you back off from getting the answers you need. You could try keeping a smaller cluster that only gets % percent of the data (eg only 2%) for finding trends, drawing heatmaps etc and that one can run in realtime but your CEO will say that's a stupid idea to your face and it's realtime-all or nothing.

You might say that's a situation you can live with provided the absolutely insane cost savings, but when the company goes at a nicer retreat for only a selective elite few that easily costs 20k, or runs an over the top kitsch open party party/recruiting that costs 80k, and you are being dragged into a meeting on a monday morning and confronted; Why did the "3k per month-3 billion requests per day" system cost 6k this month? (because we had multiple clusters in parallel computing historical data for the past 6 months that you asked for). You just get bitter you didn't give the analytics company the 1 million they asked for and be done with it.

<comment><USER14> If you're pre-calculating the metrics and dropping the raw data from your systems, you can't actually get the benefits of the ad-hoc systems. You can't ask a new question to your existing analytics data and you need to do custom development every time you need to see a new metric.

</comment></comment><comment><USER15> Here is an example Pipeline which supports loading data from an unbounded source to BigQuery in batches using load jobs (evading BigQuery's Streaming Insert cost)

See: [https://zero-master.github.io/posts/pub-sub-bigquery- beam/](https://zero-master.github.io/posts/pub-sub-bigquery-beam/)

<comment><USER16> Are you the author? OT but I'm amazed the author charged merely 100 euro for implementing that solution for the subject startup, even if they're cash- strapped. I'm not familiar with BigQuery, but I'm curious what a normal rate for solving that issue would look like.

<comment><USER17> That's shockingly cheap for the value.

</comment></comment></comment><comment><USER18> > Giving meaning to ...

Ingesting such amounts of data is a challenge indeed. But problems will become much more complicated if it is necessary to perform complex analysis during data ingestion. Such analysis (not simply event pre-processing) can arise because of the following reasons:

* It is physically not possible to store this amount of events. For example, assume you collect them from devices and sensors

* It is necessary to make faster decisions, e.g., in mission critical applications

* It can be more efficient to do some analytics before storing data (as opposed to first storing data persistently and then loading it again for analysis)

Such analysis can be done by conventional tools like Spark Streaming (micro batch processing) or Kafka Streams (works only with Kafka). One novel approach is implemented in Bistro Streams [0] (I am an author). It is intended for general-purpose data processing including both batch and stream analytics but it radically differs from MapReduce, SQL and other set-oriented data processing frameworks. It represents data via _functions_ and processes data via _column operations_ rather than having only set operations.

[0] Bistro: [https://github.com/asavinov/bistro](https://github.com/asavinov/bistro)

</comment><comment><USER14> Sounds like they over-engineered the solution. If you have ad-hoc use-case, BigQuery is great but it's quite expensive. If you just need to pre-calculate the metrics using SQL, Athena / Prestodb / Clickhouse / Redshift Spectrum might be much easier and cost-efficient.

<comment><USER19> BigQuery PM here. I'd love to genuinely understand why you have that impression.

BigQuery's on-demand model charges you EXACTLY for what you consume. Meaning, your resource efficiency is 100% [0].

By contrast, typical "cluster pricing" technologies require you to pay for 100% of your cluster uptime. In private data centers, it's difficult to get above 30% average efficiency.

BigQuery also takes care of all software, security, and hardware maintenance, including reprocessing data in our storage system for maximum performance and scaling your BigQuery "cluster" for you.[1]

BigQuery has a perpetual free tier of 10GB of data stored and 1TB of data processed per month.

Finally, BigQuery is the only technology we're aware of whose logical storage system doesn't charge you for loads - meaning we don't compromise your query capacity, nor do we bill you for loads.

[0] [https://cloud.google.com/blog/big- data/2016/02/visualizing-t...](https://cloud.google.com/blog/big- data/2016/02/visualizing-the-mechanics-of-on-demand-pricing-in-big-data- technologies)

[1] [https://cloud.google.com/blog/big-data/2016/08/google- bigque...](https://cloud.google.com/blog/big-data/2016/08/google-bigquery- continues-to-define-what-it-means-to-be-fully-managed)

<comment><USER20> The answer comes down to one line in your BigQuery Under the Hood article[0]: "The answer is very simple — BigQuery has this much hardware (and much much more) available to devote to your queries for seconds at a time. BigQuery is powered by multiple data centers, each with hundreds of thousands of cores, dozens of petabytes in storage capacity, and terabytes in networking bandwidth. The numbers above — 300 disks, 3000 cores, and 300 Gigabits of switching capacity — are small. Google can give you that much horsepower for 30 seconds because it has orders of magnitude more."

Most companies are fine letting their warehouse go underutilized, or for queries to not be solved with such enormous resources, if it means capping their monthly data warehouse bill at a fixed number, say $9,000.

BigQuery is an awesome piece of technology, but most publishers, ecommerce, and saas companies have teams of anlayts, engineers, and business folks pounding away at their warehouse all day. And it's fine if those queries aren't as fast BigQuery.

I run an analytics companies and we load billions of events into our customers' warehouses each day. Many have evaluated BigQuery and they all came back with the same answer: too expensive. Most of them are big companies, but spending nowhere near the $40K they'd have to to cap their cost on BigQuery. And with the advent of Spectrum, they're even less likely to jump ship now.

Since you're a PM, I'd be really interested to know if you guys are aware of this issue and if you're doing anything to offer a solution that competes with Redshift (fixed cost/resource). I ask this as someone who runs a ton of stuff on GCP, but we've just never found a way to make BigQuery cost effective for us.

[0][https://cloud.google.com/blog/big-data/2016/01/bigquery- unde...](https://cloud.google.com/blog/big-data/2016/01/bigquery-under-the- hood)

<comment><USER19> Genuinely thanks for the info, this is super insightful.

I'll argue that BigQuery's per-query pricing charges you JUST for the resources you use (well, data scaned), so it SHOULD be far less expensive than a model that charges you for the luxury of having a cluster sit idle (and often at only 30% utilization), correct?

Can you help me unpack this further? I think pay-per-query is ultra-efficient, but difficult to predict. However, buy-a-cluster is easy to predict but inefficient. Do you think that difficulty in planning for BigQuery spend translates into perception of being too expensive (and potentially unbounded spend?), or do you think BigQuery's pay-per-query is indeed too expensive?

Most existing BigQuery customers, even at large scale, do have the option to go from pay-per-query to flat rate and back, and choose to stay on on-demand because.. well.. its much much more efficient :)

For example, if Netflix charged you a penny per minute of watchtime, you'd have no idea if it's more expensive or less expensive, but you'd be assured it's more efficient.

Feel free to ping me offline as well.

<comment><USER20> > ...BigQuery's per-query pricing charges you JUST for the resources you use > (well, data scaned), so it SHOULD be far less expensive than a model that > charges you for the luxury of having a cluster sit idle (and often at only > 30% utilization), correct?

> For example, if Netflix charged you a penny per minute of watchtime, you'd > have no idea if it's more expensive or less expensive, but you'd be assured > it's more efficient.

The Netflix analogy is actually quite good. Using that, let's say that I have a family of six, I'm billed $0.01 per minute of viewership, but I'm getting the best/quality speed. Each person watches 40 minutes a day, for all 30 days, for a grand total of $72. Far greater than the fixed cost of $10.99 (with, say, significantly less quality, speed, and minutes per month of viewership).

In the real world, the family of six is my team of data analysts, scientists, and BI folks who are querying my database from 9-5 every weekday.

A customer of mine, a large NYC publisher, who you have heard of and probably read, evaluated BigQuery in 2017. They loaded BQ with the exact same data as their Redshift cluster, pointed their Looker instance at it, and in just one day blew through 1/4 their typical Redshift budget. All the queries were faster. Way faster than they needed to be, actually.

Going back to my original post above, the issue here is that just because Google CAN throw these massive amounts of resources at my problem, doesn't mean I can afford to use that level computation for each query, or would even want to. In the Netflix example, I'm happy if Sally and Billy get less quality or limited time watching Netflix, as long as my bill stays at $10.99.

For most companies, their Redshift cluster is optimized to be able to handle their peak workload WELL ENOUGH. That means that queries won't be as fast as BigQuery - and that's totally fine. And it means that the cluster will be underutilized for large portions of the night and weekends - again, totally fine. They just need their usage capped at a predetermined cost, and have their queries finishing in a reasonable amount of time.

I've posed this Google employees before and I'm hit with "well, you can limit how much each person can query a day." Except that isn't a acceptable solution. I can't have analysts sitting around unable to query their database because they've exceeded their daily limit. They'd rather just fire off their Redshift query and if it takes a little bit longer, so be it.

<comment><USER11> To repeat my other comments, the issue is that BQ doesn't actually charge for compute but for data scanned. You already pay for storage (uncompressed but that's a separate issue) so paying again on that metric doesn't make sense for querying since you're using CPU cores, which is a time-based metric. If I use 1000 cores for 10 seconds vs 10 minutes, it makes sense to pay exactly that regardless of how much data was scanned.

We've run into the same issue where being curious with BigQuery actually becomes problematic as users are perfectly fine waiting an extra minute to scan 10TBs, but are afraid of the $50 bill that comes with that, especially for every little query that might be mistyped or out of their hands when using BI tools that run queries of their own.

</comment></comment></comment></comment></comment></comment><comment><USER21> I see titles like this and my first thought is "more people need to watch Real Genius or watch it again."

"What do you think a phase conjugate tracking system is for, Kent?"

Great. You made a system to track a billion people a day. You're murdering privacy and then bragging about it. And bragging about it during a giant shitstorm caused by Facebook. The fuck is wrong with you?

<comment><USER22> I want to know how people use my website, fuck me right?

<comment><USER21> A hundred billion events a day isn't "knowing how people use your website". It's mass scale surveillance. It's stalking your customers.

<comment><USER11> Is there some magic scale threshold?

<comment><USER21> Are you asking permission for misbehaving by trying to determine exactly how much is too much?

Is there some magic scale threshold for excessive force? No. There is no fine line because you _shouldn 't be anywhere near the line_.

100 billion. It's not 'lots' or millions, or a billion we're talking about here. It's one hundred billion. That's 80 events per day on average for every human they've ever seen (1.2 billion, in the about section). If they see half of those on any given day that's 160 events per person, maybe 200. Per day.

Fine grained tracking you do on your own site to determine why people leave and whether they see your new content? I could see my boss asking for that. I wouldn't be enthusiastic. I might make excuses that I was too busy doing other things to help. I might even complain, but we probably aren't running that forever anyway because it tells you less and less over time but still costs the same.

But this is an ad network, not a usability study. We are currently busy handwringing about ad networks, and I'm a little taken aback by the dissonance here.

<comment><USER11> Nice spin, but you made the statement as if there's a difference so yes, I'm asking you to define the limit. You can't say "anywhere near the line" and not be able to tell me what the line is. Unless you have some understanding for what is reasonable for the business metrics, it's rather useless for any further discussion.

Those ads do have owners, paying a lot of money to both distribute and see how people are interacting with their content so it's the same thing as a website. And there are lots of tracking events to capture so it's easy to add up to billions, but as the article states it's not all user events. Only 10B come from users with everything else being backend server logs.

> I wouldn't be enthusiastic. I might make excuses that I was too busy doing > other things to help

Ok... so being dishonest to avoid doing your job is fine? If you are against then don't do it, but what is the point you're trying to make here?

<comment><USER21> Whatever let's you sleep at night, man.

        Ok... so being dishonest to avoid doing your job is fine?

You'd be a hoot as a guest speaker in an ethics class.

<comment><USER11> You haven't answered any of the questions or provided any reasonable details.

What does ethics class have to do with this? I'm sure the question of whether lying in your job to avoid doing a particular project would be far more interesting to study anyway, I'll be sure to bring it up the next time I teach one.</comment></comment></comment></comment></comment></comment></comment></comment>


Two Hundred and Fifty Things an Architect Should Know - <AUTHOR> https://www.readingdesign.org/250-things <comment><USER2> What's the opposite of click-bait?

<comment><AUTHOR> As in something that doesn't sound particularly interesting but actually is?</comment></comment>


How well do you trust your code? - <AUTHOR> http://catless.ncl.ac.uk/Risks/17.06.html#subj7

<comment><AUTHOR> Submitted after reading this: <http://news.ycombinator.com/item?id=959165>

in which I also submitted it as a comment.

The Risks digest is worth reading in its own right.</comment>