Make it less annoying every day - <AUTHOR> http://thestartuptoolkit.com/blog/Make_it_less_annoying_every_day/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheStartupToolkit+%28The+Startup+Toolkit%29 <comment><USER2> Excellent post. It applies to any area of creativity and making things.

I like the way he put a link to this comment section on his blog item as well. Smart.

</comment><comment><USER3> Wonderful. Just pure,distilled fact of life.</comment>


50M Facebook profiles harvested for Cambridge Analytica in major data breach - <AUTHOR> https://www.theguardian.com/news/2018/mar/17/cambridge-analytica-facebook-influence-us-election <comment><USER2> My problem with this 'outing' of CA is that Facebook explicitly commercially exists to harvest user data for Procter & Gamble, Johnson & Johnson, Fidelity etc etc so they can profile us. A million dollars is chump change in the crazy US election game. This all seems overly selective - it's ok for some people to profile but not for others. I'm not in favor of any of it to be clear but there is a definite political bias going on here. Let's not forget FB itself has a formal political unit that exists to push propaganda in foreign elections, 'stifling opposition and stoking extremism'

[https://www.bloomberg.com/news/features/2017-12-21/inside- th...](https://www.bloomberg.com/news/features/2017-12-21/inside-the-facebook- team-helping-regimes-that-reach-out-and-crack-down)

<comment><USER3> I could see a constitutional amendment barring psycho-graphic profiling from election advertising and intent. Not sure how it could be enforced because this is very technical.

Even so, there is a significant difference between Coke trying to sell me a flavored soft drink and a firm tweaking my emotions to get me to abstain from voting with false information or to vote against my best interests with false information.

There are definitely people who are susceptible to psycho-graphic warfare and we need to protect them in order to protect our democracy.

<comment><USER4> I don't think "false information" is the kicker here. Taking a cursory glance at Breitbart and Fox News, there isn't that much that is patently false. Furthermore, my observations of how propaganda works in general is that it doesn't usually rely information that is technically false. It relies on selective distribution of information, editorial spin, and other more subtle methods.

This makes "enforcement" basically impossible because you can't have a news outlet without editorial decisions. A better way to go about it would be to try incentivize media outlets to make a good faith effort at "both sides" journalism, but members of both political sides have been attacking the media for doing just that since the 2016 election season.

<comment><USER5> > Taking a cursory glance at Breitbart and Fox News, there isn't that much > that is patently false.

Don't you think it would be better for you to take more than a cursory glance before commenting ?

Breitbart has had significant issues with fake news over its lifetime. Far more egregious and consistent than other any other news organisation not that I would necessarily call it one.

[https://en.wikipedia.org/wiki/Breitbart_News#Notable_stories](https://en.wikipedia.org/wiki/Breitbart_News#Notable_stories)

<comment><USER6> In addition to Fox and Breitbart, the Washington Post, Buzzfeed, CNN, and even the New York Times have had to retract dozens of viral political articles over the last two years. The retractions were only seen by a fraction of the people who saw the original headlines. This is an inevitable result of journalism increasingly relying on rumor and catering to partisan audiences.

Where is this line that should be drawn, and who should draw it? Breitbart may have a higher frequency of "oops someone gave me bad info, or my reporter/editor made an unsubstantiated inference" events, but their audience is much smaller than the large outlets.

</comment></comment></comment></comment></comment><comment><USER7> I remember when the Obama campaign hired data scientists and used targeted social networking tools to pursuade voters who were on the fence and it was heralded as brilliant and the future of politics.

I worked for a company crawling Facebook data by creating viral apps the year the original API came out. By now I am sure this is done by many companies.

Why is any of this news? My understanding is that companies harvesting social networking data via viral apps and then reselling it to perform targeted voter advertising is literally a 10 year old concept. Were any laws broken here? Were there any techniques used here that were novel or done by one political party and not the other? Why are we talking about this one firm and not the many others that surely exist that are trying to do the same thing for <insert political candidate of choice>

<comment><USER8> Did you not read the article? Millions of users took a personality test that had nothing to do with politics, and that data was sold to the Trump campaign for targeting.

When did Obama's campaign ever do that?

<comment><USER9> 270.000 users.

<comment><USER10> ... and all of their 'friends', totaling over 50 million

<comment><USER9> Nope. It was not technical possible to share your friends' data in 2014.

</comment></comment></comment></comment></comment><comment><USER11> I used to make fb apps, any app gets full access to fb's user graph as long as they request the relevant permissions.

Users don't comprehend what permissions they are giving to apps they run. A quiz site getting full access is not surprising.

Once an app has any amount of access the only thing stopping them from harvesting their own clone of your data is an agreement in the ToS that you won't store PII for more than x hours.

These rules are like the bare minimum to stop good actors. If you're a bad actor fb does not do a single thing to protect users from you. As evident in this report fb is also not above blaming the users for the hostile environment fb created and placed them in.

There must be countless copies of harvested fb data out there. My employer at the time once realized we were accidentally storing some PII permanently in a derived field. If good actors can't even keep above the law what do you think the ecosystem looks like in the shadows?

IMO we aren't having the right conversation with fb over how they mistreat our PII and we should loosen the definition of that term when companies like the one in the article can infer our political preferences from the innocuous bits of our lives we tag on facebook.

We should be asking why even an authorized API that can't stop you from copying the data doesn't count as a systemetized data breach.

<comment><USER12> > We should be asking why even an authorized API that can't stop you from > copying the data doesn't count as a systemetized data breach.

Is your argument that no company should offer any developer APIs at all? It's impossible to stop apps from storing data that they have access to, given malicious intent.

This is like saying that the existence of the Google Calendar API is a "systemetized data breach" because an app could copy data from it once authorized by a user.

<comment><USER13> It's one thing to use the friend graph to show in your app who else uses it. That's pretty legit. The other use case is to store it to some database and keep it there.

FB provides since ~10 years widgets for showing who else is liking xy. I know these Social Widgets are not so customizable and thus not pretty enough to match some custom design but at least they provide some safety nets.

Maybe Facebook could just provide more Social Widgets/CSS customizability instead of letting people write their own "Facebook Extensions".

</comment></comment></comment><comment><USER14> I was curious how the figure leaped from the 270k cited in the Facebook press release to this 50M figure.

It sounds like they never had full access to the Facebook profiles beyond the 270k who installed the app, but just harvested the friend lists of those 270k. This doesn't give the app developer full access to the friends' profile data, but I guess once you have the network of friend connections you can use other public data sources to fill in or infer the gaps. And of course some of those 50M will have FB profiles that are fully public open books ready for anyone to harvest.

I will say as someone who has developed Facebook apps, the whole ecosystem is pretty much on the honor system for protecting user data. There are some seemingly random and capricious (and often erroneous) abuse detection algorithms, but once an app has access to user data who knows what they do with it and whether it was kept secure -- surely Facebook has no idea unless they perform invasive manual physical audits.

<comment><USER15> You could get access to the full friends' user profile data in Graph API earlier than v2.0. If you had 500 friends, and granted friends_* OAuth permissions to an app, the app had access to 501 user profiles.

<comment><USER16> You know where you read this?

<comment><USER17> At $dayjob-1 we relied on this to pull in your Facebook friends as contacts. Eventually FB limited the scope of this to friends who also had the app.

<comment><USER15> This was post-v2.0 afaik.

</comment></comment></comment></comment></comment><comment><USER18> Minor point of confusion -- this article refers multiple times to a "data breach". ( _"...one of the largest-ever breaches of Facebook data...", "At the time of the data breach...", "...first reported the breach..."_)

As far as I can tell, there is no data breach, right? It sounds like CA got facebook data through an app they wrote, thisisyourdigitallife, which did some shady things.

Also, _" The New York Times is reporting that copies of the data harvested for Cambridge Analytica could still be found online"_.

The link is: [https://www.nytimes.com/2018/03/17/us/politics/cambridge- ana...](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica- trump-campaign.html)

Anyone know what they're talking about? I haven't heard of any 50-million- profile data dump, and I really like collecting corpora...

<comment><USER19> It wasn't a breach as _we_ know it.

Basically FB gave the data away. Apps have access to the data but they're not allowed to give/sell it to third parties. In this case the rules were ignored. Probably many other companies with API access have also ignored the rules. In this case FB didn't make much of an effort at all to prevent it from happening so it's reasonable to assume the practice is rampant. There's likely many copies of large parts of FB data out there (left on laptops on trains or on unprotected FTP/HTTP servers, etc.).

It's a 'breach' from the users' perspective.

</comment></comment><comment><USER20> One thing other commenters haven't mentioned is that Facebook asked the other parties to delete the data and promise never to use it again and the other parties even certified that they had done so, but the whistleblower is alleging they lied to Facebook.

Maybe that's legally actionable.

</comment><comment><USER21> OK, this feels like it will bring about the end. Of something. Facebook? Massive use of data for political campaigns? Anything?

If we keep consuming news like this, and do nothing, it's going to scalate massively. Same way as when Snowden told people they were spyed on and they collectively shrugged and continued with their lives as if nothing had happened.

We, people in tech, have a massive moral burden to educate 'normals' on the meaning of news like this!

<comment><USER22> The problem is that regular people no longer have a place to communicate. It used to be that the workplace, church, neighborhood or union meetings were the place to socialize and discuss these issues and take collective action. Now we have nowhere to turn to. Modern nomadic culture alongside temporary jobs, low trust and personalized news all make sure that we cannot take collective action on anything.

We need to find a new way to communicate before this cancer becomes so widespread that the last bastillions are lost.

<comment><USER23> > It used to be that the workplace, church, neighborhood or union meetings > were the place to socialize and discuss these issues and take collective > action

People still socialize and discuss issues in the real world. Having a Facebook group for a church or neighborhood doesn't preclude anyone from going to church or physically interacting with their neighbors. People also still take collective action in the real world - Antifa, BLM and the Tea Party are three modern examples, but there are countless others which simply don't get media attention.

And, all else aside, social media is still perfectly adequate for enabling communication between most people.

I'm sorry, but your comment seems more rooted in hyperbole than reality.

</comment></comment></comment><comment><USER24> This wasn't a data breach, it was a misuse of data by a third party.

<comment><USER25> Every single definition I find classifies this as a data breach.

> A data breach is a security incident in which sensitive, protected or > confidential data is copied, transmitted, viewed, stolen or used by an > individual unauthorized to do so.

<comment><USER26> But they were authorized to access the data, weren't they? The problem wasn't that they accessed it, it was that they used it for things they weren't allowed to.

<comment><USER27> Everybody with an FB app key is "authorized to access [the] data," but the API TOU says you can't _save_ anything.

I'm imagining the crux of the CA issue is that they saved stuff.

</comment></comment></comment></comment><comment><USER28> I think I finally understand what the point of Facebook apps is and why they've always felt in some way dodgy. It's been clear for years that Facebook apps can get your user data, and that of your friends, and that Facebook designed them that way and were aware of that. The Guardian article even mentions that one of the apps used by GSR to gather data for Cambridge Analytica triggered Facebook security protocols trying to pull too much data.

What I didn't understand is why Facebook would grant this - maybe at some point they needed viral apps on the platform and giving user data away encouraged people to make them - but why did it still work a few years ago? But this article made it click: all you can really do to monetise or use millions of profiles of Facebook users is target them with ads, and Facebook is the only place you can target those ads effectively given Facebook user data, and the more data you have the more effective those ads are, the more you pay Facebook.

Facebook don't sell user data, they've long said that - and it's true. They sell the ability to target advertising to their users, and you can do that a whole lot better if you have their user data. So they don't sell it, they give an API for their users to freely give it away, knowing that once you've done all your analysis on it you'll conclude that you should spend money paying Facebook to actually deliver your messages to those users.

</comment><comment><USER29> > Facebook denies that the harvesting of tens of millions of profiles by GSR > and Cambridge Analytica was a data breach. It said in a statement that Kogan > "gained access to this information in a legitimate way and through the > proper channels" but "did not subsequently abide by our rules" because he > passed the information on to third parties.

This is exactly how Facebook was designed. You get a stupid quiz or photo frame in exchange for a copy of your friends list. It's always worked that way, and it's why Facebook OAuth was more popular than Google+ and other Oauth since 5+ years ago -- because app devs can make more money from Facebook OAuth since it comes with a copy of your friends list, so they prefer to integrate Facebook.

<comment><USER30> As of ~4 years ago when graph api 2.0 was released that's not true.

The /friends endpoint only returns friends of the user who have also already installed your application.

<comment><USER31> As of April 2016 wasn't it limited to 50 "close friends"? IIRC it was also possible to abuse iOS webview and the FB library by modifying some private methods and injecting some JS to get the info of more than 50 friends. I don't remember the details, but I saw it in the wild in a high growth top 10 iOS app (reverse engineered to see how they were getting so many users so quickly).

</comment></comment></comment><comment><USER32> Facebook: "no-one herds our sheep but us, mmmkay?"

</comment><comment><USER33> So... If I were in Cambridge Analytica's position, employed to influence the US election, one of the first things I'd do is match this data with any data I could find on voting patterns. Which reminds me, didn't some of the Russian APTs hack into state voter databases?

<comment><USER34> You don't need to hack into voter databases - most people register for the party they vote for and that is public information(along with their home address, phone number, and whether they voted in past elections).

<comment><USER35> You may not even need to cross-reference it, Facebook asks you what your political affiliation is and displays it on your profile (or did, at one point)

</comment></comment></comment><comment><USER36> I think it is much more important to focus on an investigation to make clear to the public how this data was used. That i think will lead into a much more interesting story. No one seems to want to go there and i don't understand why. Maybe because a lot of its clients are political parties/political individuals around the world and they do not want to be ousted for using "public opinion manipulation technology" on a wide scale.

<comment><USER37> Any investigation would be about as thorough as the Russian bots "investigation".

</comment></comment><comment><USER38> Think about all those apps where you connect your bank account via your online banking creds that have full access to everything you buy.

<comment><USER39> Do people do that?

<comment><USER40> Yes, I know Mint does it and most credit unions offer FinanceWorks. Intuit gets the data from both.

</comment></comment></comment><comment><USER41> I wonder how many of the "see what you'll look like when you're 80" and "find out how you'll die" quiz apps are doing this behind the scenes.

<comment><USER42> If I am reading the paper which started all of this correctly, you don't even need the quiz apps if you have the correct permissions:

[http://www.pnas.org/content/110/15/5802](http://www.pnas.org/content/110/15/5802)

<comment><USER41> The quiz apps are how you get the permissions.

Require the user to "connect with Facebook" to see their result. Give them the result, but quietly siphon off every bit of data you can with the access token.

</comment></comment></comment><comment><USER43> Whistleblower's account suspended.

[https://twitter.com/chrisinsilico/status/975335430043389952](https://twitter.com/chrisinsilico/status/975335430043389952)

</comment><comment><USER44> This kind of work combining propaganda and disinformation with AI models and feedback into them to get a progressive change of belief is fascinating. I think of this as the first of many wars democracy will fight against AI and we are currently loosing.

This comment is from the "Duped" article that has a different headline and more detail.

</comment><comment><USER45> For example, "Weev" got 3 years for downloading ATT user data. I wonder whether Bannon&Co would get anything ... So far it doesn't look like FB makes any push for CFAA case here. I wonder what would FB do if instead of Bannon it were a nobody like the above mentioned "weev".

</comment><comment><USER46> 50M doesn't strike much in FB scale, that's until...

      At the time, more than 50 million profiles represented around a third of active North American Facebook users, and nearly a quarter of potential US voters.

<comment><USER46> Sorry for the crappy formatting, can't edit now, so here's pprint version:

      At the time, more than 50 million profiles represented around
      a third of active North American Facebook users, and nearly
      a quarter of potential US voters.

</comment></comment><comment><USER47> Nothing new about Campaign Data companies. In fact knew of a South San Francisco company called 'Campaign Data' in the '90s that ran a SAS on DECUnix. They collected voter registrar data from counties for targeted voting campaigns. Usually for passing more restrictive laws or raising taxes. Like raise property taxes for schools; send flyers to renters with kids and send nothing to homeowners with no kids. It was always in a way, unfair and evil.

</comment><comment><USER48> Let's be realistic here. This headline is nothing but partisanship. The only reason this is exaggerated as a "data breech" is because of the connection to the Trump campaign.

The real scandal is that such data is so easily harvested and freely available.

I'd be interested in seeing how much of facebook's data repository was used in targeted political ads by all parties. Including Russian agitators who have been shown playing both sides.

<comment><USER49> There are at least 25 scandals surrounding the Trump administration currently that are each worse than the two pseudo-scandals surrounding HRC that Conservatives managed to drum up, I. e. E-Mails and "Benghazi". And ts not like FOX and the entire US Congress have less power than the Guardian.

So, no: "They are all the same" isn't just cynical and useless. It's also wrong.

<comment><USER48> So, in your opinion, this justifies the false headline of the article, and is reason to turn a blind eye to the same potential data abuse when committed by the other party?

>They are all the same" isn't just cynical and useless. It's also wrong.

Please do not put words in my mouth. All I am asking for is journalistic integrity. Media in the U.S. has proven repeatedly to be partisan, which, to a rational person, makes it very difficult to separate fact from propaganda. This article is a case in my point.

Unethical politicking is not an excuse for spread of misinformation.

</comment></comment></comment><comment><USER50> I hadn't thought of it like this before, but from a political POV everyone's vote, whether they are a dole bludger or a quantum physiscist, are worth the same. So really, to win an election .. take that as you will. Identifying these people is a very profitable area.

Interesting side note .. in Australia we assign school funding based on the highest education received or wage class of the parent (classes A, B ... E or such).

</comment><comment><USER51> 1) Facebook collects and builds a profile about you 2) Facebook allows third parties to target advertisements based on the profile 3) Advertisements are tracked 4) Browsing habits and advertisement tracking reconstructs who was targeted

</comment><comment><USER52> ITT: people who did not read the link Astrotrufing and conservative martyrs bleeding all over the site.

</comment><comment><USER53> Why bother protecting any data, if you can put a footnote in your ToS.

</comment><comment><USER54> I don't understand the use of the word "breach" in this headline.

</comment><comment><USER55> Can't wait for Sheryl Sandberg to write a new book now on garden soil or something.

<comment><USER56> Any reason to attack the one woman among the Facebook leadership and not, say, Mark Zuckerberg?

<comment><USER57> Because the others among the Facebook leadership are not going around writing and promoting books?

Also, do you genuinely not find it disconcerting that Facebook leadership go to great lengths to avoid discussing the privacy implications of their service? And the only person in that group who puts herself "out there", so to speak, is instead writing "success literature"?

<comment><USER49> Is there some specific evilness to writing books? Because I don't see how her writing books is reason to single her out for criticism.

> Also, do you genuinely not find it disconcerting that Facebook leadership go > to great lengths to avoid discussing the privacy implications of their > service? And the only person in that group who puts herself "out there", so > to speak, is instead writing "success literature"?

So you're angry because they don't talk about privacy. And you're especially angry at her because ...she doesn't talk about privacy?

That argument also doesn't make much sense when comparing her to Zuckerberg explicitly, who's at least as "out there" as she is. Didn't he go on a "50 states listening tour" last year?

</comment></comment></comment></comment><comment><USER58> This is hardly news... Facebook ads cannot target specific _users_ , they only target audience _segments_.

It's actually far easier to create ads targeted at segments with likely political beliefs, and Marketers have access to aggregate numbers of niche segments today.

There's no need to scrape people's profiles or get down to the individual level.

<comment><USER58> EDIT: As more is revealed about Cambridge Analytica, this clearly is "news" that requires more investigation.

My original comment was more in response to user vs segment level targeting.

</comment></comment><comment><USER59> China has more. They have enough that this is a drop in the bucket. While they might be as blatant and ineffective as Russia by interfering with an election, they want a low profile and to maximize capture of revenue, so they are more about making money than trying to put feces on the face of the American political process.

You people should pick your battles. It would help if you knew the battlefield first.

<comment><USER37> How about this idea: what if the Russian bots aren't actually Russian? How has no one considered this possibility?

<comment><USER5> Yes they have considered it. And they have identified they are Russian.

You really think governments wouldn't have checked this ?

<comment><USER37> I thought the public might be a bit more skeptical.

How did they identify with certainty that they are Russian? Is there any way for someone without top secret clearance to verify this?

<comment><USER60> How - the special counsel Mueller has subpoena authority. First link from indictments in list form. The one I saw on TV was that guy with Hillary in a cage in parades - he was paid by Russians to make it. I also remember reading about the pro and anti gun rallies at that Texas state park - the Russians made the Facebook page for _both_ sides in that little incident. [https://mashable.com/2018/02/16/indictment-russian-trolls- in...](https://mashable.com/2018/02/16/indictment-russian-trolls-interference- election/#SD5UuJloE5ql)

Presumably Mueller used that subpoena on Facebook and internet providers and the Russians didn't try to hide very hard and used mostly Russian ip addresses.

There's also the fact Facebook admitted it sold ads and post promotion to Russian agencies and told Congress the reach of those ads and posts. Recently Facebook revealed to all users in North America whether or not they had interacted with those ads/posts. Several news organizations have independently found the data from other sources including actual interviews with the people working in Russia.

One technique the Russians used was to impersonate Americans of some extreme view to stow discord and inflame the other side of some debate. [https://www.thedailybeast.com/exclusive-russians- impersonate...](https://www.thedailybeast.com/exclusive-russians-impersonated- real-american-muslims-to-stir-chaos-on-facebook-and-instagram)

There was also several different Facebook campaigns where they got Africans with pidgin English to pretend to be Americans and try to inflame white nationalists and latent racists fears.

<comment><USER37> I've heard all sorts of the juicy details, it makes a great story there's no doubt. What I'm looking for is some substantial, verifiable evidence to overcome my suspicions at how perfect of a story it is, as well as some of the inconsistencies.

For example:

> Presumably Mueller used that subpoena on Facebook and internet providers and > _the Russians didn 't try to hide very hard and used mostly Russian ip > addresses_.

So on one hand, article after article tells us how sophisticated these hackers are, yet these very same hackers didn't hide their ip addresses, and also openly posted links on twitter that were supportive of Russia. Doesn't something about that seem a little off to you? If it does, you'll be the first person I've encountered who think it does, everyone else is completely confident that this is an open and shut case. Yet, none of these same people can point to any specific evidence that could convince me. Sure, everyone has some articles full of juicy stories, usually containing confident statements from high ranking government officials assuring us crimes have been committed and they have proof, but I've never been able to find a person who could point me directly to any proof.

1/3 of the way through [https://www.justice.gov/file/1035477/download](https://www.justice.gov/file/1035477/download), rather than having my mind changed, my skepticism is stronger than before.

This feels a bit like some things we've experienced in the past, where the "facts" about an enemy, that we're are _assured_ are 100% completely true and verified, _trust us_ turn out to not be true several years down the road. But by then, we've already spent trillions of dollars and waged a war killing thousands of innocent people.

I'd rather not go down that path again, so I'm sorry if I can't join in the party vilifying the evil Russians, because based on the information I have so far, it seems like classic misdirection with the primary beneficiary once again being military budgets, and everyone is just a bit too enthusiastic to believe anything they're told.

EDIT: There is no shortage of internet downvotes for people like me who aren't willing to go along with this story, but there is a severe shortage of people who will put me in my place with actual content. You can probably imagine the effect this might have on the certainty _I_ feel in the correctness of my stance. Unlike others, I'm open to having my mind change, but for some reason no one can muster any effort beyond a condescending and intellectually lazy "let me google that for you".

<comment><USER60> They have some of the Russians who worked for the RIA interviewed for newspapers and admitting what they did, which kind of blew my mind. I guess one could claim those were faked, like the moon landing conspiracy theorists.

From your own example which I presume is about the Iraq war, there is nothing that an ordinary citizen can do to prove that powers in the US, UK, and Kuwaiti governments (and Iraqi agents working in favor of an attack on Iraq) falsified the evidence they presented to the public for a compelling argument for invading Iraq without relying on sources that cannot be verified by an ordinary citizen without access to the journalists/non proliferation experts/government agents responsible and nothing an ordinary citizen can do to prove US claims that Iraq had an active WMD program and mistreated Kuwaiti babies as claimed at the time (by what we know now was a state actor who lied for Kuwait) in congressional testimony after the Iraq invasion of Kuwait. Even the New York Times and that one reporter famously lied flat out about Iraqi weapons programs and abuses in the build up to the war. But there were plenty of people who were investigating the claims of Iraqi WMD who said there was nothing there and that led to a lot of people protesting against the war. I don't see any investigators for the FBI or NSA coming out and being a whistle blower and saying this Russian thing is faked, on the contrary, the only whistle blower who came out so far, actually showed the internal NSA documents that said the USA elections was under cyberattack by Russia. The most reputable guy who came out for the Iraq WMD was possibly Colin Powell. Less well know is that he was also the first US Army investigator who looked into the My Lai massacre and he found that nothing was done wrong, so he was kind of used to this type of thing by then. It took a second Army investigator to reveal what happened there.

Or more contemporaneously there is nothing a citizen can do to prove that the recent Russian spy who was released to the UK and attacked was poisoned with a Russian only sourced nerve agent. And if it was the Russian nerve agent (that the Russians offically proclaimed to have destroyed all stockpiles of), there's no way for a citizen to prove it was Russian government behind it or that some other power was responsible for usage of it. I didn't vote either way on your comment, it just seems like an impossible standard.

<comment><USER37> I honestly don't think wanting to see verifiable evidence before forming a decisive opinion on something is anything near an impossible standard, especially since as you've noted, our governments and newspapers are known to lie to citizens. Anyone who's done any reading on the topic outside of the mainstream knows what you see on TV in the west is a slanted version of the truth at best. And yet, look how _absolutely certain_ people are about their beliefs, but when asked to provide some of the hard-factual content they read to form this rock solid opinion, almost no one has anything other than a downvote. To me, this looks like some sort of a classic mania, and it's quite concerning.</comment></comment></comment></comment></comment></comment></comment></comment>


Python-Based Tools for the Space Science Community - <AUTHOR> http://spacepy.lanl.gov/ <comment><USER2> Fortunately Python is becoming the new programming language for science and engineering, replacing FORTRAN and Matlab. Unfortunately I do not see many opportunities for growth in other sectors

<comment><USER3> This is only partly true, and in a limited way. Fortran is still king for simulations or anything that needs to be fast. Other possibilities are C and assembler, and that's about it. Python + numpy has become very popular very fast for exploring data and the results of simulations, and it's a great environment for this. Note that when you use numpy you're calling fortran (and C?) routines. The fortran languages and compilers continue to evolve, and fortran will probably remain the language of choice for large-scale computation for some time. Other languages, such as Java and lisps, are sometimes used for big simulations, but Python is just too slow to be used for anything but prototyping.

<comment><USER4> This depends very strongly on the nature of the simulation. Lots of simulations are handled just perfectly by modelling your system as arrays. Consider, for example, nonlinear waves. Your simulation typically consists of FFT -> k-domain operator -> IFFT -> x-domain operator (repeat for i=0...T/dt).

No reason whatsoever not to do this in python, though of course the FFT is just fftw/fftpack and the x/k domain operators are numpy ufuncs (all written in C/Fortran).

On the other hand, for particle simulations, you need more complicated logic/data structures to handle multipole methods, so the python array model might not work so well.

<comment><USER3> Yes, it seems that there are some types of simulations that could be structured as numpy operations steered by a little Python code. But can this kind of code be run effectively on large multi-processor machines?

<comment><USER4> Array operations tend to be highly parallelizable by nature. Numpy operations can certainly be parallelized, and even distributed. Take a look at blaze.

[https://github.com/ContinuumIO/blaze](https://github.com/ContinuumIO/blaze)

<comment><USER3> That looks interesting. I know that numpy calculations should be straightforwardly parallelizable; my question, out of curiosity, was whether they were in practice.

</comment></comment></comment></comment></comment></comment><comment><USER5> Somewhat related, I just discovered PyEphem, and have been using it to pin down a time and location from the stars that appear in XKCD #1190

[http://forums.xkcd.com/viewtopic.php?f=7&t=101043&p=3395021#...](http://forums.xkcd.com/viewtopic.php?f=7&t=101043&p=3395021#p3394991)

[http://rhodesmill.org/pyephem/](http://rhodesmill.org/pyephem/)

<comment><USER6> I wonder if Randall Munroe also used it to calculate the positions of those stars? He has mentioned it in some other places.

(I did not wade into that 970 page thread to see if someone had already made a similar suggestion)

<comment><USER5> If you save down one of the images and up the gamma a little, you can see a tremendous amount of detail. Far more than could practically be done by hand, and afaict, more than PyEphem has in its catalogue of "bright stars".

I suspect he used something like Stellarium and then ran the images though some filters.

</comment></comment></comment><comment><USER7> This is one of the few (only?) python packages that can read and write files in NASA's CDF format - [http://cdf.gsfc.nasa.gov](http://cdf.gsfc.nasa.gov) Installing under OS X is fairly straightforward following the directions in the documentation.

</comment><comment><USER8> I cant think of another discipline where the first example from the time section will use a Julian Day:

      >>> x=Ticktock([2452331.0142361112, 2452332.0142361112], 'JD')

</comment><comment><USER9> The documentation mentions that "OSX install has become a challenge. With the Enthought transition to Canopy we cannot figure out clean install directions for 3rd party packages and therefore can no longer recommend using EPD for SpacePy." They have other directions including some using MacPorts.

Installation directions for Linux and Windows seem straight-forward.

<comment><USER10> Enthought going to Canopy as some weird hybrid of a distribution, and IDE and a package manager may have something to do with it.

I'm had many fewer issues with Anaconda, and have found that OS X behaves just as well as Scientific Linux or Mint for stuff like this.

</comment></comment><comment><USER11> "OSX install has become a challenge. With the Enthought transition to Canopy we cannot figure out clean install directions for 3rd party packages and therefore can no longer recommend using EPD for SpacePy."

Um... use Anaconda? [http://continuum.io/anaconda](http://continuum.io/anaconda)

<comment><USER12> The Python installation tool utilized to install different versions of Anaconda and component packages is called [conda]([http://docs.continuum.io/conda/intro.html](http://docs.continuum.io/conda/intro.html)). [pythonbrew]( [https://github.com/utahta/pythonbrew](https://github.com/utahta/pythonbrew)) in combination with [virtualenvwrapper]([http://virtualenvwrapper.readthedocs.org/en/latest/](http://virtualenvwrapper.readthedocs.org/en/latest/)) is also great.

</comment></comment><comment><AUTHOR> Can anyone clarify what does this means: 'License Agreement based on Python Software Foundation License'?

<comment><USER13> [http://lmgtfy.com/?q=Python+Software+Foundation+License](http://lmgtfy.com/?q=Python+Software+Foundation+License)

<comment><AUTHOR> haha, thanks ;)</comment></comment></comment>


Light and Heat: Giant Fresnel - <AUTHOR> http://www.brotherswhitney.com/2008/11/light-and-heat/ <comment><USER2> Hey, glad you like it! There are a couple of hypothetical solar energy possibilities, but neither's really worth the time.

The first is concentrated photovoltaic (CPV), based on the idea that lens is cheaper than silicon, so why not concentrate a larger area onto a smaller chip? The issue here is thermal- if a cell has 15% efficiency, then it necessarily rejects 85% of the energy as heat so significant cooling is required.

The second is solar thermal or steam generation. Heat water until a phase change occurs, then use the expansion to drive a turbine. It's not particularly safe or easy to do on a hobbyist basis.

As ars peripherally mentioned, the Fresnel essentially produces an energy rate. The thermo take-home is that there IS no maximum temperature, except perhaps the surface of the Sun. That makes the (incorrect) assumption that you have a perfect absorber with no convective or radiative losses. My current kiln design should nearly eliminate convective losses, and will greatly reduce radiative by refocusing the lost energy.

Come check it out at Maker Faire!

</comment><comment><USER3> Very impressive - it's a good thing large sheets of plastic with ridges are so expensive.

</comment><comment><USER4> Is there some kind of opportunity here for solar power?

<comment><USER3> No. The power (energy rate per area) is very high, but the total energy is not.

<comment><AUTHOR> But there _is_ opportunity in making a solar-fired kiln (mentioned in the article), and produce pure silicon from sand and carbon. The article mentions 1500C, but I'm not sure if 1800C is attainable with the method. Also, (lack of) vacuum might be a problem.</comment></comment></comment>


SSD endourance - <AUTHOR> Good day guys,
I have a 5% wear on my SSD.
At which point i have to replace it?
Is there any data showing endourance if wear is already present?
Or there is no time-frame for totall damage?
Thanks in advance <comment><USER2> I hope some tech engineer answers this question As i have same concerns about my SSDs</comment>


To keep Flash relevant, Adobe must resort to the nuclear option - <AUTHOR> http://asserttrue.blogspot.com/2010/02/to-keep-flash-relevant-adobe-must.html

<comment><USER2> It's interesting that everyone seems to accept "we can't make this open source because of H.264" from Adobe far more than they accept the opposite "we can't use H.264 because we're open source" from Mozilla.

<comment><USER3> I could make a guess. Adobe is a big corporation that can still put a lot of pressure on others in many different ways. They probably have devs who could write own implementation of H.264 given enough time. They could release their solution based on a licence that doesn't prevent linking with closed codec. Or many other things...

Mozilla is bound by the license. If you find a good enough codec, you can create a plugin yourself - but it will not be included in the main package, unless it's open.

<comment><USER4> > If you find a good enough codec, you can create a plugin yourself

No, you can't. Gecko links directly with liboggplay to support <video> and <audio>. To hack it to do something sane you'd have to fork both Gecko and XULrunner, and distribute your own builds of Firefox with unofficial branding.

<comment><USER3> There are many ways to hijack the content. Are you sure you cannot make an extension that rewrites the page before the <video> tag is interpreted by the browser and injects some custom object instead (handled by the h264 plugin)?

Actually, I just realised that noscript does a similar thing (proxy object instead of flash content) - so it has to be possible.

<comment><USER2> If you're just wanting to display video then I'm fairly certain that Firefox will hand off any video you give it to an application that registers to handle it via the object tag.

For example, I'm fairly certain that this how it reacts to stuff that Apple serves directly to quicktime (trailers and product promos), when you're on Linux.

<comment><USER4> Firefox doesn't do that, your NSAPI plugins do: on Linux there are plugins that wrap mplayer, vlc, or totem supporting the quicktime API in widespread use. Unfortunately all of them suck terribly -- seeking is always broken, they always find a way to do the wrong thing wrt progressive download, weirdass YUV->RGB artifacts after dropped keyframes, etc.

You could write a new NSAPI plugin using ffmpeg that provides the <video> API but proxied through <object>, and package it in an XPI along with some greasemonkey-style extension code to rewrite the tags and provide the default playback UI.

</comment></comment></comment></comment></comment></comment><comment><USER5> The first comment after the article, by Sh0rtWave has some good pro-Flash observations.

<comment><USER6> While video may be a small aspect of what flash provides, it is likely the overwhelming majority of what flash is actually used for.

He's right that even if flash video was obsoleted, it wouldn't die completely because it does have other legitimate uses.

</comment></comment><comment><USER7> I wonder why Flash didn't open up it's format and offer it up as an alternative?

<comment><USER8> because it's crap and everyone knows it. can we stop upvoting threads about flash now?</comment></comment>


Chaos to perfection - <AUTHOR> http://www.chaostoperfection.com/ <comment><USER2> "It appears that your web browser does not support webGL", which is bs. They seem to be pushing Chrome, for some reason.</comment>


Redshift vs. BigQuery vs. Snowflake benchmark - <AUTHOR> https://blog.fivetran.com/warehouse-benchmark-dce9f4c529c1 <comment><USER2> Disclosure: I work on Google Cloud (but not BigQuery specifically).

Cool! First, thanks for doing this.

You allude to it a little in your text, but I think your cost model:

(cost_per_hour / 3600) * query_in_seconds / expected_utilization

doesn't (often) work out. Your benchmarks only run one query at a time, so BigQuery will look pretty expensive compared to the smallest Redshift/Snowflake cluster.

Have you tried simulating say a team of 10 analysts by sending concurrent queries? (You will see the average runtime go up as you fill the box). Alternatively, even 18% utilization is probably way too much for a single person/analyst!

P.S.: You seem to have checked in export PGPASSWORD=...

<comment><AUTHOR> (author here)

Trying to compare costs between BigQuery and Redshift/Snowflake is really, really hard because of their different pricing models. I tried to be very explicit about what assumptions I am making so people can make their own judgements. You are correct that a single-user sending queries to the warehouse is not realistic, but it should still produce an accurate price comparison. Here's my logic:

* Suppose I were to send 10 times as many concurrent queries to the same warehouse. That would drop the effective cost-per-query by 1/10, which makes BigQuery look 10 times worse.

* But suppose these queries now take 10 times as long. According to my formula, that makes the effective cost-per-query 10x larger, which makes BigQuery look 10 times better.

As long as the performance of the warehouse is linear-ish with respect to number-of-users, these two effects should cancel each other out.

RE: PS: Don't worry, I made that one up just for that test cluster, which is now gone :)

<comment><USER2> I should have been more explicit: it won't be linear-ish. As you fill up any per-box sort of system, you start contending (pretend that you simulate 100 analysts) while the BigQuery results will be closer to linear. It does suck that to benchmark that "accurately", you have to make the queries different enough so that no reuse occurs (unless that's reasonable!).

Again, thanks for doing this!

<comment><USER3> When we've looked at BigQuery it seemed that if you prepay you essentially get a similar effect to what you're describing. You're given a certain number of "units" of compute, and if you exceeded your concurrent units available you end up with the same compute resource contention you would with an improperly scaled Snowflake warehouse or Redshift cluster.

If you're willing to just pay per gigabyte scanned with BigQuery you can scale near linearly I'm sure (although I haven't actually tried it), but you could accomplish the same thing using Snowflake's API to add warehouses as concurrent query load increases. That's what we do (although we just pre- allocate and suspend the warehouses because you only pay when they're on).

Redshift does suffer from this problem because the compute is tied to the data, but Redshift Spectrum is attempting to rectify that as well. I don't know anything about its performance though.

</comment></comment></comment></comment><comment><USER4> I have to consider these results to be meaningless based only on the fact that you do not use the "advanced" features like sort and distribution keys. In Redshift especially sort and distribution keys are not "advanced" features, they are fundamental factors in determining performance.

Your justification for this is that you don't know how queries will run but that's a poor excuse. With your star schema example dataset it should be easy to pick distribution keys that match the access patterns of commonly joined tables. There may be tradeoffs here but it's far from impossible and you are crippling performance by not using these features.

<comment><USER4> It's too late to edit but this comment feels like it dismisses your hard work which I do not intend to do. Having said that I am concerned that this approach leads to the ford-vs-chevy type arguments that don't help anyone make an informed decision.

What do you think about doing a series on the workloads that each database engine is best at? Surely each model has benefits and disadvantages. Finding what each system is best at will help people decide what system they really need.

You are in a unique position to answer this question because as you say, you have no interest in any specific solution and your customers use all of these systems.

<comment><USER5> Hi mulmen - I work with George (author). Dist and sort are absolutely an invaluable tool for optimizing your common BI workloads. What we are seeing with our customers is extensive use of arbitrary querying via tools like Looker or Tableau which often do not use the sort or dist key. This is becoming the most common workload for Fivetran customers b/c of the vast number of frequently updating dashboards that are powered by those queries.

<comment><USER4> Thank you for your insight. My team uses both Tableau and Redshift but we don't do a lot of frequently updating dashboards.

This sounds like a major limitation of these query tools. Are these arbitrary queries written by users or created by the tools programmatically? Since dist keys are ideally join keys why do these tools (or users) not understand the join keys? Can the queries be identified via WLM and triaged for optimization?

I am skeptical of any solution that promises to be magic and do the hard work for us. I think the "big data" industry does itself a disservice by perpetuating the myth that hard problems can be solved with mouse clicks.

<comment><AUTHOR> It's not a limitation of the query tools. The problem is that users are running so many different queries against the same schema that you can't optimize for them all. Therefore, it's most important how a data warehouse performs when the optimization "tricks" don't work.

Reasonable people can disagree on this last claim---we'll publish a v2 of this benchmark in a few months where we do more tuning [https://github.com/fivetran/benchmark/issues/1](https://github.com/fivetran/benchmark/issues/1)

<comment><USER4> There will always be trade-offs so disagreements are to be expected. My interest is largely in what models work best for a given system, specifically Redshift. What data models have you seen in use? Are the schemas dimensionally modeled or are they flat tables?

Where have you run into limitations with dimensional modeling? Is it a lack of engineering resources on the customer side or is the approach limited in some fundamental way?

Thanks for sharing your invaluable experience. I look forward to part two!

</comment></comment></comment></comment></comment></comment><comment><USER6> I'm not sure leaving out sort and dist keys is actually being fair to Redshift here.

I get the argument that you're avoiding any "special tuning", but these are basic features that the docs and most guides will tell you are not really optional. In my mind it's more like leaving out the primary key declaration on a table in Postgres or MySQL.

<comment><AUTHOR> (author here)

It's a tough call. In my experience, sort and dist keys only really work out in < 50% of queries, and it's the other 50% where you judge the performance of your warehouse.

If we had chosen to use sort and dist keys in Redshift, then we would have also used clustering keys in Snowflake, and it probably would have been somewhat of a wash. BigQuery doesn't really have an equivalent feature yet--- they have date partitioning, but that's a very narrow special case that won't work for most queries.

<comment><USER7> It looks like you had to rewrite the queries pretty significantly to get them to run on all three. Redshit/Bigquery don't support rollup/grouping sets/intersect/except etc. which the TPCDS spec queries have. Is there a reason you don't mention this in the blog post? SQL surface area seems like a pretty important thing to consider when choosing a DW.

<comment><USER8> GROUP BY ROLLUP support actually just arrived in Bigquery a week or two ago. Still a lot of things missing, but they are steadily improving.

</comment></comment></comment></comment><comment><USER9> BigQuery is one of the few products that truly feels magical. It lives up to the "no-ops" model of just working without thinking about details and handles any scale of data you have.

Only downsides would be cost-per-run pricing and low-latency queries.

<comment><USER10> Agreed.

One point that this article doesn't address (not that it necessarily SHOULD) is the fact that BigQuery achieves these results with zero database administration or schema design.

Redshift is great, but it's not "set and forget".

Also, there are a huge number of use cases out there which are daily reports on large datasets. Fixed vs. Variable cost models are obviously going to shift in favor of variable when this kind of scenario comes into play.

Last, what kind of cost factor do you think comes into account when it comes to paying someone to administer your Redshift db, vs not paying anyone to administer the BigQuery setup. I can tell you from experience that this is a huge, huge factor.

<comment><USER11> I think you need someone to run and optimize the data warehouse in any case -- figuring out data exports to BQ, table partitioning, etc don't really come for free. Perhaps BQ is easier in that you don't have to figure out how to run and scale a Redshift cluster, but as a BQ user, I do a decent amount of work mucking around with the schemas and figuring out how to do things more performantly or cheaply.

</comment></comment></comment><comment><USER12> This is super well written! I've seen a lot of benchmark queries show up on Hacker News, and I think this one is exceptional for calling out and justifying all of its assumptions, then comparing it to other benchmarks and explaining how they might be different. I really appreciate all of that clarity. That academic but easy-to-read style doesn't make for flashy headlines but it's so much more informative.

</comment><comment><USER13> I don't know; personally I feel like the scale here is much too low to be relevant. Also I think not investing the time to tune sort and dist keys makes the comparison meaningless.

But maybe that just becomes meaningful at larger data sizes and maybe most people work with less data most of the time.

</comment><comment><USER14> Do you plan to take into account the developer cost? BigQuery is more managed than Redshift so that definitely factors into the "real world" cost analysis.

<comment><AUTHOR> Yes! We're going to write a "part 2" where we talk about the qualitative differences, which are huge.

<comment><USER15> Sweet! Another wrinkle to add in: if you're comparing a redshift system that's >$35k/yr, BQ does have a flat rate pricing option which might fare better in your TCO analysis. Info here: [https://cloud.google.com/bigquery/pricing#flat_rate_pricing](https://cloud.google.com/bigquery/pricing#flat_rate_pricing) Yes, I'm a biased cloud Googler.

</comment></comment></comment><comment><USER16> I am using Redshift since two years and, as every database has its own SQL dialect and its own tricks, but it is a really good product. It is a pity that Amazon forked early PostgreSQL v8 and not PostgreSQL v9 (with array support) but if I think about databases I used before (I was admin of a Netezza instance and user of an Oracle instance, they were Data Warehouses too) I realize it is a really big step forward that now we can create with few clicks and witha really fast provisioning time, a database that took days, even month (contracts, waiting they arrive to datacenter, DBAs configuring it, installing it, etc) to have it up and running, that now thinking about Redshift or Big Query it seems to live in another era.

</comment><comment><USER17> There is a different side to the cost benchmark that's not captured by the description here. If your use case needs a lot of stored data but not necessarily a matching degree of peak CPU (even if your query load is otherwise pretty consistent), Redshift will become really expensive really fast and it will feel like a waste. BigQuery will meanwhile keep costs linear (almost) in your actual query usage with very low storage costs.

For example, you may need to provision a 20-node cluster only because you need the 10+ terabytes in storage across several datasets you need to keep "hot" for sporadic use throughout the day/week, but don't nearly need all that computational capacity around the clock. Unlike BigQuery, Redshift doesn't separate storage from querying. Redshift also doesn't offer a practically acceptable way to scale up/down; resizes at that scale take up to a day, deleting/restoring datasets would cause lots of administrative overhead and even capacity tuning between multiple users is a frequent concern.

Making matters worse, it is common for a small number of tables to be the large "source of truth" tables that you need to keep around to re-populate various intermediate tables even if they themselves don't get queries that often. In Redshift, you will provision a large cluster just to be able to keep them around even though 99% of your queries will hit one of the smaller tables.

That said, I haven't tried the relatively new "query data on S3" Redshift functionality. It doesn't seem quite the equivalent of what BigQuery does, but may perhaps alleviate this issue.

Sidenote: I have been a huge Redshift fan pretty much since its release under AWS. I do however think that it is starting to lose its edge and show its age among the recent advances in the space; I have been increasingly impressed with the ease of use (including intra team and even inter-team collaboration) in the BigQuery camp.

<comment><USER18> Redshift offers hard disk based nodes with huge amounts of storage at low cost for precisely the use case you mention. The performance of these is actually very good, especially with a little effort applied to choosing sort keys and dist keys.

Spectrum extends that even further, allowing you to have recent and reference data locally stored and keep archival data in S3 available for query at any time.

</comment></comment><comment><USER19> I've been using Redshift fairly heavily for the past couple years. I haven't had the time to do comparisons between BigTable, unfortunately. I've been too strapped for time. And, we have only just started experimenting with Snowflake recently.

I would like to try out BigTable, but experimenting with it has required too much work. I would have to move my entire data set (Several Petabytes) over to Google.

Redshift has been fine. But, with Reshift, we have two big issues: 1) Loading data takes too long. Even with efficient copy operations from S3, it takes too long to import data at scale. 2) Ineffient queries frequently overflow to disk and consume the entire SSD. I've tried to train my data science team on how to avoid inefficient queries, but it's been impossible.

So, I'm really looking forward to seeing the results from part 2!

<comment><USER11> Have you thought about using EMR to generate rollups of the S3 data and then putting that into Redshift?

<comment><USER19> We do that, for the most part. It definitely helps.

</comment></comment></comment><comment><USER20> Also, here is a rudimentary, 2-part comparison of Redshift vs Azure SQL DW (in preview) for good measure:

Part 1 - [http://bicortex.com/microsoft-azure-sql-data-warehouse- quick...](http://bicortex.com/microsoft-azure-sql-data-warehouse-quick-review- and-amazon-redshift-comparison-part-1/) Part 2 - [http://bicortex.com/microsoft-azure-sql-data-warehouse- quick...](http://bicortex.com/microsoft-azure-sql-data-warehouse-quick-review- and-amazon-redshift-comparison-part-2/)

</comment><comment><USER21> How hard would it be to add [https://clickhouse.yandex/](https://clickhouse.yandex/) to the benchmarks?

<comment><USER9> On-premise installs tend to be very different from cloud managed warehouses and not really comparable.

The 1 billion taxi rides benchmark by Mark Litwintschik is pretty comprehensive though: [http://tech.marksblogg.com/benchmarks.html](http://tech.marksblogg.com/benchmarks.html)

<comment><USER21> Yes.. that's a good starting point but the hardware used for each platform varies a lot.

Clickhouse on a single i5 compared to redshift on a 6-node ds2.8xlarge cluster

I've been throwing everything I can at clickhouse on a little VM, and the performance is amazing.

<comment><USER22> I've recently moved to clickhouse for some analytical work, and it's been awesome. It's not perfect for what I want, and I'm sure I'm using it wrong, but it's torn through what I have to throw at it and I've not spent any time doing perf tweaks.

There's a connector for redash too, so I can make nice little dashboards from the results.

Major problem is a lack of updates to data. At least loading is fast enough for my use.

<comment><USER21> You can't do updates, but you can create tables using the AggregatingMergeTree table engine.

I've been working out how to use this for tracking count/first seen/last seen for data sets. With normal sql you need to do upserts, but with clickhouse you can just create the table using countState, minState, maxState and insert:

      1, now, now

for count, first, last, and when it compacts the table it'll apply the aggregations.

Basically what you do with with [https://github.com/facebook/rocksdb/wiki/merge- operator](https://github.com/facebook/rocksdb/wiki/merge-operator) just with a few lines of SQL instead of a ton of custom code.

<comment><USER22> Interesting, I'll have an experiment with this today. I think I can see the pattern but I'm not entirely sure it'll work quite as I'm hoping.

So assuming I've got a nice primary key (which thankfully I do for the main data I care about), I could have a table something like (syntax will be wrong but):

        PrimaryKey
        Something
        Otherthings
        maxState(LastUpdated)

Then when selecting I can run queries efficiently to only get the latest version of a row?

The bit I'm not sure on is I feel a lot of this is designed to work nicely with columns, and in this case I have a fairly row-like constraint (latest row with a key).

Should be an interesting day of investigating though, thanks!

<comment><USER21> Here's a workable example that I came up with:

      CREATE TABLE mt2
      (
        whatever Date DEFAULT '2000-01-01',
        key String,
        value String,
        first AggregateFunction(min, DateTime),
        last AggregateFunction(max, DateTime),
        total AggregateFunction(count, UInt64)
      ) ENGINE = AggregatingMergeTree(whatever, (key, value), 8192)

insert into mt2 (key, value, first,last,total) select 'www.google.com', '18.104.22.168', minState(toDateTime(1498241729)),maxState(toDateTime(1498241729)), countState(cast(1 as UInt64));

insert into mt2 (key, value, first,last,total) select 'www.google.com', '220.127.116.11', minState(toDateTime(1498242729)),maxState(toDateTime(1498242729)), countState(cast(1 as UInt64));

insert into mt2 (key, value, first,last,total) select 'www.google.com', '188.8.131.52', minState(toDateTime(1498242729)),maxState(toDateTime(1498242829)), countState(cast(1 as UInt64));

select key, value, minMerge(first), maxMerge(last), countMerge(total) from mt2 group by key, value;

optimize table mt; -- compact the table and merge multiple rows with the same (key, value)

You have to add the date column, but you can ignore it.

The last query can be a view or even a materialized view.

<comment><USER22> Thanks!

I was also playing yesterday with CollapsingMergeTree.

        CREATE TABLE cmt
        (
          whatever Date DEFAULT '2000-01-01',
          key String,
          value String
          sign Int8
        ) ENGINE = CollapsingMergeTree(whatever, (key, value), 8192, sign)

Now you can 'delete' a row by sending the same row again but with a sign of -1:

        insert into cmt (key, value, sign) values ('k1', 'v1', 1)
        insert into cmt (key, value, sign) values ('k1', 'v1', -1)
        insert into cmt (key, value, sign) values ('k1', 'v1 update', 1)
        insert into cmt (key, value, sign) values ('k2', 'just delete this one', 1)
        insert into cmt (key, value, sign) values ('k2', 'just delete this one', -1)

You have to either add FINAL onto the query or optimise the table as far as I can tell for this to work, or hope it's done it in the background.

I thought you had to use the sign in the query otherwise it wouldn't work, but creating this example this morning works fine. If you're not getting the response you expect after optimising, try adding the sign column to the query.

        :) select key, value, sign from cmt

        SELECT
            key,
            value,
            sign
        FROM cmt

        ┌─key─┬─value─────┬─sign─┐
        │ k1  │ v1 update │    1 │
        └─────┴───────────┴──────┘

        1 rows in set. Elapsed: 0.002 sec.

        :)

[disclaimer - I know very little about how to make high performance things in clickhouse, tbh the Log format has been easily fast enough for my data so far]

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> Looks like Fortune picked this up. Nice! [http://fortune.com/2017/10/09/amazon-google-snowflake- data-w...](http://fortune.com/2017/10/09/amazon-google-snowflake-data- warehouses/)

</comment><comment><USER21> [https://github.com/fivetran/benchmark](https://github.com/fivetran/benchmark) Does not exist?

<comment><AUTHOR> Fixed! Thanks for pointing that out.

</comment></comment><comment><USER24> Curious, any reason you've left out MS Azure DW?

I'd be interested in seeing their numbers in there as well.

<comment><USER20> I run some benchmarks, comparing Azure SQL DW (in preview) to Redshift. They can be found here: [http://bicortex.com/microsoft-azure-sql-data-warehouse- quick...](http://bicortex.com/microsoft-azure-sql-data-warehouse-quick-review- and-amazon-redshift-comparison-part-1/) and here: [http://bicortex.com/microsoft-azure-sql-data-warehouse- quick...](http://bicortex.com/microsoft-azure-sql-data-warehouse-quick-review- and-amazon-redshift-comparison-part-2/)

</comment></comment><comment><USER25> A BigQuery partner here and a long-time user of Redshift. ([https://cloud.google.com/partners/directory/#?q=caura](https://cloud.google.com/partners/directory/#?q=caura) [https://discourse.looker.com/t/bigquery-migrating-from- legac...](https://discourse.looker.com/t/bigquery-migrating-from-legacy-sql- to-standard-sql/3568) and you can Google my creds for Redshift with "Troubleshooting Redshift performance")

So having benchmarks tests is great as a general guideline for what works under different architectures/schema designs. Unfortunately, benchmarking is highly subjective to the initial choices. I am a big fan of BigQuery (enough to go through Google's vetting process), but there are plenty of performance issues that I've run into it that would have been easily resolved with Redshift. Here are some concrete examples:

1) Running a Query across several very small tables. It turns out that occasionally querying small tables causes heavy network traffic within Google's distributed system. The solution on Redshift would be to adjust distribution. On Google, however, you don't have any control over this. You just have to hope that Google's algorithms pick up the issue based on usage (they don't).

2) Joining large tables. Avoid joining large tables in BigQuery. In Redshift the join would have been done by making sure that the sortkey is set on the column that is used for a join on the (typically) right table. Then having a common distkey between the two tables (this way the relevant data on both tables lives on the same node. BigQuery just throws resources at the problem. Well, it turns out that throwing resources at the problem is super slow (think 5-15 Redshift seconds vs. 200 BQ seconds).

Re: Snowflake. Can't speak to it as I haven't had personal experience. I have worked with Data people who had opinions on both favorable and negative sides of the spectrum. This just suggests to me that just like Redshift and BigQuery, Snowflake is not a universal solution. You really need to understand: 1) what your goals are for the usage among varying consumers 2) what skill set do the various users of the database have</comment>


Ask HN: How do you deal with haters? - <AUTHOR> Should you care about what they say? Are they important? Is it normal to have haters? <comment><USER2> Stupid people's mom is always pregnant, so yes, it is normal to meet stupid people on internet as much as in real life. The only way to deal with them that I've found is: smile and wit. Use sarcasm to smash them to the ground while being absolutely polite. It is really satisfactory (and "super effective!).

</comment><comment><USER3> Hater's gonna hate.

<comment><USER4> You are using an apostrophe incorrectly.</comment></comment>


How to Build a House on Mars (2016) - <AUTHOR> https://www.theatlantic.com/science/archive/2016/01/mars-house-concrete/423402/?single_page=true <comment><USER2> Is anyone else at least a little curious what sulfur concrete smells like (on Mars)?</comment>


Brutal Honesty - <AUTHOR> http://uncrunched.com/2011/10/01/brutal-honesty/ <comment><USER2> Brutal honesty can be powerful when used correctly, but too often it becomes a cover for an influential person who enjoys being a jerk to others.</comment>


Ask HN: How have you automated parts of your personal life? - <AUTHOR> <comment><USER2> It's gunna sound lame, but I've got a half dozen aliases to launch frequently edited files (writing a book), make database entries, and even a custom reboot to launch a selected application immediately upon reboot. I've got some IFTT alerts and Google alerts doing their thing if that counts.</comment>


Delivery drone with marijuana and phones crashed outside a S.C. prison - <AUTHOR> http://www.washingtonpost.com/news/post-nation/wp/2014/07/31/a-delivery-drone-carrying-marijuana-cell-phones-and-tobacco-crashed-outside-of-a-s-c-prison <comment><USER2> I wonder how many drone smuggling attempts have _succeeded_. I'm a drone hobbyist, and I don't think it would be at all difficult to deliver small packages with a very low chance of getting caught.

<comment><USER3> Wouldn't some kind of ballistic launcher, maybe with a parachute, be easier? Not so much for cellphones, but for light, robust items like cigarettes, dime bags, etc. You could probably do it with a slingshot.

<comment><USER2> I think your range would be much lower with any practical ballistic launcher, which makes it easier to get caught.

</comment></comment></comment><comment><USER4> I didn't even know Amazon sold marijuana.</comment>


Ask HN: How to get lucky with Hacker News? - <AUTHOR> I keep reading overnight success stories of people, who posted their new products on HackerNews and in no time those products got a great traction. For e.g.
https://crew.co/backstage/blog/how-side-projects-saved-our-startup/
I am a Hacker News member since many years now. I admit that I am not a very active one and very few times I have interacted with the community. But I believe there are many like me and still they have tasted more success than me.
What am I doing wrong? Is there a better time to post on Hacker News? Should I do something else after posting it on Hacker News for e.g. ask some member to review it, etc.? What am I missing? <comment><USER2> I've wondered along the lines of your question. So I was hoping to find more information. Totally agree with you, that there seems to be a huge factor of "luck" involved in using HN to promote products.

Since I was very curious about your situation, I checked your profile, past posts and comments. Whilst my comments don't directly answer your question, these are the things that stood out for me:

You have been on HN for over 8 years and in that time you have only amassed 107 karma. That immediately signals to other HNers that you are not a particularly involved member.

Most of your posts and comments have a "marketing slant". HN is, IMHO, a technically oriented forum. I often notice a certain level of hostility towards posts made by sales people, recruiters, etc. For me you would have more "impact" by contributing to various technical discussions.

Looking at your two products and the comments on them by other HNers suggests that they are not as "exciting" as you think. There are thousands of products (I use the term very loosely) out there. You have to make your stuff really stand out, capture interest and of course a huge measure of luck.

Finally Show HN is but one of potentially dozens of avenues to promote your products. Perhaps identifying forums that deal with your application area is useful. I know some people who are very active in domain specific forums on LinkedIn, thus they have a certain profile. They then in turn contact other active persons on the forum to canvass for business.

I hope my comments will provoke some rebuttals. My comments are just one data point. You need a broad spectrum to base any decisions upon. Good Luck.

<comment><AUTHOR> I agree. I need to become a more involved member and earn some Karma before I submit something again.

</comment></comment><comment><USER3> I have had a very short look at your submission history, and I still have no idea what product or service you're offering.

Why do you expect "success" with something nobody even notices?

<comment><AUTHOR> Well, that's a surprise. May be I am doing something completely wrong because the previous two submissions (apart from this one) were of the products, we have recently launched.

[https://www.readboard.io](https://www.readboard.io)

[http://apps.baapps.com/preview](http://apps.baapps.com/preview)

And that's what my question is, why HNers are not noticing things I am posting? :)

<comment><USER4> Well, your readboard submission got noticed, just not in a positive way... and promptly flagged, so most people can't even see it. Yay, spam filtering works.

Your other submission I very nearly missed the "example data" field, it looked like a very empty page that wants some unspecified data otherwise.

Put a page that tells people _what they are looking at_ , don't force them to click things and follow steps without knowing what the end result is. Step-by- step onboarding is nice if a user has signed up for something, but you have to show what you offer before that happens. (and most people here will close the page as soon as there is a sign-up prompt, unless you really convinced them)

<comment><AUTHOR> Thank you for your feedback. This really helps.

</comment></comment></comment></comment><comment><AUTHOR> BTW I found some great insights in following article and its comments about posting on HN.

[https://www.groovehq.com/blog/hacker- news](https://www.groovehq.com/blog/hacker-news)

Especially a comment, which says "My point is simple. Find something worth writing about that the HN community is hungry for, and come up with a good title for the post. No need to manually market your post on twitter or to your friends. Just post and watch your real-time analytics soar."

And a few things from following article...

[http://alexstechthoughts.com/post/29406022580/how-to-get- on-...](http://alexstechthoughts.com/post/29406022580/how-to-get-on-the- frontpage-of-hacker-news)</comment>


UBB overturned - Canada no longer biggest internet loser. - <AUTHOR> http://www.thestar.com/news/canada/article/932571--ottawa-to-reverse-crtc-decision-on-internet-billing?bn=1

<comment><USER2> The second half of this article reads like it was written by Bell or Rogers.

The problem with UBB is not caps, per se, but the fact that the caps were absurdly small and overage fees grossly exceeded the cost of bandwidth. Not to mention that they were introduced at the same time as Netflix launched in Canada, and the providers' own TV-over-IP services were exempt.

I dumped my $120/mo Bell Expressvu subscription last month in favor of Netflix and AppleTV. Why should I pay over a thousand dollars a year when I don't watch 98% of what it provides? UBB is just Bell saying 'screw you, we'll get that pound of flesh one way or another.'

<comment><USER3> _The second half of this article reads like it was written by Bell or Rogers._

Indeed. There were a few lines which particularly stood out to me as having been written by someone who doesn't really get the situation:

> To encourage competition, major telecom operators that have spent heavily on > infrastructure are required to lease bandwidth on their networks to small > providers.

I get that the system is maintained by these large companies, and I'm sure they have spent a lot on the infrastructure, but (and correct me if I'm wrong) I believe the system was built largely with taxpayer dollars, and it continues to be heavily subsidized by same.

> Although critics say the CRTC ruling will lead to lower download limits and > higher rates, major Internet service providers say usage-based billing based > is fair because it means heavy users pay more than those who just surf the > web and use email.

There are two problems with this. First, internet isn't like electricity. Electricity must be generated at a cost, and those who consume more cost the generating company more (perhaps not directly, since unused electricity is simply lost, but at least indirectly by requiring higher generation to accommodate peak usage). With the internet, however, the lines are already laid and it's merely a matter of sending the information down the wire. There is almost no additional cost associated with higher usage (see next point).

Second, the 'incremental cost" of bandwidth appears, from what I've read recently, to be somewhere around a penny per gigabyte, if any exists at all. Let's give the companies the benefit of the doubt and say that it's actually 2.5 cents per GB. Now let's give them a 100% markup to be nice. So now we're calling a "reasonable markup" on this service to have a price of 5 cents per GB. Bell wanted $2 per GB for folks who go over their limit. They wanted _40 times_ what I just gave as a reasonable number. Even if you want to use the most conservative estimate I found, which was the CEO of TekSavvy who said maybe it could get as high as 30 cents per GB, then Bell would still be asking us to pay 666% (<obvious joke here>) of the actual cost. That's one hell of a markup.

Also, and again using the 30c/GB estimate, Bell's offer in Ontario of a 25GB limit would equate to a $7.50 cost? Anyone want to take a guess at how much the service actually is actually priced at? I'll give you a hint: it's not $7.50.

Sorry for the rant, but I've been worried about this decision and I have a lot of pent up anger about it. It's nice that the government has actually stepped up to support the small, reasonable ISPs and the consumer.

<comment><USER4> A random nitpick that you might find interesting: in fact unused electricity is not "simply lost." Where would it go? Because of the law of conservation of energy, it would have to go somewhere, and we're talking about what might be a _lot_ of energy.

Electrical systems actually must generate exactly the required amount of electricity at all times. One particularly interesting tidbit I learned in one of my engineering classes: hydro dams are like giant capacitors. If you can't "push" more energy into the power grid, then you also can't push more water through the turbines; it gets harder to turn the turbines due to electromagnetism. And so the dam starts to fill up, more or less automatically, because of the laws of physics.

<comment><USER3> Well, now I feel like an idiot. That comment was based on a poorly recalled statement regarding the ineffectiveness of Earth Hour. If you don't use electricity, it'll flow to the next guy, so your lack of use doesn't really have any effect. Still, you need to generate enough electricity to cover everyone's potential needs. Whether that electricity is used or not (where not means that unused electricity dissipates as heat or.. whatever happens to unused electicity?), it still needs to be available, and at a cost. Unused bandwidth doesn't need to be generated. It's simply a quiet line.

Thanks for reintroducing logic to my brain. The rest of my argument mostly holds up though, right?

<comment><USER5> > whatever happens to unused electicity?

Re-orient your thinking a little and it will make sense. We're talking AC systems (not a SWER) so it is a closed-loop as far as the electricity is concerned. The electricity doesn't go anywhere, it constantly oscillates. What you are actually generating in a meta sense is ability to do _work_. You are overcoming the electrical resistance from all those users' machines doing work.

Does that analogy help?

<comment><USER3> I understand the closed loop. I did a bit of research when I wrote my last comment and when I read that line is when I realized how dumb I was.

So.. electricity gets pumped into the grid from some power station, and then [a given unit of electricity] loops repeatedly around the loop until it's either consumed by an appliance or is converted to heat by the resistance inherent to the transmission lines themselves?

</comment></comment></comment></comment></comment></comment><comment><USER6> Unfortunately for Canadians, I believe it's far from over. Business/Politics are known to make ambit claims [0] that are hard to fully remove as there's a tendency of the well-meaning of compromising is exploited. Examples can easily be seen with Obama administration with the Republicans [1] .

Canadians would have to ask for more than a reversal. It might be an overhaul of the organisation due to widespread assumptions of regulatory capture [2] that's serving the needs of monopolistic industries over the public interest.

As long the department continues to exist even though they had the sheer audacity to implement it despite public submissions; this issue is not going away.

Well done. To me, it's a small step however.

[0] <https://secure.wikimedia.org/wikipedia/en/wiki/Ambit_claim>

[1] <http://i.imgur.com/1ULKE.png>

[2] [https://secure.wikimedia.org/wikipedia/en/wiki/Regulatory_ca...](https://secure.wikimedia.org/wikipedia/en/wiki/Regulatory_capture)

<comment><USER3> Believe me, many Canadians are well aware that Bell's move was an ambit claim (a term I've just now learned because of you, so thank you for that).

I've read a number of claims on reddit, even before Minister Clement made his comments, from people who were fearful that the government would smack down this decision only for Bell to come back with a slightly less bad offer.

It remains to be seen what we'll eventually get out of this deal. At this point, however, we have the three largest political parties in the country against this, and I've heard (but haven't seen citations) that the other two major parties have stated their opposition. The Prime Minister has personally come out against the issue[1] as well, so it looks like there is a fair amount of political resistance to UBB in general.

The CRTC needs to be disbanded. This will be the second government overruling of their decisions in as many months[2], so perhaps it will make them a little more measured in their approach should they again approach the UBB issue. If the CRTC constantly needs to be kept in check by the government, then perhaps it will incentivize the government to disband the CRTC as the government will already be doing their job for them anyway.

[1] <http://twitter.com/pmharper/status/32526091855863808> [2] [http://news.gc.ca/web/article- eng.do?crtr.sj1D=&mthd=tp&...](http://news.gc.ca/web/article- eng.do?crtr.sj1D=&mthd=tp&crtr.mnthndVl=&nid=501719&crtr.dpt1D=&crtr.tp1D=1&crtr.lc1D=&crtr.yrStrtVl=&crtr.kw=&crtr.dyStrtVl=&crtr.aud1D=&crtr.mnthStrtVl=&crtr.yrndVl=&crtr.dyndVl=)

</comment></comment><comment><USER7> In some ways, I'm envious of Canada's swift reversal of this horrible idea. In the US, we wouldn't have the populist outcry to force action, and even if we did, it would be met with only a token change to the legislation that would succeed at shutting most people up.

<comment><USER8> No need to be envious, threat of an election is a powerful motivator. Liberals are itching to bring the government down and have a new election, there are rumors that the opposition is gonna bring the government down in late- march/early April. Conservatives don't want to be on the wrong side of the public opinion on a populist issue like this going into a potential election. This is pure politics at work. I hate to be cynical but if an election did not look imminent I doubt the government would have acted so quickly.

</comment></comment><comment><USER9> _Thank God._

Seriously, this restores some of my faith in bureaucrats.

[edit] Interesting information from replies. I'll not be so quick to restore faith in the future.

<comment><USER10> The bureaucrats (CRTC) were the ones who tried to force this on the population. Elected officials decided to reverse the decision. The Conservatives first said they would send the decision back for "reconsideration" and then Harper chimed in with a more definitive reversal stance. The Liberals came out earlier saying they were against UBB. The New Democrats have always been fairly progressive about the web and are also against UBB. Even the separatist Bloc Quebecois and Green Party were both voicing concern with the decision. There was no major political party in favour of the decision.

We get into these kinds of messes because the CRTC is a bunch of unelected bureaucrats playing musical chairs in and out of executive positions at Rogers, Bell, Telus, etc.. It is most perplexing because if you've dealt with other bureaucrats in Canada, most of them at least care about their jobs and try to look out for the public good. We can joke at their expense about how they have all these perks and what not. But most do uphold the laws and regulations they're sworn to. And since they tend to outlast elected governments, these are the people that actually keep programs that we depend on, like health care, running. But the CRTC? It's the lone hold out of the bunch. I pretty much have nothing good to say about that organization. I would not be sad in the least if they were dissolved.

<comment><USER11> I think it's safe to say that, with two major reversals in a year, the CRTC has lost any credibility it may have had (although it did not have much to lose).

My vote would be for dissolution - maybe then they can create a regulator that actually employs people who understand what they're regulating.

<comment><USER2> They know what they're doing. The problem is that they're biased.

It's fundamentally the same issue that plagues financial regulation. The people who are the most qualified to understand the technological and regulatory issues are the most biased, because almost by necessity they're products of the same industry they're supposed to regulate.

<comment><USER12> <http://en.wikipedia.org/wiki/Regulatory_capture>

<comment><USER13> On reading more about this I was depressed to see very little in the way of proposals to avoid the phenomenon, except (surprise! it's Stigler) don't regulate.

But I did find some interesting ideas at the end of this paper: [http://www.icgg.org/downloads/Boehm%20-%20Regulatory%20Captu...](http://www.icgg.org/downloads/Boehm%20-%20Regulatory%20Capture%20Revisited.pdf) (pp. 23-24)

</comment></comment></comment></comment></comment></comment><comment><USER14> I'm annoyed that I didn't even know about UBB until earlier today, when I stumbled across Reddit Montreal and saw something about the organization of a protest.

Then again I'm getting all my news from here and Slashdot these days - haven't watched any TV in ages.

My ISP already posted up new rates conforming to UBB - if it does get overturned, I hope they'll return to the old rates: $29 a month for unlimited bandwidth (soft capped at 100GB). I'm due for renewal at the end of this month, so there's not much time to figure out if I need to be switching to TekSavvy.

Somewhat offtopic - I never visited Reddit except when it was linked via posts on Hacker News, and I'd always heard complaints that HN was becoming Reddit, but finding out today that they have forums for my city and other interesting things like a fitness subforum - why did I never go there earlier?

<comment><USER15> HN is fairly anti-reddit because a lot of the people on here were early reddit adopters who left when it ceased to be programming and tech-centric, and when the site got big enough that the quality of the average comment dropped. That said, there are some great subreddits on there, you just have to cut out most of the main ones. I dropped r/pics, r/atheism, r/worldnews, r/gaming, and r/reddit.com. I added r/humor, r/netsec, r/python, r/java, r/math, r/depthub, r/truereddit, and a few others. It's a much improved experience, now.

<comment><USER16> > HN is fairly anti-reddit

I don't get that impression here at all, quite often you'll see Reddit articles on HN linked back to Reddit. I agree HN is more industry focused and professional and Reddit a lot more laid back but still can be professional at times.

HN and Reddit are similar but also quite different, I go to both and sometimes even digg (I know!) and whatever else I can find, I don't get this mentality that you're only allowed get to choose one social news website to view.

<comment><USER15> Any discussion of reddit here usually ends up being people complaining about it. Yes, there are often reddit links posted here, but whenever reddit is brought up, people will usually complain about how it's full of /b/tards, it's all just memes and pun threads, there's no useful comments there, etc. While this may be true for the larger subreddits, my point is simply that there are some great small ones where the quality of discussion is _much_ higher.

</comment></comment></comment></comment><comment><USER17> At the risk of being pedantic (and maybe too American), isn't this a _promise_ from politicians that it will be overturned? The title might be masking that a little bit...

(The important words in my question being "promise" and "politicians," for hopefully self-evident reasons)

<comment><USER18> In Canada you would never come out and say that a highly disliked legislation piece is going to be overturned and then not overturn it. You would be incinerated at election time.

<comment><USER19> If it were a big enough issue you'd just get recalled by your constituents and then you've got to justify it to try to get re-elected.

</comment></comment></comment><comment><USER20> Well if that isn't a ray of sunshine in a turd-sea of announcements regarding the state of the Internet.

</comment><comment><USER21> This is wonderful news. I was planning to attend my first 'rally' on saturday to protest UBB. I was shocked at how even non techies cared so much about this issue.

I hope the monopolies(rogers&bell) don't try less subtle ways to achieve their goals.

<comment><USER11> Non-techies cared, in this case, because it was pretty easy to see how this would affect you personally - you would have to pay more. Period.

If you asked the average person who is opposed to UBB (read: practically everyone), very few could actually present a nuanced argument as to why it's bad for a number of reasons. Instead, they can simply equate UBB to paying more, which it turns out is good enough for collective action.

So democracy wins, but only because this can be framed as a pretty black-and- white issue in the eyes of most Canadians.

<comment><USER22> To be more precise, it would actually be paying more if you are on an _indie ISP_ , by reducing competition.

</comment></comment></comment><comment><USER23> I don't understand what happened. Why was the government able to set internet rates at all? Is there a monopoly here or not? Why weren't we (I'm Canadian) paying per usage before -- was it a law or did a flat rate grow from convienience?

<comment><USER24> You can reference the diagram in my post here (and the more detailed one in the response to my post): <http://news.ycombinator.com/item?id=2158888>

This was mostly about Bell's ability to bring UBB to bear on the 3rd party ISPs that are leasing their lines (and thereby forcing all of those ISPs' customers onto a usage-based-billing model). Thought if you read that full thread you'll see that there's was a bit of debate over exactly _what_ part of the infrastructure most of the 3rd party ISPs are leasing from Bell.

I'm of the mind that so long as the ISPs aren't going over Bell's peering connections to the wider internet (meaning that they have their own), then Bell shouldn't be able to force usage-based billing onto those ISPs. If any of those ISPs want to use Bell's peering connections to the internet.... then I don't know.

Of note is that Bell is the only one allowed to lay new last-mile phone lines. So if a 3rd party ISP wants to hook-up customers with their DSL, they _have_ to go over Bell's last-mile lines (to the DSLAM at least).

<comment><USER12> Right, so this decision changes nothing for Bell's own customers, who will still face UBB, unless competition from 3rd party ISPs forces Bell to change its billing structure.

<comment><USER24> Right. But by allowing Bell to force UBB on 3rd party ISPs, it is effectively forcing UBB on _everyone_ , not just Bell customers. Now it's just limited to Bell customers.

</comment></comment></comment></comment><comment><USER25> Dear Hacker News Canadians:

Please continue to write your MPs. This is far from over, and we need to keep on top of it. We can't stop now that it looks like it will be overturned, because Bell will fight back. I urge every one of you to write a letter to your MP (or another one if you already have).

If we show any slowing in activity against Bell et al, they will just swoop in and come up with something just as bad.

</comment><comment><USER26> After an Egyptian protest in internet petition form, I'm glad the Canadian government had some sense in them to hold on to a future in tech innovation in Canada.

I can only imagine what kind of immediate effect this would have had on tech startups.

</comment><comment><USER27> Sometimes you have to ask yourself who are those people in charge? How did they get the job?

<comment><USER28> Depending on which people-in-charge you're referring to, generally by false- promises, cooperative back-scratching, and a proven ability to squeeze dollars from sand.

It really shouldn't be so surprising, all things considered.

</comment></comment><comment><USER29> I'm very happy, and I hope that a lesson has been learned and that people will not only be more vigilant about caps, but also realize that Canada is far behind when it comes to broadband (speed, prices, competition, etc).

</comment><comment><USER30> I wish there was UBB that was remotely based on the transfer cost as opposed to some number that a C-Suite jackass picked out to make their profits SOOOOAAAAAAAR.

</comment><comment><USER31> This has not been overturned yet! I hate misleading titles.

</comment><comment><USER32> I have to wonder if the whole thing was just a big political maneuver to make the Conservative Party look good right before calling an election.

</comment><comment><USER33> Woohoo - Australia's back on top!

<comment><USER34> You mean New Zealand, right?

<comment><USER35> New Zealand is rather isolated geographically, it's expensive to build infrastructure on the periphery of the network. This is true for the South Pacific islands in general.

This is a snapshot of the current undersea infrastructure in the Australia-New Zealand area: <http://i.imgur.com/K1t4y.png>

<comment><USER36> Right. And the expense is(/was?) typically picked up by the isolated edge node rather than shared between the two end points.

ie: There's a lot more value to New Zealand in connecting to the rest of the internet, than for the internet to connect to New Zealand.

</comment></comment></comment></comment><comment><USER37> Considering how bone headed the original caps were, I have a feeling this isn't over.</comment>


Oracle sued for alleged fraud against US gov - <AUTHOR> http://www.theregister.co.uk/2010/06/17/oracle_sued_by_us_gov/

<comment><USER2> Price discrimination is not fraud.

Selling you X widgets for Y dollars -- we both agree to those amounts -- is not fraud. And it's still not fraud no matter what I sold widgets to someone else for. It's just a negotiated, voluntary contract.

Offering discounts to one guy and not another? Not fraud. Just a style of capitalism some people get offended by (hence one might want to not mention it).

The US Government being a chump that's bad at negotiating to get stuff cheaply? Not fraud (excepting politicians who made campaign promises to do better with no reasonable expectation of being able to do better ;)

<comment><USER3> There was a similar case last year (they settled):

[http://www.washingtonpost.com/wp- dyn/content/article/2009/04...](http://www.washingtonpost.com/wp- dyn/content/article/2009/04/14/AR2009041403007.html)

"GSA contract guidelines require companies to provide the government with accurate information about the discounts they offer to non-government commercial customers, helping the GSA to negotiate its best price."

<comment><USER4> Here's an interesting tidbit I wasn't previously aware of:

      "Mr. Kapuscinski is an amazingly courageous man," McKnight
      said, according to the law firm's news release. "He saw a
      company program out of sync with designated federal
      guidelines and decided to make it right."

      He is also much richer. Under the False Claims Act, a
      whistleblower who files a fraud complaint on behalf of the
      government can share in the recovery. Kapuscinski will
      receive a 15 percent share, or $19.2 million, according to
      the law firm.</comment></comment></comment>


As Tech Booms, Workers Turn to Coding for Career Change - <AUTHOR> http://www.nytimes.com/2015/07/29/technology/code-academy-as-career-game-changer.html

<comment><USER2> Comments moved to [https://news.ycombinator.com/item?id=9964230](https://news.ycombinator.com/item?id=9964230).</comment>