The Unbearable Complexity of Climate - <AUTHOR> http://wattsupwiththat.com/2009/12/27/the-unbearable-complexity-of-climate-2/

<comment><USER2> Um, is this the same guy? CFO of South Pacific Oil?

<http://www.linkedin.com/pub/willis-eschenbach/14/a54/935>

<comment><USER3> If it was, does that make his points any less valid?

<comment><USER4> IF this person is pushing one side of an argument without revealing the large stake he has in that side winning, then he loses credibility. He wasn't honest about something one would expect him to be honest about and so one needs to look more closely at other things he says.

Journalist disclose their interests and their companies interests to avoid losing credibility this way.

<comment><USER5> > IF this person is pushing one side of an argument without revealing the > large stake he has in that side winning, then he loses credibility.

You mean like researchers who get money to "confirm" AGW?

> Journalist disclose their interests

Oh really? Do you know how many NYT reporters are married to or involved with folks they cover? (Hint - the answer is not 0. The NYT's response when asked is that they've cleared things with their editor....)

<comment><USER6> Do you have any actual evidence that climate researchers are paid "to confirm AGW"? (I take it that by that you mean that if they published work, of equal quality to what they actually publish, that cast doubt on AGW then they would get paid less or fired.)

Note 1: saying "climategate" doesn't constitute presenting evidence; pointing to something in the pilfered emails that says (e.g.) "Of course I have to fudge this because otherwise I'll lose my job" and that isn't clearly a joke would constitute presenting evidence.

Note 2: observing that very few people who get paid to do climate science are denying AGW doesn't constitute presenting evidence either. Very few people who get paid to do biology are denying evolution, but that isn't because they're paid to confirm evolution; it's because evolution is, in fact, correct, and studying enough biology to get paid for doing it usually leads people to realise this. So you'd need to rule out that sort of explanation in the case of climate science.

<comment><USER5> > Do you have any actual evidence that climate researchers are paid "to > confirm AGW"? (I take it that by that you mean that if they published work, > of equal quality to what they actually publish, that cast doubt on AGW then > they would get paid less or fired.)

What do you think that rigging the peer-review process accomplished?

> it's because evolution is, in fact, correct, and studying enough biology to > get paid for doing it usually leads people to realise this. So you'd need to > rule out that sort of explanation in the case of climate science.

Are you asserting that AGW is as "settled" as evolution? If not, the analogy doesn't hold and neither does the conclusion that you'd like to impose.

Climate researchers outside of CRU are acknowledging that the thumb was on the scale. Is denying that really the hill that you want to defend?

In science, being a "good person" doesn't count as evidence. Neither does "the folks who disagree are bad people". AGW may turn out to be real (although that's looking unlikely), but it will take us much longer to find that out because "good people" behaved badly and did cargo cult science. Which reminds me - why do we consider them good people?

In addition to rigging peer review and trying to stack grant committees, it's pretty much SOP to add some blather about "consistent with AGW" to almost anything these days, in part because that's how the money flows. Of course, this isn't new. We saw the same thing with AI and "computers".

<comment><USER6> I think it's extremely unclear to what extent anyone (1) attempted to rig the peer review process and (2) succeeded. In any case, even a very badly rigged peer review process would not make it true that climate researchers are "paid to confirm AGW" unless everyone knows that the process is rigged. I think _that_ is clearly not so.

No, I am not asserting that AGW is as settled as evolution. I'm pointing out that "everyone being paid to work on this agrees with X, which shows that you can't get on in the field without agreeing with X, which shows that there's a bias against non-X" is a lousy argument, because in the case X=evolution it produces a result that is (to reasonable people) not credible. Note for the avoidance of doubt: I was attempting to pre-empt that argument, not accusing you of having made it.

(The cases X=evolution and X=AGW are somewhat parallel, though certainly not identical. In both cases there is a robust scientific consensus challenged by a minority. In both cases at least some of the challenging minority have an obvious reason for making the challenge that doesn't have much to do with truth. In both cases the challenging minority has strong ties to a particular political group -- the same group in both cases, as it happens. In both cases the challenging minority accuses the majority of having similarly non-truth- directed motivations, and in both cases it's not terribly clear what those are really supposed to be.)

I don't know what non-CRU people you are referring to, nor what acknowledgement they've made that "the thumb was on the scale", so I can't comment on your statement about that.

The stuff about "good people" and "bad people" is a red herring; neither I nor anyone upthread made any claim even slightly resembling "these people are Good and those are Bad, therefore these are Right and those are Wrong". Eschenbach has a clear incentive to do whatever it takes to make AGW look less likely; that's true whether he's a good person, a bad person, or (like maybe 99.9% or so of the human race) a mixture of the two. And, unless he is an incorruptible saint concerned with nothing but truth, it is rational to apply some discount to things he says that make AGW look less likely: not because he is a bad person and bad people never tell the truth, but because he is a (probably quite rational) person with a strong incentive to say such things even if they are wrong or misleading.

Similarly, no one (I hope) is saying that mainstream climate scientists must be right because they are good people and good people never say false things. But I will happily make the following argument: Imagine someone in the field of climate science who is inclined to put his or her own interests ahead of truth, and who doesn't mind doing bad or dishonest science for that purpose. Do you think that person is more likely to turn to Shell or the AEI or the Republican party, or to Greenpeace or Al Gore? Where is the readier flow of money likely to be found?

Is your last paragraph just vague handwaving, or is there an actual argument there? On the face of it it looks almost like you're suggesting that the reason why climate scientists almost all defend AGW is that people writing papers on other subjects tend to say "and by the way this is consistent with AGW" whether it is or not, but every part of that looks very silly so you probably have something more sensible in mind.

<comment><USER5> > In any case, even a very badly rigged peer review process would not make it > true that climate researchers are "paid to confirm AGW" unless everyone > knows that the process is rigged.

Not at all. Peer-reviewed publications lead to grants. Shut off the former and you strongly influence the latter.

> No, I am not asserting that AGW is as settled as evolution.

If AGW is not settled, the funding disparity is "interesting". There's clearly a bias, the only question is its reason. Since it isn't science....

> The stuff about "good people" and "bad people" is a red herring; neither I > nor anyone upthread made any claim even slightly resembling "these people > are Good and those are Bad, therefore these are Right and those are Wrong".

Good people vs bad people not a red herring. It's the basis for much of the climategate defense. "Those good scientists were beset by bad people, so naturally/it's okay that they did bad things."

> Imagine someone in the field of climate science who is inclined to put his > or her own interests ahead of truth, and who doesn't mind doing bad or > dishonest science for that purpose.

You assume that money is the only interesting motivation, and that's simply not true. It isn't even the dominant motivation among folks doing bad science - true belief is.

> Do you think that person is more likely to turn to Shell or the AEI or the > Republican party, or to Greenpeace or Al Gore? Where is the readier flow of > money likely to be found?

Al Gore and Greenpeace don't fund research and neither does the Republican Party. However, the folks that do have spent far more funding AGW than anything else.

So, if you're purely motivated by money, you'd have gone with "AGW is happening" because that's where the money is, as you agreed upstream. (You only argued about why that's where the money is.)

> you're suggesting that the reason why climate scientists almost all defend > AGW

"almost all" turns out to be way too strong. Moreover, the basis for their support is often data from other fields where many of the folks involved disagree with that interpretation.

However, my point was that much of of the "AGW is happening" literature consists of quips in other fields where the connection is tenuous at best. They're studying something else and threw in an AGW comment because that's an easy way to get some points. (I'm ignoring the bogus cases like the islands off India that have been sinking pretty since anyone started paying attention.)

As I wrote, we see this all the time. When something looks like a winning story, folks adopt it. Remember "turbo"?

<comment><USER6> Either you have misunderstood my point about peer review and being-paid-to- confirm, or I have misunderstood what you mean about being "paid to confirm AGW". I would only say that someone is paid to do X if they understand that their job depends on their doing X. A secretly rigged process would not have that effect.

AGW is not as settled as evolution, but it's still pretty damn settled. I think that is sufficient to account for the disparity in publications. (As for disparity in funding: disparity in total funding is a consequence of the fact that the great majority of climate scientists are AGW believers; if there's a disparity in funding per person then I haven't heard of it, and I'd rather suspect it's in the wrong direction for your argument.)

I don't believe I have ever heard anyone defending the CRU people, or the climate science community in general, or the proposition that the climate is warning and much of that is the result of human activity, say anything like "Those good scientists were beset by bad people, so naturally/it's ok that they did bad things". I have heard "Those scientists were being harassed, so naturally they did bad things", but that isn't a claim about good people or bad people. In any case, the discussion here wasn't (last time I looked) about Things People Have Said About The CRU Emails, but about whether it's correct to say that climate scientists are "paid to confirm AGW".

I do not assume, or believe, that money is the only interesting motivation. The question was whether climate scientists are "paid to confirm AGW"; I was not the person who framed the matter in terms of money. I do think, though, that money is an extremely common motivator for dishonest work, and that when people claiming some kind of AGW fraud conspiracy actually deign to suggest why there would be one it's usually money that they suggest.

The question about funding wasn't "who funds research on this stuff?", but "who would fund unscrupulous research on this stuff?". If it's easier to get your dishonest research paid for by one side than the other, then that's the side that will likely attract a higher proportion of dishonest research.

(So, drawing together two of the above points: what determines which way an unscrupulous person who cares only about the money will jump is not the total amount of funding obtained by people on each side, but the amount of funding per person available to unscrupulous people on each side. I don't see any reason to think that that disparity works in favour of AGW; I suspect it works strongly in the other direction, but that's just a guess.)

I dispute your claim that "almost all" is way too strong. See, e.g., <http://tigger.uic.edu/~pdoran/012009_Doran_final.pdf> which surveyed a bunch of earth scientists and found ~90% agreement among the whole population and ~97% agreement among those who (1) list climate science as their area of specialization and (2) have climate change as the subject of more than half their recently published papers. (I don't know where those criteria came from, and therefore can't rule out the possibility that they were dishonestly chosen to get the result the authors wanted; but I have no reason to think they were.) Or <http://scienceblogs.com/strangerfruit/Oreskes2007.pdf> which looks at the abstracts of 928 papers found by searching the literature for "global climate change".

I'm afraid I still don't really understand your point about "quips in other fields". Maybe it's because I don't spend much of my time reading scientific literature from randomly chosen fields (and the fields I'm competent to read scientific literature in aren't terribly close to climate science) that I haven't seen this alleged torrent of quips that (it seems like you're saying) is somehow responsible for climate scientists' belief in AGW. I really don't see how that could work, but perhaps I'm being dim.

I don't remember anything called "turbo" that illustrates that "when something looks like a winning story, folks adopt it"; sorry.

<comment><USER5> > Either you have misunderstood my point about peer review

You're "forgetting" that the AGW folks fixed the peer review system. I know that that's an "inconvient truth", but ....

> AGW is not as settled as evolution, but it's still pretty damn settled.

Then you're not paying attention. The tree data "trick" means that the "before thermometers" estimates by the AGW folks are wrong, which guts the trend argument. The "no warming since 2000" means that the models are wrong. The accounting for how temperature stations were moved is broken. The AGW folks mis-represented the Russian data. It goes on and on.

> I would only say that someone is paid to do X if they understand that their > job depends on their doing X.

Since one can get the desired effect without doing that, that's an absurd standard. You pick people based on whether you think that they've bought in and are not going to rock the boat. You never have to say anything.

That's part of why I mentioned "true believers".

> I haven't seen this alleged torrent of quips that (it seems like you're > saying) is somehow responsible for climate scientists' belief in AGW.

I didn't say that it was "responsible", I said that it constituted much of the supposed consensus. The amount of research that actually concerns AGW is fairly small. However, there's lots of research where it can seem relevant. Since AGW is "hot", there's a huge temptation to just throw in a "consistent with AGW" comment that isn't actually supported by the research being reported.

This happens all the time in every field. Making a tenuous connection to a hot topic is a common way to jazz up a paper.

<comment><USER6> No, I am not '"forgetting" that the AGW folks fixed the peer review system', I am explaining why _even if that's true_ it doesn't mean that AGW people are "paid to confirm AGW". Also, so far as I can tell it isn't true for any sensible meaning of "fixed". But, no doubt, if you repeat the claim often enough without providing any evidence, it will become true.

The tree data trick does not mean what you say it does; see [http://www.pnas.org/content/early/2008/09/02/0805721105.full...](http://www.pnas.org/content/early/2008/09/02/0805721105.full.pdf+html) (key finding from conclusions section: "We find that the hemispheric-scale warmth of the past decade for the [Northern Hemisphere] is likely anomalous ... This conclusion appears to hold for at least the past 1,300 years ... from reconstructions that do not use tree-ring proxies, and are therefore not subject to the associated additional caveats."). Temperatures vary a lot on a timescale of a few years, so the much-ballyhooed "no warming since 2000" doesn't in the least mean that the models are wrong; see [http://www.realclimate.org/index.php/archives/2009/10/a-warm...](http://www.realclimate.org/index.php/archives/2009/10/a-warming- pause/) and also figure 4 in [http://www.columbia.edu/%7Ejeh1/mailings/2009/20091216_Tempe...](http://www.columbia.edu/%7Ejeh1/mailings/2009/20091216_TemperatureOfScience.pdf) (and the text below that figure). Your claim about "accounting for how temperature stations were moved" is too vague to respond to. The AGW folks did not "misrepresent the Russian data" so far as I can tell; see <http://news.ycombinator.com/item?id=1000732>.

Do you have any actual evidence that climate scientists "pick people based on whether [they] think that they've bought in and are not going to rock the boat"?

I simply do not believe your claim that throwaway "this is consistent with AGW" remarks in papers on other topics "constitute[s] much of the supposed consensus". Are you suggesting, e.g., that a large fraction of the papers in Oreskes's study were really about other topics and had only throwaway references to AGW (it sure doesn't seem like it from her description), and that if those papers were excluded then the study would find much less acceptance of AGW (it sure doesn't seem like it from her description)?

I'd like to point out that you have a nice two-way argument here. If papers by climate scientists support AGW, that's because climate scientists are all "true believers" who have been picked based on whether they've bought in and aren't going to rock the boat. If papers by other people also support AGW, that's because they've just slapped brief pro-AGW remarks onto papers about other things. So, I wonder, is there anything the peer-reviewed literature _could_ contain that you would regard as good evidence that careful scientific study supports AGW?

<comment><USER5> > I am explaining why even if that's true it doesn't mean that AGW people are > "paid to confirm AGW".

What part of "you get grants for peer-reviewed publications" don't you understand?

> Also, so far as I can tell it isn't true for any sensible meaning of > "fixed".

The folks who wrote the "ClimateGate" e-mails disagree, but then they did the fixing.

> So, I wonder, is there anything the peer-reviewed literature could contain > that you would regard as good evidence that careful scientific study > supports AGW?

This fixation on "peer-review" is interesting. You claim that the "peer"s all agree, so their review is subject to all that that implies. You might remember <http://www.wired.com/magazine/2009/12/fail_accept_defeat/> .

The climate people have threw away raw data, so we can't know whether "normalization" was actually reasonable. They're still saying "trust us" even though every time someone finds an error in their work, the error goes the same direction....

BTW - Scientific truth isn't defined by consensus.

<comment><USER6> Everything you have said here I think I have already answered earlier, with the following exceptions.

It was you, not me, who introduced peer review into the discussion; I've no idea what your basis is for talking about a "fixation".

Most of the raw data that the CRU threw out in the 1980s are, AIUI, still available elsewhere. CRU's model produces results extremely similar to everyone else's models.

I don't think it's true that every time an error is found it goes in the same direction. (Specifically, I think I remember seeing a counterexample. I don't remember what it was, but then you haven't mentioned any specifics of anything at any point so I don't feel too bad about that.) It might be true that every time an error is found and trumpeted about by disbelievers in AGW it's in the same direction, but there is an obvious explanation for _that_ , no?

Of course scientific truth isn't _defined_ by consensus, and of course I neither said nor suggested that it is. But for someone who isn't an expert in any scientific field, the consensus of experts is typically pretty much the best estimator they have for the truth.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> Saying, "I don't understand what's going on" does not mean that nobody understands what's going on.

PS: If you actually carried out the experiment using the picture as a guide you would measure a rise in body temperature. The human body trys to maintain constant temperature, but it can't ignore the laws of physics.

<comment><USER8> Amen. This is a long opinion piece, with no evidence cited that the Constructal Law is a better predictor of the climate's response to increased CO2 than existing techniques. The one relevant paper linked to is an intriguing but underdeveloped model:

[http://homepage.mac.com/williseschenbach/.Public/Constructal...](http://homepage.mac.com/williseschenbach/.Public/Constructal_Climate.pdf)

The paper _doesn't_ back up the author's claim that "there is no physics-based reason to assume that increasing CO2 will make any difference to the global temperature". It includes an assumed factor y=0.4 for the infrared reflectance 'greenhouse factor', and any increase in that value _will_ affect the expected results.

If anything applying the Constructal Law would be more worrying than current models, since there's potentially a more chaotic response to increased CO2 levels, increasing the risk of dramatic climate changes.

<comment><AUTHOR> From a recent paper by two German Physicists

(a) there are no common physical laws between the warming phenomenon in glass houses and the fictitious atmospheric greenhouse effects, (b) there are no calculations to determine an average surface temperature of a planet, (c) the frequently mentioned difference of 33 degrees Celsius is a meaningless number calculated wrongly, (d) the formulas of cavity radiation are used inappropriately, (e) the assumption of a radiative balance is unphysical, (f) thermal conductivity and friction must not be set to zero, the atmospheric greenhouse conjecture is falsified.

<comment><USER4> I think you got voted down because you don't name the physicists or the journal they published in.

Also, the phraseology of your points couldn't get into any scientific journal, so what are you quoting, your summary?

<comment><USER4> Actually,

I think you are talking about

<http://arxiv.org/abs/0707.1161v4> \- It's referenced by Alexander Cockburn in his most recent nation article.

It's arxiv with no journal listed as publishing it. You do correctly quote the abstract but the language of this abstract is pretty overwrought - it doesn't read like something that would be published in a serious journal whether it's correct or not. Part of the problem with the language might be a poor translation.

Climatology is a complex thing and I'm actually sympathetic to exploring skepticism about global warming. The problem is that the skeptics seem to often be their own worst enemies.

</comment></comment></comment></comment></comment><comment><USER9> "Because climate is a flow system far from equilibrium, it is ruled by the Constructal Law. As a result, there is no physics-based reason to assume that increasing CO2 will make any difference to the global temperature, and the Constructal Law gives us reason to think that it may make no difference at all."

Sorry, but that is stupid. Just because changing one thing (ie making a short cut in the river) doesn't change one other thing doesn't imply that no matter what you change, nothing will change. Also, the author never seems to have seen a redirected river in a bed of concrete.

It's been nice to have learned about the constructional law, but other than that, I have to take this article with a grain of salt. It clearly doesn't seem to be neutral in motivation.

</comment><comment><USER10> is this kosher?

it starts out fine, explaining how complex systems are, well, complex. but then it starts to seem a bit odd because it ignores statistical properties (you can't predict the movement of every gas molecule in a box, but you can know the average temperature pretty damn accurately). and then suddenly it jumps to "constructal law":

> The Constructal Law applies to all flow systems which are far from > equilibrium, like a river or the climate.

wtf? how can something as vague as "the climate" follow a law? and what's the constructal law? and so you go to wikipedia - <http://en.wikipedia.org/wiki/Constructal_theory> \- and it all starts to look a bit flakey...

[edit: less flippant]

<comment><USER11> Adrian Bejan, the guy who created/discovered Constructal Theory doesn't seem to be flakey at all: [http://hcr3.isiknowledge.com/author.cgi?&link1=Search...](http://hcr3.isiknowledge.com/author.cgi?&link1=Search&link2=Search%20Results&AuthLastName=bejan&AuthFirstName=adrian&AuthMiddleName=&AuthMailnstName=&CountryID=-1&DisciplineID=0&id=533) and <http://www.mems.duke.edu/fds/pratt/MEMS/faculty/abejan/>

<comment><USER11> Although the "James Harry Potter Gold Medal" sounds kind of weird, it is a real award: [http://www.asme.org/Governance/Honors/SocietyAwards/James_Ha...](http://www.asme.org/Governance/Honors/SocietyAwards/James_Harry_Potter_Gold_Medal.cfm)

<comment><USER12> "For outstanding snitch catching on the Quidditch pitch, I present to you..."

</comment></comment></comment></comment><comment><USER13> Heh.

The human body is a 'flow system far from equilibrium.' So while it's true that if you dunk the feet of a healthy person in hot water and wait, the temperature of their forehead may not increase by that much, it does not imply that the temperature of their feet won't increase, it does not imply that the forehead temperature of a hypothermic person won't increase, and it does not imply that the forehead temperature of a healthy person exposed to a flamethrower will not increase...

Does the Constructal law say anything about the environment were the Sun to suddenly disappear?

</comment><comment><USER14> I wonder if Mr Eschenbach realizes that the effectiveness of America's entire nuclear arsenal relies on the computer models he so casually dismisses. Nuclear weapons testing is virtual, and has been since 1992.

Contrary to the article, climate models don't assume a relationship between CO2 and global warming; rather, they reflect what we know about the physical properties of CO2 in the atmosphere, as born out in laboratory experiments and field measurements. The relationship with global warming emerges from the models, not the other way around.</comment>


Google's smart city dream is turning into a privacy nightmare - <AUTHOR> https://www.engadget.com/2018/10/26/sidewalk-labs-ann-cavoukian-smart-city/ <comment><USER2> If you cut out the editorializing and just focus on the example presented, it seems totally reasonable:

1\. As a private company, they don't believe they should be responsible with setting public policy. They want a separate board to be set up for that.

2\. By default, everything should be private.

3\. They're proposing that if an applicant has a compelling reason to access raw data, they need to have it approved by the governing board, and raw data is automatically destroyed after a short interval.

They gave the example of video feeds, which are both common in public spaces and are not anonymous (even if you blur faces). If they're being thoughtful and responsible, they need some mechanism to handle desirable exceptions. This sounds like a proposal to do so.

In the article, the person who resigned is presented as a respected expert on the subject. Maybe that person has more context that didn't make it into the article. But the facts presented - "group designing public space is proposing policy to handle aspects that can't easily be made anonymous" \- don't warrant the clickbait title or the editorialization.

Sounds like Sidewalk is doing it's diligence - thinking about the things a group designing a "city of the future" _should_ be thinking about.

<comment><USER3> The problem with #3 is that they _already_ doing that with the fisa courts and it's a shit show. In our current system, they rubber stamp every warrant that is presented to the court with no oversight[0].

[0]:[https://www.eff.org/deeplinks/2013/06/president-obama- called...](https://www.eff.org/deeplinks/2013/06/president-obama-called-fisa- court-transparent-despite-it-being-shrouded-secrecy)

<comment><USER4> Toronto, the city mentioned in the article, is in Canada. "Fisa courts" are a U.S. entity which has nothing to do with what is being done in Canada.

<comment><USER5> Until the data gets sent to US servers.

<comment><USER6> ^This. "5 eyes" Canada, US, UK, NZ, and AUS

<comment><USER7> and so even if the fisa courts said no, the data would still arrive

</comment></comment></comment></comment></comment></comment><comment><USER8> To be honest, I don't quite understand what smart city even means. This term is so board and include so much different aspect of life that I am not sure what it includes. It is a cool marketing term, but does not explain too much what is happening.

So, I have read some of the use cases. One of them is putting a sensor in a garbage bin, so that the sensor could take picture of the garbage and determine where they should go. I don't think this is smart. Japan has already been very efficient on garbage collection, because Japanese people just separate different type of garbage themselves, and fine people who don't. A policy could just fix this in a cost effective way, not some sensor. But the spirit here is placing a camera to see what it could do.

Some other project like improving the accuracy of water meter to detect water usage and pinpoint leaks. They admit it already exists, and only improvement is required.

I could conclude that it is a debate about who could put an IoT or sensor in the public area. Google just wants to place those IoT in the city to see what they could do with it. Basically a web crawler in physical world. The government may also want to create a hype so that people would approve infrastructure funding to improve existing infrastructure people don't think it's that smart.

Am I missing something here?

<comment><USER9> I just assumed it would be a more Orwellian version of the old "Company Towns." In addition it would be a big laboratory to test what people will accept and how much personal information can be gleaned from those sensors and related analysis.

</comment></comment><comment><USER10> To me, the word "smart" applied to anything but a human has now gained a negative connotation: "it's smarter than _you_ and can work _against_ you." Plenty of dystopian science fiction to support that view too, but scary that now it's actually becoming reality.

<comment><USER11> I think there's two ways things can be smart without being invasive. Either it can be smart, but not personalized, such as monitoring general traffic flow and making changes to accommodate (some express lanes on highways, for example, switch direction based on the time of day), or it can be personalized, but private to you. Most smarthome devices wouldn't be a problem if they remained entirely onsite in your possession. Obviously the former is a way a city could be smart, but the latter is really not.

I want a smart city to shut off lights that aren't illuminating anything or to optimize trash collection routes based on efficiency. I don't want a smart city watching where I, personally, go and suggesting I do things based on that.

<comment><USER10> _some express lanes on highways, for example, switch direction based on the time of day_

 _shut off lights that aren 't illuminating anything_

These are examples of existing technology which most people don't consider "smart" \--- the word is reserved more for the new technologies that also contain pervasive data collection.

<comment><USER11> You could automate these sorts of processes using data collected live, just in aggregate. Rather than say, a simple light sensor on a street lamp, you could note where cars are on the road, and illuminate a series of them ahead of any given vehicle so they can see in front them. Or a road could dynamically reconfigure itself based on traffic flow, rearrange lanes around accidents, etc. These are processes that would require data collection, processing, and automated control of a equipment along the road.

My point isn't in the specific examples but that there are smart, computerized features that absolutely could be built that may improve lives or create efficiency that don't involve personal profiling.

</comment></comment></comment></comment><comment><USER12> It's curious that there hasn't been any backlash against non-tech companies for this stuff. E.g. something like airmiles, where your entire purchase history is sent to advertisers. No one gets up in arms, despite how sensitive that data is.

That's not to say people shouldn't be concerned with their privacy, but it's a weird double standard.

<comment><USER13> The problem is that if airlines ever takes away your airmiles points because your bought a leonardo di caprio fake poster from aliexpress I wouldn't miss them at all. I would just delete their email and move on.

The issue with Google is that they can ban you from all their services for "trying to act oversmart". I know startups that had their whole gmail/drive/docs suspended and everything purged because one of their dev had a banned account. My colleague had his google account banned because he resold the pixel ([http://fortune.com/2016/11/17/google-pixel- ban/](http://fortune.com/2016/11/17/google-pixel-ban/))

I had one account banned because they did not allow paypal after they launched google wallet. And they banned a second account for being "related to first account"

Because google does not have customer support for gmail/drive they run amok with the ban hammer. If you don't mind being suspended for all google accounts for absurd reasons then yes you are right, this is a non-issue.

</comment></comment><comment><USER14> I don't think it's possible for a private company to have the public's interest at heart, at least in the long term. Especially for Google, who have shown a lack of respect and privacy for its users.

<comment><USER15> > I don't think it's possible for a private company to have the public's > interest at heart, at least in the long term. Especially for Google, who > have shown a lack of respect and privacy for its users.

While I agree in general with this sentiment, do you think non-profit corporations are in the same boat? Or something like the FDIC?

<comment><USER16> >do you think non-profit corporations are in the same boat?

There are really bad non-profits, I've worked at them. However, endless search for endless profit by shareholders, CEO's, and executives is usually the cause of a loss of morals and ethics. I think it's nearly impossible for very large corporations making profits to do actual good for society except give us endless growth with products and services we really don't need. This doesn't mean a small local bakery taking in profits isn't doing the community good, but extremely large corporations don't benefit the same way.

<comment><USER17> [https://www.npr.org/2014/10/29/359365276/on-superstorm- sandy...](https://www.npr.org/2014/10/29/359365276/on-superstorm-sandy- anniversary-red-cross-under-scrutiny)

The Red Cross is worse than many for profits. If a for-profit doesn't deliver the goods, they don't get paid: they are incentivized to do what their customers want. The Red Cross is incentivized to bring in donations even at the expense of their stated purpose. Look at the salary of the Red Cross CEO — a salary paid by tax deductible donations. Look at the corrupt nonsense engaged in by many universities. To suggest non-profits have any better or worse ethics than for profits is a vast over simplification. The Clinton Foundation even paid for Chelsea Clinton's wedding as well as private jet transport by principles. Even Greenpeace's leader was commuting by airplane.

<comment><USER18> Well, it's actually quite easy: if you care about profit, your morals are necessarily compromised. You are right that non-profits have other incentives as well, but expecting a for profit organization to have morals is inherently going to disappoint your expectation every time.

</comment></comment></comment></comment></comment><comment><USER19> I'm pretty much convinced that the greatest threat to true freedom is pervasive monitoring. It is impossible to resist any powerful force when they know everything you say and do. They can simply undermine any activity. I realized this when my buddies were telling me about their work on facial recognition for WalMart stores. And then read about how WalMart finds reasons to fire everybody who had contact with union organizers. That is just the tip of the iceburg, as the powerful will get better and better at both monitoring and control - without our awareness that we are being controlled.

<comment><USER20> Also the studies on how pervasive monitoring causes self-censorship. Probably everyone here has had the experience where they've typed out a comment, then thought about whether they want that on their "permanent record", then decided not to post. It's insidious.

</comment></comment><comment><USER21> The whole point of these devices is data harvesting. The Times op-ed last September titled These Are Not The Robots We Were Promised [0] covers the discrepancy between our expectations of personal assistants and the reality they came to be.

[0] [https://www.nytimes.com/2017/09/09/opinion/sunday/household-...](https://www.nytimes.com/2017/09/09/opinion/sunday/household- robots-alexa-homepod.html)

<comment><USER17> "These devices.."

Except HomePod. Alexa and Google Home harvest data; HomePod does not. The purpose of HomePod is to sell HomePod, other Apple devices and subscriptions to Apple Music. The purpose of Alexa and Google Home is to learn about you in order to sell or exploit that information.

It's a critical difference. People complain about how Siri isn't as good as XYZ, yet simultaneously complain about the privacy implications of the "good" assistants.

</comment></comment><comment><USER22> Any "smart city" is going to turn into a privacy nightmare as soon as it goes beyond the depersonalized level. I.e. it should analyze traffic patterns and optimize traffic lights but should never know who exactly is and is going where. Personalized features should not be offered even on voluntary opt-in basis as this will make the people who don't opt-in a target for prejudice.

</comment><comment><USER23> Google isn't really evil, they just do what they can get away with. In the European Union that isn't a whole lot. In the US there are no limitations. And god only knows what China will allow.

<comment><USER24> "doing whatever they can get away with" feels like a pretty reasonable definition for at least one type of evil in my book.

<comment><USER20> law-abiding evil.

</comment></comment></comment><comment><USER16> People go to work in what are essentially private authoritarian governments, now they want to fully live under one too? This reads like an Orwell book. The fact that such a major privacy issue is arising so soon should doom the project for good.

<comment><USER25> Also, the fact that people blindly accept that companies somehow still have to work like authotarian goverments from old ages is mind boggling.

Especially in the US, there is a massive lack of employee discourse inside the company.

<comment><USER16> When you have almost 80% of the population living paycheck to paycheck [1] with practically nobody in that percentage with a significant savings [2], and workplace laws where you can be fired for literally anything, you can easily stifle workplace unionization efforts and discourse. People literally cannot afford to do anything that would benefit them.

This article is about Canada, but it's really becoming a worldwide issue.

[1] [https://www.cnbc.com/2017/08/24/most-americans-live- paycheck...](https://www.cnbc.com/2017/08/24/most-americans-live-paycheck-to- paycheck.html)

[2] [https://www.cnbc.com/2018/01/18/few-americans-have-enough- sa...](https://www.cnbc.com/2018/01/18/few-americans-have-enough-savings-to- cover-a-1000-emergency.html)

<comment><USER25> It also seems like a cultural issue to me. Americans seem so opposed to the idea of labour unions and workplace councils, because it somehow "hurts companies"?

it is probably for historical reasons of the cold war, but fascinating nonetheless.

<comment><USER16> It's definitely somewhat anti-left sentiments built into our culture/society through means such as mccarthyism and anti-communism, as well as purposeful lack of education. It's also our media and entertainment. These tactics can be further learned about in Chomsky's 'Manufacturing Consent'.

</comment></comment></comment></comment></comment><comment><USER26> "Smart" stuff is great, but really requires third party governance and regulation.

It's possible, just based on what you can buy for commercial smart building systems. But it adds cost as you cannot sell devices at a loss and farm revenue based on selling data.

</comment><comment><USER27> So it's pretty safe to say this vid from The Onion made a few years ago has become reality.

[https://www.youtube.com/watch?v=lMChO0qNbkY](https://www.youtube.com/watch?v=lMChO0qNbkY)

What a time to be alive.

<comment><USER28> The Onion was also prescient about the end game for surveillance based companies:

[https://www.youtube.com/watch?v=w8c_m6U1f9o](https://www.youtube.com/watch?v=w8c_m6U1f9o)

(this video is especially interesting in light of Google's recent involvement in Chinese censorship)

<comment><USER27> Holy shit, thats a good one aswell. Bookmarked. Thanks!

</comment></comment></comment><comment><USER29> Here's an interesting discussion involving many of the players directly involved in the project. Ann Cavoukian from the Engadget article is on this.

[https://youtu.be/-2C2_ulRrXE](https://youtu.be/-2C2_ulRrXE)

Actually, what's really interesting is to compare her outlook from earlier this year [https://youtu.be/oEyBjYNgaMY](https://youtu.be/oEyBjYNgaMY)

</comment><comment><USER30> > An independent trust, the team argues, would be better equipped to make > these decisions -- even if they allow other companies to collect personally > identifiable data.

Why should anyone trust an "independent trust"? That trust will be staffed by humans, subject to all human failings and conflicts of interest.

</comment><comment><USER31> Don't many cities already have govt cameras with no anonymous-ization? Traffic cameras feed into CalTrans in California.

For cameras in public places, the video shouldn't be used for profit. But should the police have access to the raw footage? Amber alerts? Terrorism?

</comment><comment><USER32> turning into? it was doomed from the start. this kind of issue was completely predictable.

</comment><comment><USER33> A google led smart city project and they were expecting privacy by design?

Either very naive or very stupid...

</comment><comment><USER34> I read the title and immediately thought: 'duh!'

</comment><comment><USER35> Wow, what a surprise with a privacy focused company like Google.

</comment><comment><USER36> Turning into? That's assuming Google didn't intend it to be that _by design_.

</comment><comment><USER37> That's what Google does. They're spending money trying to find the next thing...their interests do not align yours.</comment>


Show HN: MediDate – Yoga Dating app - <AUTHOR> http://www.medidatewith.me/ <comment><USER2> I count at least 11 different styles of text on this small page, with titles and text changing color. Looks unprofessional.

You might also want to use something other than blue color for text, as it looks like a link.

</comment><comment><USER3> The styles on this page looks like a fanpage for britney spears in 1997</comment>


Will successful Self-Taught Programmers please stand up? - <AUTHOR> I'm sure John Carmack and a few other famous programmers are not the only cases of self-taught programmers that have been successful (whatever the degree). Please tell us how it is being a self-taught programmer. Do you have a startup? Do you work for Google / Facebook / Apple / ...? Do you work on your own, or in someone else's company, and are as successful and acknowledgeable than your CS colleagues?<p>Please tell us your story. <comment><USER2> I'm self-taught, starting with C on MS-DOS at age 9 (I'm now 32). I moved onto Visual Basic (ugh) at 12 because my C compiler wouldn't do windows programs, and later moved to Borland C++ on Windows. I did later do a BTEC course in computer studies, but that was just to hedge my bets.

I Switched to Linux in 1995/96, and have spent the last 15 years alternating between systems administration and development roles. I've worked for large online retailers, government departments, multinational manufacturers as well as smaller content portal websites, and I'm currently at a search startup in Cambridge, UK. I can program in C, Perl, and PHP, but the language I love most is Python. I also love relational databases (PostGreSQL in particular), and have quite a strong DBA-streak.

I've never really noticed much tension between the CS-taught and self-taught programmers that I've known. I've often considered that a team is stronger for having both types represented. The self-taught programmers, if they started early, often have an additional 5-6 years commercial experience over the degree-holders (for their age), but it all balances out in the end.

In terms of success, I'm gainfully employed, I'm well regarded by my peers (it's one of the things that motivates me), I can still learn things, I can still teach things and I'm not in management. Win.

I hope that's a useful response for someone!

<comment><USER3> I'm in college, 20 years old. I've just started teaching myself ruby and python. Do you think I would be able to develop crude prototypes of apps by the time I graduate college? (in about 2 years?) I've been dedicating around 6-10 hours ever week so far and plan to dedicate more once summer starts.

<comment><USER2> I would certainly think so (the two languages you've mentioned are both regarded as being quite easy to get started with), but everyone is different and is interested in different things. If the demo app you're writing (and the framework you're using, if any) allows you to try something and see the results quite quickly, that's a good way to avoid getting discouraged.

Although frameworks (Rails, Django etc) are a good way to avoid writing boring or non-unique stuff, they do add to the amount of up-front learning you have to do before you can write _your_ programs (since you have to learn the language _and_ the framework at the same time). Trying out some noddy programs that just use stdin/stdout is a good way to learn the language features first and then the framework can follow afterwards.

Early code in any language is always horrible when you look back at it a few months later, don't beat yourself up :)

</comment></comment></comment><comment><USER4> I'm a self-taught programmer without a degree of any kind. I also code PHP on a Windows box. As far as fellow developers are concerned, I'm lower than scum :)

I'm successful in the way that I'm now earning a good living working for myself, have happy clients and get lots of referrals. Let's just say it moved me... to a bigger house.

<comment><USER5> Hello from a fellow self-taught, PHP-on-Windows developer. And I'm a girl, which automatically makes me special ;)

</comment></comment><comment><USER6> I'm self-taught, without a single credit hour of college. In high school a friend took a programming class and showed me a bit on a demo machine at radio shack, and that hooked me completely.

I've worked mostly for smallish companies, and did consulting for a while. A couple of years ago I had quit my job to work on my own project (a way too ambitious startup idea), but almost immediately I got an interview for my current job, which was too cool to pass up (space exploration). There's more than one way to make a difference in the world. Now I head up the dev team here.

Over the years I have spent a lot of time and money on books, and continue to do so. There's always something to learn, and many people way smarter than me continually creating/discovering new things to learn. If I had earned my degree in the normal time frame that would have graduated in the late 80s, and my education would now be woefully out of date, so I'd _still_ have to continually learn.

Am I recommending _not_ going to college? No. It's probably better to go. But certainly you _can_ have a career without it. But either way, get used to learning and pushing yourself.

</comment><comment><USER7> I'm primarily self-taught, and at the age of 21 I'm comfortable listing 8+ years of C/C++ development on my resume. I built my first web site when I was 11, and I started learning C when I was 13 so that I could add features to a multiplayer text-based game (MUD).

I spent a few years running a MUD, doing some coding competitions, taking on small freelance projects, and learning about data structures, networking, graphics, and other things. When I was 15 I created a full Win32-based 2D game engine for a school project. It even had a terribly designed scripting language for defining levels and game behavior. After that project I vowed to never touch Win32 development again.

When I was 16 they allowed me to take CS courses at our local university, and I thought I'd found my home. Unfortunately I realized most college students at our university don't really care about programming and are just there for the degree. We had no hacker culture, and I tutored several seniors who couldn't fix their own syntax errors.

I've been happily employed since I was 16. I've done freelance, small projects at our university, computer technician work, and embedded systems software development. When I was 19 I got recruited by Lockheed Martin to write flight control software for the FAA for a summer. It was amazing, but I didn't want to work at a big company in the long run, so I turned down the offer to return.

Last March I dropped my job and began working on my startup full time :) I should also note that I'm in college part-time as a CS/Economics double major.

I have to admit that most epic programmers I know didn't get a formal education in CS. Some have degrees in other fields, and some have no degree at all. I think passion for software and building things takes people further than a degree. However, most of the programmers I've worked with that had CS degrees did quality work and had a good understanding of CS concepts. They just rarely seem to be coding demons.

</comment><comment><USER8> I took one java class in college, but that was about it when I started learning javascript 3 years ago (along with html, and css). I carried around the OReilly javascript book for the summer - read it on the bus, on the beach, wherever.

For the last couple years, I have worked as a freelance web designer and front-end developer. Now I'm pleased to have just gotten a job with the title webmaster (for a section of a major news organization.)

I wouldn't call myself a JS ninja, but I'd say I can make a browser do some cool stuff with some pretty clean code. And I'm getting paid pretty well for it. And 3 years ago, I didn't know HTML. So, for anyone who wants to learn to code from scratch, Go for it! Go to your library's programming section, checkout something that looks 'elegant, yet friendly', and take it to the beach. Or if you want something right now, this should probably do nicely: <http://jqfundamentals.com/book/index.html>

Along the way: Listened to a couple podcast lectures, read a couple books, read hacker news every day (thank you!), fell in love with jQuery, picked up a little python, real a little about lisp, and found some start-up partners in New York to start hacking on some fun stuff.

</comment><comment><USER9> Self taught is a little difficult to define. I was a math major and took a few CS courses: intro, data structures and algorithms, and then math classes with a programming component (that often counted for elective credit toward CS): graph theory, numerical analysis, linear programming.

That said, I pretty much consider myself a "self taught" programmer, because I didn't take most of what I would consider the core of CS.

Can we define "self taught" as someone who didn't take a formal course in:

algorithms and data structures compilers operating systems ( _)

(_) must include a low enough level language that you are manipulating memory

By that standard, I'm self taught, as I'm missing the second two. I promise myself I'll get the book and read them carefully, but I still haven't done it.

That said, I suspect that someone who did take the entire full "core" would be pretty much as "self taught" where it comes to most professional programming. However, I still think they'd be better prepared than I was for a programming career, as compilers and operating systems is more relevant to professional programming than real analysis and abstract algebra.

</comment><comment><USER10> Primarily self taught. Started in sales. Since becoming a programmer I have received training from various companies that I've work for. I am working on my own projects and have been employed as a developer for just over a decade. So far, I've survived a few layoffs and the "all software will be developed off-short" panic. If I had to do it over again though, I'd probably be a CS major just to have had the earlier exposure to my peer group that has since gone on to do some pretty cool stuff. I live in the Bay Area, so I'm spoiled in that I have proximity exposure to some of the most active minds in software is often readily available through meetups and other venues.

</comment><comment><USER11> I started playing around with programming around the age of 13, but never anything too serious. After getting a degree in mechanical engineering, I applied for jobs in booth mechanical engineering and software development.

The software development jobs just seem like more fun and I went that direction.

I have been a software developer for 9 years now, the last 2 years as a successful freelancer and at the moment partial owner of a software consulting firm (couple of employees, but growing fast)

The lack of a CS degree never has been a problem. All the developers I work with value a person by knowledge and skill, degree is not that important.

</comment><comment><USER12> 1\. Define self-taught. By the time I got to college I had already been programing for five+ years. With respect to coding in general, college was mostly a waste of time, but it did forced me to look at things I would had never look at otherwise. (I got a Math/Comp Sci degree)

2\. Define successful. If it is about money, most IT workers in the US are already on the top 90% money earners in the planet, plus (in my experience) the more boring and un-challenging the job, the most it paid.

Maybe you mean famous? Lots of open source rock stars are not to rich in money terms.. are they successful?

</comment><comment><USER13> Does "successful" mean that you passed an interview and got a job as a programmer? Or something else? I guess I don't quite understand what you mean.

</comment><comment><USER14> I am primarily self taught( with some minor help from my father, when I was a child ). I don't work for any of the "cool" places you mentioned, but I do have a reasonably successful ecommerce business.

I did go to school, but I began programming tiny asm programs at ~8 years old out of my dad's 8086 book and my obsession with software expanded from there.

</comment><comment><USER15> If 'successful' means 'can code an application more secure than Windows', I guess I'm in there. Self-taught ColdFusion (go ahead, laugh), PHP and Python (my favorite). My staff uses the tools I've written to do their jobs. And we're still in business. Woo hoo!</comment>


The Man in Seat Sixty-One - <AUTHOR> http://seat61.com/

<comment><AUTHOR> It occurred to me that people might not know about seat61.com. If you're doing any rail travel in Europe this site is completely invaluable; one of those in- depth single subject sites that the web started off as.</comment>


The Dotty compiler for Scala bootstraps - <AUTHOR> http://www.scala-lang.org/blog/2015/10/23/dotty-compiler-bootstraps.html <comment><AUTHOR> Called "Dotty" because it is based on Dependent Object Theory (DOT). DOT has also just been given a big boost because researchers have found a way to prove type soundness (long term open problem). See [http://lambda-the- ultimate.org/node/5270](http://lambda-the-ultimate.org/node/5270)</comment>


Create an off canvas menu with CSS only - <AUTHOR> http://forthehackofit.com/create-an-off-canvas-with-css-only/

<comment><AUTHOR> Popularized by apps like Facebook on mobile, the off canvas menu allows to save valuable real screen estate by toggling the navigation into view only when needed. We will dive and explore one way of achieving the effect using some overlooked CSS properties.</comment>


Jedberg on Reddit's most recent outage - <AUTHOR> http://www.reddit.com/r/announcements/comments/gva4t/on_reddits_outage/ <comment><USER2> Reddit's software is not very impressive. They should really focus on consolidating to a more manageable setup. I'm aware that big sites often have complex environment to facilitate scaling, but frankly, reddit is a big monstrous mess. It'd be a lot simpler to move hosts if they would get some of those extraneous dependencies removed and put some simplicity and _sanity_ into their data storage, instead of the half Cassandra/half PgSQL thing they do now.

I wrote a long post about it here: [http://www.deserettechnology.com/journal/reddit-the-open- sou...](http://www.deserettechnology.com/journal/reddit-the-open-source- software) .

</comment><comment><USER3> Actually, there was a heated discussion on such an issue a month ago at Reddit [1], when AWS started acting up back then. Someone was asking why cann't Reddit buy beefy servers and co-lo them. The arguments were that considering the traffic Reddit is having, it's very difficult to DIY technically if not possible. Looking back, it looks like co-lo is still the way to go, especially for big sites like Reddit. Anyway, if StackOverflow can make it, why cannot Reddit, both have comparable traffic?

[1] [http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...](http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l76ok)

<comment><USER4> Whether you co-lo or run in the cloud it seems to me that you would still want the same types of redundancy/failover processes in place. The advantage of EC2 is you can easily set up servers in multiple places across the globe (or at least across the country if you just want to host in the US.)

<comment><USER5> and there are many downsides to EC2, dependence on a third party to a greater degree, shitty disk IO, instances go down more frequently than real servers, inflexibility with hardware (no SSDs), etc, etc.

There are co-lo provides that have data centers in different places.

</comment></comment></comment><comment><USER6> I've discovered through a bunch of similar lessons that buying really good hardware and hosting yourself sometimes requires the least amount of your time of any option. It provides a stable base on which to build additional redundancy such as shared network storage or database replication if you wish (and this is generally how expensive "enterprise" solutions work).

In practice and on cheap hardware, networked storage is flakey and has umpteen failure modes. Database replication is even worse. Both require babysitting by developers or sysadmins and hours to repair when it goes wrong. What is the point outsourcing hardware and scaling to EC2 if you end up with even more work to monitor and keep fixing the infrastructure you build on top?

</comment><comment><USER7> Be interesting to see what hosting offers they get .... lot of benefits to being the hosting company that keeps reddit online, if they manage to.

<comment><USER8> Its actually very easy to beat the price/performance of ec2 using real hardware. The draw of ec2/aws is in the "everything is a monthly charge and an api call away" operational instant gratification. If you need to auto-scale by the hour there's few other places you can do it. If you can generally capacity-manage your setup a month or more into the future, then a 1-week latency on RunInstances isn't actually a problem.

</comment></comment><comment><USER9> Does anyone else think that Reddit's usage of EBS might be the culprit?

Looking at their past outage response:

[http://blog.reddit.com/2010/01/why-did-we-take-reddit- down-f...](http://blog.reddit.com/2010/01/why-did-we-take-reddit-down- for-71.html)

Money quote: "In response, we started upgrading some of our databases to use a software RAID of EBS disks, which gives drastically increased performance (at a higher cost of course)."

RAIDing EBS disks seems like a really really BAD idea. There is a non-trivial failure rate of any single EBS disk, and if you RAID them together, your failure rate of the RAID will consequently increase. Am I understanding that correctly?

If they fix that, could that be a 'silver bullet' to fix these outages?

<comment><USER10> They certainly think there's a problem with EBS: "Since that last failure, we have been doing everything we can to move ourselves off of the EBS product. We're about half way there. All of our Cassandra nodes are now using only local disk, and we hope to have all of postgres on local disk soon."

</comment></comment><comment><USER11> I just love how someone sent them a bacon pizza during the outage: <http://imgur.com/dunO2>

</comment><comment><USER12> Jedberg last year stating that a startup buying physical machines today is foolish: [http://www.reddit.com/r/IAmA/comments/a2zte/i_run_reddits_se...](http://www.reddit.com/r/IAmA/comments/a2zte/i_run_reddits_servers_and_do_a_bunch_of_other/c0fm2we)

<comment><USER13> Reddit is not a startup anymore.

</comment></comment><comment><USER14> > All of our Cassandra nodes are now using only local disk, and we hope to > have all of postgres on local disk soon.

Since local disk is temporary and can be lost any second, don't they still have to use EBS for persistence?

<comment><USER15> They probably would use replication to a different region/zone and then they implement their own snapshot/backup mechanism on those replica machines so that they can use them to re-create instances when they lose them. Also PostgreSQL ships logs that you can store in S3 and bring up an instance from another area starting from a known backup point.

EBS is a pretty bad product regarding reliability and consistency of performance unfortunately, it's better to design your system without it.

<comment><USER16> Re-building by reading logs out from S3 can be very slow. It should be seen as a last ditch effort. Not really a hot swap, or fail over solution (depending on data size obviously).

If you can't use any persistent storage, then these machines become pure processing nodes. Which in this case, I feel like it would be better to not design your system with EC2 at all. :-(

To me having EBS made EC2 a very powerful solution compared with their competitors. Not having durable and consistent EBS otherwise makes no point in differentiation and serves purely as non-functional fluff we end up paying extra for.

<comment><USER17> If I were them, I'd look at an approach that was based on either SimpleDB or S3. I've been working on a couple of prototypes of systems similar to theirs for my own stuff at work, and I've been toying with a system that uses S3 to store what I'm calling "absolute fallback" sources of data for conversations (basically, JSON documents) and SimpleDB as a front-line store.

My general take on this issue is that if you're running your app on EC2 and your persistence medium is something that's also on EC2, you really have no ideal high availability scenario. Of course, even in my case, if SimpleDB and S3 go down, I'm still in trouble, but at least I have the option of throwing Akamai in front of it.

</comment></comment></comment></comment><comment><USER18> Has anyone else noticed the Amazon status page says there are still problems today but on the status history show no problems yesterday only on the 21st?

<http://status.aws.amazon.com/?a>

<comment><USER18> They have corrected it now.

</comment></comment><comment><USER16> At any time an instance may become un-responsive and so will local storage, no?

I don't see how this helps?

<comment><USER19> postgresql replication to other instances?

i believe the point is that instance storage is more reliable than ebs.

<comment><USER16> Why would you want to take a chance on that? Every instance should be thought of as expendable. They can go down at anytime.

EBS is really just like any other NAS/iScsi vol and it wouldn't be such a problem if they did what they're supposed to. That is, be consistent in read, write, and durability.

<comment><USER19> What do you mean by "take a chance on that"? If the instance goes down, you can failover quickly to another one that you've been replicating to.

<comment><USER16> If you accept that any of your instances can go down at any time, you accept the fact all of them can be down at one (the same) time.

You might think your safe with reserved instances too, thinking you've reserved the dedicated time with EC2. Well what happens when the entire network stack goes down or block storage the reserved instance rack depends on goes down? So does all 10 or 20 reserved instances you had up, at once.

Also, it was this very replication/snapshot mirroring feature of EBS that cascaded into (network,etc) congestion.

Check [http://joyeur.com/2011/04/22/on-cascading-failures-and- amazo...](http://joyeur.com/2011/04/22/on-cascading-failures-and-amazons- elastic-block-store/)</comment></comment></comment></comment></comment>


Ask HN: Freelancers, which project management software do you use? - <AUTHOR>

So I'm starting out with freelancing now, and am kinda lost. I definitely don't want to use email to communicate with clients.
I tried looking for a lot of options, but couldn't find any that would fit my needs. What I need is this:
- Adding projects (duh) - Adding client(s) to projects - Invoicing and payment (optional, can use another solution) - Milestones (I like to break my projects into smaller chunks) - Milestone reviews. (Client should be able to leave reviews on milestones before we sign them off) - Discussion with client (So we can share files, brainstorm etc)
It's just me so teams' features aren't required.
Any suggestions? <comment><USER2> In my experience, you can't dictate the use of project management tools to a customer. If your customer likes email, that's what you're going to use.

I've tried Basecamp with limited success. But that's just my clients. Maybe your situation is different.

(Also, Basecamp gets astonishingly cluttered and hard to navigate when you get slightly busy. I have a group of 10 people creating a nonprofit organization from ground zero -- including legal work, website, etc. -- and it's pure shit to try to find stuff in Basecamp. Highly NON-recommended. So beware the curse of success in persuading people to use a tool like Basecamp.)

This means that you need to keep yourself highly organized in order to remain sane. For me, I am a happy IQTELL user. [http://www.iqtell.com](http://www.iqtell.com)

But you might find something else that you like.

See, the problem isn't the tool you use. It's you and your brain. Get that right and everything will be fine.

A piece of paper with a list of stuff on it is fine. Right now I have a list of 9 things on a piece of paper. If I'm not working on one of those, I'm off course.

The usual disclaimers: David Allen is God, etc. etc.

But seriously. Keep it simple. I would say get a Moleskine and a pen, but I lost my paper organizer in a cab in Riyadh and that kinda sucked the farts out of dead seagulls. So now it's all in the cloud and in my phone, courtesy of IQTELL.</comment>


Italy on Lockdown - <AUTHOR> https://www.corriere.it/ <comment><USER2> Autotranalation of the main article [https://translate.google.com/translate?sl=auto&tl=en&u=https...](https://translate.google.com/translate?sl=auto&tl=en&u=https%3A%2F%2Fwww.corriere.it%2Fcronache%2F20_marzo_09%2Fcoronavirus- l-estensione-tutta-italia-restrizioni-spostamenti-solo-comprovati-motivi- salute-o-lavoro-ddea9980-623d-11ea-9897-5c6f48cf812d.shtml)

<comment><AUTHOR> How probable that something similar will happen in other countries in few days?

</comment></comment><comment><AUTHOR> It is happening right now. PM Conti havin press conference. Full lockdown beginning in few hours just anounced.

</comment><comment><AUTHOR> The entire country of Italy is quarantined, starting from tomorrow morning.</comment>


Judge OKs class-action suit against Apple, Intel, Google, Adobe - <AUTHOR> http://www.mercurynews.com/business/ci_24390480/judge-oks-class-action-suit-against-apple-intel-google-adobe <comment><USER3> This seems like a really big deal. If Silicon Valley companies are colluding to suppress programmer salaries, perhaps it is time for programmers to team up to fight back, using a union or a guild.

Now, before the usual tropes about the software industry's meritocracy and rugged individualism lead everyone to laugh at the idea of a labor movement for software engineers, let's all just take a moment and have a look at what happened to the Hollywood visual effects industry. It is one of the very few parts of the movie production process that is not unionized. Because of this lack of protection, studio execs openly brag about putting them out of business as a normal part of making movies (see the quote "If I don't put a visual effects shop out of business (on my movie), I'm not doing my job." in the first link below):

[http://www.variety.com/article/VR1117965871.html](http://www.variety.com/article/VR1117965871.html)

[http://www.npr.org/2013/03/19/174703202/visual-effects- firms...](http://www.npr.org/2013/03/19/174703202/visual-effects-firms-miss- out-on-a-films-success)

[http://www.visualeffectssociety.com/node/2425](http://www.visualeffectssociety.com/node/2425)

Something similar could quite easily happen to the software industry. Now is a perfect time to start a labor movement, as the settlement and pending class action suit clearly show the need for such a thing.

 __EDIT __: Just to be clear, the reason for the comparison with the VFX industry is that it is a solid example of a bunch of technocrats who had a chance to unionize but didn 't, and now they are being ground into the dust. The exact same thing could easily happen to the typical Silicon Valley programmer. This case with Apple, Intel, Google, and Adobe shows that they are already trying.

<comment><USER4> Interesting to see this idea as the top rated comment here on HN. I wrote a post about a year ago about a similar concept (for different reasons - <USER2> API limits, Apple rejections), and most of the HN crowd disagreed back then. [http://www.techfounder.net/2012/08/30/developers-vs- bigcorp/](http://www.techfounder.net/2012/08/30/developers-vs-bigcorp/)

<comment><USER5> But just using some companys api is not the same as having a employee /employer realsionship.

What is the Union issue here? If your product is freeloading on the back of another well you should make sure you have more than one source of income.

This is like complaining that Google has moved into my area and is eating my lunch for example flight or hotel bookings

</comment></comment></comment><comment><USER6> Title of the article is _" Judge OKs class-action suit against Apple, Intel, Google, Adobe"_, indicating this matter has not even yet been brought to trial. Just a quick heads up. The posted HN title, "Steve Jobs conspired with Tech companies to limit wages" seems to be declaring a conviction in the case.

<comment><USER7> Actually, the companies involved have already admitted wrongdoing and settled with the DOJ.

> The allegations prompted an investigation in 2009 by the U.S. Department of > Justice to determine if the companies had violated the Sherman Antitrust > Act. A year later, the agency announced a settlement in which the companies > acknowledged having had agreements to not "cold-call" employees at certain > firms. They agreed to refrain from such no-poaching-pacts for five years, > but the deal provided no compensation for their employees.

This new case is a civil class action on the same issue.

<comment><USER8> Hang on, how can they agree not to _break the law_ for a period "5 years" instead of... never again?

If they did the usual "we are not admitting we broke the law but will settle anyway to make this go away" then the DoJ had a duty to make it clear that they did break the law.

Otherwise after 5 years some other shmuck is going to have to go after them again when they start doing it again.

It is like the DoJ just washed their hands of the whole thing. (Let the proles deal with it.)

<comment><USER9> agreeing to not hire from the group of companies in the deal is not 'not breaking the law' it is the crime in question.

<comment><USER8> I think you misunderstood me, the "agreeing" I was referring to was them agreeing with the DoJ to not break the law instead of the DoJ doing their job.

</comment></comment></comment></comment></comment><comment><USER10> _In the settlement with Pixar, Lucasfilm and Intuit, the three companies -- which employed about 8 percent of the affected workers -- agreed to pay a total of $20 million, according to Kelly Dermody, the plaintiffs ' lead lawyer._

Basically, this means that it was profitable for the companies to collude.

8% of 64,000 is about 5000 employees. $20 million is $4000 per employee. That's in all probability significantly less than the annual salary bump that a typical employee in the class would be likely to negotiate.

So the companies not only reduced their costs in the first year, they reduced it in subsequent years and avoided compounding the costs via raises and via subsequent negotiations with employees whose salaries would have been higher due to previous negotiations and compounding with raises.

But on the bright side, at least the attorneys got paid.

<comment><USER11> > But on the bright side, at least the attorneys got paid.

The attorneys probably took 40% of the $20 million, so the actual employees affected only got $2,400, which is a really bad deal. I'm surprised the class voted to accept that settlement.

</comment></comment><comment><USER12> "Judith Zahid, a San Francisco lawyer specializing in antitrust matters, said the sued companies face an additional problem if the case goes to trial because, under antitrust law, any damage award against them would be automatically tripled."

Automatically tripled?! Wow. Good on the legal system, and you would think this knowledge would've prevented this sort of scheme, but they either didn't know or didn't care.

</comment><comment><USER13> This ruling grants class status to the employees, allowing them to sue as a class instead of individually. While class actions usually have the deck stacked against them, in this case the DOJ already got a settlement out of the companies, which tends to have a halo affect on any corresponding civil suit.

</comment><comment><USER14> The actual lawsuit docket is available here:

[http://www.plainsite.org/flashlight/case.html?id=1860459](http://www.plainsite.org/flashlight/case.html?id=1860459)

</comment><comment><USER15> Who?! Jobs, the saint, this is impossible!

</comment><comment><AUTHOR> Interesting, the story isn't at the top of programming subreddit anymore. I guess the moderators got to it.

</comment><comment><USER16> Hmm? This is basic practice, and if it's in the contract, what's so bad?

They /agreed/ to these terms in advance as far as I'm aware, who are you to tell the other employees(who didn't complain) that their contract should've been avoided?

<comment><USER17> Most non-compete clauses are invalid in California. This "agreement" was an attempt to get around that law.

Also, the reason it's a federal case, this "agreement" was done amongst the employers without the consent of the employees. If it were not California, and there actually was a non-compete in their contracts then you may have had a point.

There's a major difference between you agreeing to not compete and your employer actively preventing you from obtaining alternative employment.</comment></comment>


BBC Micro Bot – Runs your tweet on an 8-bit computer emulator - <AUTHOR> https://twitter.com/bbcmicrobot <comment><USER2> It's a pretty fun way to code. There's the #tweetcart hashtag which has been used for PICO-8 programmers to showcase their tweet-sized code and the results as a GIF recorded from PICO.</comment>


Introducing the Ross Ulbricht Legal Defense Fund - <AUTHOR> https://www.rossulbricht.org/introducing-the-ross-ulbricht-legal-defense-fund/

<comment><USER2> Unfortunately, the creator of this site doesn't explain truly why he feels Ross Ulbricht needs a defence fund. Has he publicly stated that he is unable to mount his own defence?

If he truly is DPR, then he has funds available for his defence, and if not, the US government has an obligation to provide an attorney for him. At the same time, I'm sure some legal gun who wants the publicity would take on the case.

This whole thing doesn't make a lot of sense to me. I think there are probably other people more in need of a defence fund.

<comment><USER3> Added to my to-do list for the FAQ (I operate the website). No, no such statement has issued, but then neither did Bradley (now Chelsea) Manning make such a declaration when she was arrested, and I registered bradleymanning.org and eventually help bootstrap what is now the Private Manning Support Network for her defense.

</comment></comment><comment><USER4> Didn't he make $80 million from selling drugs online?

Weirdest defense fund ever.

<comment><USER5> Also completely perplexed by this. The author seems to rush to put up a defense fund websites for high profile tech cases. The same author started a Manning Defense fund quickly after the story broke.

<comment><USER3> After being informed of the news and reading about it, i realized that this was something that needed doing (your mileage, of course, may vary), and that supporters might be vulnerable to scams. I quickly found that there was already a "freedpr" website out there asking for Bitcoin donations with no information whatsoever identifying the operator, so I said to myself: "Gotta do it."

Incidentally, the Bradley Manning work I did was hundreds of hours without me taking a dime. I resigned my membership of the Support Network steering committee about a year after getting the thing booted up, confident that the rest of the members had things well in hand.</comment></comment></comment>


Koch Brothers Are Cities' New Obstacle to Building Broadband - <AUTHOR> https://www.wired.com/story/koch-brothers-are-cities-new-obstacle-to-building-broadband/ <comment><USER2> Is it a big surprise that Republicans want to perpetuate a classist system? Anything owned by the public, any portion of it owned by the public, makes it difficult to have the haves and the have nots, and this is an affront. And not literally have not, because it's a utility, so most everyone will still be able to afford to have, but because they will pay a much higher percentage of the disposable income to have access to a basic utility they have have not of something else.</comment>


To Know, but Not Understand: David Weinberger on Science and Big Data (2012) - <AUTHOR> https://www.theatlantic.com/technology/archive/2012/01/to-know-but-not-understand-david-weinberger-on-science-and-big-data/250820/ <comment><USER2> At what point do we shift our investment in time and energy from building models like those mentioned in the article to the bigger picture? Maybe it's just my perception but it doesn't seem like we have very many people thinking deeply about what models we should build and to what ends. Instead we are just building the models and hoping we can put them to good use afterwards.

For example, what's the end game for the cellular signaling modeling outlined in the article? It seems like the result isn't valuable in and of itself, and it can't be much more than that because the scientist "doesn't understand it, and doesn't think any person could." So we now have an equation that expresses constants within a cell and that's it. We don't understand it and we can't put it to good use. So was that time and effort well spent? Do we just put this work in a drawer so we can pull it out if it could be useful at some point in the future? Is that what we're doing with all the similar advances in modeling?

There's nothing wrong with knowledge for knowledge's sake, but I think we've way over indexed on the tools and predictions side of the system. If we continue to constantly create new tools/models/predictions we might find a use for them by chance. It just seems more efficient to focus on what outcomes we really want and then put the models to work in pursuit of those outcomes. Perhaps we focused more on the outcomes in the past because we didn't have the technological horsepower to constantly churn out new models.

Maybe I'm wrong and there are people working on the big picture. Are there modern day philosophers doing this work? Do they make up a significant portion of the work being done? If not, why?

<comment><USER3> _> what's the end game for the cellular signaling modeling outlined in the article?_

Pharma.

Most of the modeling work that people do is fairly well motivated. Going from models to working technology is indeed a huge leap, but everything starts with the basic scientific understanding.

 _> Maybe I'm wrong and there are people working on the big picture._

You can usually find the "big picture" behind a paper by reading the recent grant applications from the PI who funded the research (or the funding lines explicitly mentioned in the paper, if any).

<comment><USER4> Pharma may be the intended target for the signaling work, but as a data scientist who works in pharma, I can say with certainty that no biologist or chemist here would entertain for a minute any model that can't explain its mechanisms of action. Nor would the FDA, who wants any model not only to accurately predict the intended outcome but also reflect awareness of the contextual circumstances that surround and lead to it.

No competent physician would be satisfied with a disembodied diagnosis. The constituent symptoms and assay metrics that support that diagnosis are essential to know, especially as disease is often complex and dynamic, and no single diagnostic label should ever hope to supplant a deeper understanding of each patient's unique mix of normality and abnormality. A diagnosis using ML may be a useful starting point in treatment, but never should be the endpoint.

<comment><USER5> >no biologist or chemist here would entertain for a minute any model that can't explain its mechanisms of action.

Entertain? Who even really knows what means at this point. But I'm fairly convinced that you'd be quite happy to have a theory-free "intuition pump" that could tell you "if you slow down binding with the following 3 membrane proteins, you see roughly double that effect on overall energy use by the cell".

The tool that generates this prediction may be completely unable to give you a "theory" about why this should be so, but then neither will the experiment(s) you do that confirm it to be true.

So, while indeed, ML-style stuff "should never be the endpoint", they can act as a incredibly useful intuition pump/launchpad for ideas and approaches that would otherwise remain inaccessible.

<comment><USER3> That's the mode of use for ML in _most_ industries -- flagging stuff for follow-up by humans. Basically anything that's not real-time works like this.

Most uses of ML in real-time settings look more like hybrid systems -- a little dusting of ML on top of a whole heap of more traditional mathematical modeling/software engineering.

Outside of a few very niche settings, we're still a long way off from "trusting" ML in any meaningful sense.

</comment></comment></comment></comment></comment><comment><USER6> From my point of view there are 2 possible ways:

\- we simply acknowledge we don't understand something enough and keep looking into it until we do. I mean everything we now understand (at an acceptable level by our standards) has gone through an intermediary phase - see alchemy for example.

\- we declare some things (prematurely?) as forever escaping our grasp and accept we may never have a simple model of them.

What bothers me is the 3rd way:

\- we don't know why but the computer model gave this result so let's go ahead with putting it into production. We make money, the user/consumer may have a nice experience or die, fingers crossed.

<comment><USER7> I think there's another (perhaps worse?) way:

\- we think we understand a complex model completely and we actually don't

</comment></comment><comment><AUTHOR> Bottom line:

"With the new database-based science, there is often no moment when the complex becomes simple enough for us to understand it. The model does not reduce to an equation that lets us then throw away the model. You have to run the simulation to see what emerges. For example, a computer model of the movement of people within a confined space who are fleeing from a threat--they are in a panic--shows that putting a column about one meter in front of an exit door, slightly to either side, actually increases the flow of people out the door. Why? There may be a theory or it may simply be an emergent property. We can climb the ladder of complexity from party games to humans with the single intent of getting outside of a burning building, to phenomena with many more people with much more diverse and changing motivations, such as markets. We can model these and perhaps know how they work without understanding them. They are so complex that only our artificial brains can manage the amount of data and the number of interactions involved."

<comment><USER3> _> The model does not reduce to an equation that lets us then throw away the model. You have to run the simulation to see what emerges._

This is true of simulation in general, not just data-drive models. E.g., a lot of applied mathematics uses PDE models that don't have closed-form solutions and so you just run a ton of simulations sweeping a parameter space.

 _> For example, a computer model of the movement of people within a confined space who are fleeing from a threat--they are in a panic--shows that putting a column about one meter in front of an exit door, slightly to either side, actually increases the flow of people out the door._

The crux of this type of science is that you don't know whether the computer simulations are telling you anything about reality. You just have to run real- world experiments and see what happens. And even if the experiment turns out to work, you still don't know for sure that your model was reasonable.

<comment><USER8> So you are suggesting we need to do a double blind test, by means of throwing a molotov cocktail at a few people gatherings?

<comment><USER9> Double blind so that neither the thrower of recipients of said cocktail doesn't know wether it is real or placebo? It's the only way to know if they are panicking because of a bottle or a fireball...

</comment></comment></comment></comment><comment><USER10> Completely off topic -- one of my favorite stories from a friend at Google was that they saw someone writing what looked like a giant AWK script, and they went over to the guy and told them "look over at that desk, that's Brian Kernighan, the 'K' in AWK" only to be met with a scornful "I'm the W" from Weinberger.</comment>


Can/do windows users move to mac? - <AUTHOR>

I have been  a windows user for the last seven years.I have used Ubuntu but not mac. Do windows users ever feel a need to move to mac? is it beneficial?<p>I am currently doing my undergrad cs studies,and so program in c/c++ ,vb ,matlab etc <comment><USER2> I'm a life-long PC user who has toyed with Mac briefly from time to time and been somewhat impressed. The price points have always kept me away however.

Back in March I picked up a MacBook from MacMall (recommended) for my fiancee and after using OS X a bit more regularly I can safely say that as soon as I have a reason to pick up a new computer it will be a Mac.

I'm not a coder so I can't comment on using either OS for that purpose. What I can say from my perspective is that it just works. Her MacBook doesn't freeze or bog like every PC I've ever owned has on an almost daily basis. The layout is smart and functions are user-friendly. Everything makes sense to her (relative novice). As a result, she asks far fewer questions of me than she did when she was using XP.

Of course everything is a matter of personal preference. I would suggest that you take a good hour (at least) in an Apple Store and play with whichever model caught your eye. My guess; you'll end up liking it.

</comment><comment><USER3> I think that Apple's growing market share would indicate that at least SOME Windows users see the benefit in switching to Mac. I also wouldn't be shocked to find that some Mac users have switched to Windows for whatever reason.

"Beneficial" is a matter of personal taste and requirements from your OS.

</comment><comment><USER4> You've got to troll better than that. These waters have been fished out.

I'm sure Google can lead you to a couple hundred thousand examples. Try searchyc.com as well. Try "paul graham mac". Find some mac developer blogs.

</comment><comment><USER5> Here is what Paul Thurrott has to say:

<http://www.winsupersite.com/paul/whatiuse.asp>

 _I think it's important to keep any eye on the Apple side, though I've always found Windows-based PCs to be more productive, given my work style (not to mention my job). But it's not really about work: I just prefer Windows, sorry. That said, Apple makes good stuff, and of course I'll be evaluating Leopard soon._</comment>


Best personal wiki after voodoopad and gitit - <AUTHOR> https://github.com/tankywoo/simiki <comment><AUTHOR> I'am a big fan about how to struct my note. Personal wiki is my ultimate choice. After Voodoopad been selled and not updated, I believe in 'simiki' meet all my need.

\- pure markdown. \- easy deploy to github pages. \- easy preview in local broswer.</comment>


The Secrets of Effective Technical Talks - <AUTHOR> http://www.msteched.com/2010/NorthAmerica/SIA334

<comment><USER2> Silverlight required - guess I won't be watching that, then.</comment>