Trump's cell phone use is security "nightmare" waiting to happen, lawmakers say - <AUTHOR> https://arstechnica.com/tech-policy/2018/04/trumps-cell-phone-use-is-security-nightmare-waiting-to-happen-lawmakers-say/ <comment><USER2> I see this, as well as Hillary Clinton's use of a personal email server, as a manifestation of a problem with the way we do security: when IT security restrictions are a pain in the butt to comply with, people are going to try to get around them.

A trivial example: IT requires users to change their passwords every N days. So users start with "password", and when they are required to change it, they use "password1", then "password2", and so on.

A non-trivial example: someone in a high government position is told they have to use a crappy, outdated, locked-down Blackberry device to access their official email, so they start using a personal email account or a personal phone for official business.

Security doesn't work if it's hard to use, because people will find a way around it, and some of those people will have enough power that you can't force them to comply.

<comment><USER3> > as well as Hillary Clinton's use of a personal email server, as a > manifestation of a problem with the way we do security: when IT security > restrictions are a pain in the butt to comply with, people are going to try > to get around them.

Wasn't that more of an effort to get around government records-retention and transparency laws than an attempt to get around annoying and onerous security requirements?

<comment><USER4> My reading, at the time, led me to believe it was primarily a matter of access and facility, for Clinton. She wanted her email "her way", and State technical resources -- perhaps also dependent on cross-agency resources -- could not easily nor readily make it happen.

So, she had one of her own staffers take it rogue, with this server deployment.

Like a lot of senior executives, Hillary Clinton seems to have a pretty big ego and a commensurate attitude that she knows what she's doing.

In a larger context, something that really bothers me about all this: The Chelsea Manning problem arrived around the start of her tenure at State. State, the President and administration, pretty much all the powers that be were outraged over this event.

Again, from my reading, I gathered that the system these data were in was fundamentally deficient with respect to the security needs that were outline. Basically, two levels of acccess, with no compartmentalization nor, IIRC, active auditing -- perhaps much of any auditing or audit trail at all.

Clinton was head of State for the better part of eight years. They remained outraged at Manning and never ceased their push for punitive action.

But she very apparently never tackled and fixed the systems deficiencies that allowed the problem, in the first place.

As her campaign proceeded, I read numerous stories about its fundamental disorganization and corresponding disfunction.

To speak politically for a moment, I'd take this in a second compared to the malicious disaster that is Trump and his crony clown car of an administration.

But neither did she acknowledge, accept responsibility for, and actually work to fix the very tools her and State's work are now founded upon.

Go back twenty years. The intelligence services had a better scoped information collection and analysis system that showed real promise, while also better respecting privacy. Thin Thread.

Dick Cheney gets into the VP seat, and he wants to steer as much business as possible to his cronies. Thin Thread gets shut down, and in its place we end up with various proposals and tweaked formulations for what is called -- I can't remember the program names, right now -- "total situational awareness". That drowns in its own inefficiency, imprecision, and over-collection.

And in the gap, we also get 9/11 and a mis-lead if not dis-ingenuous ramp up to a multi-front war on the other side of the globe.

These people don't care about good, proper systems. They and their agendas vary; nonetheless, their agenda always takes primacy. And, invariably, the shortcuts or biased choices they force through end up costing, down the line.

<comment><USER5> "These people" such as Hillary, Bush, Cheney, usually started off a mail service outside the Whitehouse to get their campaigns organised. Over time their entire entourage is using that mail server.

Then they get into Government and we expect them to transplant their entire support network into the new environment. That is not going to happen.

What is needed is for the mail server that someone starts up to support their campaign to be secure be default with Whitehouse-level security made easier than not using it at all.

The hard part is getting people to use S/MIME certificates so that messages are encrypted end to end and you never have to quarantine cleartext on your cloud service.

Surely there is an entrepreneur in the HN crowd ready to take this on?

</comment></comment></comment></comment><comment><USER6> Oh my if that's a "nightmare" then what is the rest of what is going on considered?

<comment><USER7> I think that died when people realized that email is inherently insecure unless you jump through a LOT of hoops (e.g. pgp)

</comment></comment><comment><USER8> I remember when email security was a top concern to folks.

<comment><USER9> Well what happened arnt we concerned about clasified info being hacked?

<comment><USER8> It can't get hacked by our adversaries if we just tell them when they visit the oval office!

</comment></comment></comment><comment><USER10> Oh I don't think it's waiting.

<comment><USER11> Yea, seriously.

>or the fortuity of foreign agencies not knowing his personal cell number

I find it incredibly difficult to believe that professional intelligence services don't have his cell phone number already.

</comment></comment><comment><USER12> I'm not the most clued up on US politics, but is it his personal phone? If it is, this could give rise to a situation not dissimilar to the one caused by Hilary Clinton's private email server!

<comment><USER13> Only if he's illegally conducting official business on it.

</comment></comment><comment><USER14> Thought: it would be possible to make the phone reasonably secure by having one of the SS agents (who are always around him) carry a secured WiFi access point, which the phone connects to (and it doesn't connect to anything else).

Then the access point is VPN-ed and firewalled back to an endpoint that is secured, from which phone calls are then connected directly to e.g. Verizon / ATT / Level3 for termination to the dialed number.

i.e. network topology is

TrumpPhone <\--> WifiAP <\--> VPN to <\--> Secured endpoint <\--> telco/bandwidth .

More security could be added by e.g. configuring a virtual phone number which lives "on the switch" then forwards to a securitized softphone. This would mean that the phone number attached to a physical phone would never be used; and multiple phones configured identically could be set up ahead of time, audited, upgraded, etc.

</comment><comment><USER15> An hypothesis: There are idea bubbles, just like there are financial bubbles. In fact, arguably a financial bubble is just one application of idea bubble - for example, 'real estate always goes up'; 'tulips are worth more than gold', 'crypto-currency doesn't need regulation', etc. That is, the evidence for idea bubbles seems obvious: they are widely known in finance, technology (hot trends, etc.), angry mobs, and elsewhere. I'm just classifying them as a way of thinking about them.

More specifically, I hypothesize that there is a bubble right now in political thinking: It doesn't matter what evil, incompetent, or highly risky things we do, as long as our side 'wins' in the immediate term. We can safely ignore the longer term consequences. A successful businessperson who relies heavily on international trade told me recently: 'Trump is vile, but I voted for him and I'd vote for him again - because I'm a businessperson and business is good.' In that statement they believe there is a problem but choose to ignore its consequences, even the ones that will directly affect their business (e.g., trade problems).

And that property seems to apply to all idea bubbles in all areas: Ignore the consequences. I can see how bubbles work up close: The mechanism seems to be that when everyone joins the bubble, 1) it's exciting and engaging, and 2) nobody is making us think about the consequences; the social pressure to exercise judgment band behave is gone, because nobody will judge you for it. The parents are gone - let's party! Pick up your pitchforks and torches! This is gonna be Awesome!!!

 _Reality is that which, when you stop believing in it, doesn 't go away._ \- Philip K. Dick[0]

[0] Attributed; I don't have time to find the source.

</comment><comment><USER16> >Last year, Trump reportedly had an iPhone with just one app on it: Twitter.

Can someone with more knowledge explain how is this a security nightmare? If he's not using it for email, if it's just for him posting his opinions onto Twitter, I'm not sure I follow why it's so bad?

<comment><USER17> It's a smart device with known (and unknown) vulnerabilities that's always near the president. It probably has a browser, a camera, gps, and a microphone and it is always connected to Apple's cloud. It's a large attack surface if you're a nation state. Oh, it's also a phone. It makes un-encrypted calls, and is capable of sending and receiving messages.

<comment><USER18> I thought it was determined (well, alleged, with data) that Trump uses an Android device[1]. Has that changed?

[1]: [http://varianceexplained.org/r/trump- tweets/](http://varianceexplained.org/r/trump-tweets/)

</comment></comment></comment><comment><USER19> Headline is misleading. It should say _smart phone use_ , not _cell phone use._ A lot of this is about his use of the internet via his phone, not about voice calls per se. The ability to pull a phone out of your pocket and call someone, almost any time, any where, is just another layer of the issue. But his use of twitter would be problematic even if it was all done from a PC.

When I had a job at an insurance company, where we had to comply with HIPAA et al, most people in the department hated making phone calls. I was there a fairly long time before I got any training on handling phone calls.

Phone call themselves are an information security nightmare waiting to happen in part because it is live conversation. It is hard enough to write a letter that is HIPAA compliant. Certain kinds of letters, like those advising customers of a HIPAA breach related to their policy, had to be written using a form letter and then reviewed by the legal department to make sure it was in compliance and this all went through your boss.

In the claims department, it was common for people to speak colloquially of 'paying claims' because most claims were paid, not denied. But the correct term is _processing claims._ I had a coworker get in trouble because she called a customer, said something like "We need this information so we can pay your claim" (instead of saying "so we can process your claim") and then the claim was actually denied.

President Reagan helped bring the presidency into the video age. He was a former actor and was constantly aware of surroundings and what was in the background behind him, what was framing his image. This changed the way the presidency was portrayed in visual media, both pictures and film. If you go look at presidential images preceding his administration and those following it, they are dramatically different.

My impression is that Trump made a concerted effort to go where the people were and adopt the channels of communication they used, including twitter. It wasn't his thing, personally. No surprise that he has no clue what he is doing.

Perhaps this is the presidency where we need to figure out how the president uses the internet and social media. Perhaps they need to develop some protocols around it. I don't believe there are previously established good protocols.

This is another venue for communicating with the people. Hopefully we woo't throw out the baby with the bath water in trying to resolve the issues this presents.

</comment><comment><USER20> Agreed, but didn't Obama use his famously during his presidency?

<comment><USER21> I think it's quite the opposite. He (reportedly [1]) didn't have a smartphone until 2016, and it was pretty locked down.

[1] [http://fortune.com/2016/06/10/president-obamas-new- smartphon...](http://fortune.com/2016/06/10/president-obamas-new-smartphone- is-more-like-a-toddler-phone/)

<comment><USER20> I remember when he got elected in 08 he didn't want to give up his "Crackberry"

<comment><USER22> They made sure his blackberry was secure. There was no security problems with him using it.

</comment></comment></comment></comment><comment><USER23> This is such a yawner. Everything about Trump is a disaster or a nightmare or incompetent, blah, blah blah. (Hillary had a barely secured email server in here closet!!)

And here we are, decent economy, relative peace, no collusion, functioning government. Trump may be unconventional but there's nothing wrong with that (in fact many, including detractors, praise that quality) and it seems to be working OK so far.

Disclaimer: I loathe Trump but believe the criticism he gets is frequently unwarranted.

<comment><USER24> Didn't he recently just wake up and decide to fire some rockets on Syria? You call that relative peace?

Why are you trying to downplay how incompetent one president it is by highlighting another candidate? Maybe it is true that they would both have been relatively incompetent, but that is not the point being made here. The example you're giving also does not serve to put things in perspective, as it is also an egregious offense.

<comment><USER23> I was going to say peace but qualified it with relative. The Syria thing is pretty minor war-wise, imo.

I only threw in the "Hillary email" thing because the subject is communications security.</comment></comment></comment>


Introducing Cloud Inference API - <AUTHOR> https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-inference-api-uncover-insights-from-large-scale-typed-time-series-data <comment><USER2> The author of the blog, Emanuel taropa, is one of the most brilliant software engineers I've met.

<comment><USER3> Didn't he write the predecessor to Effingo ([http://www.cs.huji.ac.il/~dhay/IND2011.html](http://www.cs.huji.ac.il/~dhay/IND2011.html)), more than a decade ago? I still remember his name and email because the service was great, but also difficult to tame. And you'd end up asking for his help.

<comment><USER2> I can't really share that level of detail, but what I'd say is, if you noticed your searches were fast, and the results were fresh and accurate, then Emma probably had a hand in it.

</comment></comment></comment><comment><USER4> Here's a neat use case analyzing GDELT data: [https://blog.gdeltproject.org/peering-into-the-visual- landsc...](https://blog.gdeltproject.org/peering-into-the-visual-landscape-of- half-a-billion-news-images-with-googles-cloud-inference-api/)</comment>


Mining Bitcoin with pencil and paper: 0.67 hashes per day - <AUTHOR> http://www.righto.com/2014/09/mining-bitcoin-with-pencil-and-paper.html <comment><USER2> A fascinating article. There was extensive discussion when it was first submitted:

[https://news.ycombinator.com/item?id=8380110](https://news.ycombinator.com/item?id=8380110)

Of course, comments there are closed, so if you have anything to add, you'll have to do it here.

Other submissions of this story don't seem to have created any further discussion:

[https://news.ycombinator.com/item?id=11174816](https://news.ycombinator.com/item?id=11174816)

[https://news.ycombinator.com/item?id=10836366](https://news.ycombinator.com/item?id=10836366)

[https://news.ycombinator.com/item?id=9635131](https://news.ycombinator.com/item?id=9635131)

[https://news.ycombinator.com/item?id=9596444](https://news.ycombinator.com/item?id=9596444)

[https://news.ycombinator.com/item?id=8387682](https://news.ycombinator.com/item?id=8387682)</comment>


Finally we have Threads in Node 10.5.0: a practical intro - <AUTHOR> https://medium.com/dailyjs/threads-in-node-10-5-0-a-practical-intro-3b85a0a3c953 <comment><AUTHOR> Good to know that Node finally has multi-threading. Excited to try it out.</comment>


Cassette Tape UI - <AUTHOR> http://www.schillmania.com/projects/soundmanager2/demo/cassette-tape/

<comment><USER2> I wouldn't call this "an exercise in skeuomorphic excess," since it doesn't actually re-create any real-world interactions.

The value of skeuomorphism is that it uses visual representations of familiar physical objects to help people understand how to interact with an interface. It's not always appropriate, but it has its place when it's done right.

In this case, there's a cassette tape, which is not itself an interface, but a storage medium. Unless the user mouses over the cassette, there are no controls visible at all. Once revealed, the hidden controls turn out to be simply a set of traditional play and seek buttons, albeit oddly arranged (I'm not sure why the play button is backwards). It took some experimentation on my part to learn that clicking the cassette itself also toggles playback, and that clicking on the label reveals a hidden input field.

One of the biggest criticisms of "flat UI" is the lack of visual affordances that show users what is clickable, and what it should do. This ersatz "skeuomorphic" interface has the missing affordances of a flat UI, without the trendy design.

I guess it's funny, but I would had more fun if it had gone all the way, such as re-creating a classic cassette deck, or a Walkman, or something.

<comment><USER3> Interfaces are a two way deal: input and state representation. I would say communicating playback progress through tape reels is skeuomorphic.

You're right in that it's not really "excess".

<comment><USER2> True, it has some skeuomorphic elements. I suspect the creator was trying to make fun of skeuomorphism through excess, but missed the mark by misunderstanding the concepts involved.

</comment></comment></comment><comment><USER4> What concerned me the most was whether the diameter of the spool with less 'tape' on it grew faster than that of the spool with more tape. It doesn't (but it should)

Mathematical accuracy aside, while I think this is definitely a cool visualization, it would benefit from some kind of time display and an easier way to navigate through the tracks.

<comment><USER5> The right wheel, the one being driven, appears to go faster as it plays, which is incorrect. The drive wheel is always moving at a constant rate of rotation. the left side should be speeding up as the amount of tape shrinks.

In fact, it appears that the right side is doing what the left side should be doing. Seems like a simple mistake.

<comment><USER6> No, that's not right. The tape always moves at a constant _linear_ speed, so both reels will go at a faster angular speed as the amount of tape shrinks.

<comment><USER5> I'd forgotten about the rubber drive wheel. Then the simulation needs some more work.

</comment></comment></comment></comment><comment><USER7> Instant feedback, as I imagine it:

age <= 16: "What is that?"

16 < age <= 30: "Gram-o-phone? Isn't that what they used to call these? Like, in the olden days? Wait... why can't I skip?"

30 < age: "If you think I'm going back to using cassettes, you can stop RIGHT there and fuck RIGHT off."

<comment><USER8> I find it entertaining how people think that younger generations don't know what things are. The majority of teenagers right now could tell you what it is and probably even how it worked.

<comment><USER9> Majority?

One reason they know about it is because there's some retro stuff happening, and cassettes are appearing on t-shirts and iPhone cases. Also, "mix tapes" are probably still big, even if they're in a different format.[1]

But I doubt that most teenagers really know much about a cassette. I guess it depends how much we're talking about. They might know it's something that held music. But would they know about C60? Or the prevent-record tab? Or spinning a cassette on a biro to rewind it (or gentle twiddling the biro to wind tape back on after it had despooled)? Would they know about chrome? Would they know that cassettes were double sided?

I'm making a list for my son (he's 2 1/2) of things we do today (and things I did when I was younger) that he might find weird. Cassettes definitely go in that list. "Physical thing? For sound? What bit rate? Wait, what, analogue?". I'm also including the fact that I used to have to chop wood for the fire and that we had a 'party line' (a phone line shared between a number of households with phase based signalling).

<comment><USER10> What's the [1] for?

<comment><USER9> Whoops.

[1] What is the modern equivalent of a mixtape? MixCDs are almost equally archaic. A mix-playlist??

</comment></comment></comment></comment></comment><comment><USER11> I dig it. Thanks for putting it up here. Graphics could be nicer I guess but all the talk about 'skeumorphism' and accuracy (although this is helpful feedback and if I'd code something like this, I'd also want to strive for accuracy) kind of weirds me out, once more. Can't you just enjoy it for a minute there?

Also, always a pleasure to hear fragments of 'The Edge' by David McCallum (also used in one of my favorite 'true school' hip hop tracks, 'No Regrets' by Masta Ace).

</comment><comment><USER12> It's kinda neat but the Play button is facing the wrong way.

<comment><USER13> He's probably a leftie. It doesn't play anything on my PC, though.

<comment><USER12> Doesn't matter, play buttons should always point right. And in this case specifically the tape is spooling from left to right. :)

<comment><USER14> There were non-flip decks with a play button for the Side B side. That play button pointed left. But this tape direction is playing the Side A track.

Teac double auto reverse deck:

[http://cachepe.zzounds.com/media/quality,85/CASS_202mkIII_SP...](http://cachepe.zzounds.com/media/quality,85/CASS_202mkIII_SP-2a7d890a47c506964d9833ace07fc31a.jpg)

</comment></comment></comment></comment><comment><USER15> soundmanager2 is awesome - I use it for my project tunes.io This interface brings back memories of Mixwit! (YC W08) <http://mixwit.wordpress.com/>

</comment><comment><USER16> Reminds me of the iOS Podcasts App, except not quite as bad.

</comment><comment><USER17> very confusing... when you change the controls position it loses all the model that we have of a play control</comment>


Steps to Turn Off the Nagging Self-Doubt in Your Head - <AUTHOR> http://www.wsj.com/articles/steps-to-turn-off-the-nagging-self-doubt-in-your-head-1465838679 <comment><USER3> This article hits home. Five years ago, I created a SaaS app that did well, followed by one that did less so, and another that flopped (all eBay selling software).

New projects weren't having as much an impact and making me feel good, so I sat around reading books.

A turning point came when I read two: Flow by Mihaly Csikszentmihalyi and On Writing by Stephen King. The first teaches we get optimal experience out of what we spend our attention on. It really got me psyched about chasing that feeling of being in the zone with my work. On Writing did that too. I saw another HN poster refer to King's "blue collar work ethic" \-- totally true. He works hard because he loves to.

I recently started working again for pleasure, less encumbered by insecurities. I'm still working on it, still practicing. I created an open source framework for Amazon Echo development, and it's starting to get some traction. I think I was able to produce the work to get the traction because I've had a clearer mind.

<comment><USER4> The flip side of being in the flow is that you're not learning much (the task is literally so easy for you that you can lose yourself in it).

<comment><USER5> My understanding of flow is that it is triggered when the task is neither too easy or too hard. It's a sweet spot that maximizes your engagement with the task at hand.

<comment><USER6> Why can't _so easy for you that you can lose yourself in it_ fall within _neither too easy or too hard_ range?

<comment><USER7> Because it fails to emphasize that there is any difficulty at all.

</comment></comment></comment></comment></comment><comment><USER8> It seems to me that the more you have a propensity to think (or over-think), the more likely it is that you may be able to construct rationalizations (excuses) before/during the moment as to why you might _not_ be able to do something. More insidiously, you can construct these excuses after the moment passes, like an endless negative Nancy TiVo gone berzerk in your mind.

There is no one-size-fits all solution to self-doubt, lack of confidence, etc. Really, the only solution for nagging self-doubt is to generate contradictory evidence to self-doubt. That is, do something.

Motivation is not ideal either because usually it makes you _think_ you can do something.

The more I work on meditation and visualization and other techniques - <USER2> more aware I am in the actual moment of when I am resisting doing something out of my comfort/risk zone. There is a lot to be said for Zen and Stoicism, as well. And in the end, only you (or your closest friends or partner) know when you are not pushing yourself.

<comment><USER9> Can you give any resources, which influenced your thinking in this context?

<comment><USER8> While there are some things that I am very good at naturally, there are many things (especially in the social sphere) that I struggle with. The problem is that I've developed camouflage that makes my acquaintances and even some of my friends think that I am better in some of these areas than I know that I am.

Having bought (and eventually given most away) pounds and pounds of self- help/self-improvement books over many years, there are common patterns in each book. They are all marketing to fix X problem with Y techniques. All parts of every technique Y may not work for you. However, like agile/scrum, there are parts of it that may be worth using for you, personally.

Having someone that knows you help keep you accountable (even better if you are both keeping each other accountable - or a small group) may work for you. You can also hire a personal coach. The accountability doesn't have to be every day (that is micromanagement and would get exhausting on both sides). Something like once or twice a month can work well. A regular 8am meeting twice a month, for example. It doesn't have to be face to face, it can be over Skype.

Finally, you may encounter the strongest resistance from yourself. Simple stuff like going to a yoga class instead of, say, a tech meetup. Or trying to meditate or visualize every day (the key tip here is do it before you turn on your laptop or your phone - do it right away, in the morning, as you roll out of bed). Saying hi to that cute person or not - and regretting it until it fades. You can make excuses until "the cows come home". And in the end, just doing it is what matters.

Stoicism 101: A Practical Guide for Entrepreneurs [http://fourhourworkweek.com/2009/04/13/stoicism-101-a-practi...](http://fourhourworkweek.com/2009/04/13/stoicism-101-a-practical- guide-for-entrepreneurs/)

Detaching emotion from action. Many articles talk about this, this is just an example. [http://briankim.net/articles/important-remain-emotionally- de...](http://briankim.net/articles/important-remain-emotionally-detached- results/)

Jia Jiang 100 Days of Rejection [http://fearbuster.com/100-days-of-rejection- therapy/](http://fearbuster.com/100-days-of-rejection-therapy/)

</comment></comment></comment><comment><USER10> I talked to some psychologists at a conference a while back and they were mildly opposed to "positive psychology" on the grounds of crowding out healthy negative thoughts with positive ones being potentially detrimental to learning and potentially harmful with non-understood side effects. They said the research isn't exactly conclusive and noted that there's a lot of snake-oil salespeople branding themselves with "positive psychology" (which makes sense as it's a prefect self help topic which tends to attract some of them). I never followed up on this by reading current studies.

The criticism section for the "positive psychology" article seems to suggest something similar. I'd be interested in more informed opinions.

<comment><USER11> That's interesting, I'd love for somebody with more knowledge to chime in.

As a layperson my response to a psychologist who was critical of positive psychology would be "okay fine, let's assume you're right - what would you recommend people do to help themselves get better if not mindfulness, CBT or some other form of mental programme to improve their mental wellbeing?".

I totally agree that there are people selling snake oil off the back of positive psychology becoming more mainstream. That's life unfortunately...

Personally I'd take "non-understood side effects" over feeling like there's nothing I can do in the event I suffered from depression/anxiety/low self esteem or one of the dozens of other mental health issues that society on the whole still aren't very good at recognising and treating.

<comment><USER12> I am not sure mindfulness falls into 'positive psychology'. Accepting things for what they are, seems neutral to me.

<comment><USER13> This. At least the mindfulness I practice (rooted in Buddhist teachings, not tied to them however) is definitely not driven by an expected outcome of "positivity." Oddly enough, this has led me to a more positive outlook, generally speaking.

<comment><USER14> That's a great insight. When you experience the truth of yourself, you experience the joy in it. Not because you are "trying to find joy". It's more realizing the joy you already are. Or that's been my experience with meditation and mindfulness at least.

</comment></comment></comment></comment></comment><comment><USER15> I like to think I'm at least partially genius ;) - I had no idea about CBT or any of the steps mentioned in the article but I had developed a very similar process process mostly on my own.

The first step is to keep awareness of your thoughts and feelings. Next is to remove partiality towards thoughts - you learn to take every thought as is and you learn to not have feelings towards the negative ones. That is you realize that you're partial to negativity - for some it's an excuse or explanation, for others it is deeply engrossed beliefs in certain ideas. Whatever it is you have to realize that you're more gullible to some type of thoughts than others.

Next is evaluating the impact of the thoughts on your real actual life in terms of steps you can take. Often times there isn't anything you can do about those thoughts and you drop'em like hot coal. Sometimes you find you can do things differently or avoid certain circumstances to not let those thoughts recur. Some other times it's a longer term thing. Point is to map thoughts to concrete actions and drop the ones that cause you negativity but have no action possible to counter. In yet another words drop the madness and keep the actions.

Rest is practice - you'll initially fail but so long as you notice it it'll get better the next time. You can also use distraction initially if your thoughts are too overwhelming.

It's a slow process especially for adults but it works!

<comment><USER16> Genius? Nah.

Anyone who brings awareness to their own thoughts and feelings will figure this out.

<comment><USER15> It was a tongue in cheek remark obviously but nonetheless bringing awareness is only the first step - then you're stuck being aware of your own limits - overcoming those are the real effort and it requires quite a bit of it to become habitually impartial to your own thoughts - the process, the making of that habit of detachment - that's where real progress is.

<comment><USER16> Ok there. You sound so smart there bud. You guys are such pretentious losers.

</comment></comment></comment></comment><comment><USER17> There's two really good approaches to this issue that I've learned. The one in the article is cognitive behavioral therapy (also, dialectical behavioral therapy, depending on the type of ruminations). Another approach that has been gaining more traction lately is mindfulness, which is basically using meditation to clear the mind of ruminating thoughts.

I've read bits of Feeling Good Handbook, which is an often recommended book for CBT, but I really felt like I connected more with the mindfulness approach. Mindfulness seems to work faster for me since I can use meditation to quickly clear out negative thoughts rather than trying to reason about them over time as with CBT. For mindfulness, I'd highly recommend Hardwiring Happiness, which goes into depth about using neuroplasticity (the ability to rewire the brain) to pave more positive neural paths using mindfulness techniques.

I'm sure both CBT and mindfulness are good approaches and can be used in conjunction, and there's likely other good alternatives to these as well. Ultimately the goal is to balance our evolved negativity bias with more positive thinking.

<comment><USER18> I totally agree about the mindfulness. I started to use the app, [https://www.headspace.com](https://www.headspace.com). Which walks you through mindfulness meditation.

I've noticed a stead improvement after using it for a while.

<comment><USER19> That app on android needs some strange permissions, access to photos and storage and access to device id and call information plus some other stuff. I can't see why it would need to know what numbers I call.

<comment><USER20> The actual permission is 'read phone status and identity', which I guess is used to pause a meditation session if you get a phone call. 'Access to photos' is needed to get access to external storage, presumably to download audio files.

Disclaimer: I'm not an Android dev, this is just what I've picked up from the internet.

</comment></comment></comment></comment><comment><USER21> I once heard Sam Harris talking about a similar process. Most people sort of _are_ their thoughts. That is -- this is tough to describe plainly -- their thoughts and feelings are primary, driving their actions as if mechanically linked. But you can in a certain sense become aware of your thoughts and view them as an outsider, as something that's happening that you can see and choose how you react to. It's not just about self-assessment. It works during acute "attacks" too.

It's crazy, but hearing him say that made a huge impact on an anxiety problem that had been getting worse for me. Thinking of it this way has without exaggeration changed my life.

<comment><USER22> That sounds fascinating. Do you happen to have a link to anything he's written or recorded on the subject?

<comment><USER23> I've listened to Sam Harris' stuff too not long ago and I'm not sure if it changed my life, but it definitely blew my mind and is part of a broader interest in this type of stuff that _is_ changing my life. I don't agree with all his opinions, but his interview with Tim Ferriss and his book Waking Up are both fantastic.

Tim Ferriss Interview: [http://fourhourworkweek.com/2014/06/18/sam- harris/](http://fourhourworkweek.com/2014/06/18/sam-harris/)

Waking Up (narrated by Sam himself, which I prefer): [http://www.audible.com/pd/Religion-Spirituality/Waking-Up- Au...](http://www.audible.com/pd/Religion-Spirituality/Waking-Up- Audiobook/B00M9KCT12)

<comment><USER21> Harris was initially my least favorite of the Four Horsemen -- Richard Dawkins, Christopher Hitchens, Sam Harris, and Daniel Dennett -- but my opinion has completely changed over the last year. With Hitchens gone, he's playing the role of public intellectual better than anybody else I'm aware of right now. I highly recommend his podcast.

<comment><USER24> I lost some respect for Harris when he tried to lay into Noam Chomsky and failed miserably [http://www.alternet.org/books/what-happened-when-sam- harris-...](http://www.alternet.org/books/what-happened-when-sam-harris-tried- and-failed-embarrass-noam-chomsky)

<comment><USER21> I was able to make it through about half of that before the irrelevant mewling about "privileged powerful male elites in U.S. society" became too much for me.

If anybody is interested in what Harris actually had to say, they should go read it for themselves:

[https://www.samharris.org/blog/item/the-limits-of- discourse](https://www.samharris.org/blog/item/the-limits-of-discourse)

<comment><USER24> yeah, because that's a totally neutral take on what happened.

</comment></comment></comment></comment></comment></comment></comment><comment><USER25> This article reminds me a lot of my personal experience with cognitive behavioural therapy.

I've long suffered from social anxiety and after years of anti-anxiety drugs and some therapy, I finally tried CBT and it was essentially:

\- write down the thoughts

\- identify what triggered them

\- find alternative (positive) conclusions

At first I thought it was ridiculous but it actually works after a while.

<comment><USER26> And some people pay thousands to have them those things explained

<comment><USER27> Well when you are depressed/anxious or just in a bad state of mind it is very difficult to follow simply steps; But thankfully I didn't need to pay since my university offered free mental health support

</comment></comment></comment><comment><USER28> I have extreme anxiety, caused by OCD where the compulsions are mental rather than physical rituals. Looking through this article, all of the tips are tremendously powerful.

Unfortunately, they're things that have taken me over 5 years to really become aware of, accept, practice--oh man, is persevering difficult--make mistakes, and finally get better at. Unfortunately, as with many mental habits, it's really, really hard to implement advice when other people just tell it to you. It's a much longer process than that.

Still, all the points are great.

<comment><USER29> Hi, fellow OCD sufferer here.

Are you using dietary supplements to treat your OCD/anxiety?

I've found some helpful pointers here [1][2][3], but I'm only starting to explore the field.

[1] Overcoming Ocd & Depression: My Personal Journey and Recovery, White, David B.

[2] [http://www.alternativementalhealth.com/natural-healing-of- ob...](http://www.alternativementalhealth.com/natural-healing-of-obsessive- compulsive-disorder-ocd/)

[3] [http://www.ocd-free.org](http://www.ocd-free.org)

</comment></comment><comment><USER30> The world, and Silicon Valley startup culture in particular, is filled to the brim with oblivious narcissists. You can pry my nagging self-doubt from my cold, dead fingers, thank you very much.

<comment><USER31> North American culture has a tendency to redefine all criticism, external or internal, as 'negativity' \-- to the point that, from my observation, organizations will a) fail without realizing it b) fail unnecessarily due to inability to deal with criticism c) define unrealistic goals on the assumption that "I think I can" will get them there.

I think this is worse in California than other places, and infects Silicon Vally the worst.

I think it leads to a 'shotgun approach' where if enough people "try try try" then yes some do succeed spectacularly while everyone else around them crashes and burns. But for people who just want to get things done properly, being on the hurtling train of "YES WE CAN" is frustrating.

Growing up with an immigrant European (German) father, I have a hard time shutting off the critical voice, and it has led to lots of culture conflict for me within workplaces.

I think there is definitely a distinction to be made within oneself between needless and mean negativity and critique, but I think the crucial point is that when the latter is repeatedly ignored, it leads to the former.

<comment><USER32> Are you sure that this is an America/California/Silicon-Valley feature? That your own internal critical voice arose as a result of growing up with your father (a German) seems to indicate that you got it from either having an immigrant parent, a German/European parent, or a parent who was just a negative nancy. I don't mean to be critical of you or your father; I mean to point out that the criticism=>negativity idea doesn't seem to be a strictly American thing.

> I think it leads to a 'shotgun approach' where if enough people "try try > try" then yes some do succeed spectacularly while everyone else around them > crashes and burns.

This reminds me of the Orks in Warhammer 40k, wherein the sheer psychic power of an Ork can make a war machine function; the machine would collapse if a non-Ork attempted to operate it.

I think the negativity about / criticism of oneself and one's identity (e.g. the self-effacing comic) is unique to Western civilization. Do people outside the West (e.g. Africa, Middle East, Asia, and perhaps South America) view themselves as backwards for how their lives work? Plenty of people, whether on the right or left, are critical of Western policies, especially regarding military and economic campaign in non-Western countries.

I am genuinely curious if people in non-Western countries see themselves as Westerners do (e.g. more bad than good).

</comment></comment></comment><comment><USER33> I now actually re-read the book

A Guide to the Good Life: The Ancient Art of Stoic Joy /by William B. Irvine/

It describes a couple of practical techniques to overcome negative emotions. I highly recommend it for everyone, even for the ones not suffering from self- doubt. Reading it was life changing for me.

A sort summary: [https://sivers.org/book/StoicJoy](https://sivers.org/book/StoicJoy)

<comment><USER34> Here is another summary, which I recommend: [http://becomingeden.com/summary- of-a-guide-to-the-good-life/](http://becomingeden.com/summary-of-a-guide-to- the-good-life/)

I tried to read the book. It's a long read and the author shows the reader all of his thought processes. Example: The chapter about _negative visualization_ goes back and forth for a while, almost as if you needed even more convincing.

But the concept of _negative visualization_ isn't really that hard to grok, the benefits are obvious to a somewhat intelligent reader and quite intuitive and easy to implement from a summary. No lengthy convincing required.

Edit: Clarification.

</comment></comment><comment><USER35> This sort of thing has been around for a long time. See "The Power of Positive Thinking" (1952), by Norman Vincent Peale.[1]

The short version: "Insecure", by Joan Jett.[2]

The century-old version: YMCA Salesmanship Training Manual, 1920.[3]

[1] [https://www.amazon.com/dp/B000FC0SXM](https://www.amazon.com/dp/B000FC0SXM) [2] [https://www.youtube.com/watch?v=SAVeAVvJIoo](https://www.youtube.com/watch?v=SAVeAVvJIoo) [3] [https://books.google.com/books?id=CBFPAAAAYAAJ](https://books.google.com/books?id=CBFPAAAAYAAJ)

<comment><USER36> Then there's the millennia-old version in the form of Buddhism. I've been looking into Buddhist 'psychology' and it's shocking how much of this was already practiced and theorized about way back then.

</comment></comment><comment><USER37> Every time you doubt yourself, ask yourself if you're on the right path. If not, make the adjustments, sometimes drastic. If you are and just are doubting yourself, put your head down and get back to work. Not busy-busy-work, actual work. Best quote I know is "If you think you can't, you can't". Hope this doesn't come as preposterous, it's common sense, but at some stage I needed to hear it too and that helped me. WORK is HEALTH.

</comment><comment><USER38> Self doubt is a good thing. If more people doubted and considered their actions rather than maintaining the "I AM SO GREAT" attitude, and believing they are gods great gift and their actions are infallible, perhaps fewer shitty things would happen in this world.

<comment><USER39> As someone who has dealt with depression and self doubt for the majority of my short life, I will say I would be much more terrified of a world where everyone had the outlook on life and self worth that I hold than one where everyone had an "I'm so great" attitude.

</comment></comment><comment><USER40> > Imagine that you have a friend who is exactly like you in every respect. > Give him a name. Then pretend he is telling himself the same destructive > thoughts you tell yourself.

You may also be interested by the IFS therapy method which allows one to stays in Self and discusses with his inner voices/personas with curiosity and goodwill. You can read [http://www.selfleadership.org/](http://www.selfleadership.org/) and the self- therapy a step by step guide.

</comment><comment><USER41> I didn't get through the article so apologies if I missed the point.

I've run into a fair bit of this stuff recently: people who encourage me to "think positively" or "remind yourself how awesome you are."

I think it's all nonsense. I believe that self-doubt and self-interrogation are usually healthy. Obviously there's a line to be drawn. At some point self- doubt will impede your ability to act. But, in general, I think a skeptical internal dialogue is a healthy thing.

<comment><USER42> There's a huge difference between being skeptical and being self-deprecating. By all means one should play devil's advocate to their own impulses, but without practicing self-awareness, it's exceptionally difficult to come up with useful criticism of one's own thoughts and to unravel the complex interplay between thoughts, behaviours, and emotions. Because of this, many people give themselves poor/irrelevant feedback and form habits of thought that cause the same kind of negative self-evaluation to recur.

In addition, it's common for people to feel a sense of threat from their own negative thoughts, and the sympathetic nervous system doesn't very effectively differentiate psychological threats from physical. Subsequently, attacking oneself can lead to fight-or-flight response, which can cause a cascade of changes throughout the body, including the brain, which can manifest as cognitive biases, for instance attentional bias towards negative stimuli or interpreting ambiguous stimuli to be negative.

Hopefully you can see how the combination of these effects could create a negative feedback loop, causing typically rational skeptic to veer off into unfounded, inconsistent thoughts blaming oneself and generating further negative emotion.

So ultimately, I'd agree with you 100%, there's a line to be drawn. The unfortunate thing is that it's very fuzzy from within one's own mind to discern where that line should be drawn, and negative feedback loops can set up instability that overshoots that line, even when meaning well. For individuals with this kind of pathological negative self-examination, framing things positively can have a huge impact on how they are actually perceived and acted upon. And for people struggling with depression/anxiety, it can make a world of difference in outcome to simply reframe a thought like: "I should have worked harder on that work deadline. Now everyone hates me because I missed it and I'm going to get fired." as "I'll feel better about my work and my relationships with colleagues if I give myself extra time and raise concerns early about deadlines."

At the same time, an excess of positive thinking can absolutely create it's own share of psychological effects and reactionary feedback loops. For example, in bipolar disorder we see a grandiosity of thought and excess of confidence that leads to underestimating risks and inflated expectations, which collide with reality (often disastrously) leading to a large down swing in emotion, and the resulting post-manic depression.

</comment></comment><comment><USER43> What did it for me was choosing to behave as if I lived a charmed life. I'd make my plans under the assumption that it would work out. Things would very rarely go wrong (thus reinforcing the charmed life angle), and if they ever did, it wouldn't be that hard to fix it anyway. People will forgive an amazing number of things, and the worst case scenario just doesn't happen in practice.

It's amazing how much baggage you can shed with this approach. And shedding the baggage makes you more pleasant to be around (even to yourself), thus giving you better opportunities and even more ready forgiveness.

<comment><USER44> > What did it for me was choosing to behave as if I lived a charmed life.

And on a global scale (and a historic view), we actually and undoubtedly do.

"The Ancient Art of Stoic Joy" discusses this in depth (and in an actually applicable manner).

</comment></comment><comment><USER45> This is so true, I'd like to add sometimes I get a bit frazzled or swept up in something I don't need and I mentally stand aside and ridicule and laugh how silly and pathetic that part of me is. And then we laugh together and suddenly everything is trivial funny and not worth any worry at all.

</comment><comment><USER46> and on the flip side..the other challenge is to keep your hubris/ego in check. obtaining the equilibrium is the art of life

</comment><comment><USER47> Now that you know you can, should you?</comment>


The Big Data Backlash - <AUTHOR> https://plus.google.com/104306562254349871206/posts/BAaHRiEroj5

<comment><USER2> I feel Dr. Goodson is fighting a straw man here. Nowhere does the original article make the statement that Google used an "unbelievably complex model that no one could ever understand", or that we had _no_ understanding of why GFT worked (merely that the assumptions were not made explicit).

What it _did_ say though was that "Big Data practitioners" (whatever that means) too often tend to make the mistake of ignoring threats to external validity [1] because dealing with big datasets gives researchers a false sense of confidence in the assumption that "N = all". This is a very valid point and IMHO the most important takeaway of the original article, but it is not addressed here.

On a side note, I feel the pothole example (where using a mobile app to crowdsource pothole detection in Boston unknowingly led to a bias towards younger and more affluent areas) would have been more relevant towards discussing that thesis than focusing on the details of GFT.

What Tim Harford (the author of said article) denounces is the recent trend that sees people claiming Big Data is "the end of theory" [2], where traditional theories and hypothesis testing are being replaced by theory-free procedures such as validating metrics against a hold-out set. The problem is that while previously the relative scarcity of data lead social science researchers to carefully consider their data sources, the abundance of passively generated data we have now tends to cause them to forget to do their due diligence when assessing the threats to the external validity of their findings. Normally such concerns naturally arise when you're deciding on your data collection methodology or your experiment design, but Big Data analyses do tend to be different in that they focus much more on finding latent relationships in existing raw data. This also increases the risk to unknowingly fall prey to the multiple comparisons problem [3], which is the other important point Harford touches on but isn't really addressed in this post.

In that respect, yes, Big Data analyses can be more prone to this kind of problem. Even if you dismiss it as a case of bad scientists and not bad science, it remains that the general population is much more prone to overly trust findings that came out of such analyses, and articles such as Harford's are important to remind both practitioners and laypeople to be careful.

[1] [http://en.wikipedia.org/wiki/External_validity](http://en.wikipedia.org/wiki/External_validity)

[2] [http://www.theguardian.com/news/datablog/2012/mar/09/big- dat...](http://www.theguardian.com/news/datablog/2012/mar/09/big-data-theory)

[3] [http://en.wikipedia.org/wiki/Multiple_comparisons_problem](http://en.wikipedia.org/wiki/Multiple_comparisons_problem)

<comment><AUTHOR> _Nowhere does the original article make the statement […] that we had no understanding of why GFT worked (merely that the assumptions were not made explicit)._

It's very difficult to read the folllowing line from the FT article and reach that conclusion:

"The problem was that Google did not know – could not begin to know – what linked the search terms with the spread of flu."

But the authors of the science paper that Tim Harford refers to did 'begin to know' how Google Flu Trends worked [1]. That's how they developed several reasonable suggestions for what caused the over-prediction problem. In particular, the suggestion that changes in the google search algorithm caused a bias in the flu trends results, could easily be tested [2]. Perhaps we would use a large dataset to do that. And that's ok.

The suggestion that statisticians suddenly forget all of their training when _n_ reaches a certain threshold is a misrepresentation of the facts. There are bad analyses based on large data sets just as there are bad analyses based on small data sets. We have tools to deal with large datasets and multiple comparisons [3]. We don't need to throw our hands in the air and panic.

[1] [http://www.sciencemag.org/content/343/6176/1203](http://www.sciencemag.org/content/343/6176/1203)

[2] For instance we could check that to see if the over-predictions of flu cases started on the same day as the search algorithm change.

[3] To pick a random example: [http://en.wikipedia.org/wiki/False_discovery_rate](http://en.wikipedia.org/wiki/False_discovery_rate)

<comment><USER2> Thanks for weighing in, but you are completely misrepresenting my argument by reducing "Big Data" to "data that reaches a certain n" and responding to claims I did not make. As you are probably aware, doing these kind of analyses require more than pure statistics; they also require solid understanding of good experiment design, and this is precisely what I was arguing is at higher risk of breaking down in these types of analyses.

To clarify my previous post, what I was referring to (and what I believe is what is commonly referred to) when I was talking about Big Data is a specific albeit vaguely defined trend of analysis that tends to focus on:

a) mining data out of large, _unstructured_ existing datasets

b) leveraging data that has been passively generated, ie. are byproducts of normal activity and not a conscious experiment design decision

c) maximizing predictive power as opposed to validating a theory

And yes, there are many ways to do bad analyses on small data sets, but that is not the point I (or, I believe, Harford) was making. The point is these types of analyses, because of their nature, tend to require additional care regarding external validity, because:

a) many make the mistake to think that big n means you don't need to worry about sampling biases, ie. what Harford was referring to as the "N = all" fallacy and what I believe was his main point; you'll agree it tends to be more common in "big n" analyses

b) since data collection is not an issue, the real challenge is in data _cleaning_ , which requires special care because you need to think about potential biases in the way the data was generated (a process you had no control about); this can be trickier than it sounds when exploring large datasets that were passively generated, because _every_ feature is potentially subject to these biases and some are less than obvious. The Boston case was a good example, but now consider that in many real-world datasets almost all features are subject to similar considerations (and may all be subject to different biases)

c) the focus on predictive power when using theory-free metrics leads to a risk of overfitting when the possible sources of heterogeneity are not understood (ie. the assumptions are not made explicit)

d) since they've been optimized for predictive power, they give a false sense of security ("it worked on the validation set!"); this is compounded by the fact these models will often work for a while (as is the case in GFT) before breaking down [1]

e) since there is a stronger focus on exploratory analysis, addressing the multiple comparisons problems is not as trivial as you make it sound; the issue is not the statistical tools we have at our disposal [2] but making all your assumptions explicit, which is trickier in the exploratory phase (because by doing this initial phase, you are already implicitly dismissing or selecting relationships to study)

f) since these analyses tend to be very application-oriented and to function at a large scale, mistakes have the potential to be much more destructive (for example, false positives in the Target example). This is compounded by the fact that due to the technical challenges in handling complicated data, and because applications are often found in tech companies, many people who do these analyses come from a computer science background and are not necessarily well trained in statistics or econometrics

Again, none of these are insurmountable; no one is actually dismissing Big Data analyses as a whole, but they present some unique opportunities for screwing up.

[1] Incidentally, this is precisely why I said earlier that discussing the details of GFT seemed only tangential to the point: yes, the Google researchers were well aware of the limitations of the method, and so is Harford ("Google Flu Trends will bounce back, recalibrated with fresh data – and rightly so"). The relevant point is _not_ whether the Google researchers were right, but the false sense of certainty it instills for the consumers of the research, something I also made explicit at the end of my last post.

[2] Although some make a pretty good case that it is, but this is beyond the scope of this comment: [http://www.nature.com/news/scientific-method- statistical-err...](http://www.nature.com/news/scientific-method-statistical- errors-1.14700)

Edit: I forgot to address the first part of your reply. While of course the researchers knew that people looked for these terms because they are concerned by their health, what is missing is why these _specific_ queries are important: you may well find that some of these queries are more related to general concern, while some are specifically about treatment options, and others about vaccination. These may not evolve at the same time and in the same direction, and while they may have been indistinguishable in the past, it's entirely possible the first type of queries will be disproportionately affected by changes in the Google algorithm vs. others, or that some of the assumptions are only valid for one type of query and not the others. In economics (which is Harford's background), this is often considered insufficient when deciding whether to add a variable to a model.

<comment><USER3> Thanks for this respectful, detailed, and analytical contribution.

I agree that "Big Data" is a tendency that is worth talking about as if it is a new thing. There are edge cases that reside on the border between conventional moderate-n statistical analysis, and large-n "vacuum up lots of data and try to extract correlative information" approaches. Examples like the pothole-location collection and GFT are emblematic of something new, that's well beyond this fuzzy boundary.

So it's not surprising that there are new issues. Some of the fixes may be old-fashioned, but some may not.

We should also face the fact that the people doing this work often don't have any formal statistical training, so it's on the community to highlight the important pitfalls.

</comment></comment></comment></comment><comment><USER4> " _I suggest that 'Big Data' analyses are no more prone to this kind of problem than any other kind of analysis._ "

The notion that 'big data' is just as susceptible to bad statistical analysis if you ignore dynamics in the incoming data is entirely true. In that sense 'big data' is the same as every other kind of analysis. But there is another notable difference between 'big data' analysis and any other kind: the marketing. 'Big data' solutions are often sold as something you can just plug in to your infrastructure, set up some data feeds, and out pops an insightful trend analysis telling you things about your customers that you could never have understood with Excel alone. That is what's wrong with 'big data'.

Used properly, by an intelligent statistician ("data scientist") who knows their field well, big data tools are very useful for doing the things that statisticians do only faster. _But that is all._ Big data tools don't magically do clever statistical analysis on their own, and they are not a replacement for statisticians, despite what some people seem to think.

<comment><USER5> It just makes practible analysis that before would have been to costly before. I suspect most of the backlash is from MBA students who could make a spread sheet draw purty pictures but crash and burn when required to do any real work.

By that argument Lyons tea shops should have never have built LEO - after all they had a manual system that could track the cost of a bun down to fractions of a farthing (1/4 of a penny)

</comment></comment><comment><USER6> "I suggest that 'Big Data' analyses are no more prone to this kind of problem than any other kind of analysis."

To an extent, large data volumes make it more difficult for the statistician to be as nimble. Trying different algorithms, different specifications, different ways to approach the data is part of the statistical workflow; not everything can be easily parallelized and run on a Hadoop cluster.

There are insights a statistician can quickly obtain (few hours) from a carefully selected random sample of a few million observations, in memory, in a single R or Python process. The same analysis for the complete, multi- terabyte data would be rather more painful or costly to obtain.

Of course data scientists such as Martin Goodson know that (though their bosses do not always) and are used to doing exploratory analysis or prototyping on sample that fit in RAM.

<comment><USER7> It's not just volume but variety too. Most big data solutions are intended to handle large varieties of data as well as large volumes.

Once you get there, all bets are off.....

</comment></comment><comment><USER8> The "Big Data Backlash" isn't anything new: Nassim Taleb's conversation about it in his last book, in which he presents a mathematical proof that noise-to- signal increases exponentially with data, is scathing.

"Big data" means anyone can find fake statistical relationships, since the spurious rises to the surface.

[http://www.wired.com/2013/02/big-data-means-big-errors- peopl...](http://www.wired.com/2013/02/big-data-means-big-errors-people/)

</comment><comment><USER9> But all big data algorithms are not simple.</comment>


Ask HN: What's the best way to handle rejections? - <AUTHOR> In classic HN spirit, does anyone have a optimal method, theory or formula for knowing when review your strategy? This would be applicable in many forms of day to day life, weather its business, asking for pay-rise or even seeking your potential partner.<p>- how many times should you pitch before you give up or look at changing your proposition? - how do you know if the problem is your proposition or the total concept?<p>I'm sure there's a correlation between the number of times you do something & at which point you start to get rewards for your effort. I just wanted to know if there was a more educated way of going about it rather than constantly guessing. <comment><USER2> Rejections should teach you something.

I think of it like the sculptor: Rejections should be like the process of removing all the material that is not part of what you are sculpting. Each rejection should chisel another chunk off.

If it seems like it's been a while since you learned something from a rejection it's time to review.

Even after refining to the point of having a finished sculpture, you still need to consider the possibility that you've got something good, but simply haven't pitched to the right person yet. Look for comments like, 'that's a really compelling|interesting|novel|worthy idea, but it's not what we get involved in'. In those cases, ask for referrals to people they think might be interested.

The point to cease then, would be when you're not learning anything new, and you're not getting references to someone who might be interested.

Each of the examples you listed would have a different protocol. If you get a no when asking for a raise, or pitching investors, you should be able to get an explanation for why. But if you get rejected asking for a date, you'll likely not get a reason you can use. So some intuition is always going to be required.

<comment><AUTHOR> Thanks 11thEarlOfMar, I liked your point about learning, it's defiantly a good indicator..

What about cases that you get no response? For example I've sent an email or attempted to followup but there's no way for me to get real feedback other than knowing they clicked on the link or they opened the email because they simply don't reply or answer.

It's like going finishing & you know there's fish because they bite but you never manage to hook them. It's so frustrating even though I've changed rods, hooks & bait.

The irony is that you start to second guess as to why you sensed the bite. Was it a false positive? How many could be false positives?

<comment><USER3> there will be (many) times when, for your own emotional/mental sanity, you have to let go of attempting to comprehend someone else's behavior, and just get on with your life.

</comment></comment></comment><comment><USER4> Do you know Jack Ma? His current net worth is 28.8 billion USD. Before he became successful he applied at 30 jobs when he was looking for work and was rejected from all of them, including a job at KFC where 24 people applied and they hired everyone but him. He wrote to Harvard for a total of 10 times and was knocked back every single time.

Goals + perseverance + time = your magic formula.

<comment><USER5> Do you think Environment has anything to do with it?

<comment><AUTHOR> It's probably the right balance of nature & nurture that comes into play. The only thing to note is that both those learnings happen subconsciously but the real masters seem to be making the right decisions consciously which leads me to think that they tend to have overwhelming self discipline & self awareness.

</comment></comment></comment><comment><USER6> I optimize along the way. Each experience is a chance to look at what worked and what didn't work.

Eventually you'll break in a very interesting way. The rejections will just stop stinging, and they just become another part of the journey to your goal.

Your number one rule as an entrepreneur is finding your true believers.

<comment><AUTHOR> thanks lowglow, What do you do when you don't have a feedback loop. Let's say that I've sent 10 emails all personalized with a tailored message & propositions. I've tracked them & all recipient have open the email. Not one person has replied but I can see that they all looked & considered my offer because they clicked on the link.

At this point, I don't know if it's working or if it's failing. Do I stop & pivot because I didn't get a response or should I peruse to try and get a response from the previous 10 before moving along.

<comment><USER6> Here's the secret to sales: Keep it simple, relevant, and keep the conversation going by ending everything with a question that leads down the funnel.

If you're doing that then you have to investigate where down the funnel you might be failing. If they're opening your email but not engaging, then the body of the email:

1\. Doesn't give a clear CTA

2\. Makes the reader feel they're not your demographic

3\. Just feels spammy

4\. Is overly verbose.

Another tip is NEVER try and close right away. I usually begin with a leading question about them/their service, then hint about what I'm doing and ask how they're currently doing something similar, and then propose alternatives.

Unless your product is something they don't already have but need, you're competing with activation energy to switch. That's hard. I guess it depends on what you're trying to accomplish.

In the end remember:

1\. That business is a relationship, and a relationship is built on trust.

2\. Trust is earned.

3\. People want to feel like people.

<comment><USER6> I see I've been downvoted to -1, if you disagree with my advice let me know why you disagree.

The advice I've offered is gained through my personal experience, if you've learned a different approach please share it and help the entire community. :)

<comment><USER7> I enjoyed your advice and on the whole rarely vote for anything on HN, but there's is a certain irony in you not handling this rejection well.

Have confidence that you said what you think, there's nothing wrong with it.

Voting systems feed on ego and self-worth. And I lost respect for your original comment after seeing the reply.

<comment><USER6> Oh, I'm not asking for votes, I'm asking for optimizations. If someone sees a flaw, or knows some better techniques gained through experiences that are counter to my own, they should share them. We'd all like to know. :)

Otherwise I was wondering if it was just bot activity, which is why I sent a call out for human engagement! :^)

</comment></comment></comment></comment></comment></comment><comment><USER8> change your approach.

Every rejection gets your closer to a yes. Learn from your mistakes (if any), optimize and try again with the expectation of getting a yes.

I know its hard but try to be more robotic and take emotion out of it as much as you can.

Lastly, know your audience. Cant use the same pitch with every person.

<comment><USER9> And if you are learning from mistakes and trying again after fixing those mistakes, then also firmly believe, "If what I wish doest not happen, then something better is going to happen" to psychologically better your odds!

</comment></comment><comment><USER10> > how do you know if the problem is your proposition or the total concept?

No pitch or message is going to get to the logic center of the other person's brain without passing through the survival filters of the crocodile brain system first. And because of the way we evolved, those filters make pitching anything extremely difficult.

ON this subject, Oren Klaff is brilliant > [https://audioboom.com/posts/2273663-oren-klaff-pitch- anythin...](https://audioboom.com/posts/2273663-oren-klaff-pitch-anything)

</comment><comment><AUTHOR> I think this is a huge problem for people who struggle to find their purpose or place in society. Considering that in the last 45 years suicide rates have increased by 60% worldwide. Suicide is now among the three leading causes of death among those aged 15-44 (male and female). Suicide attempts are up to 20 times more frequent than completed suicides. As a society we really need a better way to teach & handle rejections.

</comment><comment><USER11> Rejection is easy to handle in comparison to self-inflicted fuck-ups. I still think about some of the big mistakes I made years ago, while I am hard pressed to remember the rejections I got yesterday.

Every rejection is finding one way that doesn't work. Get enough and you will end up succeeding through a process of elimination.

</comment><comment><USER12> The best way is to be happy. Your situation will change given time, the road is often a bumpy one.

</comment><comment><USER13> cant we just decompile their android app and have a look?</comment>


Ask HN: What would you do if you could take a break from your career? - <AUTHOR> I'm at a point where I'm considering taking a break from my career as a developer. I have no children but it's something I want to happen at some point so fancy doing something before I settle down. What would you do if you were in my position? <comment><USER2> Travel definitely, but also explore your hobbies. Always wanted to hike the pacific crest trail? Go do it. Always wanted to make handcrafted musical instruments? Go find an apprenticeship. Think taking a class on woodblock printing sounds interesting? Do it. As someone married with three kids and on the cusp of 40 who never took that moment to explore anything outside work, I'd strongly encourage you to go explore those hobbies and creative interests - you might find you don't want to come back to development.

</comment><comment><USER3> Spend more time with family and friends, particularly the older ones. There are stories trapped in those bodies, and they won't last forever. You could pass these things down in the future.

</comment><comment><USER4> Travel. Extensively. You will never regret it, you will always remember it, and you will always have a story or 10 to tell people when socializing. Parts of the developing world (India, Thailand, parts of South and Central America) are still cheap enough, and interesting enough, that you can get by on $1,000 or so a month while still living relatively well. I spent several months traveling >4 years ago while transitioning between both job and countries, and still talk and think about it often.

<comment><AUTHOR> Thanks for the suggestion. I've never felt compelled to travel although I've been to Japan and would like to go again. I feel like going abroad to do something might be more up my street. Do what? Is the question.

</comment></comment><comment><USER5> As others have said in this thread, travel is one sure way to get some interesting experiences (including of the Chinese kind [1] :)

I once stayed in a hut on a beach, 5-10 seconds away from the water's edge (depending on the tide), for 3-4 weeks. Could hear the roar of the ocean all night (and day). Good experience.

[1] [http://en.wikipedia.org/wiki/May_you_live_in_interesting_tim...](http://en.wikipedia.org/wiki/May_you_live_in_interesting_times)

</comment><comment><USER6> I left my programming job in 2004 and spend the next 5-6 years studying, first a year of theology and then rest on Greek and Latin. It was probably not the best move career-wise, but it's something I had always wanted to try. I learned I'm happier coding, and that Greek is more fun as an amateur than a professional. Now I'm back in development, this time as a freelancer, and things seem to have worked out just fine. I have no regrets.

<comment><USER7> For learning Greek - were you a grad student?

<comment><USER6> I took two years of night classes while working, then I spent another year reading Greek texts with a friend while I took Latin night classes. Then another couple years reading on my own (overlapping with the theology year), then a postbaccalaureate year, then grad school. :-)

</comment></comment></comment><comment><USER8> Whatever you do, follow the _The Moth 's_ advice and make it "story worthy"

Try to imagine yourself later telling the story:

"Oh I started a company" then it might not be that story worthy vs "I applied to an accelerator in another country, moved there, and started a company." Instead of "I cooked everyday," make it "I cooked something new everyday, no exceptions."

<comment><USER9> Starting a company is not story-worthy enough?

<comment><USER8> Maybe that's not the best example.

What I mean is something like...for example, the guy who has five companies and starts a sixth one isn't nearly as story-worthy as the gal who quits their full-time job and moves cross country to start their first company.

Of course it isn't an absolute rule by any means but it might be helpful. At some level it is probably in the same family as sayings like "Get out of your comfort zone" and such.

</comment></comment></comment><comment><USER10> Develop my hobbies, outside of my career (my career is also my hobby.)

In addition to tech, I like working with wood, electronics, and firearms. Would like to be able to build cooler things with electronics, wood and be a better shooter. I do all of these things a lot in my spare time but would like to do so more often.

</comment><comment><USER11> There is no substitute for travel. Experience the world while you have the freedom. It will change your life.

</comment><comment><USER12> Primarily: \- motorcycle touring (within India) \- read \- meet people

and whenever I can, I would: \- teach and mentor (programming, electronics, critical thinking) to my nephews, son and some school students \- hack (code, diy, electronics) to build things for my home, motorcycle, and kid

</comment><comment><USER13> I'm a teacher by career. If I had the opportunity today, I'd go do an internship in SV for the experience.

But when I was in your shoes I spent 13 months living on a bicycle. I was 26 when I did that, and it's kept me grounded for the last 16 years.

Go have an adventure!

</comment><comment> jags-v Learn a new language and take care of your health.This would be a good time to evaluate your financial well being too.You could volunteer too.You would be able view the same things from a very different perspective.

</comment><comment><USER14> I'd travel the hell out around the world, then start a company with (most likely) better human skills and start a family with amazing stories to tell my kids when they grow up.

</comment><comment><USER15> I am currently in between jobs. I spent most of my time catching up on TV shows, Movies and learning dancing.

</comment><comment><USER16> Maybe a course in something totally different in another country? Art or philosophy for example

</comment><comment><USER17> do the thing you really like, I like to start a project, private, or open- source, I spend on it full-time, never worry about someone will disturb you, that is wonderful, and I think it will helpful to you next career

</comment><comment><USER18> Go live in another country for 6-12 months

</comment><comment><USER19> Stained Glass & Woodworking.</comment>


A Visual Guide to State in React - <AUTHOR> https://daveceddia.com/visual-guide-to-state-in-react/ <comment><USER2> > This object describes the entire state of this app. And that, in a nutshell, > is what React's state is for.

Isn't that still confusing? When someone says React state, I think component state. React state is usually reserved for UI interaction state.

In the example, let's assume the source of data is from an IoT service. Fetching data would be be pushed/pulled then the application state updated outside of the components. Application state changes then propagate down to components (PowerUsage, IndoorTemp, OutdoorTemp) as React properties not React State.

<comment><USER3> State in React isn't exclusively reserved for UI interaction state like modals and dropdowns. It's also common to store server data in there if it should persist between renders (which it usually should).

As another example, imagine that a chat app is receiving messages from a websocket. Each time it receives a message it should re-render the list of messages. The "current list of messages" is state. Each received message gets pushed onto an array inside state, which triggers a re-render.

That's not really "UI interaction state" in the sense that the user triggered it, but it's definitely state the UI needs to track unless it wants to fetch the entire list of messages every time a new one arrives.

<comment><USER4> I don't have experience using websockets with React, but right now I handle absolutely everything that comes from the server by putting it into app state. Is there a reason for changing that paradigm in the chat example?

</comment></comment></comment><comment><USER5> The article starts off well, but, like almost every single other article ever published on the theory of react, it stops short of discussing the use of state outside of a toy example.

Redux and state in an app with multiple components is given one or two sentences at the end.

Discussing where to apply state (to containers?) Is given one or two vague sentences at the end.

Meh... ...disappointed. The writer ia just rehashing the same content from other articles. Albeit somewhat less vague.</comment>


Lazyhub: Terminal UI Client for GitHub - ryo-ma https://github.com/ryo-ma/lazyhub <comment> ryo-ma Binary releases are available. You can try lazyhub from [https://github.com/ryo-ma/lazyhub/releases](https://github.com/ryo- ma/lazyhub/releases)</comment>


Google age bias suit reaches California Supreme Court - <AUTHOR> http://www.mercurynews.com/breaking-news/ci_15148451 <comment><USER2> I wasn't around Google 8 years ago. But I can verify that today Google is perfectly fine for older workers. The guy who sits next to me is in his 50s. Absolutely _nobody_ doubts his abilities. Nor does he have any complains about how he is treated. And the last guy I helped interview who got hired (I'm not sure for what team, hopefully for ours), is of similar vintage and I believe will work out well.

The one warning I'll give to older programmers who want to come to Google is that everyone codes here. There is an attitude that some people get after decades in the working world. That attitude is, "I'm so senior and experienced that it makes no sense for me to actually do something so menial as write code." If you have that attitude, then Google won't have a place for you. There is no such thing as a non-coding software architect here.

But if you have experience, ability, understand algorithms, and are willing to code, then Google can be a very friendly place indeed. People here are very aware of the difference between one year of experience 10 times, and 10 years of experience. The former has no place here. The latter is highly valued. And 20 years of experience is even more highly valued.

(Disclaimer: I'm only 40, and have only been in the software world a dozen years.)

<comment><USER3> Maybe it differs by location, but it is not true that everyone codes.

Within engineering, about 1/3 of tech leads do not code beyond mocks, and no one in engineering management codes. Staff and above also have diminishing coding activity; which is necessitated by the requirements of group leadership and planning.

You can speak to SRE coding activity. But to continue, many in test only manage contractors and their coding is no more than small python scripts for tool automation.

And most importantly, Program Management is the single quickest growing group in terms of influence and employee percentage. I do no believe most even remember how to code.

<comment><USER2> I know that in my corner, everyone codes unless their job is clearly a non- coding position. For instance managers don't code, but you're not going to get an engineering interview as a manager. You likely know the broader organization better than I do, so I'll take your word that there are non- coding people out there.

But that said, I'd be very, very surprised if anyone got through the engineering interview process without being asked to write code. Multiple times. So even if you did wind up not writing much code, you're not getting the job unless you're both willing and able.

</comment></comment></comment><comment><USER4> Sometimes I wonder if laws like this actually increase discrimination. I had a professor who hired a female minority who had good qualifications and interviewed well, but turned out to be incompetent. He was fanatical about documenting everything, noted down when she made mistakes and actually tried a lot to help her fix them, but no luck, so he let her go, and then she sued. The suit was eventually dismissed, but he said it was very stressful and costly for him.

So I wonder if situations like these make people say, "Screw it, I'm hiring someone that can't sue me if they don't work out." Not a good thing, but I think a very possible secondary effect of this sort of thing.

<comment><USER5> I'm surprised that she would sue. After all, she must realize that she was treated fairly and that she's going to make it incredibly hard for the next female minority to get hired, who might be qualified _and_ a nice person to boot. Stuff like that tends to linger, even if it is subconscious and against the law.

<comment><USER6> _I'm surprised that she would sue_

She may have never heard criticism of her abilities in her entire life, was working for a boss who had nothing but positive feedback for her for an extended period of time, and then was suddenly canned with no warning for being incompetent. Every professor and teacher she's ever known is in unanimous agreement: she's awesome. _Clearl_ y, the boss must be racist.

</comment></comment></comment><comment><USER7> Some background in his wikipedia entry at [http://en.wikipedia.org/wiki/Brian_Reid_%28computer_scientis...](http://en.wikipedia.org/wiki/Brian_Reid_%28computer_scientist%29) when they fired him it was nine days before the IPO and he lost out on $10M.

The opinion from the appeals court is here <http://www.gmsr.com/writing/scw- reid.pdf> he offered a statistical analysis of promotion and compensation that indicated a pattern of discrimination against older employees, Google did not offer their own statistical analysis. He has more than hearsay or "stray comments" driving the case.

<comment><USER8> > offered a statistical analysis of promotion and compensation that indicated > a pattern of discrimination against older employees

How could statistics possibly indicate causality?

<comment><USER5> I don't think he has to prove causality, it is more that if he manages to sow enough doubt that he was wrongfully terminated by showing that in the statistics he is an outlier that it will bolster his case.

<comment><USER8> That saddens me. The law should be based on fact and reason.

<comment><USER5> The law _is_ based on fact and reason. Judges are there to interpret the law and apply it to real life situations which are gray instead of black or white.

<comment><USER8> In that case, why bother showing statistics demonstrating correlation unless there's actual evidence to back up some kind of causation?

A high percentage of people with certain characteristics being fired or having poor bonuses is factually not equivalent to those characteristics being the basis for those people being fired.

There are well established relationships between membership of particular groups and academic performance. It is generally not accepted to say that those people are being discriminated against when marking due to their ethnic groups unless there is specific evidence to prove that such wrongdoing has occurred. In fact, generally accepted reason behind the statistics in one not of discrimination, but of high rates of poverty amongst those groups causing other problems.

<comment><USER5> The statistics are not the only evidence, they are part of the evidence, and together with that other evidence present the case.

It's like being blindfolded and strapped to a chair and having to decided if something is on fire.

You don't have any direct evidence. But you can smell smoke, you can feel the heat on your face and you hear a crackling noise.

Now none of those by themselves are proof, but together they add up to a preponderance of evidence, and you (the judge) may (rightfully) conclude that there is indeed something on fire.

Either that or someone is going to great extremes to trick you.

Lawsuits like these are rarely decided on absolutes.

Now, the other side can play the same game, they can show partial evidence to the contrary. They can suggest that the noise you hear is made by the wind playing with a plastic bag, the heat is the sun on your face and the smoke smell is because last week there was a fire but there is none today.

It wouldn't take a smart judge very long to ask some follow up questions to figure out who has it right and who is fibbing.

When it's a really close call or when both sides 'have a point' the judge may order mediation.

More likely the parties will then settle because neither will want the case to be decided on such a slim margin because a ruling 'against' could be much more costly than a settlement.

<comment><USER8> Thinking about it, you're right.

My statistics brain shouted 'eeek, correlation != causation' but you make a pretty compelling argument with the fire analogy.

Using a programming analogy, you could think of a bayesian spam classifier. The word 'viagra' might not on it's own constitute spam, but 'viagra' 'hard p3n1s' and 'meds' scores enough to be safely considered spam.

Consider my mind changed.

<comment><USER5> Bayes actually deals with evidence so you're spot on.

Here is an interesting wikipedia article on that subject:

<http://en.wikipedia.org/wiki/Evidence_under_Bayes_theorem>

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Let me ask a possibly unpopular question because the answer is not immediately obvious to me:

Why are there laws against age discrimination?

For example, if I want to fill my company with pretty young women, competent or otherwise, why should the law prohibit me from doing so?

<comment><USER10>
        Let me ask a possibly unpopular question because
        the answer is not immediately obvious to me:

Good enough reason.

Here's an explanation I've heard: Yes, it's your business, your property, and in most things you're free to do as you like with it. But it doesn't exist in a vacuum. You can have a business and a good life because you live in a particular society, so you have some obligations back to that society. That can take the form of, well, taxes (for example), but also as special protections for select groups of people.

<comment><USER11> I don't think an "obligation" or obeying the law is necessary a good qualifier.

Let look at this way. Each law we made will have potential side-effects, which is most likely to be negative. Thus, the law of unintended consequence.

For example, a business deciding who to hire might decide on only the merit/performances of employees. However, this end up with lot of white people, because white people are disproportional represented in the tech industry. Any minority in the tech industry have to swim against lot of talented white people. So what we end up with is a firm with mostly white people.

However, discrimination against race is prohibited in our law and it _looks_ like the firm is engaged in favoring white people over other type of colors. So people either sue the firm for racial discrimination, or the firm decided a quota of people from each race to hire.

Thus, we end up with the strange situation of a color-blind company being sued for discrimination that it never engaged in, or forced to take a racist route to avoid lawsuits.

So we better be careful of how we make our laws. A law with good intentions and made with great moral conviction may end up doing the opposite.

"It is a saying among Divines, that Hell is full of good Intentions, and Meanings." -- 1654 R. Whitlock Observations on Manners of English 203

<comment><USER10> "So we better be careful of how we make our laws. A law with good intentions and made with great moral conviction may end up doing the opposite."

I absolutely agree, and offered the explanation as one I've often heard, but it isn't one I can get my heart into. But even if it's true in general principal, the idea of trying to do good by roundabout methods is indeed tricky, and if you cannot accurately measure the results you're doing witchcraft not statecraft.

</comment></comment></comment></comment><comment><USER12> So if someone is a sub standard employee, either in their own output or in the team environment and their age could be a contributing factor are you unable to fire them?

I mean a younger employee could also be slow or not on the same page as their coworkers and they would be being fired for the same reason.

<comment><USER13> He seems to be claiming that it was _mainly_ his age, which would be less permissible. In particular, as far as terminating him goes (as opposed to merely changing him to a different role), the reason Google gave that they couldn't find him any future role in the company was that he was a bad "cultural fit" for the company, which is a lot more vague and questionable- sounding than saying he was a "substandard engineer" or anything more performance-oriented like that. Reid is more or less alleging that "not a cultural fit" is a euphemism for "old guy", and he claims he has some statistics to back that up, showing that Google fired old people disproportionately often.

<comment><USER14> > He seems to be claiming that it was mainly his age

If the reasons he was let go were all typically legitimate reasons but also possibly directly _due_ to age, would that be discrimination?

I can't think of solid examples for programmers, but to use another field: Suppose a firefighter is let go because his age has robbed him of his ability to perform physically. Is this ageist?

<comment><USER13> It's a complicated area, but I think generally the answer is that that's allowed, but with a fairly high bar for the employer to show that what they're testing really is a "bona fide occupational qualification". One of the important cases in the area is _Dothard v. Rawlinson_ (1977), where the Supreme Court held that Alabama's height and weight requirements for prison guards, which disproportionately impacted women, were not justified, because there was no evidence that these height and weight cutoffs correlated strongly with prison safety: <http://en.wikipedia.org/wiki/Dothard_v._Rawlinson></comment></comment></comment></comment>