Apache Spark Scale: A 60 TB+ production use case - <AUTHOR> https://code.facebook.com/posts/1671373793181703/apache-spark-scale-a-60-tb-production-use-case/ <comment><USER2> I tried (and failed after a month of work) earlier this year trying to do ~20TB shuffles with Spark. I felt both relief and frustration reading this post.

Relief: I'm not an idiot, and the problems in the shuffle were likely in Spark and not just me being a beginner user.

Frustration: I wanted to group about 100 billion x 200 byte records into 5 billion groups. This seems like exactly the problem Spark is designed for and is advertised for. I had great difficulty even getting example Spark SQL code (or my own RDD based code) working. I hear so many great things about Spark as a tool for big data, and also "your data isn't big". I considered 20TB on the "low end" of big data, but a seemingly popular and widely used big data tool can't shuffle it without numerous bug fixes and pain on the part of the user. Shuffling 90TB was worth a Facebook blog post! This makes me ask: To all the people using Spark for "big data", how painful is it and how much data are you handling? It appears the answer for >=20TB is "very painful", and for <=5TB I think you're generally in "handle on single node" territory.

<comment><USER3> One of the hardest parts for me was getting the cluster sized appropriately so that all data stayed in memory. Overflow to disk slows things down a lot. But, sizing the cluster can be tricky if you're generating a lot of data structures in the tasks.

I only use RDDs, so I put a lot of thought into the processing flow so that unnecessary data wasn't shuffled. If you're reducing PairRDDs, make sure that the data is evenly distributed. Also, I'm guessing you read the optimization docs, but a huge amount of network I/O can be reduced by choosing the right types and collections and optimizing serialization. And, of course group within partitions first, then within nodes, then across nodes. And, of course, go for fewer bigger servers with lots of network bandwidth.

There are a lot of tricks, unfortunately. And, since I don't know your experience level I won't bore you with things you probably already know.

<comment><USER4> I have no practical experience with Spark, but I was wondering where Alluxio fits into what you're describing.

Is it only applicable once the "cluster [is] sized appropriately so that all data stayed in memory" as you mention?

<comment><USER3> Alluxio can cache objects in memory that are used downstream in other steps or even other jobs.

In Spark, the biggest performance gains are realized when all data stays in memory. But, the Spark memory architecture is kind of opaque. It's pretty well documented, and there are some good blogs out there, but there are a lot of factors in play.

For example, let's say we start with 1 TB of compressed data and take a guess that it will uncompress to 10 TB. So, we do a naive calculation and guess that we'll need 20 nodes, each with 500 GB of memory.

For our example, Spark will be managed by Yarn. Yarn is going to take a portion of each node's memory. I forget how much, but let's say 10%. Then, depending on how many executors we have per node, each will also have memory overhead. Again, I don't know exactly, but let's just say another 10%. Then, some memory will be dedicated to system processes, etc. Then there's the Spark driver. So, after all of that, let's say 25-30% of memory is used outside of the executor. So, that leaves us with 10 executors per node, each with ~35 GB of memory. For maximum performance, we have to ensure that all 10 TB fits across all executors. So, we may actually need 29 nodes, or a lot of data is going to spill to disk.

Then, let's say that our processing job doesn't evenly distribute data across partitions. Some partitions get 5x as much data as other partitions. Again, data spills to disk - or the VM runs out of heap. Or, maybe our 10% compression estimate is off by 5% - more spills to disk.

</comment></comment></comment></comment><comment><USER5> I've been working on Databricks for a month now and it is a very unique and satisfying learning experience. I generated a billion n-grams (1-6 word) from 300k docs and thinking in RDDs and Dataframes reminds me of how I felt when I first learned functional programming and GPU coding. RDD flatMap solves problems I didn't even know I had.

I ran a 'small' cluster of 24 nodes (300GB+ of RAM) and while I know I could have done the same thing with super optimized code on a single machine, not having to worry about each node's performance but instead thinking in pipelines is refreshing. My goal was not to use/make optimal tokenizer or stop-word remover but instead to make sure every part of the chain could be done in parallel.

My biggest complaint? Repeatedly having to restart the cluster because nodes stop responding or throw arcane errors. If these reliability fixes were in place last week, I would have easily been 20-25% more efficient with my time. Can't wait till Databricks team deploys this for their users.

</comment><comment><USER6> What I find most exciting here is that they were able to improve performance by combining the three previous steps into one, in other words asking directly for what they wanted rather than trying to trick Hive into executing it in a certain manner.

Presumably the task had been split up in the past to work around performance limitations of Hive; the fact that they got better performance in a single step shows that the tooling is improving and much less complexity is required to implement this kind of job.

</comment><comment><USER7> This is very intriguing, I cannot help but wonder how this operation would have performed on something like Google Big Query. I know that it is highly unlikely that Facebook would ever load their data to the Google Cloud Platform, but it would be an interesting comparison.

<comment><USER8> While both Spark and BigQuery do the shuffle step in-memory, there are some differences[1]:

\- BigQuery's execution is pipelined (don't wait for step 1 to finish to start step 2)

\- BigQuery's in-memory shuffler is not co-tenant to compute

And, of course, it's one thing to have software and hardware. BigQuery provides a fully-managed, fully-encrypted, HA, redundant, and constantly seamlessly maintaned and upgraded service [2].

[1][https://cloud.google.com/blog/big-data/2016/08/in-memory- que...](https://cloud.google.com/blog/big-data/2016/08/in-memory-query- execution-in-google-bigquery)

[2][https://cloud.google.com/blog/big-data/2016/08/google- bigque...](https://cloud.google.com/blog/big-data/2016/08/google-bigquery- continues-to-define-what-it-means-to-be-fully-managed)

(disc: work on BigQuery)

<comment><USER3> Would pipelining help much when the processing job is CPU bound (all cores maxed out)?

Sorry - what does co-tenant to compute mean?

<comment><USER8> Pipelining helps with hanging chads, tail latency of work steps. If you have a slow worker (due to, say, data skew), entire job slows down. All other workers are sitting idle, waiting for the one worker to finish their piece. Read [0] to see what Dataflow does, and BigQuery/Dremel do very similar stuff to deal with this issue. BigQuery also doesn't have to wait for ALL workers to finish step 1 before proceeding to step 2.

By co-tenant to compute, I mean that processing nodes themselves handle the shuffle in Spark. This can cause non-obvious bottlenecks. BigQuery handles shuffle outside of the processing nodes [1].

[0] [https://cloud.google.com/blog/big-data/2016/05/no-shard- left...](https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind- dynamic-work-rebalancing-in-google-cloud-dataflow)

[1] [https://cloud.google.com/blog/big-data/2016/08/in-memory- que...](https://cloud.google.com/blog/big-data/2016/08/in-memory-query- execution-in-google-bigquery)

</comment></comment></comment></comment><comment><USER9> Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?

<comment><USER10> Looks they are using the same library from Presto. You would just add the Java agent line to your Spark config i.e. spark.executor.extraJavaOptions.

[https://github.com/prestodb/presto/issues/4004](https://github.com/prestodb/presto/issues/4004)

Although you have to distribute the library across to all of the data nodes (if using YARN) or Spark nodes (if in standalone).

<comment><USER11> Thanks! I was going to ask where I might find it since Google results were a little thin on the ground.

</comment></comment></comment><comment><USER3> "While running on 20 TB of input, we discovered that we were generating too many output files (each sized around 100 MB) due to the large number of tasks."

I could be completely missing something here, but to decrease the number of output files you can coalesce() the RDD before you write. For example, let's say you have a 20 node cluster, each with 10 executors, and the RDD action is split into 20000 tasks. You may end up with 20000 partitions (or more). However, you can coalesce, and reduce the number down to 200 partitions. Or, if necessary, you could even shuffle (across VMs but within the node) down to 20 partitions, if you're really motivated.

What am I missing?

<comment><USER3> "Remove the two temporary tables and combine all three Hive stages into a single Spark job that reads 60 TB of compressed data and performs a 90 TB shuffle and sort."

"As far as we know, this is the largest real-world Spark job attempted in terms of shuffle data size"

I'm far, far from a world class engineer, but I regularly do 90 TiB shuffle sorts. I must seriously be missing something, here.

<comment><USER12> Are you using Spark? That's the context.

<comment><USER3> Yes

</comment></comment></comment></comment><comment><USER13> Juding by the JIRA tickets most of these changes/fixes landed in 2.0.0 and some landed in 1.6.2.

I have been debating if I should go ahead and start switching to 2.0.0 and this is pushing me in that direction.

</comment><comment><USER14> Great write up. I used Hadoop a lot for two consulting customers, and Google's map reduce as a contractor. I found map reduce to be a natural way to process data. That said, Spark is a huge leap forward. I like both the developer workflow using a repl and the machine learning library mllib is excellent.

<comment><USER15> > I found map reduce to be a natural way to process data.

It is a natural way to process certain types of data in certain ways. While there are lots of use cases like that, I have seen too many efforts to force things into Hadoop MapReduce/YARN or similar platforms just because it is the parallel processing hotness of the year.

</comment></comment><comment><USER16> Amazing work and nice article! Congrats to the Facebook and Databrick teams!

</comment><comment><USER17> 60T sadly isn't that much. We tried (and failed) to use spark on a several PB dataset and it failed miserably.

<comment><USER13> Do you have anymore details? What version?

<comment><USER17> I want to say it was the highest 1.6.x around Feb/March of this year with a few PB (a sample of the real dataset) over an infiniband network. It just broke miserably. Also, java was never terribly good when you want to speak native ibverbs as the jni stuff is just slow.

</comment></comment></comment><comment><USER18> Couldn't they have just used Postgres on one 4/8-socket server with RAID?

A 60TB dataset can fit in one server. Isn't Spark intended for massive clusters, like dozens or hundreds of servers, over petabytes of data?

<comment><USER19> Note that it's 60TB _compressed_

Mostly it's about speed. With multiple machines you can bring more cores to bear for the processing, and have more RAM to cache partial results. Postgres could certainly do the job, but I'd be surprised if it would run within an order of magnitude of these results.

<comment><USER20> IO is pretty huge as well. You can spend lots and lots and lots of money to buy 1 machine a hard drive that can read 60 TB fast. Or you can have 100 machines with the cheapest possible hard drive and smoke the total IO.

</comment></comment></comment><comment><USER21> they never mention hadoop and yahoo but mention they built hive a dozen times in one paragraph. is this how PR people think open sourced tech should be mentioned/credited? so lame. they can't think beyond their belly buttons.

<comment><USER10> Not sure what you mean. They only mentioned building Hive once.

And why would they mention Yahoo ? They stopped being particularly relevant in the Hadoop space quite a few years ago.</comment></comment>


Why Introverts Are Better at Sales Than Extroverts - <AUTHOR> http://www.inc.com/geoffrey-james/first-90-days-the-surprising-reason-introverts-are-better-at-sales.html <comment><USER3> This is a weak argument and frankly the introvert extrovert juxtaposition is trite at this point. The article sounds like frustrated happy hour conjecture by somebody who has never actually worked in sales. Cherry picks a lot in order to make its case. Yes selling takes a lot more than a smile and persistence. But why can't extroverts be good at research and listening? What if you're driven and enjoy a good balance of quiet time and social time? It's rare that anyone is this binary and thinking that way is ignorant and kind of rude. You're essentially saying "extroverts are dumb they don't know as much as they should". Come on. And also there will always be some form of cold calling in sales. The point is not literally the call, its finding a way to introduce yourself to a complete stranger and create a relationship. Whether thats through email, the phone, or walking up to someone at a conference, there will always be some version of this. If you're relying on landing pages and conversion optimization to sell at all scales of software, someone else is going to beat you by knowing a guy who knows the CTO, or being the nephew of a VP, or having the guts to walk up to someone at an event.

Theres obviously a lot to learn from archetypal human behavior, but when it's presented like this it just falls flat.

<comment><USER4> Exactly.

I have yet to meet a good salesperson who didn't research, listen, and react. That's part of the sales process.. to understood who you're selling to and why.

This is Sales 101 and has nothing to do with being a intro/extro-vert.

<comment><USER5> The best sales people I've worked with (in a software/services B2B context - specifically when the customers are large companies) are into long-term relationship and trust building.

They analyze the hell out of the customer from an organizational perspective and really try to figure out what problem they need to get solved.

They are definitely extroverts. They are interesting people to be around, have great conversational and wining and dining skills, etc.

They know their limits when it comes to technology and see themselves as facilitators of deep technical discussions between the prospective customer's tech people and the internal staff. They are happy to take the backseat when it comes to tech discussions - while still listening, taking notes, etc. (That they'll then come back with to i.e. me to try to understand and figure out the next step.)

<comment><USER6>
      The best sales people I've worked with (in a software/services B2B
      context - specifically when the customers are large companies) are
      into long-term relationship and trust building.

Adding nuance, I was intrigued by an argument [0] that "relationship builders" weren't the out-performers in sales. In fact, they were the least likely to outperform in complex B2B sales.

[0] [https://hbr.org/2011/09/selling-is-not-about- relatio](https://hbr.org/2011/09/selling-is-not-about-relatio)

<comment><USER5> It obviously differs quite a lot depending on the type of customer.

An example where I have seen the type of relationship-building salesperson outperform the aggressive kind of sales person: sales to large, stable incumbent companies in Europe.

<comment><USER6> For sure! I think the nuance (and if you read the article and the study) is the best performers are consistently "assertive" \-- or in the middle between passive and aggressive.

<comment><USER5> Yeah. They have to be able to carefully thread that needle between pushy/assertive vs annoying quite carefully and intelligently. And counteract the pushy factor with charm/hosting skills.

.. and that's why this job is so highly paid (the kind of people I'm talking about: ~300k/year or more).

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Every good salesman I've ever met has either been an extrovert, or acted the part to a tee.

As an anecdote, my heavily extroverted brother in law is the best salesman I know. He went door-to-door in San Diego selling pest removal services. He would net over $80,000 in commissions over a summer.

Maybe it's different in B2B, but in B2C the best salesmen I've seen have always been extroverts.

<comment><USER8> How do you know if someone selling you something is introverted or extroverted, unless of course they're your friend? All i see is someone trying to pitch something to me. They could be extroverted and naturally good at it or they could be introverted and acting the part out. How would you know?

I'm an introvert myself and if you see me in social situations you'd never think that I am and most people don't realize it until they get to know me personally. This is why this article makes little sense to me.

<comment><USER9> If an introvert is quiet and no one is there to hear it, is he still an introvert?

</comment></comment></comment><comment><USER10> This seems to be a bit of click bait, but when Mark Roberge at HubSpot analyzed the performance of their sales people, they found that the most important traits were conscientiousness and coachability. I've seen this come in multiple interviews on Sales for Nerds-- if you can get away from the notion in your own head that you have to be extraverted, being introverted is fine, and _may_ be an advantage because you may be more process-oriented, more willing to take rejection as a chance to fix process vs personal, etc.

Mark Roberge's book The Sales Acceleration Formula. [https://www.amazon.com/dp/B00T7Q9E2S/](https://www.amazon.com/dp/B00T7Q9E2S/)

<comment><USER11> Props Reuben, I'm a fan of your podcast.

For others that don't know: [https://www.salesfornerds.io](https://www.salesfornerds.io)

<comment><USER10> Thanks for the kind words. Selling introverts unite! ;-)

</comment></comment></comment><comment><USER12> I can't take this article seriously. I've never met a (good/great) salesman who was introverted. They are always super comfortable meeting new people and shooting the shit. Saying an introvert is better just because they research and listen doesn't exactly prove a point. All of the aforementioned great, and extroverted, salesmen I mentioned prior researched and listened - that's why they are great.

<comment><USER13> I have met several good, introverted sales-people (and like to think I was one for a couple years). But _most_ good salespeople are extroverts. The exception that proves the rule.

My definition of introvert is "drained by other people" (vs. extrovert as "energized by other people"). I'm not _shy_... I'm outgoing, friendly, good at shooting the shit. But I get home and I'm _tired_.

Extroverts, by virtue of that energy-boost, are _more likely_ to get into a enjoy sales... and do what it takes to be a _good_ salesperson (research, etc.). So that means that the _average_ introvert in sales is a "better" salesperson than the _average_ extrovert, because the extrovert population is so much larger/deeper. There's no such think as a low-performing introvert in sales, because that introvert wouldn't be in sales in the first place.

<comment><USER14> To say that introverts are drained and extroverts are charged by interactions with people doesn't seem right to me.

It feels like saying "Power lifters are energized by weight lifting; frail people are drained by weight lifting."

Interacting with people (like lifting weights) is just work. People who are good at it make it look easy. People who are bad at it get tucker'd out.

<comment><USER15> I'll offer two theories:

1) Introverts may have a higher default level of arousal, so that social situations overstimulate them and make them tired. In this case extroverts would be the opposite, i.e. their default arousal is lower, so they seek social situations to raise their arousal level.

2) Introverts process social situations more deeply, and therefore expend more energy in the same social situation than an extrovert would.

</comment></comment></comment></comment><comment><USER16> "Why classifying people into categories is bullshit"

I hate these articles that assign a label to an individual. We are complex beings that constantly evolve as we go through life and explore various experiences.

You know who's better at sales? The person that has had a wide-range of experiences, understand the domain he's selling in, and especially one that can put themselves in the clients' shoes because he has been on their side of the aisle. That person can sell, because he/she understand the value proposition of what they are selling better than anyone else.

Any time I see an article that throws people into groups "Millennials, baby- boomers, white, black, introverts, extroverts..." I just cringe at how stupid it is to categorize people based on a single trait.

The day we learn to stop putting labels on everything, is the day humanity will change forever.

We are more than a label. We are human beings.

IMO articles like this should be flagged with reason: "label bullshit".

Just my Friday $0.02 rant :)

</comment><comment><USER17> > Customers hate being cajoled or manipulated into buying something they don't > want.

I, personally, even hate when I actually buy something that I wanted, but was pressed by the seller at the same time. It takes away my joy of buying, of deciding, of choosing.

<comment><USER18> I rarely wear hats, but found a nice hat shop in Charleston while on vacation with my girlfriend. I found a hat that I liked, might have even bought it (I could use one, hair's thinning, nice for some weather conditions, hate baseball caps which are all I have).

A salesperson started hassling me, pointing me at other hats, practically put one on my head before I took it from her hands. We left, didn't buy. They lost a sale because they were too pushy.

Show that you're available to the customer, listen to what they actually say (don't pull out the items they aren't looking at unless they are actually similar and/or better by some metric - less expensive, better material, something), and get out of the way.

<comment><USER19> A salesperson helps you find what you want, and buy it from them. These people are useful occasionally, and only when they are actually knowledgeable about the products.

An advertiser tries to make you want something that you were not previously interested in. These people are the bane of my existence. I don't go shopping unless I already want something, and whatever that may be is extremely unlikely to match the thing that the advertiser is trying to push that day.

Combining the two is a guaranteed-certain way to make me leave immediately, and avoid walking in to your store forevermore.

</comment></comment></comment><comment><USER20> I never really understood the introvert vs extrovert sort of definitions. They are always measured in absolutes meaning you can't be both, yet I'd wager that most people are.

I'm a good example of how unfitting the definitions are. In my professional life I'm an extrovert. I refine my ideas by running them by other people or thinking out loud, I engage in office talk every time I think I can contribute (it's not always the case), I wastly prefer face-to-face communication, I couldn't function outside the open office space and I'm a member of every sort of club and social activity we have as well as being friends with pretty much everyone I've ever worked with.

In my private life I'm almost he exact opposite. My ideal Friday night is alone on my couch with a good book, and maybe my fiancée. It's not that I don't like spending social time with my/our friends, but it's usually a taxing experience that I need to rest after.

I score as ESTP on the Myers Briggs related tests professionally, but if I answered in regards to how I live my private life I'd be a clear ISTP. I've thought about whether one side was dominant or not, but I frankly can't say that one is.

</comment><comment><USER21> If you're an introvert like me and want to explore sales more I highly recommend "The Secret of Selling Anything" by Harry Browne Check out this book on Goodreads: The Secret of Selling Anything [https://www.goodreads.com/book/show/17622965-the-secret- of-s...](https://www.goodreads.com/book/show/17622965-the-secret-of-selling- anything)

He lays out the sales process in a very logical manner that I have not seen elsewhere. Before I read this book I too had the impression that sales was about manipulation and this book completely changed my view on it.

</comment><comment><USER22> I wouldn't necessarily say that technology has cold calling obsolete. We sell a SaaS platform, and despite doing lots of online marketing, we still get a lot of business by having salespeople ring up companies who we think would benefit from using our software and pitching it to them.

Can't say it's a task that appeals to me, but it does seem to be fairly effective.

<comment><USER23> +1

B2B SaaS here too, and targeted cold calling is our most effective lead gen channel.

</comment></comment><comment><USER19> Some introverts, maybe. I couldn't sell an extinguisher to a man on fire. Actually, I could, but it would likely be at cost, rounded up to the nearest whole dollar.

Because come on, the guy was on fire.

What if he tried entering my personal space like that?~

</comment><comment><USER24> In B2B, there is a sales activity called Appointment Setting. Appointment Setters do nothing but cold calling and scheduling appointments. They don't attend meetings or, in many cases, even work for the company that has something to sell. They learn just enough to know whom within a company to target. Appointment Setters typically are paid in part based on the number of appointments they set. You don't hire them as much as you turn them loose... These are the purest hunters and good ones are relentlessly effective.

Once the appointment is set, the person who actually makes the sales visit is a specialist in selling the product.

If the product or service is highly technical, the sales person may be backed up by sales engineers, who join the sales process at the point that the prospect is making a more serious inquiry. Sales engineers may just respond to technical questions, or, they may actually develop prototype or demonstration implementations of the product.

Pairing up Appointment Setters with introverted technical sales personnel can be an effective strategy to maximize sales production.

In the end, companies that succeed have learned how to best sell their products and services, whether it's a smartphone cover and no humans are involved in a sales transaction except the customer, or whether it's a weapon system sold to the Air Force that keeps a sales team of 20 busy for years, or your local Toyota dealer.

There is no one-size-fits-all, but if you need to figure out how to sell your stuff, you can look at successful companies that reflect yours to learn what worked for them, and then assimilate what looks effective for your business, instrument the process, and iterate.

</comment><comment><USER25> I'd argue the author is half right: people want their sales person to have these qualities (which there is plenty of evidence that introverts are better at). What he's wrong about is that people are willing to give up the social skills they are used to in good extroverted sales people. You still need those social skills to get your foot in the door. 9/10 what people want is extroverted sales people who've put effort into developing these other (often introvert-attributed) skills to round things out. Yes, the cartoon stereotype of the sleazy extrovert salesman is out. Everybody is on the lookout for them. Consequently, in the meantime, real extrovert sales people have gotten the memo and adapted. I find it very easy to classify people I meet into the different personality categories (introvert/extroverted, thinkers/feelers,serious/casual,etc.) I'm an introvert and I can spot these well-rounded extroverts salespeople a mile away. I appreciate their efforts to tune into my wavelength and I can tell when they are operating outside their natural strengths to do so. Anyone willing to adapt like this I am happy to do business with and be sold to by them.

</comment><comment><USER26> As an introvert, I'm skeptical of the conclusions of this article. Sales will always include a huge amount of customer interaction, and someone who _gains_ energy from these interactions will always have an advantage over someone who finds these interactions tiring.

If anything, I've seen a greater separation of responsibilities, where sales people are supported by technical presales folks who can do a lot of the behind the scenes work.

</comment><comment><USER27> Not only does this idea sound ridiculous (extroverts can't listen? WTF?), it sounds like the author is missing the point of the big difference between extroverts and introverts: how much they enjoy being around and meeting new people. To introverts, this is tiring, whereas to extroverts it's enjoyable and refreshing. Sales is naturally a job for extroverts because of this factor.

As an aside, this makes me wonder: what jobs are a good fit for introverts, aside from maybe 3rd-shift security? Software development, for instance, is most definitely a job for extroverts. Introverts would never be happy with all the noise, talking, commotion, and lack of privacy involved in a modern software development job in an open-plan office.

</comment><comment><USER28> What defines being "good at sales"? Is it selling more things, or is it making your customers happy? It seems that in today's corporate world being good at sales means you sell lots of things, regardless of whether your customers are happy or not.

<comment><USER29> "Good at sales" means selling and bringing in money.

</comment></comment><comment><USER28> It's a great thing that we have all these brilliant minds who just sit at home theorizing about life. How else would I know how stupid I am and how I'm doing X wrong and why Y is better than Z.

</comment><comment><USER30> Good sales people are pushy - They know how to close. Introverts are often too considerate of other people to actually get the deal over the line.

Closing the deal often requires a blatant, highly assertive statement.

When I was a sales person in a computer shop, my best closing line was:

"So you want me to wrap it up for you?"

I would say this before the customer expressed any interest in buying the computer - It actually works though - It turns out that a lot of people prefer to fork out $1000+ than to create friction by saying "No".

</comment><comment><USER31> I'll need to see some data on that.

The reason I doubt it: extroverts are 'present' focused on the person they are with. Introverts are more likely to be 'inwards' in their thoughts - <USER2> attentive to the social situation, not paying attention to emotional queues, not engaging.

Sales is mostly relationship management. Extroverts are usually better at dealing with people, finding common ground etc., introverts more likely to tackle intellectual issues.

</comment><comment><USER32> I'm so tired of this introverts versus extroverts stuff.

Seriously. You're good at what you do because you're good at what you do. Generally, the people who rise to the top or perform better are the ones who really put their minds to doing whatever it is they do, which usually means taking careful study of their field and spending some time alone to focus.

</comment><comment><USER33> FLASHBACK: "Ambiverts achieve greater sales productivity than extraverts or introverts do," > [http://www.inc.com/larry-kim/ambiverts-are-more- successful-a...](http://www.inc.com/larry-kim/ambiverts-are-more-successful- amp-influential-than-extroverts-here-s-why.html)

</comment><comment><USER34> And Scorpios make good scientists.

</comment><comment><USER35> I think the internet will virtually elimate sales for the generation that grew up with Amazon. I know it has for me, I consider any product that forces me to call someone for a quote likely too expensive for my needs.

</comment><comment><USER36> Introversion-extroversion probably is an interesting axis for analyzing socialization, but I wonder if enough people are really far enough from the center for analyses like this one to mean anything.

</comment><comment><USER37> Extroverts can also research, listen and adapt.

</comment><comment><USER38> Introverts are better at hide and seek.

</comment><comment><USER39> Is there any evidence here or is Toby just making this stuff up?

</comment><comment><USER40> and yet another article that makes things black and white... I'm sure _again_ the sweet spot is somewhere in between.

</comment><comment><USER41> this "article" is 484 words. it's so vacuous I'm surprised it wasn't published as a tweet storm.

</comment><comment><USER42> Why HN? Why? Why to bring an article to the front page and then beat the shit out of it in comments??

</comment><comment><USER43> They listen.

</comment><comment><USER32> I'm so tired of this introverts versus extroverts stuff.

Seriously. You're good at what you do because you're good at what you do. Generally, the people who rise to the top or perform better are the ones who really put their minds to doing whatever it is they do, which usually means taking careful study of their field and spending some time alone to focus.</comment>


What storytelling does to our brains - <AUTHOR> http://blog.bufferapp.com/science-of-storytelling-why-telling-a-story-is-the-most-powerful-way-to-activate-our-brains <comment><USER8> About a year ago I starting becoming incredibly interested in making ideas spread. I decided to read books on how to present myself and my ideas in a way that builds my own credibility, trustworthiness, and helps people to take on my ideas.

If you are interested in these things, I highly recommend _Made to Stick_ by the Heath Brothers [1]. It focuses on the SUCCESs framework:

S - <USER2>

U - <USER3>

C - <USER4>

C - <USER5>

E - <USER6>

S - <USER7>

They elaborate in excellent detail on each of these ideas and the acronym that they coined is a perfect example.

In terms of building your own credibility, I suggest reading _How to Win Friends and Influence People_ by Dale Carnegie. [2] This book is instrumental in understanding the basic concepts that can have a material impact on your life. I think that Autobiography of Ben Franklin [3] teaches a few key lessons in the use of diffidence that harmonizes well with Carnegie's ideas.

[1] [http://www.amazon.com/Made-Stick-Ideas-Survive- Others/dp/140...](http://www.amazon.com/Made-Stick-Ideas-Survive- Others/dp/1400064287)

[2] [http://www.amazon.com/How-Win-Friends-Influence- People/dp/06...](http://www.amazon.com/How-Win-Friends-Influence- People/dp/0671027034)

[3] [http://www.classicly.com/download-autobiography-of- benjamin-...](http://www.classicly.com/download-autobiography-of-benjamin- franklin-pdf)

</comment><comment><USER9> Tyler Cowen has a great TEDx talk (from a while back!) that hits many of the same points, in an arguably clearer way.

<http://www.youtube.com/watch?v=RoEEDKwzNBw>

</comment><comment><USER10> _Our brain learns to ignore certain overused words and phrases that are used to make stories awesome_

This applies tremendously in marketing, and is one of the reasons why marketing speak has to change constantly. Otherwise people start ignoring the cliché ad speak.

<comment><USER11> If only the author used this insight to look at the "what X does to our brains" framework that he organized his post with.

</comment></comment><comment><USER12> "Make others come up with your idea. / The next time you struggle with getting people on board with your projects and ideas, simply tell them a story, where the outcome is that doing what you had in mind, is the best thing to do."

Sounds like Inception!

<comment><USER13> I have a friend who does this to me occasionally. It pisses me off. Just get to the point, man! ;) When you know the magic trick, and when you can see the motivation while the story is being told, it's a lot less impressive.

<comment><USER14> well, story telling has its place and timing. If all he does is telling stories people. It becomes annoying, unless he is really good at it.

</comment></comment></comment><comment><USER15> I decided to abandon bullet-based decks a few years ago for images and narratives (even when dealing with programming), followed by a live-coding demonstration; anecdotally, the response to talks I've given in this format has been great.

Years later, people will spontaneously write and reminisce about such and such a talk on e.g. extending Roxygen.

This is, incidentally, the same format as my daughter's bed-time stories; coincidence?

</comment><comment><USER16> There's actually a book on writing that came out somewhat recently that talks about this topic in great detail. Was an interesting read from a writer perspective as well as general thoughts on how storytelling impacts people.

If you're interested you can search for Wired for Story.

<comment><USER17> Thanks for sharing. I've been looking for a book like this for a few weeks.

</comment></comment><comment><USER18> So, we're vulnerable to persuasion by storytelling. Try to stay skeptical when someone tells you a compelling story.

<comment><USER19> Also pay attention to the story that you tell yourself. We often create elaborate stories to justify to ourselves why we can't do something.

</comment></comment><comment><USER20> [http://www.amazon.com/Wired-Story-Writers-Science- Sentence/d...](http://www.amazon.com/Wired-Story-Writers-Science- Sentence/dp/1607742454/)

Wired for Story is a great book about the science of story. I think it applies to a lot more than writing.

<comment><USER21> Bought the book on your recommendation. Thanks.

</comment></comment><comment><USER11> OK, we know language is good communication medium for dealing with humans, hopefully. Metaphor, drama, sensory-rich descriptions and so-forth are good tools for the language. These may or may not work well put together into a story.

And consider, a "hook" like "what X does to your brain" indeed combines interest-in-your-favorite-person-yourself and the crunchy authority of science. Those can work well till they get old and start to sound thin, formulaic and manipulative.

My next blog post will be: "What crude simplifications do to ours brains"

</comment><comment><USER22> " Not only are the language processing parts in our brain activated, but any other area in our brain, that we would use when experiencing the events of the story are too."

It's good to be reminded of this...

</comment><comment><USER23> _Try for example to reduce the number adjectives or complicated nouns in a presentation or article and exchange them with more simple, yet heartfelt language._

But... what if one has developed a rich vocabulary, and truly does feel those more complicated sentence structures and Five-dollar words from the heart? Is it better to dumb down my storytelling language so the listener can more rapidly assimilate the message, if by doing so, it is no longer my linguistic passion that ignites the tale?

<comment><USER24> Are you writing for you, or for your reader? Which do you think matters more?

Also, see <http://en.wikipedia.org/wiki/Purple_prose>

<comment><USER23> Not writing; speaking. Now it's not as if I deliberately employ sesquipedalian graphology for the express purpose of obfuscation. But I've spent a lifetime developing a rich, meaty vocabulary, and I'll be damned if I'm going to talk like a public high school sophomore with a sub-20 ACT score.

Just sayin'.

<comment><USER25> I am (or at least _was_ ) an experienced assembly programmer. I'm familiar with numerous obscure x86 opcodes that take dozens of machine cycles to execute.

This knowledge comes in handy when reading other peoples' code, but I don't use those instructions myself. Why? Because I know how to achieve the same effect through carefully-scheduled combinations of smaller, faster opcodes.

Exactly the same principle applies in English composition. Use an awkward or unusual word, or a flowery modifier? That's an L1 cache miss. It'll cost a few cycles of wasted execution time, if viewed in terms of the reader's attention span. How about a word that sends the reader to the dictionary? That's an L2 miss. Those can waste thousands of cycles.

You can tolerate a few L1 misses here and there, but those L2 misses that have to be fetched from DRAM will make your "code" run like shit.

<comment><USER23> One of my team members actually ran out on her lunch break to buy a dictionary. She told me this a month or two after the fact. She needed to look up something I'd written in an e-mail. I apologized for causing her to go to the extra effort--and expense--but she wouldn't accept it. She enjoyed the verbal challenge and the sense that came with it of becoming a more literate person.

Good communication is not always about passing the message in the optimal way.

</comment></comment></comment></comment></comment><comment><USER26> When I read this I realized that ministers have been using this tactic effectively for hundreds of years (if not more--I have only read from people from the last 400 years or so). I think this component is what is missing in my own presentations. I have never liked telling stories but maybe it is time to learn a new tactic.

<comment><USER27> Even if you look at Jesus as a secular historical figure (Thomas Jefferson- style) it's pretty clear that he was a master storyteller.

</comment></comment><comment><USER28> "Picture Work" <http://www.gutenberg.org/files/41505/41505-h/41505-h.htm>

</comment><comment><USER29> Don't you like it when a blog post about storytelling finishes with the obligatory 5 lines on "oh by the way, we want comments to make it seem like our blog is hugely popular. tell us your comments on the subject!"

Looks like they forgot to tell the story of why comments would enhance their content. Fail.

<comment><USER30> Wow. The first comment on HN in a long while that made we wish for a down arrow.</comment></comment>


New Windows Is a Lucky 7 - <AUTHOR> http://www.nytimes.com/2009/10/22/technology/personaltech/22pogue.html?_r=1&ref=technology

<comment><USER2> >What kind of operating system doesnt come with an e-mail program?

The right kind? The kind that requires people learn there's a difference b/t an OS and an app, and that there are free apps for everything, both from MS and others, and that such programs are not hard to find or use?

</comment><comment><USER3> > And if the programmers at Microsoft have any strength left at all, they are > high-fiving.

      I can picture these wonderful high-5s ;)</comment>


A year ago, we decided to make the game we wanted to play - <AUTHOR> https://imgur.com/gallery/iJ38yeA <comment><USER2> Wow looks cool

<comment><AUTHOR> Thanks!</comment></comment>


Donald Trump says any US-UK trade deal has to include NHS - <AUTHOR> https://www.ft.com/content/0cc78564-86d0-11e9-a028-86cea8523dc2 <comment><USER2> Apparently not. [https://metro.co.uk/2019/06/05/donald-trump-backtracks- nhs-t...](https://metro.co.uk/2019/06/05/donald-trump-backtracks-nhs-trade- deal-comments-piers-morgan-interview-9812294/)</comment>


How a Silicon Valley Investor Views a Post-Facebook World - <AUTHOR> http://www.nytimes.com/2010/10/04/technology/04investor.html?hpw <comment><USER2> "Mark's biggest fear continues to be that Facebook will not get the chance to change the world."

Ugh, what marketing fluff. Don't worry Mark, if you don't do it, somebody else will. Or wait, is that what he's afraid of?

<comment><USER3> Is it unreasonable to want to "change the world" (sick of that phrase) yourself instead of someone else doing it?

<comment><USER2> No, not at all. But I'm weary of seeing this cast as a noble, altruistic desire to help society, rather than a businessman's desire for wealth, power and glory.

<comment><USER3> The great thing about many businesses is that one can gain wealth, power, and glory WHILE helping society. Whatever the motivations, the end result is positive.

<comment><USER2> What I'm talking about is the nonsense we keep hearing about the motivations. I'm very skeptical about this.

<comment><USER3> I find that even people who do the real save-the-world type stuff (doing unheralded AIDS relief work in Africa, for example), are in it for some sort of "selfish" motivation (feel good about themselves, appreciation from others, etc.) I find it counterproductive to speculate about people's altruistic motivations since everyone, in some way or another, is working towards their own benefit.</comment></comment></comment></comment></comment></comment>


The Large-Scale Structure of Semantic Networks (2005) [pdf] - <AUTHOR> http://web.mit.edu/cocosci/Papers/03nSteyvers.pdf <comment><USER2> This isn't good news for Word2Vec and gloVe approaches to word representation.

<comment><USER3> I wrote a thing previously noting that some of the same fractal phenomena happen in word2vec ([http://howonlee.github.io/2016/02/05/Fractal-20Wordvecs.html](http://howonlee.github.io/2016/02/05/Fractal-20Wordvecs.html)).</comment></comment>


Toyota Gives University of Michigan $22M for Robotics, AV Research - <AUTHOR> http://www.freep.com/story/money/cars/2016/08/10/toyota-gives-u-m-22m-robotics-av-research/88483712/ <comment><USER2> That's a good invest. Ryan gets more things done with 22M than the oh-so- disruptive kids with 1B in the valley...

</comment><comment><USER3> The new rav4 2016 SE has great electronics and sensors. I was quite impressed with Toyota bringing some automation. However the user interface is horrible. For a $35k car, I expect some serious software. They could have gone for android auto. But they made some half assed system. entune is ad.

I wonder if Toyota will get eaten up like Nokia.

<comment><USER4> Did they ever fix the problem of their trucks rusting out after 5 years in the snow belt?

<comment><USER5> Yes [http://wheels.blogs.nytimes.com/2008/05/07/toyota-offers- to-...](http://wheels.blogs.nytimes.com/2008/05/07/toyota-offers-to-buy-back- rusty-tacomas/)

</comment></comment></comment><comment><USER6> i'm curious why these two professors will _remain_ as adjuncts, despite having pulled in what appears to be significant research funding.

<comment><USER7> Both have tenure according to their web pages so I think the article is just incorrect.

</comment></comment><comment><USER8> Maybe they could use a tiny bit of this funding and build a replacement for Entune.</comment>