Progress on addressing online abuse - <AUTHOR> https://blog.twitter.com/2016/progress-on-addressing-online-abuse <comment><USER2> Unsubscribing from notifications for a specific conversation is the only useful feature here. Muting words/hashtags from showing up in your TL would have been useful. Regarding "online abuse", shared blocklists were much more useful than locking or suspending accounts, that's not going to help at all.

Also I wonder what exactly falls under 'behavior that incites fear about a protected group' and 'repeated and/or or non-consensual slurs, epithets, racist and sexist tropes, or other content that degrades someone'. Will Twitter enforce politically correct tweets? Is NSFW content going to be banned because it offends some people?

I think it's reasonable to ban calls for violence against individuals, but I don't think banning all sorts of hateful content is a good idea and it's not going to help removing it anyway. If people don't want to see certain content, by all means, let them hide it from their feed and notifications.

<comment><USER3> As long as Twitter applies a double standard to abuse against white men, it'll be a useless political echo chamber. Allowing "#killallmen" posters to get away with their drivel while bringing the banhammer on anyone challenging the SJW narrative is itself taking a strong and IMHO obnoxious political stance. Twitter is unbearable

<comment><USER4> All you need to do is look at the people twitter invite to their conferences, and associate with at their executive level to know that Twitter will only get worse in these regards.

</comment></comment></comment><comment><USER5> One type of abuse that seems oddly popular here in Mexico is via a trending topic. I suspect that some kind of mafia group with a lot of followers are testing their weapons (since creating a trending topic is a very good business) by insulting women, minorities and even disabled people.

Blocking, muting and reporting users isn't the answer when you have right in your face a hateful hashtag. You could look for the one who started the trending topic and report it, but with thousands of mentions, you'll have to scroll a lot.

I actually thought this "progress" was about reporting trending topics, or finding the one who started it. And then, once again, we see twitter turning the blind eye.

<comment><USER6> I don't even know how to see the trending topics while mobile. To say that a list of topics on a web version of twitter only is worse than targeted abuse from users seems silly to me. I think it should be trivial to block the trending topics list from loading in your browser, even.

<comment><USER5> On mobile, you go to search to see the trending topics.

<comment><USER6> Oh, there it is. When I tap the search icon, I'm tapping again on the input box immediately to bring up my saved searches, so I'm not focusing on that trending list (or seeing it most times) because I'm waiting for my saved searches to appear.

</comment></comment></comment></comment><comment><USER7> "We're enabling you to mute keywords, phrases, and even entire conversations you don't want to see notifications about"

So, instead of fostering more exchange of ideas and information, we're just going to have a bigger echo chamber to help perpetuate groupthink, misinformation and convenient lies?

I'm not trying to be rude but does no one else see why this is a problem?

<comment><USER8> No one deserves attention merely because they are speaking.

<comment><USER9> No one is saying that.

</comment></comment></comment><comment><USER10> I'm glad to see that Twitter is responding to what users want and need, and acknowledging that there's no easy answers here.

Any sort of blocking solution is hard, I think akin to Facebook's "fake news" problem. There's no easy way to determine what news is "fake" or not. But giving people tools that they can use to manage their own space in their own way is an important step forward.

I hope that this will encourage more people to speak out on twitter about injustice and abuse, since it gives them some ability to shield themselves from any backlash.

<comment><USER6> Facebook has un-follow and see less from tony. Twitter is just launching mute, which is similar to un-follow. I'd like both services to offer "don't show me links from tony" because that's really what I want when I choose "see less from."

<comment><USER11> I've been clicking "hide all from xyz" on article shares for a few years now and almost have all the most popular link sources covered. A blanket setting to mute article shares from some or all of my friends would be most helpful. But something tells me that these article shares are a big part of FB's engagement strategy.

<comment><USER6> Now that I've thought about it, Twitter also has "turn of retweets from Tony" which is _almost_ turning off link shares.

> for a few years now

Each time I see that option, I think how crazy I could get blacklisting every FB under the sun. It sounds like you've tried!

</comment></comment></comment></comment><comment><USER12> I find the easiest way to addressing online abuse, is not using social media, and installing comment section blockers for browsers. I get the content I want, and I don't have to be distracted by the nonsense.

<comment><USER13> This addresses nothing, but merely hides it from your eyes.

It's like fighting neighborhood crime by drawing your curtains.

<comment><USER12> The anonymous nature of the Internet will make this endeavor impossible. No one is going to be able to get rid of online abuse, ever.

Ignoring/hiding online abuse, is very effective at shutting down the culprits. If no one is effected, they can't do anything.

<comment><USER13> I'd wager that you are probably not the target of a lot of this abuse, though - so you likely wouldn't have seen much of it anyway, and wouldn't be as affected by it. (For instance, I'm betting @imadefood doesn't get a lot of tweets from eggs calling you any four-letter profanities that begin with "cu".

<comment><USER12> You'd be surprised. imadefood is one of many entities I deal with. There's always someone that wants to start something. Best just to ignore.

</comment></comment></comment></comment></comment><comment><USER14> It's a start. There will be ways around the mute filters by using alternative spellings, substituting in similar unicode characters, etc.

There needs to be a "quality" score for a Twitter user and an option to mute tweets from "low quality" users. Quality could be determined by utilizing an algorithm similar to Google Page Rank. Retweets, likes, and follows by "quality" users would be considered in a user's quality rank. Reports of hateful conduct would be weighted in as well.

It's too bad that it has taken this long for Twitter to even -start- to address the abuse. They are lucky they have no direct competitors.

<comment><USER6> Is Klout still what it used to be? I can't tell by looking at their homepage. [https://www.crunchbase.com/organization/klout](https://www.crunchbase.com/organization/klout)

<comment><USER15> At least in my circles I've never seen klout be relevant in any way other than for shady "sign in with twitter to post your score" spam

</comment></comment></comment><comment><USER16> Twitter is an anonymous comment section on steroids. How about some kind of real user verification beyond email addresses? (i.e. Phone verification) It could be an optional step, but then provide the ability to ignore any Twitter content that doesn't come from these more-verified accounts?

<comment><USER5> As stated previously, there's no easy answer. Even when validating a telephone number, then everyone would use Twilio to create multiple "validated" accounts.

<comment><USER16> Using Twilio to get around the verification may be obvious if you are a HN reader, but most people have no idea what Twilio is. Phone verification is certainly not a bulletproof solution, but I have to feel it would help a lot.

And it doesn't have to stop at phone, either. You could multiple levels of verification, like Airbnb. Then as a user you could further filter how strict you want your tweet exposure to be.

<comment><USER17> > Using Twilio to get around the verification may be obvious if you are a HN > reader, but most people have no idea what Twilio is

4chan isn't a bunch of computer illiterate folks, either.

<comment><USER16> Yep. Phone verification certainly isn't bulletproof. But reducing the deluge of abusive posts by troll accounts featuring Pepe memes as their profile image even 10 or 20% would be tremendous, in my view.

</comment></comment></comment></comment></comment><comment><USER18> They are right to focus on notifications. That's where the annoying or hostile crap accumulates (abuse, spam).

I think it would be valuable to split notifications into two areas. Low and high value notifications. High-value would tend to be stuff associated with accounts you follow, and maybe unfollowed, verified accounts judged as 'high quality'. Everything else - interactions from 'randos' \- gets filed as low- value.

Nothing needs to get hidden from users who want to see every time anyone interacts with them.

<comment><USER18> To elaborate: only high value notifications activate OS notifications (i.e. your phone beeping.)

Low-value notifications simply accumulate in a separate chronological list. There would be a visual signification in the Twitter UI when you have new unseen stuff there.

This will make spam accounts that 'like' random tweets, to get people's attention, less effective.

The downside is it raises a barrier to possibly valuable new connections being made between strangers. But not a hard barrier. It'll take _some_ extra effort for a message-recipient to engage - they need to proactively check their low- value-notifications page.

Twitter would prefer a solution that preserves a high engagement level. So I don't think they'll adopt my great idea.

Should the Twitter app on my device serve me, or Twitter Inc.? This seems like a conflict of interest!

Imagining a superior, more harmonious social networking ecosystem: separation between developers of client interfaces and message-delivering infrastructure. This is the approach Urbit is trying...

Twitter's free, ad-supported model, is not financially successful. It may not survive, whether it's disrupted by an alien computing ecosystem or not. In the course of evolving the microblogging concept, other desirable aspects may die too, like Twitter's monopoly status (and thus its near-universal reach). Maybe we'll end up with hundreds of separate networks, with varying user policies and performance qualities.

</comment></comment><comment><USER19> I don't really use twitter so I can't picture some of these interactions, but how can people be getting abusive notifications?

To get abusive notifications, doesn't the unfortunate recipient have to follow the abuser first?

Does Twitter really allow anyone to @mention anyone else without the recipients having to confirm it? Or does it in fact notify the recipient with the full text of the comment, so harassers are exploiting the approval mechanism?

<comment><USER20> You can @ anybody. They don't get to confirm / approve. They do see the full text in notifications.

<comment><USER19> I'm asking because this sounds like a recipe for disaster, which is frankly what they've been having.

If we accept the notion that Twitter resembles a public bulletin board, then unpleasant (and potentially, hate) speech can and will come at anyone, and just like in real life, often the recipient has no methods of recourse other than trying to disregard it -- not making a value judgment whether this is fair, but this is how it actually happens. Blatant threats to life and property can be reported to the authorities, but most real-life verbal harassment cannot be pre-emptively filtered.

On the other hand, if we respect that each user's timeline, notification space, and knowledge graph is controlled by the user, unsolicited, unapproved @mentions are disruptive by nature. But there doesn't seem to be a way in Twitter right now to disable unsolicited @mentions or make them require the mentionee's approval.

So Twitter, by its behavior, seems to take the first stance. Why, then, does any Twitter user have an expectation that abuse targeting them can be limited by the company?

</comment></comment></comment><comment><USER21> It might seem small, but it's a big step in the right direction. Every block/report on a user/tweets could be rolled up and adjust a spam/abuse score for that user. That would allow twitter users to set a threshold for users that show up in their feed (tweets blocked for that user, akin to google's safe search).

One challenge of this is attempts by groups to harm a user's score by blocking/reporting non-hate/spam. This requires a block or report to be weighted by the blocker/reporter's credibility. It's definitely doable, * Warning Shameless plug* a product I've worked on [https://www.inversoft.com/products/profanity- filter](https://www.inversoft.com/products/profanity-filter) handles these situations well. Without some type of credibility tracking it's usually incorrect to blindly trust reports/blocks.

<comment><USER4> No matter what system is put in place, their are large, determined, and even some well funded groups which will game any and all systems to modify, censor, or punish others if you allow them the power to 'group moderate', especially in the political space - which is a very large part of twitter.

</comment></comment><comment><USER22> Twitter really brings out the worst in people. While accounts aren't ID verified, this sort of abuse won't be affected. Let's just face it, people really suck behind an anonymity curtain.

<comment><USER23> I think the key is that _some_ accounts are verified and _some_ people choose to use their real names. Bullying seems lower for the average Facebook user because everyone can see exactly who's calling you a fat loser. And bullying on 4chan and the like seems pretty low because everyone's anonymous so no one knows if you're actually fat or not. Twitter is this weird overlap where some groups share who they really are, but the harassers are free to hide behind anonymity.

</comment></comment><comment><USER24> Very happy to see some progress being made. I've been very disturbed by the rise of coordinated hate group activities like this: [https://www.washingtonpost.com/news/morning- mix/wp/2016/11/1...](https://www.washingtonpost.com/news/morning- mix/wp/2016/11/11/get-some-of-them-to-kill-themselves-popular-neo-nazi-site- urges-readers-to-troll-liberals-into-suicide)

<comment><USER25> Just the combination of the headline and the image below it are enough to get me to close the tab on this one. I'm getting really tired of the constant accusations of racism, and this immediately set off my 'upcoming bullshit' detector.

</comment></comment><comment><USER26> 1) All anti-bias statements should include body-weight / shape. Far too many people are abused/discriminated against on the basis of their weight.

Referring to this sentence: > Our hateful conduct policy prohibits specific conduct that targets people on the basis of race, ethnicity, national origin, sexual orientation, gender, gender identity, religious affiliation, age, disability, or disease.

2\. What steps are you taking to address the seeming divide between left/right abuse enforcement? For instance, AssassinateTrump was trending for a time. Will you offer evidence showing that you are banning abusive political speech from the left and right?

<comment><USER27> Regarding (2), I think that would be counterproductive. The whole point here is to signal submission to the establishment.

</comment></comment><comment><USER28> Muting words is dumb. It's not really complicated to write n1gger instead of nigger.

Suspending accounts is the dumbest, though—the account of an abuser is likely to be in the block/mute list of hundreds of people. By suspending his account, the user will just create another and continue annoying the very same people that had blocked him already.

I wonder who in Twitter, Inc. uses Twitter at all.

<comment><USER29> A crowdsourced "blocked list" similar to ublock origin for twitter?

<comment><USER30> then you could harass people by falsely getting them on a block list.

<comment><USER9> I'm already on some block lists. I'll run into some journalists who have me blocked. Never interacted with them. I hardly interact with anyone, but some block lists just take the entire list of followers from people and add them.</comment></comment></comment></comment>


Nest learning thermostat gets refreshed with a slimmer design - <AUTHOR> http://www.engadget.com/2012/10/02/nest-learning-thermostat-gets-refreshed-with-a-slimmer-design-i/

<comment><USER2> Matt, the founder of Nest, is doing an AMA right now: [http://www.reddit.com/r/IAmA/comments/10tpxx/i_am_matt_roger...](http://www.reddit.com/r/IAmA/comments/10tpxx/i_am_matt_rogers_founder_of_nest_and_former_apple/)

</comment><comment><USER3> Any updates to the software? I'd really like to see some kind of API available over the network.

Radio Thermostat (<http://www.radiothermostat.com/>) makes a line that's not as pretty, but they provide a web/JSON interface that looks pretty interesting.

API doc: [http://www.radiothermostat.com/documents/RTCOA%20WiFI%20API%...](http://www.radiothermostat.com/documents/RTCOA%20WiFI%20API%20V1_0R3.pdf)

<comment><USER4> although not necessarily a public api, you can control the nest via rest.

here is a reference implementation <https://gist.github.com/3808559>

</comment></comment><comment><USER5> Will Nest work with a dual zone system? (Also does anyone have one of these? I have questions)

<comment><USER6> I have it in 8 zones. They cooperate well.

<comment><USER5> Are you saying you have 8 zones, or 8 zones and a nest for each one??

<comment><USER7> You need a nest for each zone.

</comment></comment></comment></comment><comment><USER8> Nice, but I'm not spending $250 on a thermostat; I have 10 thermostats in my house.

<comment><USER9> So... you're wealthy enough to own a house that requires _10_ thermostats, but don't want a premium system to control it? :-)

<comment><USER8> A 4 bedroom 2-story house with radiators in each room. I'm not poor, but we're not talking top 1% here.

<comment> brown9-2 I'm not a homeowner - is it normal to have this many radiators in a house this size? My 2BR apartment has one thermostat and my parent's 4BR house has one as well.

<comment> jack-r-abbit IIRC, if you do not have central heat, then you have a radiator in every room you want heated. Each radiator is then going to have its own thermostat to control its heat level for that room.

</comment></comment></comment></comment></comment><comment><USER10> My home has separate heating and cooling systems. (The heating is done with radiators and the cooling is a conventional AC.) Does anybody happen to know if Nest works in this type of system?

<comment><USER11> I believe it should. If the systems have separate thermostats you'll need two nests though I think (they apparently can communicate with one another).

<comment><USER12> >If the systems have separate thermostats you'll need two nests though I think

You shouldn't, unless you can't run the wires to the same location for some reason.

<comment><USER10> It's a >70 year old house, so it's a miracle when I can get wires anywhere.

What we have in our favor, though, is that the boiler is in the unfinished basement, the air handler is in the unfinished attic, and our electrician ran some pull string through a chaseway that connects the two. It may actually be possible to get both sets of wires to the same spot.

</comment></comment></comment></comment><comment><USER13> Can anyone comment on whether buying one of these is worth it and if they've seen any savings?

<comment><USER14> I'm wondering if they provide any meaningful savings vs a cheap programmable thermostat. Putting your HVAC on timers so it runs less will obviously save energy, but can Nest actually do much better than a unit from Home Depot?

<comment><USER15> Nest has an awesome feature called AirWave.

When you're running the AC, it actually learns how long it can continue to cool by just running the fan due the the lingering coolness of the coils after the compressor has been running. This causes the AC compressor to run less while you're still able to be kept cool, at significant cost savings.

Your standard thermostats currently don't do this.

</comment></comment></comment><comment><USER5> What happened with the patent issues we heard about a few months ago?

<comment><USER16> Nest went after the patents [http://allthingsd.com/20120926/nest-takes- honeywell-patents-...](http://allthingsd.com/20120926/nest-takes-honeywell- patents-back-to-the-drawing-board/)

</comment></comment><comment><USER17> I'm curious to know if some of the software features of the new Nest will make it into an update for owners of the first-generation Nest.

<comment><USER18> Yes, 1st gen are getting the upgrade as well.

From: [http://www.nest.com/blog/2012/10/02/the-next-generation- nest...](http://www.nest.com/blog/2012/10/02/the-next-generation-nest- thermostat/)

<comment><USER17> I just got an email from Nest. Subject? Oh, just "Experience 3.0 Software and a whole new Nest".

I am pleased.</comment></comment></comment>


What I learned building a career driven by "everything should be free" (2013) - <AUTHOR> https://medium.com/@1marc/what-i-learned-from-building-a-career-driven-by-an-everything-should-be-free-idealism-be97720fedee#.zd34p29w0 <comment><USER2> A blog, datepicker, and meetup events are trivial to me. They're like the freebies you'd expect in a party favor bag. It sounds more like he doesn't want to work hard, just hang out and talk, make a few small things, and somehow expects that to translate to survival.

It's a common fallacy shared by hippies: "let's all get together, feel good from drugs, sex, and music, make a few bead necklaces and paintings, and we'll live happily ever after as a peaceful family."

It's ironic he's continuing to do the same he's warning against by blogging about it on Medium.com.

Here's the secret about those "free" ideological communities... it's only free in terms of dollar bills; really their currency is love.

They want to be loved, and accepted. To hang out with fellow kind who stroke and groom them. They don't want to go out and hunt or fight. They're like the weak, infants of the tribe.

That's why someone like Stallman pushes so hard for the "free" thing, because in a highly competitive economy, he just isn't equipped to function. He can produce a few things, but all the other stuff necessary to thrive in the harsh outer world just isn't there. It's either find a way to get some love somehow, in a less competitive "free"verse, or be a loser in misery.

This guy might have more potential if he's managed to reproduce and have a spouse. Or there might be some critical flaw there that's blocking his ability to work at Google, and he's stuck without much opportunity.

Unfortunately, some people are sucked into that "free" religion when they didn't need to be. It's a disservice to society, but an advantage for the guys who can convince people to join their hippie circle. Stallman is a selfish greedy top 1%er like people accused Bill Gates of being; the only difference is, he's a lot more tricky in his deception.

<comment><USER3> > A blog, datepicker, and meetup events are trivial to me. They're like the > freebies you'd expect in a party favor bag. It sounds more like he doesn't > want to work hard, just hang out and talk, make a few small things, and > somehow expects that to translate to survival.

You're grossly underestimating the work that was put into open source, writing and running events. I worked 60-80+ hours per week throughout my entire twenties. 20-25 was largely given to coding open source, speaking and running events...all for free.

> This guy might have more potential if he's managed to reproduce and have a > spouse. Or there might be some critical flaw there that's blocking his > ability to work at Google, and he's stuck without much opportunity.

You're missing the point here. I've had an abundance of opportunities, but decided against working at these large tech companies. I'm from the Midwest was insistent on doing everything the hard way...consulting and giving as much of the time I could to open source, blogging and speaking for free.

Ultimately this article was just meant to be a brain dump of some things I wrestled with along the way. Right now I run a company which sells education, so I've changed my tune in that free is not always doing better for humanity. Arguably I'm making even more of an impact now and also being paid for it...

</comment></comment><comment><USER4> > it turns out being relied upon for mission critical software for multi > million dollar companies has it's own pressure

I thought that was really very funny "revelation" this fellow had.

Welcome to the real world.

<comment><USER3> Right but the point I was aluding to was there's a difference in pressure from being just a developer on a team like I was to being catapulted to "the guy" who's supposed to lead delivering mission critical software bug free when millions of dollars on the line. It was a simple comment saying being given more prestige will lead to even greater expectations of you...

<comment><USER4> Believe me, I get it. I've been in the position of principal engineer responsible for multi-million dollar projects multiple times. It's always interesting to see someone go through revelations when stepping into new shoes. We've all done it. In my case multiple times. Interesting how quickly one's perspective can change with the advantage of a new frame of reference.

</comment></comment></comment><comment><USER5> I have a solution for you: Copy Fair licensing - give OpenSource to OpenSource, charge businesses that make money. We're also working on a collaborative CopyFair ownership: [https://github.com/CopyFairCorp/copyfair](https://github.com/CopyFairCorp/copyfair)

</comment><comment><USER6> "Built a blog to 100,000+ unique visitors per month", but yet posted on medium.com.

<comment><USER3> I killed my blog and stopped writing around 2008 and it peaked at near a few million page views in 2010. See: [http://marcgrabanski.com/grab/3c8cd1.png](http://marcgrabanski.com/grab/3c8cd1.png) \-- this is just web traffic. Not email subscribers or RSS readers.

Thanks for stopping by to question my authenticity!

<comment><USER3> Oops...missed 30 days... [http://marcgrabanski.com/grab/40fa28.png](http://marcgrabanski.com/grab/40fa28.png)</comment></comment></comment>


RIM PlayBook to benefit Quanta and Foxlink (delay due to panel shortage) - <AUTHOR> http://www.digitimes.com/news/a20110406PD224.html <comment><AUTHOR> from the article: "Sources from touch screen panel makers also pointed out that PlayBook shipments were postponed for about a month from the original schedule due to a delay in software testing as well as shortage of touch panels because Apple already booked up most of the available capacity."</comment>


Google's Eric Schmidt: 'The math is that the American economy is doing well' - <AUTHOR> http://www.cnbc.com/2016/11/10/eric-schmidt-of-alphabet-discusses-artificial-intelligence-at-dealbook-conference.html <comment><AUTHOR> The highlight, quote:

"Another high priority for the incoming government should be cybersecurity, Schmidt said. He said disclosures from WikiLeaks and insiders like former NSA analyst Edward Snowden will become ever easier.

"An obvious thing that the next president should do is have a division that audits the security of the data of our citizens," Schmidt said."

You complained about Thiel, but here we are, just two days in and we have Schmidt, one of the top Clinton people, already recommending his services to Trump's administration in suppressing whistleblowers. A nice game of spot the fascist we have here.</comment>


Let's implement a Bloom Filter - <AUTHOR> https://onatm.dev/2020/08/10/let-s-implement-a-bloom-filter/ <comment><USER2> Since the submitter is the author:

> 6 days after I started to write this post, it is proved that the math behind > false positive probability was wrong for 50 years

This isn't true.

The "Bloom math is wrong!" clamor started at least 12 years ago with [1], which brazenly cites Bloom, then proceeds to give an analysis of Knuth's subtly different filter from TAOCP instead.

There are issues with [1], but its exact false positive formula was eventually proven correct. Its argument is incompatible with Bloom's original filter, though, and Knuth explicitly states that the false positive formula he gives is approximate.

In other words, nothing was proved wrong. The analysis doesn't apply to Bloom and doesn't contradict Knuth.

Now, [1] may be the first paper to give the exact false positive formula for Knuth's filter (and Bloom's was given in 1979 by [2]), so it's not entirely without merit. But in practice none of this matters anyway. At useful sizes the approximations are accurate. Knuth's is a lower, Bloom's an upper bound, and the two become functionally equivalent very quickly.

1\. [https://cglab.ca/~morin/publications/ds/bloom- submitted.pdf](https://cglab.ca/~morin/publications/ds/bloom-submitted.pdf)

2\. Algorithm-A in [https://sci-hub.tw/10.1109/proc.1979.11543](https://sci- hub.tw/10.1109/proc.1979.11543)

<comment><AUTHOR> I believe I jumped into conclusion without properly reading the article I linked and the first reference you gave. I will update my post not to give any false information. Thank you!

</comment></comment><comment><USER3> The article has a good implemenetation, but when reading about Bloom filters, I often see claims about optimality, but it does not say optimial for what. In this example, an optimal Bloom filter for 1M elements does 7 lookups in a bitvector of 1.14MB, which is usually 3 to 5 L2 cache misses (with 256KB or 512KB cache).

But looking up the 1M elements directly in a BTree will usually lead to 1-2 cache misses and depending on the data element size, may even fit in the L3 cache. Then, what is this bloom filter saving me from?

<comment><USER4> Assuming your keys are short strings, 1M elements is 32MB plus btree overhead. If your keys are longer strings, (URLs are typically too long to benefit from short string optimization) 1M elements is still 32MB, but now it's also a pointer to somewhere else in memory with more data at the other end. So there's that; probably more than just 1-2 cache misses.

That being said, in the case of just 1M elements, a Bloom filter won't buy you much. For a bloom filter to be truly effective, you need all of the following:

1\. Many queries come back negative; that is, you spend a lot of time searching for keys that are not in the DB. This is not the typical case. When the typical web app searches for a _key_ the key is typically in the DB. (as opposed to "find all sales on November 7th" or whatever, which Bloom filters cannot assist with)

2\. The actual query is disproportionately expensive. Unless a Bloom filter can save you a disk/network call, it's generally not worth the extra overhead.

3\. You rarely/never remove items, and your data grows slowly.

Many problems that can be solved with a Bloom filter can also be solved by simply buying more RAM. Many problems that can be solved with a Bloom filter can also be solved by using a faster lookup. (everybody overlooks hash tables. I don't know why.) Bloom filters are _not_ a general purpose algorithm. You really need to grok your data before pulling the trigger on a Bloom filter. (don't forget to benchmark)

<comment><USER5> > (everybody overlooks hash tables. I don't know why.)

It is probably worse than you think since a hash table can also replace the Bloom filter in its "false positives allowed" filtration role. Not sure if that's what you meant.

If all you save is the "fingerprint" (or a B-bit truncated hash code of the keys) then you get a very similar probabilistic data structure. Further, if you put these small integer fingerprints in an open addressed hash table with linear probing then you get a (very likely) single-cache miss test for membership. This alternate structure was, in fact, also analyzed by Burton Bloom (in the very same paper he introduced his famous filter). He found fingerprint filters to be slower only on very unrealistic memory systems with neither cache nor "word fetch" structure where access to every bit had unit cost no matter its location. If you can grab a big batch of bits in unit cost the analysis changes a lot. Modern systems grab 512..1024 bits at the same cost as 1...

This fingerprint filter alternative does take 2..4x more space depending on scale and allowed false positive rates, but that space is organized better in terms of caches. Most would choose to spend that space to save k-1 cache misses of time in today's era of 60-120 ns memory and multi GHz CPUs. It is slightly easier to implement a large bitvector than an array of B-bit numbers, depending on B. { It's pretty trivial for B=8*m, though. ;-) } It's kind of crazy that hash tables for this purpose fell into "also ran/obscure" status...

<comment><USER6> If you switch from linear probing to cuckoo hashing, then you end up with a cuckoo filter, which is more compact than a bloom filter for a given false- positive rate. The compactness comes from the fact that Cuckoo hashing allows load factors of over 90% which are impractical for linear-probing.

<comment><USER5> Well, sure. You can also switch to Robin-Hood linear probing to work well at high load factors and retain the almost never >1 DRAM hit. These are just various hash table options to claw back some load factor ratios of space usage.

</comment></comment></comment></comment></comment><comment><USER7> Bing's BitFunnel paper is a pretty interesting read. The parts about scaling bloom filters by flipping their axis and using multiply filter layers is particularly of note if you are actually interested in production use.

[https://danluu.com/bitfunnel-sigir.pdf](https://danluu.com/bitfunnel- sigir.pdf)

</comment><comment><USER8> "This looks really promising! A Bloom Filter that represents a set of million items with a false-positive rate of 0.01 requires only 9585059 bits and 7 hash functions."

My immediate thought here was that this wasn't very space efficient as it requires over 9 bits per item?

I'd have to dig more into this though to confirm or deny my intuition!

Edit: this page agrees with the blog post and shows some interesting graphs [https://hur.st/bloomfilter/?n=1000000&p=0.01&m=&k=](https://hur.st/bloomfilter/?n=1000000&p=0.01&m=&k=)

<comment><USER9> Yes and no. It requires 9 bits per item _in_ the filter, but the total keyspace may be much larger. If your items are keyed based on 64-bit identifiers, you've compressed things by a factor of 7.1 .

<comment><USER10> Furthermore, most used of bloom filters I've seen are indexing something notably larger than 64-bits, and usually variable length. Arbitrary names, URLs/URIs, etc.

If you are looking for a key value then chances are it _does_ exist in the DB so the bloom filter won't save you anything as you are then hitting slower storage to read the item that the key represents anyway.

Bloom filters are suited for situations where you are looking for larger non- keyed data such as URLs, where there is a fairly good chance that what you are looking for isn't in the data yet so the "no false negatives" property is important for saving further accesses (which is why cache arrangements is where you'll often find these structures used).

</comment></comment></comment><comment><USER11> Bloom filters confused me at first, but they're really cool. I like to think of bloom filters with a (oversimplified) physics analogy. When you collide two particles together, other smaller particles come flying out of the explosion. You measure their tracks with a film. If you keep smashing particles together, eventually the film will be saturated and you will basically lose the ability to ascertain with confidence whether or not two specific particles were smashed together.

</comment><comment><USER12> I've read that some old factories used bloom filters to quickly look up where a worker is. At the morning they pressed three buttons on a wall that represented a workshop, and with a knowledge of these numbers one could check if that person was not in the workshop.

The similar technique but in a reverse I've seen in a factory myself, when 5-7 closest people at the gate queue pressed their buttons, and a controller collected their outstanding passes from the other side and then checked against their faces very quickly. That greatly saved time in a queue.

</comment><comment><USER13> > hashers: [DefaultHasher; 2],

Bikeshedding a little, but that probably shouldn't be an array. hashers[0] and hashers[1] aren't semantically equivalent and the only usage that array gets is just getting element by index. Something like `startHasher` and `stepHasher` would probably better convey intent.

</comment><comment><USER14> Very nice read. I knew bloom filters but the detailed explanation and the rust code (almost Greek to me) were very nicely explained. I also read through the linked page of Kiran.

</comment><comment><USER15> I think most people who took CS classes in college are familiar with hash table implementations, but perhaps not bloom filters.

Bloom filters are just chaining hash tables, without the chain.

</comment><comment><USER16> I like hand drawn diagrams and use them a lot, especially since work gave me an ipad, but whenever I see them in 'high-production-value', well researched and edited blog posts like this, I am reminded that it still seems to be a hard problem to quickly produce good quality vector diagrams (with foss software).

Has the state of the art moved on recently, or is it still a case of fighting TikZ?

<comment><AUTHOR> Those are not hand drawn diagrams. I used [https://excalidraw.com](https://excalidraw.com)</comment></comment>


If you were going to design a paywall, is this how you'd do it? - <AUTHOR> http://www.90percentofeverything.com/2010/06/14/if-you-were-going-to-design-a-paywall-is-this-how-youd-do-it/

<comment><USER2> I have another issue with the paywall.

With NoScript on, the links on The Times site just don't do anything (they just redirect back to the homepage). Surely some sort of noscript option is possible...

</comment><comment><USER3> One of the things they did do was move comments on stories over to the new site before the full transition, encouraging heavier online users to transition. That seemed quite a good move on their behalf, it almost made me sign up so I could troll^W write my informative comments there.

</comment><comment><USER4> Anyone know how this experiment is going?

<comment><USER5> I hear it's solved all their scaling problems at one fell swoop ;)</comment></comment>


Ask HN: What languages or tech should a freelancer learn for maximum ROI? - <AUTHOR>

I started out about eight years ago programming in PHP/MySQL and I've been freelancing for the past six. Over the last couple years, I've moved first to iOS and lately added Rails to my arsenal. I feel really good about my current situation, and I'm charging about $150 / hr.
However, I doubt this will last and I'm thinking about a few years down the road and what I should be investing in learning right now. I'd like to maintain or continue to grow my hourly rate, and while I know there are Wordpress developers who can charge $250 / hr, they're the 1% of the 1%. There must be niches where $250 / hr is normal. What are they?
What technologies, platforms, languages, etc. do you see where the supply / demand curve is the furthest out of whack AND the demand side has $$ to spend? <comment><USER2> With the rates you're able to charge, I think you already have it as figured out as anyone. You could answer this question yourself and anyone else out there would have no better idea than you.

Personally, I feel like the once and always most valuable skill is knowing how to ship from A to Z and have a proven track record of this ability. In other words, you have pushed out a quality product which has had some success (probably not enough to quit freelancing or you wouldn't still be doing it.) I believe that most agencies out there don't have this skill / experience. There really isn't much room for it in most dev shops because it's not their specialty, the overhead is high and with a healthy client base, they just don't have the resources. But there there are handful who do this (many of the *labs types of agencies.) Being able to build a product is valuable, but being able to direct (consult) the client to success after the product is built is far more valuable IMO. As an extreme example, just think of how much 37signals could charge if those guys were still doing client work. They probably wouldn't even be building products, but rather consulting for astronomically high rates (but they would still be a bargain for those who could afford to pay the tab.)

Another huge value would be to put together an A team which would be able to pull off the above. There are lots of great independent freelancers out there, but not so many crack-shot teams. If you need evidence for this, then just go look at all the aquihiring which has been going on just to grab talented premade teams.

Of course, the end game is to make valuable products, and this can break you out of the dollars for hours grind. Build a great product and you can raise your income at internet scale.

Do all this and the actual technology / platforms that you use all of a sudden doesn't seem to be of much importance.</comment>


Why Taxpayers Pay McKinsey $3M a Year for a Recent College Graduate Contractor - <AUTHOR> https://mattstoller.substack.com/p/why-taxpayers-pay-mckinsey-3m-a-year <comment><USER2> Former McKinseyite here (and one reasonably skeptical of the firm as a whole)... nobody hires McKinsey for fresh college grads. That's just something they tell fresh top of class Ivy League college grads to make them feel important. Clients hire Directors to give them peer level strategic counsel. How they technically allocate payment for time across consultants is just backfillling to get to a number already agreed to in advance.

<comment><USER3> I'm trying to parse the phrase "peer level strategic counsel".

Does it mean advice as good as the advice we're giving other firms like you?

<comment><USER2> I meant that C suite folks see McKinsey Directors as people at their level of judgement and skill who they can talk to in an open and honest manner to get advice. I'm not saying this is good/bad or right/wrong. Lots of CEOs want a consigliere with good advice who is on their side and that's what they pay to get with a Director.

<comment><USER3> Thanks for the clarification. I think my mistake was interpreting what you said as the perspective of a hypothetical company and thus assuming you meant a peer-level company. It seems in general McKinsey is rather targeted at the C-Level.

<comment><USER4> It's typically targeted at someone of sufficient seniority that the cost of McKinsey is negligible when compared to the cost of doing the wrong / less optimal thing.

Or, in other words, if a manager makes a wrong decision then it costs the company $x00,000. If an SVP or C-level screws up, it costs the company $x0,000,000+.

</comment></comment></comment></comment></comment><comment><USER5> Have been on the receiving end of several McKinsey engagements.

The work itself was generally not all that good. The knowledge of "experts" brought into meetings rarely contributed more than what a reasonably intelligent person could dig up on Google search results in an hour. They were also often farmed out on random staff augmentation functions that just annoyed the hell out of people. "Hi I need you to fill out this excel spreadsheet with 35 columns so we can put a presentation together... oh and if you could do that by 6 PM tonight that would be great." That sort of nonsense so they could produce some nonsensical 50 page PowerPoint deck that nobody read.

There were a few decent people there but by and large value was not generated. As others have pointed out a major motivator seemed to be to provide some C-level exec with CYA coverage to claim that programs being implemented were based on the advice of outside "experts."

In the two main cases I saw the McKinsey strategy ended up being a total disaster that seriously damaged the company and the C-level exec that hired them in both cases got canned as a result so in the end even the CYA concept didn't really work.

<comment><USER6> CYA = Cover Your Arse

(I just found about that one, sharing if it can help others)

<comment> blaser-waffle CYA is SOP, you FNG

(Cover Your Ass is Standard Operating Procedure, you, uh, "Friendly" New Guy)

</comment></comment></comment><comment><USER7> Sort of a buried lede here. There's nothing shocking about overpriced, underdelivering consultants. But the bit about how the GSA is essentially _profit sharing_ with said overpriced consultants at the expense of the rest of the government and taxpayers, via the IFF incentive structure, is pretty mind blowing. Another brilliant financial "innovation" by the Clinton Administration that's been quietly burning billions of taxpayer dollars to the end of significantly less efficient government for decades now...

<comment><USER8> McKinsey, Bain, BCG, et. al. are all extremely close to state and industry in all countries.

I'm from Norway, and every now and then someone will publish articles about the governments exorbitant use of consultants (McKinsey at that) for seemingly menial tasks which should be handled internally.

Truth is, these management consulting firms have some very good advantages to keep their business model going:

1) Their alma matter is spread all over the world, usually in the upper management.

2) They get highly detailed information from all sides, and can tweak their best practices accordingly.

3) They take the blame, so as others have mentioned, they're basically a million $ CYA insurance for both politicians and executes alike.

You could remove incentives like those mentioned, but I'm sure they'll find some other way to make money. They're so tightly integrated, and in so many places around the world, they probably have hundreds and thousands of strategies up their sleeves.

<comment><USER7> Yea, I don't doubt they would remain closely entangled with the US Government even if we changed the IFF incentive structure. But even so, the present incentive structure is particularly perverse because it's creating a positive feedback loop. The GSA has ~$40 billion yearly revenue because of the profit sharing model, which it is incentivized to continually increase, and the consulting firms don't even need to lobby for bigger contracts, the GSA itself lobbies for them.

<comment><USER4> The article gives short shrift to the reason this was originally put in place: to incentivize the GSA to aggressively outsource the government.

The GSA, like the departments they're serving, are filled by government employees.

Absent an incentive structure, a radical new executive commandment like "Outsource all the things" would get a yawn, papers filed, and absolutely no mass action.

And unlike private industry, I believe performance bonuses aren't really an option for government employees.

The flip side of job security is decreased initiative.

</comment></comment></comment></comment><comment><USER9> Had a lot of experience with McKinsey doing projects at my company at different levels I've been in throughout it the years. When it's highly technical things like implementing a new technology you know nothing about or a new manufacturing footprint that is tangible to perform, they do good work. Anything "Strategic" or fluffy I find a waste of time. They are excellent salespeople and do a lot to calm the C-suite but I rarely see these fluffy items driving employee happiness or the bottom line after spending millions.

<comment><USER10> Doesn't seem surprising. If it's implementing a specific technical process, there are measurable outcomes that must be met, and your client will know whether or not you met them.

Senior leadership contracting an outsider to tell them how their own industry works and explain their place in it, though, is right up there with asking the Psychic Friends Network for career advice. They're just begging to be taken for a ride.

<comment><USER10> I'm realizing that I should temper my cynicism.

"Senior leadership contracting an outsider to tell them how their own industry works and explain their place in it" might be a straw man. Strategic consulting could just as easily be, "We're considering expanding into this new market that we don't know as well, and we're hiring a consultant who specializes in that." Or, a previous employer hired a consultant for guidance on how to prepare the company to be sold.

</comment></comment></comment><comment><USER11> I was a technical project management consultant for a long time. The value that most orgs get from a consultant isn't really in the advice the consultant gives them, it's the political cover to make changes they knew they should make all along, but didn't have the social capital or the focus to make those changes until they had a person in a chair across from them.

<comment><USER12> It's _exactly_ this.

Within any organization, you can have 3 or 6 important people pushing in completely different strategic directions. And it can be clear to any unbiased observer that there's only one good, responsible choice, but within your org that gets met with essentially "but that's just your _opinion_ , man".

So you call in a consulting firm and they deliver what most of the people already know. The consultants don't need to be rocket scientists.

But afterwards, _nobody can say it 's just your opinion any more_. It's now expert analysis that you paid good $$$ for, and everybody who disagreed before now basically has to get on board.

So it's not even necessarily political "cover", but almost like a referee that brings enough credibility to settle otherwise intractable internal disputes.

Also, this lets the CEO appear unbiased to all the people who "lost". So they can get on board with the new policy but not feel like the CEO shot them down personally, which is bad for morale, can lead them to quit, etc. (Just because they believed in the wrong strategy doesn't mean they still can't be super- valuable in the future in executing the right strategy.)

<comment><USER13> >Also, this lets the CEO appear unbiased to all the people who "lost". So they can get on board with the new policy but not feel like the CEO shot them down personally, which is bad for morale, can lead them to quit, etc.

The flip side is that it is also bad for morale for someone to be recommending a solution for some time, the recommendation is ignored, a consultant is hired, the consultant recommends the same solution, and then the recommendation is implemented. This is exacerbated when the consultant's fee is some large percent or maybe even a multiple of the salary of the person who originally recommended the solution. It makes the original employee feel worthless and doubt the leadership's ability to properly evaluate solutions.

<comment><USER14> Wouldn't one enjoy some benefits from being proven right?

<comment><USER15> Right up until you realize the 23 year old snot who made the recommendation billed ~$3 million against your ~$100k salary.

<comment><USER12> If you look at entry-level consultant salaries at McKinsey and then divide by the fact that they're often working 7-day, 100-hour workweeks... you'll quickly discover that your "23 year old snot" is actually making around minimum wage on an hourly basis. They could be flipping burgers instead with a lot less stress.

The consultants aren't the ones taking the money home, the firm is. So don't blame the 23 year old. He or she is honestly just doing their insane best to pay off their huge student loans.

<comment><USER16> 100 hours a week at federal minimum wage is less than $38k. These people are not making anything remotely close to minimum wage.

<comment><USER17> But federal minimum wage ($7.25) is a lot smaller than, say, the NYC minimum wage ($15), which is more relevant. At the NYC minimum wage, 100-hour work weeks get you about 80k a year, which is about the base pay for an undergraduate new hire at McKinsey.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> $1M for a six week "EM plus two" engagement is on the high side, but it's about par for the course for most McKinsey contracts. The standard is usually $500k-$1M for that kind of engagement. I think this is kind of a non-story for anyone who is actually familiar with the consulting business model. The brand is all McKinsey really has, and they are very quick to offer free engagements to protect the brand if true value isn't being delivered.

</comment><comment><USER19> From the linked article[1]:

I find it absolutely INSANE that highly specialized organizations like the NSA rely on McKinsey to direct their future operations.

These people have intimate knowledge of their organization and culture, shaped and hardened by years of challenges and learned experiences. What can they possibly think bringing an outsider in to do a complete overhaul will solve!?

Why can they not apply their own experience to solve problems in their own organization!? The proliferation of consulting firms _baffles_ me. It seems like you're just paying people for critical thinking.

Unrelated: All of my classmates that went into consulting joke about how absurd the field feels to them.

[1]: [https://www.politico.com/story/2019/07/02/spies- intelligence...](https://www.politico.com/story/2019/07/02/spies-intelligence- community-mckinsey-1390863)

<comment><USER20> >It seems like you're just paying people for critical thinking.

That's _exactly_ what it is. Unfortunately most organizations, probably even ones like the NSA, really struggle with doing any kind of critical thinking on their own. This is especially true when trying to decide on a future direction for the org: the org itself is most likely biased to just doing things the way it's always been done. In that regard, having an outsider's perspective, especially one that has also done work at other similar organizations and has taken note of all the various ways other orgs do things and knows what works and what doesn't, can be quite beneficial.

<comment><USER19> Getting an outsider perspective - sure. That I can get behind, it can obviously be helpful.

But getting an outsider to tell you what to do? A complete overhaul? That's a leap. You're going from getting perspective to getting directed by an outsider. A recipe for disaster, and from what we're seeing here, that's exactly what has happened.

<comment><USER20> >You're going from getting perspective to getting directed by an outsider

Yes, but it's not like this outsider has no idea what they're talking about. This outsider has decades (sometimes centuries) of knowledge working with organizations exactly like yours, and they have seen what works and what doesn't. When McK goes to consult for the NSA, they likely bring along with them experience from doing similar projects for the CIA, or DIA, or DoD. Your own organization likely doesn't have much insight into the successes and failures of your fellow (sometimes competitor) organizations, while a consulting company does, and they can lend you that knowledge.

At least, that's what is _supposed_ to happen (and indeed it does happen if you are diligent about the specific team of consultants you hire). I have increasingly witnessed a lot of deception and bullshit in the consulting industry, so it's completely understandable to me why people are skeptical.

<comment><USER19> Makes sense. Thanks for pointing this out.

</comment></comment></comment></comment></comment><comment><USER21> The IFF referenced in the article seems like a great example of how good- intentioned changes to incentive structures can have very warped outcomes, potentially years later. These kinds of effects keep popping up for me, even in industry contexts (e.g. stack ranking).

Are there accepted mechanisms for systematically identifying these knock-on effects, and if so, what are they and how can they be more broadly applied? How many "hops" of influence can you get away from the change before the effects are impossible to predict?

Or does it just boil down to "ask very smart domain experts to think about the problem very hard for as long as you can afford to pay them"?

<comment><USER22> In terms of lawmaking, the usual way to deal with this problem isn't with front-end analysis, but rather by adding a "sunset provision" (see [https://en.wikipedia.org/wiki/Sunset_provision](https://en.wikipedia.org/wiki/Sunset_provision)). That's a clause in a bill that requires it to be re-authorized periodically in order to stay in effect. If a law with a sunset provision ends up causing unintended consequences, then lawmakers can let it die just by doing nothing. That's an easier lift than you get in a bill without a sunset provision, which can only be killed if you can convince a majority of lawmakers to actively kill it.

Sunset provisions aren't nearly as widely used as they probably should be.

<comment><USER23> If I were writing the constitution for a representative democracy today, I think it would include mandatory sunsets for all legislation.

Yes, this would lead to the legislature spending a lot of time just reauthorizing existing legislation, but I would argue that the majority of new legislation being passed in modern representative democracies would be better off as revamps of existing legislation anyway.

<comment><USER24> This assumes you don't have gridlock in the House/Senate, so it's a bad idea.

We already have govt shutdowns when we can't pass a budget, imagine if a law that legalized gay marriage was allowed to expire, or one that granted people health care or immigration status.

Even if a non-discrimination law only expired for a short period of time, that would be enough to do real damage to people's lives.

<comment><USER25> Other countries have solved this issue simply by automatically forcing an election if a budget is not passed. Their representative are somehow able to overcome ideological differences and pass legislation once their jobs are on the line.

<comment><USER24> That doesn't solve the problem because the idea here is that some laws should sunset, some shouldn't.

When a civil rights law sunsets, lots of bad can happen in a short period of time.

I'm using the example of the government unable to agree on a budget as an analogy.

What happens when we can't agree on how to renew the Civil Rights Act and it sunsets for a period of time?

<comment><USER25> If the civil rights act requires over 1/2 votes to extend it and it doesn't gather that much support, maybe it should sunset. The budget votes are different because they require 2/3 of the votes which is what allows the minority party to block it from passing.

<comment><USER26> The problem with this is that there may be laws that are highly valuable for poor people (e.g. minimum wage, workplace safety, building/fire codes), but irrelevant for rich people/politicians and thus end up way way down on the priority list compared to, let's say, budget laws or the next big tax cuts package.

A mandatory sunset provision can only work if the legislators act with the interests of their country and constituents first and their individual pockets second, which isn't the case any more in many Western democracies. The US are just the most obvious example where Republicans blocked everything Obama tried to pass through.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> I don't remember seeing HN articles about McKinsey before this month or last. Now there are three articles about McKinsey on HN, including one that says that the Houston Astros' sign-stealing exemplifies the decline of a McKinsified America. (Or maybe it's the Astros that are M'd.)

How did it become the flavor of the month?

<comment><USER28> Pete Buttigieg is surging, McKinsey is in the air.

[https://www.huffpost.com/entry/pete-buttigieg-secret-work- mc...](https://www.huffpost.com/entry/pete-buttigieg-secret-work- mckinsey_n_5de83018e4b00149f73aef4e)

<comment><USER29> Sundar Pichai is also McKinsey alum, so in a way McKinsey has Google now.

People are also starting to take note of some of the bad things McKinsey have been involved in like their role in Enron (CEO was McKinsey and it was their 'sandbox'), role in U.S. Immigration and Customs Enforcement (ICE), role in Saudi clampdown on dissidents, support of authoritarian regimes and the Great Recession 2008 financial crisis and many, many more scandals or questionable actions. [1][2]

> _McKinsey is said to have played a significant role in the 2008 financial > crisis by promoting the securitization of mortgage assets and encouraged the > banks to fund their balance sheets with debt, driving up risk, which > 'poisoned the global financial system and precipitated the 2008 credit > meltdown'..._

Overall, like the other management consulting firms, McKinsey and the like can be used to justify some anti-competitive/anti-consumer/anti-employee but pro- board/executive or authoritarian policies and introduce plausible deniability for the directors that employ McKinsey.

Some have said McKinsey is a "culture of corruption". [1]

> _McKinsey 's fingerprints can be found at the scene of some of the most > spectacular corporate and financial debacles of recent decades. — Ben Chu, > The Independent (2014)_

> _Defenders of McKinsey claim that the firm merely advises, and is not a > decision-maker._

Plausible deniability on both the company and consulting firm, they deny liability for all parties involved in a McKinsey action.

> _Nevertheless, since the end of the 20th century, McKinsey has been either > directly involved in, or closely associated with, a number of notable > scandals. Reuters describes these incidents as indicating "not bad apples, > [but] a culture of corruption"._

[1] [https://en.wikipedia.org/wiki/McKinsey_%26_Company#Role_in_c...](https://en.wikipedia.org/wiki/McKinsey_%26_Company#Role_in_corporate_accounting_scandals)

[2] [https://en.wikipedia.org/wiki/McKinsey_%26_Company#2008_fina...](https://en.wikipedia.org/wiki/McKinsey_%26_Company#2008_financial_crisis)

</comment></comment></comment><comment><USER30> This is true for most NOVA contractors. In 2010-2012, Accenture and KPMG was pretty new to Federal IT Auditing and most of their consultants had never done anything IT related in their lives.

It's ridiculous that accounting and tax firms like Arthur Andersen was able to rise out of Enron's scandal and were able to create catch-all IT divisions and win those contracts from the US government when they had no expertise.

</comment><comment><USER31> This type of stuff is endemic to all bureaucracies.

Managers are too scared to make decisions so they look outside in order to justify any decision.

Its maddening as these are the very people who are paid to make these types of decisions, yet they dont. And due to their employer being a public entity, they are not held to any real standards nor limited by such pesky matters as profits and productivity.

It happens in local government,it happens at the state level and its absolutely enormous at the federal level. The waste is astounding, as are the underlying causes.

Its so simple to solve, yet as long as we have the fox's guarding the hen houses, it will remain.

</comment><comment><USER32> > The consulting firm's sway at ICE grew to the point that McKinsey's staff > even ghostwrote a government contracting document that defined the > consulting team's own responsibilities and justified the firm's retention, a > contract extension worth $2.2 million. "Can they do that?" an ICE official > wrote to a contracting officer in May 2017.

Nope. This is illegal under the FAR. If they're doing it, they should be barred from Federal contracting.

For what it's worth, I think this author gives the GSA a bad wrap based on the analysis of one individual. GSA has to do a LOT of work to award and manage a large number of contract vehicles that allow everyone in government to purchase goods and services at negotiated rates (which also saves them hundreds of thousands on hiring their own contract staff). The fact that the funding fee incentivizes GSA to help people spend more money is not necessarily a perverse incentive - it gives them the incentive to ensure their contract vehicles are effective vs. a normal procurement.

</comment><comment><USER33> McKinsey have messed up government policies all over the world when it comes to McDonaldization of Society. They are using simple measurements to model how complex societies should work, unfortunately this is only works for McKinsey and their partners.

<comment><USER34> Sounds a bit like a lot of "machine learning" actually (when it's done blindly).

But do you have any specific references for the kind of thing you're talking about? It sounds plausible, but if I had a dollar for every time something plausible-sounding was completely wrong...

<comment><USER35> Sounds like someone doing linear regression, claiming it's artificial intelligence, and charging $150k salary in a year for said data science...

</comment></comment></comment><comment><USER36> It's not just governments. Large corporations too frequently use McKinsey to make recommendations for things they want to do anyway (e.g. layoffs or restructuring). From the client executive's point of view, the money spent on a McKinsey contract effectively purchases credibility for these decisions.

</comment><comment><USER37> My favorite McKinsey story: Back in 2005 UBS (big Swiss Bank) brought McKinsey in to figure out how to expand more rapidly in the US. The consultants suggested, "Hey, you have almost no sub-prime exposure. Everyone's trading this stuff! You gotta go in, and in a big way! It will certainly be a pillar of your US strategy."

So UBS did... The result was one of the most severe debt writeoffs of the Great Financial Crisis.

<comment><USER38> Is the point that McKinsey didn't know the Great Recession was coming / it's causes ahead of time?

<comment><USER37> The point is that they suggested a strategy that other big banks were already doing. There was no creativity in their suggestion in this regard.

</comment></comment></comment><comment><USER39> Lost me at the end ... took an otherwise detailed and nuanced analysis and concluded it with a broad generalization that smooths over all of the detail and nuance.

"At any rate, at some point decades ago, we decided that most political and business institutions in America should be organized around cheating people. In this case, the warped and decrepit state of the GSA leads to McKinsey- ifying the entire government. Mr. Clinton, you took a fine government that basically worked, and ruined it. McKinsey sends its thanks."

</comment><comment><USER40> As another comment here mentioned: outsourcing of decision making (consulting) is endemic to bureaucracies, whether that's in government, education or elsewhere.

Has research been done on the amount of money saved/wasted on these consulting firms? Have any economists tried to evaluate whether or not these consulting firms provide any value at all beyond political cover for bureaucrats afraid to make or incapable of making decisions?

From my brief experience working in a large University's bureaucracy and seeing how consulting firms took advantage of the school, I believe the public is being swindled by companies like McKinsey and others. If private corporations want to spend money on consulting firms without any tangible benefits, fine. But taxpayer money should not be wasted on these corporate vultures.

</comment><comment><USER20> There's a lot to unpack here, but the general gist is that McKinsey is overpaid for mediocre services. It's hard to argue against that, and McK in particular does have some insanely unjustifiable rates, but I do think the author has a bit of a misunderstanding of why consultancies are hired in the first place.

To be clear, when a consultancy like McK puts a 23 year old analyst on a project, the $50k/week bill isn't just paying for the knowledge in the analyst's head. The analyst is really just a vehicle to deliver the expertise of the actually-experienced consultants and knowledgebase of the entire firm. The experienced consultants don't have the time to sit and write out the powerpoint deck, so instead they throw as much knowledge as they can at the analyst and the analyst is the one who synthesizes it into something digestable by the customer. Is it worth $50k/week? Fuck no. But it is at least worth recognizing that you are getting more expertise than a sole bachelor's degree.

Second, as someone who has both worked as a consultant at major companies like McK and worked on the other side of the table, and as someone who has long despised the consulting profession because of its overpriced bullshit, I think the author would be surprised at how effective even a measly 23 year old can be. As a consultant, it would drive me _insane_ that my company was billing me out at $700/hr to help implement an IT system when all I was actually doing was reading and regurgitating the software's documentation and making sure that the client didn't ignore it. Surely the client could do this themselves and save the $700/hr, right?

It took me a long time to realize that the answer to this question was actually _no_. The typical HNer will probably be surprised (and saddened, if you're like me) to realize that the average corporate worker bee is in fact not competent enough to do the most basic tasks, such as refer to documentation, without hand-holding. It is an unfortunate reality that sometimes a well-educated and well-vetted 23 year old analyst can do things much better than an 20-year industry veteran. Companies do know this, and this is what they are paying for, much to the chagrin of the rest of us.

As a last note, the author mentions the possibility of just hiring an IT professional to do the work instead of hiring a consultant. This is something I also see come up a lot in consulting discussions. The reason this isn't done is because hiring an employee full-time brings on a lot of risk to the hiring company. It can take months to hire someone, many more months to train them, and then if they don't perform up to snuff, it can take _years_ to build up a case to fire them, all the whole they are sitting on your payroll taking up budget. A consultant, on the other hand, can be hired in a day, trained in a week, and if needed, can be fired in a minute. You pay a premium for the agility to hire and fire them quickly, but that's the entire point.

<comment><USER41> It seems like people do really well for themselves after they leave McKinsey, do you think this is mostly a function of the cache having McKinsey on your resume brings? What skills are really developed as a management consultant?

<comment><USER42> McKinsey data scientist (basically a consultant who codes). Can answer this after following some colleagues who have left and seen many technical teams at clients for my general role.

Most data scientists... don't know what to do. And most clients... don't know what to do with them. So they do little data science projects and build dashboards that few people use. Coming out of McKinsey you can quite confidently say you've learned how to identify top problems, organize a project around it, and solve it in a fast timeline with realistic solutions. That's pretty much gold. It's really useful to be able to say you've worked with c-suite leaders.

Finally, it's probably a function of just fast growth internally. You gain responsibilities much faster at McKinsey than at most most companies. The company invests in your development more. And when you leave, you get weeks of paid time off to search for a new opportunity.

<comment><USER41> I feel like I do data investigations pretty often in my role as a PM at a tech company, how does this differ as a consultant?

<comment><USER42> This only really applies to really large companies mind you, but I would say:

1) You don't need to work your way up the approval chain. Your idea is already going to people with the authority to act on it.

2) You have significant ability to just call people and get them to share their data and domain knowledge

3) You have little risk of long term problems from saying something that someone doesn't like, so long as its not the person who hired you.

4) The pace is probably very very fast relative to the speed of most large orgs. Probably not sustainably fast tbh. It's fine because it's several weeks of hard focus and then a bit of a break.

If you feel like you're doing fine and are answering important questions then you probably are doing just fine and your company's data culture is healthy. IMO people who build analyses that few people want or need are acutely aware of it.

</comment></comment></comment></comment></comment><comment><USER43> I'm sorry but no case is made to support the idea that taxpayers are buying generic advice. Let's agree that the service buys a fresh college grad assigned full time. The grad might be a customer contact collecting and organizing customer issues before presenting them to internal subject matter experts, and then presenting those findings back to the client. I'd like to hear evidence to support how the service is fresh college grads simply using what they learnt in college and their chutzpah, which is what the article implies.

<comment><USER44> > _The grad might be a customer contact collecting and organizing customer > issues before presenting them to internal subject matter experts, and then > presenting those findings back to the client_

This might describe how technical consulting is done at big firms where the output is a product, but when the output is strategic documents... this is grunt work done by the front line analysts with a generic playbook. There is no "behind the scenes" strategy work like there is for dev. Maybe there is a lower analyst or intern that doesn't interface with the customer.

<comment><USER20> FWIW this is the direct opposite of my experience working in both strategy and technical consulting. When it comes to dev work, it often actually _was_ just a 23 year-old new grad writing code with little oversight (when I was 23 I actually wrote a lot of code that was reviewed by no-one, tested by only myself, and is probably still being used across a couple of F500s). Strategy work on the other hand was almost always reviewed and revised by several tiers of higher-up, more experienced consultants before being delivered to the customer.

</comment></comment></comment><comment><USER45> I am guessing nothing will change unless there is significant outrage... I imagine McKinsey kicks back money into election campaigns as well.

It seems worth it to become a govt contractor though..

</comment><comment><USER46> Not surprised. This happened in South Africa.

[https://www.dailymaverick.co.za/article/2017-09-14-scorpio-a...](https://www.dailymaverick.co.za/article/2017-09-14-scorpio- amabhungane-how-mckinsey-and-trillian-ripped-r1-6bn-from-eskom-and-planned-to- take-r7-8bn-more/)

</comment><comment><USER47> I guess it doesn't change the value proposition much, but the whole "23 year old recent college grad" thing is a little overblown. Sure, a college grad like that might land somewhere with a title of "Business Analyst", but the higher the consulting contract the more likely it's some "distinguished" or at least older person with a shoe shine and 30 years of absorbing all of the management buzzword trends.

I just had a round of them come through where I work, and 3 of the 4 met this description. The 4th was the younger "college grad" and they made him actually click the power point slides forward.

</comment><comment><USER48> On glassdoor it seems like that recent grad business analyst is taking home ~100k [1]. One of the articles earlier this week about McKinsey as "Capital's Willing Executioners" [2] talked about how the firm spreads a specific capitalist ideology.

Given that (in this instance), the company is charging a 30x markup for the labor of that recent grad business analyst ... should we be thinking of McKinsey analysts as misguided exploited workers?

[1] [https://www.glassdoor.com/Salary/McKinsey-and-Company- Busine...](https://www.glassdoor.com/Salary/McKinsey-and-Company-Business- Analyst-Salaries-E2893_D_KO21,37.htm) [2] [https://www.currentaffairs.org/2019/02/mckinsey-company- capi...](https://www.currentaffairs.org/2019/02/mckinsey-company-capitals- willing-executioners)

<comment><USER49> I worked as a management consultant. When my firm billed my time out at ~$10K/day and paid me ~$150K/yr (a ratio of ~20:1), I didn't feel exploited. That ~$10K/day covered way more than just my time.

Here's what that billing had to cover:

* My normal work hours [the thing we billed for]

* My overtime hours

* My annual bonus

* My benefits & retirement savings & overhead & employer-side taxes

* My time spent on business development & sales (most of which don't pan out)

* My time spent on discounted or free projects performed to seed future sales

* My time spent in training & development (probably ~4 weeks in the first year)

* The time of managers spent training & developing me

* My support staff (industry analysts, Powerpoint designers, internal experts)

* Multi-million-dollar IT investments to build tools and databases to support me in my work

* A couple first-class flights a week

* A few nights a week at expensive hotels

* Lyfts & Ubers everywhere

* All meals expensed

* The firm's partners working the case

* New employees for whom we don't bill

* Fees for market reports

* Fees for expert interviews with industry executives (in some cases we'd do dozens of interviews a week, paying ~$1K/hr each)

* Office overhead (rent, power, insurance, etc.)

* Office support staff (HR, finance, janitors, etc.)

* Profits for the firm's equity holders

(Now a few of these expenses were billed separately I believe, but it shows the wide breadth of costs that a firm has to cover.)

None of this opportunity would have existed without the firm, the network, the brand, and the partners. It's amazing to think that all I had to do was show up and work, and in exchange, I would get paid a high salary, get experience working with senior executives, get a prestigious brand on my resume, and accelerate my career relative to most career tracks. I don't regret the job, nor do I think it's surprising that there are crowds knocking on the door trying to get in.

Looking at other industries, I wouldn't be surprised if a cashier at a grocery store made less than 5% of what they put in the register each day. And I wouldn't be surprised if a oil driller got paid 5% of oil that they extracted each day. Even if we're the ones touching the revenue, we're still a small part of what it takes to run a big business.

</comment></comment><comment><USER50> I don't know much about this article, but Matt Stoller wrote one of the most garbage articles about Netflix a while back, where he claimed Netflix has a monopoly on online TV show production.

I think he lacks analytical rigor and basically bangs on a keyboard with strong emotions.

Having said that, McKinsey consulting services really are 90% a finishing school for Ivy League liberal arts majors so they can move onto real work.

</comment><comment><USER51> from the article >>> "In other words, the agency of the government in charge of bulk buying isn't paid for saving money, but for spending too much of it. The IFF also incentivizes the GSA to get the government to outsource to contractors anything it can, simply to get more budget. The IFF has been creating problems like the McKinsey over-payment for a long time."

<comment><USER52> And yet there are people refusing to accept that fact, I met one of them. And that guy also failed to realize what an immense cash flow advantage government has against private sector companies. Government _wants_ to spend money as soon as possible. That alone is a huge lever in procuring hardware. Not sure why they don't use it.

</comment></comment><comment><USER53> And let's not forget the outcome of those illegally overpriced contracts: [https://www.propublica.org/article/how-mckinsey-helped- the-t...](https://www.propublica.org/article/how-mckinsey-helped-the-trump- administration-implement-its-immigration-policies)

</comment><comment><USER54> Matt Stoller wrote a really interesting book giving a history of monopoly power in America: [https://www.amazon.com/Goliath-Monopolies-Secretly-Took- Worl...](https://www.amazon.com/Goliath-Monopolies-Secretly-Took- World/dp/1501183087)

On my winter reading list :)

<comment><USER55> Thanks for the recommendation, I didn't read the article either.

<comment><USER54> Darn my pinning in google keep and reading the thread the next day has backfired on me! :)

</comment></comment></comment><comment><USER56> Ugh...

I work at a much, much smaller niche-focused management consultancy. I'm really glad we don't do this or play political cover...

but also it would be fantastic to make what they make. The perils of _actually having expertise that clients need_ and _not playing pork games_ , I guess.

</comment><comment><USER57> Who's going to be the tax payer's hero in that government department? This squeaky wheel would effectively defund his colleague's cush jobs and likely his or her own. Not going to make a lot of friends but it will save the tax payer a fraction of a cent.

</comment><comment><USER58> > In other words, the agency of the government in charge of bulk buying isn't > paid for saving money, but for spending too much of it.

I'm shocked SHOCKED to learn the U.S. government is not a well oiled machine free from fucked-up incentives or outright corruption.

</comment><comment><USER27> Many years ago, David Owen wrote about this--though the customers mentioned were in the private sector--in an essay called "Punks in Pinstripes". It suppose that it may have been collected in _The Man Who Invented Saturday Morning_.

</comment><comment><USER59> The power point thing is so important though and consultants make amazing power points. It's so important I'd hire a power point master just for explaining to executive all the concerns and issues me and my team have over our IT practices.

</comment><comment><USER60> It's depressing to think that there are people who aren't already aware that corporate consulting is pure BS, but I guess there are and so I guess it was worth writing it.

</comment><comment><USER35> Pete Buttigieg worked at McKinsey after his college days...which was only 10-15 years ago. I wonder if/how he was exposed to unethical practices and grafting like this...

<comment><USER45> If he was I expect he understand the issue and as President would work with Congress to reform the practice.

<comment><USER61> Why would you expect that? He's been given opportunity to acknowledge the problems with McKinsey and he has avoided the issue.

<comment><USER45> Do you have a reference? I'd be interested to read more about it.

<comment><USER61> Here it is: [https://nymag.com/intelligencer/2019/02/a-long-talk-with- dem...](https://nymag.com/intelligencer/2019/02/a-long-talk-with- democratic-2020-candidate-pete-buttigieg.html)

I originally came across these comments of his when reading the piece written about him in Current Affairs.

<comment><USER45> The latest article in the NYTimes says he is under an NDA so that may explain it.

<comment><USER61> He might be under an NDA with respect to his specific involvements at McKinsey, but he would not be stopped from commenting generally.

He really just doesn't have a problem with McKinsey's various immoral practices. That's the simple way to see it.

</comment></comment></comment></comment></comment></comment></comment><comment><USER62> Only recently discovered Mr. Stoller's media/reporting, and I have been absolutely impressed with his weekly mailing. Can highly recommend subscribing.

</comment><comment><USER63> Hmm. HuffPo ran a front-page hatchet job on Buttigieg disclosing that he's ex- McKinsey. Coincidence?

</comment><comment><USER64> One of those "we're in the wrong business.." moments.

</comment><comment><USER65> Consultant-ceptio

</comment><comment><USER66> Title's terrible, I read it as taxpayers pay $3M a year per U.S. college graduate. It's not even the original title, "Why Taxpayers Pay McKinsey $3M a Year for a Recent College Graduate Contractor."

<comment><USER67> The actual article title is "Why Taxpayers Pay McKinsey $3M a Year for a Recent College Graduate Contractor," don't know if it was edited or the poster just chose something different.

</comment></comment><comment><USER68> TL;DR organizations respond to incentives and the incentive laid out in the legislation for the organization responsible for setting prices the government pays is to spend money, not save it and McKinsey takes advantage of that.

Considering that a big player like McKinsey can't systemically overcharge the government and get away with it due to how noticeable it would be in terms of sheer dollar amount there are likely be other smaller players that are making even more absurd profits on their services and I wonder who they are since IMO their existence is more worthy of investigation than some BigCo walking right up to but not over the line drawn by the law (as BigCos do). This is not to say that the incentive structure shouldn't be changed to categorically prevent all this.

<comment><USER69> Okay, I'll ask the next question. How do I get to be one of those smaller players. Ha!

<comment><USER22> You'd need to get your business listed on the GSA's schedule of approved contractors (see [https://www.gsa.gov/buying-selling/purchasing- programs/gsa-s...](https://www.gsa.gov/buying-selling/purchasing-programs/gsa- schedules)).

Being "on the schedule" means you've already been pre-vetted and deemed a legitimate vendor by the GSA. For government agencies, doing business with a contractor on the GSA schedule lets them skip a whole bunch of approvals and paperwork that they'd otherwise have to do if they just pulled in a vendor off the street.

Getting on the schedule can be a long and bureaucratic process, but once you're on it you have a privileged position when it comes to winning government business. So if you want to sell to the Feds, you really want to be on that schedule.

</comment></comment></comment><comment><USER70> If you know anyone personally who works at McKinsey, you know them to be brilliant, humble, hardworking, and unfailingly likeable. Think Pete Buttigieg. It's hard to square this with the company's public reputation in anti-establishment screeds.

<comment><USER71> I guess you meant this to be sarcastic, right? Doltish, vainglorious, lazy and unfailingly boorish is what comes to my mind when I review the McKinsey people I've met.

<comment><USER72> That'd all go for about 80% of my exposure to same, can confirm. Doltishness and lack of self-awareness re: the same sorts of things the consult on for others were especially surprising to me.

</comment></comment></comment><comment><USER73> > ... Work of a 23-Year Old

This really bothers me. I understand the point of the article is that they're charging a lot of money for an inexperienced consultant and no bias (discrimination) was meant, however, age is irrelevant. There are a lot of very "experienced" 23 year olds, as long as you're asking them about the work they've done. There are a lot of people in their 50s with very little experience (as their actual experience is irrelevant to the topic at hand or their actual experience has been very repetitive).

I think we need to be a bit more conscious of unconscious bias towards age (in either direction).

<comment><USER74> By definition a 23 year old can't have 25 years of experience in anything, even tying shoes.

They can't legally have worked full time in the United States for more than 7 years.

Unless they are a wild outlier, they can't have an advanced degree and significant years of professional experience doing anything.

You kinda have a point but sensitivity to age discrimination towards the young in this sort of statement is a bit of a stretch imo. It's also shorthand for absolute realities of time and space.

Edit: Age discrimination also isn't perfectly symmetrical because:

If you're young, you will very likely get to be old one day.

If you are old, you never get to be young again.</comment></comment>