How 2K Killed Irrational Games - <AUTHOR> https://hackernoon.com/how-2k-killed-irrational-games-a09d8865fd8b#.68bjzkfxw <comment><USER2> ugg, how did this make it onto the front page of HN. It's been floating around the internet for a few days.

I worked at Irrational Games / 2K Australia from Freedom Force right up until shipping Bioshock 2. I was a team lead, producer, and even ran the Australian office for about year before I left to make Neptune's Pride.

To me there is nothing really controversial in this article. Running big studios full of people with big personalities is really hard. People didn't get along. Things fell apart.

<comment><AUTHOR> I just saw it tonight on Twitter and thought it was interesting. I know nothing in it is terribly unique compared to what other studios have gone through, but I still thought it painted an interesting picture of the inside of the industry and what happens to the individual parts that make up the mega-studios we're all used to.

This person is obviously unhappy, but I'm not suggesting there is a major scandal here.

<comment><USER2> haha, yeah, I guess in my mind there were major scandals, but none of them are in the article. ;)

<comment><USER3> oh you're just gonna put that line out there for us to bite are you? i bet we're getting nothing

<comment><USER4> of course we're getting nothing, would you risk your career for a few points on here?

</comment></comment></comment></comment></comment><comment><USER5> This article quotes sales numbers in the millions and asserts that this means games like Bioshock Infinite were "hugely profitable".

That's not necessarily true.

Bioshock Infinite in particular was "disappointing" compared to projections:

[http://www.p4rgaming.com/take-two-in-retrospect-ken- levines-...](http://www.p4rgaming.com/take-two-in-retrospect-ken- levines-200m-projected-bioshock-infinite-sales-was-fairly-unrealistic/)

2K posted losses in the hundreds of millions in the year leading up to Bioshock Infinite's release (Bioshock would have been a significant percentage of that) and continued to lose money in the year of its release:

[http://www.gamesindustry.biz/articles/2013-05-13-bioshock- in...](http://www.gamesindustry.biz/articles/2013-05-13-bioshock-infinite- hits-3-7-million-boosts-take-two-sales)

People often ignore that these games may cost over a $100 million to develop and the same again to market (Bioshock Infinite ads were _everywhere_ ). Selling 5 million copies might only net $100 million back again (after taxes and other distribution costs).

<comment><USER6> I'm pretty sure that's a parody.

The market has only a specific size. If you sell as many games as the most successful games ever you really can't expect more. Besides, Bioshock Infinite had no special message, it was utterly trivial in that respect. The article is a joke on both fronts: That the sales projections must have been overblown since they indeed regarded the millions of copies sold as disappointing, and that they thought the story with its twist was great writing.

<comment><USER5> Oops, you're right. In my haste, I really did link to a parody piece. Very embarassing for me.

I was trying to illustrate the point that Bioshock Infinite's _early_ sales were disappointing. Early projections had 2K selling 3 million copies in the first month:

[http://www.gamesindustry.biz/articles/2013-03-06-bioshock- in...](http://www.gamesindustry.biz/articles/2013-03-06-bioshock-infinite- forecast-to-ship-3m-in-march)

But it actually sold less than a million:

[http://www.gamespot.com/articles/npd-bioshock-infinite- march...](http://www.gamespot.com/articles/npd-bioshock-infinite-march-us- sales-hit-878000/1100-6407267/)

and only hit 3.7 million after 3 months.

HOWEVER... Don't cry a river for 2K. Bioshock Infinite _eventually_ sold 11 million copies, becoming the biggest Bioshock game:

[http://www.gamesindustry.biz/articles/2015-06-01-bioshock- in...](http://www.gamesindustry.biz/articles/2015-06-01-bioshock-infinite-has- sold-11-million-units)

</comment></comment></comment><comment><USER7> >> the team was so huge and unwieldy that he (Ken) wanted to go back to basics and build a game with a skeleton crew like in the old days

That happens not only in the gaming industry. An approach that works on the small scale may not work on the large scale. If you are small you are forced to take care about your product, customers and employees (maybe not in this order). If you are at a big corp you have to deal with people who only care about company politics and the ego-game, even if you do not care about these things.

</comment><comment><USER8> A minor issue: Borderlands 2 sold 12 million copies (as of 2015). Borderlands: the Pre-Sequel was quite disappointing compared to it (both sales and reviews) and it seems to be mostly based on Borderland 2's code. So at least part of its short development time and small team can be attributed to it not having been written from scratch.

<comment><USER9> Yeah. I've been a fan of the Borderlands franchise since the first game and I couldn't believe the guy had the courage of pointing to the pre-sequel as a success. It felt over-extended, and it should have been cut to a third, at most, and sold as a DLC for Borderlands 2.

While the Borderlands 2 and most its DLCs have been really fun and great value for the money, the pre-sequel has left me wary of the 2k shenanigans around the franchise. Won't make the mistake of pre-ordering again.

</comment></comment><comment><USER10> SWAT 4 didn't sell that well tbh, Tribes Vengeance was pretty bad and Vivendi stopped supporting the game 6 months after it's release, it was primarily a multiplayer only game and received only 1 patch, and Freedom Force was a joke.

It's also should be noted that Ken Levine is somewhat known to be hard to work with, there was a fall out with Vivendi/Sierra and there were issues with employees leaving Irrational after having fights with him (you can find various posts on industry forums) which also lead to Nate Wells (Executive Art Director) leaving Irrational for Naughty Dog.

<comment><USER11> _and Freedom Force was a joke_

What was wrong with Freedom Force? City of Heroes, it was not, but it was a fun little game. It also had a great modding community before the cease-and- desist orders from Marvel started.

Scored well critically, too: [http://www.gamerankings.com/pc/340353-freedom- force/index.ht...](http://www.gamerankings.com/pc/340353-freedom- force/index.html)

<comment><USER12> Scored well, but of the few people I know who owned it no one finished more than a few levels. It lacked a compelling goo, for me.

<comment><USER13> I played through the entire game in 2004 (the original, not Third Reich.) The ending works pretty well emotionally. There were a few levels that almost got me to quit, though, particularly the underground/ant levels.

</comment></comment></comment></comment><comment><USER14> NOTE: Don't make the company you work for your life. It will end in sadness.

<comment><USER2> Terrible advice.

The highs and lows are what make life intresting. Be passionate about what you do and give it your all. Make it personal. Give a damn. It's the only way to live.

<comment><USER15> Terrible advice!

Unless you have equity (and even then it must be a large amount), you are a cog that can be replaced at will, trading X hours per week of your life that you won't ever get back for your currency of choice.

Give a damn and make it personal: about _your_ growth, _your_ compensation, and _your_ career path. If you want passion, work towards financial security and _then_ work on something fulfilling, where you control your destiny.

No one is looking out for you but you.

Source: 15 years in tech, also watching a friend lose everything (EDIT: friends from 80+ hour weeks, job and house when let go) when High Voltage Studios (a game studio in Illinois) started its decent into failure

<comment><USER16> I have seen too many Programmers lose everything. I'm not sure why.

I think part of it's the big money up front, free perks(food & drink), and all the positive energy that the founder/founders tell the employees. It seems like family--kinda. It seems like a dream come true.

Then your not. A bigger company buys yours, and heads roll. Then the real world rears its ugly head. You are being interviewed by a kid.

Since the barrier to entry is not great--there's usually someone to fill your shoes. The ego is not killed though. You had a really good run. You have skills. You start to regret all those pep talks you gave people. I seen some turn to drugs to fill the void.

I have seen twenty somethings go from the fast food industry to pretty good jobs. Their attitudes changed. They aren't cynical about the "system" anymore. They have a great time at work. They work hard. Some save that money, but a lot of it is just blown. The cute girlfriend worships him. It all seems kinda easy, but hard enough to not worry. "No one can do what I do, or as well." There's company dinners. The awards. There's trips to Europe, with LDS steeped into the luggage. Fun times! Life couldn't be better.

The company quietly losses money, and it's back to just getting by. Too many become homeless.

I really believe unless your the Rock Star Programmer; save your money because their will be a time when the parties over. In this industry, it seems like it's in your late thirties.

I hate to be that guy, but I'm tired of hearing, "I used to be a programmer." from a homeless person. Don't think she will stick by you either. Don't estrange the people you once knew when you get that shiny job. Don't let the "haves, or rich kid's father fill your head with stars, and cliches. He has a huge bank account to fall back upon. He has assets to fall back on. He will take care of his kid, but you are expendable.

<comment><USER17> > I hate to be that guy, but I'm tired of hearing, "I used to be a > programmer." from a homeless person.

Where are all these homeless programmers located?

<comment><USER18> San Francisco.

A few post on Hacker News, occasionally.

</comment></comment></comment></comment></comment></comment><comment><USER19> I m trying to see the other side of this whole affair. Usually buisness- leaders, expect a royal household to form around them, who is rewarded on perceived loyalty.

Now creative personal, can be "unsettling" from that perspective, it might be not that extreme in the video games industry as on the theater-stage, but from the CEO-medieval perspective, the constant fluctuation of signals might seem like betrayal and back-stabbing.

Reaction would be the exactly wrong approach- getting more involved and keeping a tighter control on everything, resulting in content, where the more "extreme" fluctuating personal is leashed. Now the resulting product would miss the extremes, which is exactly what attracts the audience in the first place.

</comment><comment><USER20> Why is it so hard to run a tech company in Australia? Some days it feels like I either need to start my own company to find out or just leave this country to find job with a career.

<comment><USER2> What makes you think it's any harder than anywhere else?

Part of 2K Australia's problem was that we grew very big when the Aussie dollar was low, then when it came back up we were a very expensive studio. We had very little turnover and a lot of well paid, senior developers made us look even more expensive per head.

We were working closely with 2K China who were a lot cheaper per head.

<comment><USER21> Given this direct experience, in this instance do you personally think that the well paid senior developers provided value compared to the cheaper her head talent?

<comment><USER2> Err, in this instance I don't think they were directly comparable.

We always had the impression somebody far up the chain would be looking at the spreadsheet and thinking about where they want to invest further. It may have been our imagination.

In fact, now that I think about it, probably the opposite was happening because a lot of effort when into growing the Marin Studio in California which I'm sure would have been even more expensive per head than us.

<comment><USER22> Meh. This is what overseas management do to Australian subsidiaries all the time. They don't worry about the profits made by the employees, what they tend to believe is "It's Australia, we get tax incentives to work here, there's no way Australia workers could be as productive as U.S. workers."

Then they try to introduce business practices that only work in the U.S. and consequently fail. I've seen it time and time again.

<comment><USER23> What is a business practice that only works in the US?

<comment><USER24> While I'm a US worker, I managed a team in Bulgaria. Initially I synchronized via email, chat with each team mate. Met once a month for architectural review over video.

Management hoisted that we had to do stand ups over video chat daily. It was done 9PM EST, and 5AM their time. The time was picked to match my managers PST schedule. There was also the weekly backlog grooming, biweekly poker plannning, etc.

Working with that team, a European Sales team and on site European vendors. The two things I've seen is our meeting culture grates on them. Secondly my American team mates, I don't want to say looked down. But consistently commented, on lack of devotion and effort of other counterparts.

</comment></comment></comment></comment></comment></comment></comment><comment><USER25> This was written 4 days ago and says Bioshock Infinite sold 4.3 million units, yet the Wikipedia page says it has sold 11 million copies. Where is the disconnect? Seems like a huge discrepancy.

Is "units" normalized to full price or something where "copies" just total games sold even at sale price? That is the only guess I could come up with.

<comment><USER26> The number cited in the article is probably limited to a certain release-based window.

</comment></comment><comment><USER17> Funny how the article mentions Bureau as the point where it all came crashing down.

I actually quite liked the game. The problem I think is that there was a market mismatch with that title.

XCOM 2012 caught a lot of flak for it being too different from the 1994 original. Only brilliant execution made it possible for it to succeed.

An XCOM TPS (Which is what Bureau is, to a degree) is just too much of a stretch, most of XCOM's audience are not that open minded unfortunately.

I believe that ultimately the aforementioned market mismatch was too difficult to overcome, even without the publisher's interference and micromanagement.

<comment><USER27> I grew up with the original x-com and terror from the deep, and regularly play them to this day. I personally didn't mind the FPS/third person shooter bit, and I think it even felt a lot more "under invasion" and scary than the x-com remakes.

So I actually liked it, but because it was so poorly made technically I never bothered to Finnish it. Maybe I'll force my way through it one day, because of the story and atmosphere, but it's frankly one of the worst pieces of gameplay I've ever tried.

<comment><USER28> > So I actually liked it, but because it was so poorly made technically I > never bothered to Finnish it. Maybe I'll force my way through it one day, > because of the story and atmosphere, but it's frankly one of the worst > pieces of gameplay I've ever tried.

I'm a bit late to the party here, but this is where "Let's Play" videos on YouTube are really great. You can get all the bits you want without any of the frustrating gameplay. The trick is just to find a YouTuber you can stand for the duration of the video (Nerd Cubed is one of my favourites, but he would be an acquired taste).

</comment></comment></comment><comment><USER29> Still waiting for Bioshock Infinite to come out on GOG (or any Bioshock games for that matter). But 2K aren't interested in more sales it seems.

<comment><USER30> How much extra sales can you really get from DRM-free sales years after the fact?

<comment><USER29> Any amount means profit. Is there any sensible reason not to release them DRM- free? Besides psychological / political ones (like 2K being scared of DRM-free conceptually), I don't see any reasons.

<comment><USER31> You don't matter to 2K. The number of people who would ttly pay for a DRM free version is way smaller than the number of people who would be stopped from just pirating the thing by the DRM.

This is why DRM is a fact of life. Adjust or do without.

<comment><USER32> I just had a look, and found Infinite + all DLC very easily in a torrent index. Dozens of seeds, too.

The number of people stopped from pirating the thing is effectively zero. The number of people willing to pay for a DRM free version demonstrably exceeds that.

</comment></comment></comment></comment></comment><comment><USER33> Usually acquired studios pay for the mistakes of their parent companies. I don't know to which point it is a good idea to acquire a profitable company for millions of dollars, to set it for failure and then close it.

</comment><comment><USER34> Starting a studio in Canberra seems like a peculiar choice. Canberra doesn't have the population, nor the standards of living needed to attract tech talent. It isn't even a cheap city to live in.

<comment><USER19> Hipster on the faultline Hills, you pay so much for so little, at least you can look down on the rest of us, whose existence is worthless and brittle.

</comment></comment><comment><USER35> List of studios bought by ea. Orange ones are the ones they closed down officially but a lot of them exist only in name. [https://en.wikipedia.org/wiki/List_of_acquisitions_by_Electr...](https://en.wikipedia.org/wiki/List_of_acquisitions_by_Electronic_Arts)

Buying a studio 9/10 is the end of its existence in any meaningful way.

<comment><USER36> Large publishers like EA and Ubisoft always operated differently compared to other companies. Smaller companies usually just hire contractors or use 3rd party companies to work on their projects since they can't afford to employ someone to work on concept art, audio or voice over full time.

Large publishers able to keep employees even if they have months between different projects. Still they move people around their subsidiaries and projects a lot since in game development most of staff only work on certain part of project for limited time.

So closing isn't always mean someone was fired at all and in same time any acquired studio might be soon only exist on paper since all people might be working in different studios on different projects.

</comment></comment><comment><USER37> As much as I love the Bioshock series, I take all of the figures with a pinch of salt considering I, presumably not alone in this, own at least 3 copies of each on various platforms. I'm also about to buy the remastered for PC, just because.

The sad part is that the real people who made these great games will see none of this continued income.

<comment><USER38> Why would that make you take the numbers with a pinch of salt? It doesn't matter that you bought 3 copies, they sold 3 copies. The numbers are still accurate for copies sold, they aren't claiming active players or something, there's no subscription fees here.

</comment></comment><comment><USER39> From the comments:

"...Now to correct one thing, what keeps 2k and Take 2 afloat is Rockstar Games. They produce multimillion dollar games that go on to produce over 700 million dollars from their online mode alone. Their games are the top sellers in their years and even decades. Firaxis Games as of now is a failure of a studio. Years back yes they did develop the wildly successful Civilization 5 which went on to sell over 9 million units and Xcom Enemy Unknown that went on to sell over 3.4 million units. [1] Then they developed Beyond Earth which tanked at 1.4 million and Xcom 2 at .8 million, a game so bad by even it's supporters admission it needs mods to be playable. Civlization VI is off to a rough start and will probably match Beyond Earth. It hasn't even broken its first million yet.

Given how poorly they are doing, my money on the next studio closure is going to be them if they are not overhauled instead. I have no clue how their capacity to develop has declined so fiercely but sadly it has. On to your question as to how a game can sell 1.7 million units and not be profitable. I'm going to break this down for you, on console sales the publisher only gets on average $27 for every $60 sold. Digital is more profitable at 70% of total sales. For that 1.7 it turned out $45.9 million, give or take a few million. After development and marketing the fact it sold significantly less than Borderlands 2 is why it is considered a failure. You sold 14% of what Borderlands 2 did.

Worse and I am going to give you the consumers perspective on this, the game is god awful. For starters it story is loaded with virtue signaling, but that's only a minor annoyance. What truly cans it is the fact it is either the Pre Sequel is canon or 2 and the books are canon. That's how atrociously written the presequel was. Even the voice actor of Handsome jack sounded miserable spewing out those out of character lines. The game was received so poorly that within months it was being bundled with Borderlands 2 to desperately sell units.

While I understand that development on The bureau wasn't your fault it also was atrocious. I must say I loved their original version of the game that was first shown off. The final red flag for the games low quality should have been the YOLO trailer, but sadly I still purchased it, much to my regret.

Autralias other projects aren't exactly well recieved. At best they are controversial, Bioshock 2 is hated by half the community at least, CodShock was an embarassment that alienated the entire original audience for the game and came with a story only the greatest of idiots thought was intelligent that ultimately damage the continuity of any future Bioshock games going forward in the manner of public perception. It sold well on the hype of Bioshock in the Sky, instead it was Call of Duty with color. I understand that people put a lot of passion and work into those games, but that doesn't make them by default good or profitable.

Thanks for giving us insights into what transpired though. Even if they are through tinted lenses it is always great to see insights into what transpires behind closed doors inside the industry."

<comment><USER40> "virtue signalling" is the politically correct way to say "political correctness" these days, right?

<comment><USER41> Not really.

[https://en.m.wikipedia.org/wiki/Virtue_signalling](https://en.m.wikipedia.org/wiki/Virtue_signalling)</comment></comment></comment>


ISPs and Tunnel Vision (2019) - <AUTHOR> https://cy6erlion.keybase.pub//tunnel-vision.html <comment><USER2> Once upon a time, we had a functioning FCC that was contemplating the difference between content and infrastructure. The goal of an infrastructure business is to provide the best service to end-users. The goal of a content business is to get end-users to look at their stuff.

When you combine the two into one business, that business inevitably increases profits by reducing the quality of all services that do not lead directly to their own content.

Make it illegal for content businesses to own infrastructure businesses (and vice versa) and the need for other market regulations drastically decrease.

<comment><USER3> Man, this is the best explanation of net neutrality that I've read. Thank you.

</comment></comment><comment><USER4> Counterpoints:

\- Before the Internet was handed to the masses in the mid 90's, there was still a lot of creativity. In the past 50 years we have numerous new styles/genres of music, art, movies and other types of entertainment.

\- Technically oriented, educated people that tend to be outside of popular social spheres were the ones that tended to understand the Internet the best, use it, communicate with others of a like mind, and were there in the initial rush in the 90's. I'll call these "Internet people". This is not most people. The same people that can't tell the difference between Facebook and the Internet still won't without zero ratings.

\- Small websites were always hard to discover unless they happened to be in the right place at the right time or you were an Internet person and spent a lot of time exploring and looking. Google helped small websites for a while, but is now tuned to the normal user, not Internet people.

\- Mass manipulation of society through Internet services doesn't need zero rating, it needs a self-feeding network effect and a site accessible enough to non-Internet people. Facebook has achieved this and didn't need zero rating to do it.

\- Non-Internet people with their inability to see the Internet as anything other than television 2.0 + Facebook is causing ISPs to consider zero rating as something to capture customers with. Remember when various groups in the 50's and 60's said TV was evil? They were right.

<comment><USER5> I miss the 90's. Too bad the thing that ultimately made the Internet sustainable also ruined it: the masses.

</comment></comment><comment><USER6> I'm not quite sure if this needs a [2019] in the title, but it's from November of last year. I don't know where the cut off is for posts needing a [YEAR] tag added, 5 months ago might not be long enough?

<comment><USER7> I think 5 months is long enough. Added.

</comment></comment><comment><USER8> This has been going on in earnest for almost ten years now (I configured zero rating for several services as far back as 2008, back when WAP was still a thing).

In Europe it's a staple of many prepaid mobile data plans. The big difference now is that (some) telcos are actively doing it for OTT services (like streaming).

</comment><comment><USER9> Merely tangential, but I'd missed or forgotten that you can use Keybase as a static hosting platform! Very cool.</comment>


Ask HN: Who Is Hiring Remote Workers? (December 2010) - <AUTHOR>

Just like last month but all over again:
Similar to the "Who is Hiring" thread but, please, remote positions only.
Remote and co-lo'd work posts should be kept separate. Many of us who work remotely tend to work exclusively remotely. <comment><AUTHOR> I have a client (stealth startup) who I've been working with for a little over a month. I'm consulting with them while they get an MVP out the door. After that, I'm shepherding them to stand on their own. I'll let them speak about their funding themselves; however, for reference, they've had zero problems paying my invoices promptly.

They are seeking an _EMPLOYEE_ (remote is A-OK) for a PT/FT Ruby on Rails developer. Particularly interested in people with experience using: Heroku, Test::Unit, Capybara, Rails 3, and developing/maintaining RESTful APIs in Rails.

Contact me through the email provided on the last link in my profile.

<comment><USER2> Seems like there are lots of people hiring _employees_ but not Rails contractors right now!? :(

<comment><AUTHOR> Colin Bartlett? ;-) Evan Light here, BTW.

There are definitely Rails consulting gigs out there. Engaging consultants is what startups tend to have to do when they can't find hires.

Marketing, for any consultant, is of prime importance. You have to find the work to get paid to do the work. ;-)

</comment></comment></comment><comment><USER3> This thread was started/answered at the beginning of the month: <http://news.ycombinator.com/item?id=1957364>

<comment><AUTHOR> Oh, no kidding? And I missed it too. I'll be damned...

</comment></comment><comment><USER4> We are..<http://miami.craigslist.org/pbc/eng/2092075120.html>

</comment><comment><USER5> Sentry Data Systems is hiring, looking for Technical Analysts. Depending on the person could be remote.

<http://www.sentryds.com/company/employment/>

</comment><comment><USER6> Hiring part-time for technical writers. drop me an email.

</comment><comment><USER7> would be interested in too</comment>


Sexist Prices? - vezzy-fnord http://www.overcomingbias.com/2015/12/sexist-prices.html <comment><AUTHOR> >> The largest price discrepancy emerged in the hair care category: Women, on average, paid 48 percent more for goods like shampoo, conditioner and gel.

Sexism or market forces dictated by fashion? Is it sexism to accept that women value their hair, and therefore their hair care products, more than men? They literally have more of it to manage. Shampoo doesn't count for much if you are bald or at least short-haired. So it is totally understandable that men place less value on their hair care products. Is it also sexist that men pay less on average for haircuts than women?

The OP author strikes me as someone looking for something to be angry about.

<comment><USER2> Perhaps detaching "market forces" from sociology is problematic here.

"Is it sexism to accept that women value their hair, and therefore their hair care products, more than men?"

Yes.

<comment><AUTHOR> Good luck trying to market health and beauty products. Pretending that men and women share the same values/needs/wants/tolerances when it comes to hair products will only end in ruin.

But maybe I'm wrong, and those selling beard and mustache trimmers are missing out on the untapped women's facial hair market. I smell startup.

</comment></comment></comment><comment><USER3> Demand-based pricing.</comment>


AWS: Network Connectivity issues affecting EC2 in US-EAST-1 - <AUTHOR> http://status.aws.amazon.com/?HNURLFix

<comment><USER2> Again with the green tick with an 'i' icon as a status, rather than a yellow/red icon, jeeze.

</comment><comment><USER3> Had a bunch of timeout alerts. Some machines of an application server array have the packet drop issue. ELB says that everything is peachy. Folks, we're experiencing yet another "EC2 flavored SNAFU™".

Initial though was: f#*k, I ran out of network I/O, since EC2 simply states "low, mid, high" as performance specs, therefore some proper planning is out. Turned out that with load avg under 0.02 on all machines, the I/O wait wasn't to blame. Average response time, per Pingdom, went up from 170ms to 450ms. New Relic isn't happy either. I guess we should all thank Amazon. Again.

<comment><USER4> I see the same thing. Pingdom reports higher response-time, but no downtime (meaning no alert). Also no alerts from AWS Cloudwatch. I first became aware of the issue when internal api-tests started failing at 9:56am CET. I see users accessing the site, but I don't know how many it's failing for.

<comment><USER3> No issues in the internal EC2 network. At least, none that I could find. I guess that's the reason why ELB doesn't shift any traffic. The whole issue seems to be on the Internet facing network. Failing routers, maybe.

Pingdom still claims 100% uptime, but New Relic (which includes an equivalent pinging service) reports downtime from time to time. Around 25 timeout alerts into the last couple of hours.

</comment></comment></comment><comment><USER5> Health Dashboard updated the status to "Resolved":

 _5:17 AM PDT Between 12:51 AM and 4:52 AM PDT we experienced elevated packet loss affecting instances in the US-EAST-1 region. Some of our APIs also experienced increased error rates and latencies. The issue has been resolved and the service is currently operating normally._

<comment><USER6> Bullshit, I am still down.

</comment></comment><comment><USER7> This started a little before 01:00 PDT (08:00 UTC), so we're approaching the 3 hour mark now. FWIW, that's about 1h15m before there was any sort of indication of a problem on status.aws.amazon.com

</comment><comment><USER8> This got me wondering for quite a long time: I'm pretty sure almost all EC2 issues happen on the US east region -- why is that? Is it because it's the most used region?

<comment><USER9> Think so... it's the cheapest, and is the default choice. Sounds like it would have an order of magnitude more usage (and therefore problems) than the other regions.

<comment><USER10> The US-East-1 is also spread across more than 10 datacenters, which doesn't make things easy. Might not be true for other, more expensive, regions.

</comment></comment></comment><comment><USER11> My site is affected. I don't know how to handle this situation because the issues affect whole region.

<comment><USER12> You should have duplicate infrastructure in another region which you fail over to automatically or manually.

<comment><USER13> Duplicating across regions is not well supported by AWS unfortunately. Any idea doing that without completely rebuild the infrastructure in an other region?

<comment><USER12> I assume you are using the AWS dashboard to manually deploy instances and setting them up by hand? First step is to get your infrastucture to the point where it deploys and scales up and down by itself. Once you manage that, moving to or keeping hot spares in another region is pretty easy.

Look into automated deployment tools like Foreman and configuration management tools like Puppet.

<comment><USER13> You assumed right.

We are learning the hard way that the EC2 Availability Zone separation is not enough, and EC2 is lacking some key features offering multi region tools.

Thanks for the hints, I'm going to check them out.

<comment><USER14> I echo this... although I don't personally use Foreman. You can do scaling type stuff without the tool and I just use POP (plain old puppet). For me personally it took longer to get on the config-management bandwagon but once I did I have never looked back.

Learning to use Puppet, Chef, Salt or similar will only bring benefits!

</comment></comment></comment></comment></comment></comment><comment><USER15> Some of my us-east-1 servers are affected, others are happy so far, so it's not the entire datacenter. However, connectivity on affected servers has gone from just flaky to completely gone.

</comment><comment><USER9> Odd... my site is on Heroku and there seems to be no trouble.

<comment><USER16> git push seems usable. :(

<comment><USER17> i can't git push to heroku...

</comment></comment></comment><comment><USER18> US-EAST-1 seems to have more issues than their other data centers... anyone know if this is really true?

<comment><USER19> It does, but not for nefarious reasons -- it (until the more recent history) the cheapest region in the world for AWS -- it was only right before Oregon rolled out that Ireland, US-EAST and US-WEST-2 all became the same price point, but for the 4 years prior to that, it was always the cheapest so that is where most customers rolled out most of their infra.

Now that the prices has normalized I think the load is distributing more evenly, but for historical reasons I think that region sees a lot more churn (starting/stopping/deploying/etc.) -- just more grinding on the hardware at that region that others.

</comment></comment><comment><USER17> Looks like this is the reason I can't push to heroku either.</comment>


London's Twitter Traffic Mapped As Contoured Landscape - <AUTHOR> http://londonist.com/2010/06/londons_twitter_traffic_mapped_as_c.php

<comment><USER2> I don't get it; is there a convention for contours? What do the colours stand for? Also, what is the scale of traffic that is represented?

<comment><USER3> The blog post ([http://urbantick.blogspot.com/2010/01/new-city-landscapes- in...](http://urbantick.blogspot.com/2010/01/new-city-landscapes- interactive.html)) has more information.</comment></comment>


What to do when users tell you one thing and data tells you another? - <AUTHOR>

My startup launched 4 months ago. We have software that online retailers (our customers) place on their product pages for their shoppers (our users) to interact with. So far, we've received great feedback from our users - they love our application. We're also positively impacting our customers through increased conversion rates and reduced returns.
Today, though, we discovered that the sales of the retailer are negatively impacted by one particular piece of our application. That is, once users engage with this particular feature, they become less likely to purchase (though on the whole the application is a net positive). The problem is that, from collected written feedback, this is clearly users' favorite feature. It has also been the source of a lot of buzz/press for us because it's something new/cool that has never before been part of the online shopping experience.
So what do you do when your users are telling you they love one part of the application, but you have data that says your customers might be better served by getting rid of it? PG says to do what's best for the user. Usually I agree with this, but our users aren't the ones paying us.
We've started by doing user interviews about the offending feature to see if we can better understand what it is about their interaction that may be deterring them from purchasing. It's not clear, though, that there's a tweak we can make that will change their behavior without ruining what they like about the feature. What should we do? <comment><USER2> "once users engage with this particular feature, they become less likely to purchase"

"this is clearly users' favorite feature."

Your description is a bit vague, but here is what I am thinking. I am wondering if these folks were not going to buy anyway, but they stop to "engage" this feature because it has gee-whiz appeal. So basically you are presenting non buyers with a toy and since they are not busy with the task of buying anyway, they play with the toy.

Be careful. Correlation does not (always) imply causation.

<comment><USER3> This is what I was going to say.

Your data may be full of noise with people coming expressly to check out your new fangled feature. This is especially true if you've been getting press about it.

Personally, I'd interview the people who DID buy. Ask those who used it why they used it (and what they thought of it), and those who didn't why they didn't.

If the net effect is still positive for your client, then I don't see what the problem is. Maybe even just give it some time to see if the situation changes once the buzz has stopped.

</comment></comment><comment><USER4> Well, I do suspect that your startup is the one you have linked from your profile? Well, then I first want to granulate you do a damn clever idea. (The 3d model is damn sweet information)

But I can see how it can have an impact on your retailer have a negative impact, as you are delivering information to their customers, that they probably had to make a purchase to figure out before.

With that said, I don't see a problem with you providing this information to the customers, but I think your retailer need to find new ways to interact with it's customers and encourage them to buy.

I tell you what, fire a mail to mattias at plux dot se and I will make a suggestion, for free that you can use however you want. :)

</comment><comment><USER5> "We've started by doing user interviews about the offending feature to see if we can better understand what it is about their interaction that may be deterring them from purchasing."

This. Something sounds fishy. You should collect more data and interview more users. Are you asking the right questions? If you can't find anything about the "offending feature" that is driving people away from sales, then there might be some other hidden issue.

</comment><comment><USER6> Review your data capture process. Maybe your data is telling you the wrong information.</comment>


Fakespot – measure the legitimacy of reviews on popular sites - <AUTHOR> https://www.fakespot.com/ <comment><USER3> This just isn't reliable. I tested in on the Amazon Echo 2nd Gen, it gave it a D grade, 45.8% low quality reviews, half a star.

The two unreliable reviews I checked were two-star reviews, they were not high quality reviews, but they weren't "fake". For example one complained that they thought it was wireless, but it requires a power cord.

I checked the product's two star reviews (6% of reviews), the reviews I saw were all legit. The were long reviews, mostly complaining about the sound quality compared to the 1st gen. For reference, the top 5-star review also says sound quality was poor, until after the third firmware update.

Amazon isn't paying for fake reviews of the Echo, and no one would pay someone to purchase a $79 product (verified purchase) and give it a two-star review.

<comment><USER4> I use Fakespot religiously since 2016 and I noticed the same thing so I emailed them asking them, why would Amazons' products have fake reviews? This was their reply (from co-founder, Ming):

"We have noticed trends where fake review clubs/bots seed their reviewer profiles by leaving reviews on various products, specifically Amazon best selling products to create a profile that looks legitimate (it is done to fool anti-fraud systems that Amazon utilizes). Our engine detects these seed reviewer profiles and therefore penalizes them accordingly by examining their profile and our AI algorithm (one of numerous) was trained to distinguish these patterns."

I hope this is helpful. I love the website and I think it is not to the benefit of Amazon, Yelp, or TripAdvisor to do anything about fake reviews as it only helps their platforms generate more $.

</comment></comment><comment><USER5> Most people don't realize this but lot of products I'm seeing these days are priced almost 20% more on Amazon than many other places. For example, Nest was on sell on Amazon about $100 more than in Costco during Christmas period. Amazon has successfully trained people in to thinking that everything there is lowest priced and you just need to click button to get the goods. Further more, many of the items now are actually fake even when branded. In many categories like cables, batteries, cameras - you are actually very likely to buy either Chinese branded or fake branded items from Amazon then the real deal. For example, virtually 1 out 4 product listing on Duracell are actually fake Duracell. It will still say "Sold by Duracell" and available on Amazon prime to make it completely legit but its fake supplied from Chinese seller. The funniest thing is that during Black Friday, many of these product prices went up while Amazon advertized sale on the site, truly cashing out the trust they have built over the years.

Amazon is becoming jungle in truest sense where your money would be eaten away if you are not careful. Bezo keeps talking about "customer obsession" but he seems to complete ignore all of the above.

<comment><USER6> I remember a few years back there was a post on HN about algorithmic price changes of books on Amazon. I wonder if this was broadened to other products?

<comment><USER7> They do it on a bunch of their products - maybe all of them, I'm not sure.

You can see pricing history (as well as set yourself price alerts) at a couple of different websites such as [https://camelcamelcamel.com/](https://camelcamelcamel.com/)

</comment></comment></comment><comment><USER8> I wonder how they address this problem, which affects all attempts to spot fake reviews: How do they know how accurate their results are?[0] What are the false negative and false positive rates? To measure your accuracy objectively, you'd need an independent method to verify, with high reliability, the legitimacy of the reviews. (You may think you can spot them manually, and maybe you can spot some obvious ones, but false negatives are perilous: You don't know if the reviewer is simply better at hiding their trick than you are in discovering it - and they likely have far more expertise and experience than you do. Consider: Could you write a fake review that would trick someone like yourself?)

Fakespot's FAQ says the following. I think the use of machine learning is a good example of the problem: The machine needs a good data set from which to learn, which means they'd have to flag reviews in the training data as legitimate or not - but they have no way to determine that. Basically, they are training the machine to make the same judgments, of unknown, accuracy, as the humans (in fairness, I'm making some presumptions about their use of machine learning).

 _What criteria are used by Fakespot when analyzing reviews?

Fakespot utilizes numerous technologies to validate the authenticity of reviews.

The primary criteria is the language utilized by the reviewer, the profile of the reviewer,correlation with other reviewers data and machine learning algorithm that focuses on improving itself by detecting fraudulent reviews.

The technologies include: profile clusters, sentiment analysis, cluster correlation and artificial intelligence intertwined with these functionalities._

[https://www.fakespot.com/faq](https://www.fakespot.com/faq)

[0] As Richard Feynman said, "The first principle is that you must not fool yourself -- and you are the easiest person to fool."

<comment><USER9> Statistics.... No, absolute ground truth is impossible for this problem. There has been some research that estimated the percentage of fake / spam reviews at 2 - 6% but that was from a few years ago. I'm sure the percentage has increased greatly since then [great, another dissertation... My MSc dissertation was on this topic]

Text-only based features are not fabulous, just useful (like 75% correct, using a dataset where ground truth is known). So there is / can be linguistic differences in the writing that indicate the writer is not 'truthful'. Sentiment analysis however, by itself, is abysmal - and the behavior of spammers has changed over the years so they are more knowledgeable how to craft reviews.

But other signals, like relationships to other reviewers, IP addresses, submission time of review... they have been shown to be more accurate - <USER2> not in the 90+%. Establishing who is in a spamming group seems to be reliable though, so once that is known, you can more confidently label their reviews as spam [but not 100%, of course, to be fair and objective]

I guess Fakespot takes a stab at estimating correctness and hopes the false negative rate is acceptably low. Yelp OTOH cranks things up so the false positive rate is high....

<comment><USER8> Thanks; that kind of contribution is why I read HN.

> using a dataset where ground truth is known

How is this dataset created?

</comment></comment></comment><comment><USER10> I really can't stand this website.

We had them say ~40% of our Amazon reviews were fake/unreliable... and I'm pretty damn sure we'd know if we were faking our own reviews.

I don't know how their system works, but the false positive rate in my experience is absurd.

<comment><USER11> Their long-term monetization strategy probably involves you paying them to "verify" your reviews.

</comment></comment><comment><USER12> Maybe I'm naive but isn't this something Amazon, for example, should be doing? Why aren't they concerned about the integrity of their community?

<comment><USER13> > Why aren't they concerned about the integrity of their community?

Amazon has virtually no competition, ergo is not incentivized whatsoever to be concerned with the integrity of their community. Further, Amazon will charge _you_ for return shipping so, again, they couldn't care less if a positively reviewed product actually sucks. Now that I think about it, the whole system is pretty brilliant because they're double dipping: charging the customer for returns but charging the vendor for warehouse space. It's the definition of a win-win.

<comment><USER14> Return shipping is free if the product is defective or the website description is inaccurate.

<comment><USER15> Most people choose to bin and forget the $10 shit they got, instead of sending back and asking for refund.

</comment></comment></comment></comment><comment><USER16> I'm skeptical Fakespot works well. Consider this product:

> > [https://www.amazon.com/gp/product/B0064EKNKI](https://www.amazon.com/gp/product/B0064EKNKI)

Where do they get the idea that 25% of the reviews are unreliable/deceptive? I bought it and it works just fine, which is pretty consistent with 87% of the reviews being 4+. Why should I think otherwise?

<comment><USER17> It's attempting to spot sponsored reviews, not bad products.

<comment><USER16> To quote itself, it's spotting "unreliable" and "deceptive" reviews, not merely sponsored ones. And even in the latter case: do those look like sponsored reviews?

<comment><USER17> No, but I imagine sponsored reviews aren't supposed to look like sponsored reviews.

The three it flagged as suspicious for the ethernet cable above were all very short, like "Great product!" or "A++++". I'm not sure what to think of that. I write reviews (of flashlights) as a hobby and wouldn't call something that doesn't describe how the product was tested or include any technical or performance analysis a review.

On the other hand, the average buyer doesn't usually have the knowledge to write a useful technical review and just wants to report that the product met their expectations.

<comment><USER16> > No, but I imagine sponsored reviews aren't supposed to look like sponsored > reviews.

I mean, yes, but that wasn't the point. If the sponsored reviews seem to blend in with the normal reviews and and seem to be at least as accurate as normal reviews then how in the world are they "fake"? Where is the tangible distinction for me as a customer? (Why in the world would I avoid such products?)

> I write reviews (of flashlights) as a hobby

Funny you mention that! Flashlights seem like just about the worst-marketed products I've seen on Amazon, so they indeed definitely need great reviews. At the risk of going off-topic, do you have any tips on figuring out what the lumen output of a flashlight actually is (or even what LEDs they use)? I keep seeing Cree LEDs being claimed to output 2-3x as many lumens as they're even capable of.

<comment><USER17> Flashlights on Amazon are mostly horrible. Thrunite and their offshoot Wowtac are the only brands I'd buy there, because they sell direct. It's usually better to order from specialty dealers. It's often cheaper too, as they tend to do outreach on forums, /r/flashlight, etc... and offer standing coupon codes.

Visual identification is usually the best way to tell what LED a light really uses. Here's a chart with photos of a bunch of common ones: [http://budgetlightforum.com/node/26665](http://budgetlightforum.com/node/26665)

While the lights you find on Amazon making extreme claims about output are almost certainly lying, LED manufacturer datasheets tend to be conservative about how much power LEDs can handle. Typical mains-powered fixtures don't have the sort of heatsinking and thermal paths good flashlights do, and LED manufacturers base their limits on a service life of tens of thousands of hours. You will not use your flashlight for 50,000 hours. Overdriven LEDs aren't rare, especially in enthusiast-oriented lights.

Measuring lumens is hard because you need a way to collect all the light, or at least average it, then put that average on a sensor. Commercial integrating spheres for this purpose cost thousands of dollars. It's possible to _estimate_ using a shoebox and a smartphone though, assuming you have a light source of known output with which to calibrate it. I wrote an app for that: [https://github.com/zakwilson/ceilingbounce/](https://github.com/zakwilson/ceilingbounce/)

<comment><USER16> That's awesome info, thank you! :)

<comment><USER17> I should also mention that a lot of reviews of flashlights found on Amazon praising things I would rate as absolute junk are probably entirely honest.

If your last flashlight used alkaline batteries and an ancient LED, or even an incandescent, anything using an LED from this decade and a Li-ion battery is going to look pretty impressive. That Li-ion cell might well be taken from a laptop battery sent in for recycling, and the charger it comes with might burn your house down, but it's still impressive to the average person.

On the other hand, I've seen a one-star review for not including a battery. It never said, or even suggested that a battery might possibly be included. My conclusion is that Amazon reviews for most kinds of products aren't useful for much more than determining if a product has an unreasonably high failure rate.

<comment><USER16> Yeah, I definitely did find these impressive for the same reasons you mention... hopefully my house won't burn down with the charger I got (it hasn't yet)!

<comment><USER17> Just make sure it's not this one: [http://lygte- info.dk/review/Review%20Charger%20Bowei%20HC-10...](http://lygte- info.dk/review/Review%20Charger%20Bowei%20HC-103W%20UK.html)

And ideally, see if you can find your charger tested here: [http://www.lygte- info.dk/info/indexBatteriesAndChargers%20UK...](http://www.lygte- info.dk/info/indexBatteriesAndChargers%20UK.html)

If it is the first one, break it so nobody else finds it and tries to use it, throw it away and replace it. The Xtar MC1 is the cheapest safe charger commonly sold in the US; they're about $5.

<comment><USER16> Haha, thank you for the heads up :) my charger is not on that list apparently... it's this one (which is no longer being sold; sorry for the low- res image): [https://imgur.com/VpPNswX](https://imgur.com/VpPNswX)

But some of the batteries are on there with not-great reviews... I guess I might have to replace them?

<comment><USER17> Yeah, those batteries are junk. Try the Sanyo NCR18650GA from illumn.com.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> It might be worth clicking the "run analysis again" button if it's offered to you. On the headphones product I tried (and have purchased myself), it flagged some genuine reviews as having been created by automated accounts. Seems it thought the surname Akcicek was just random letters generated by a computer. But after running the analysis on the page again, it changed its mind about that review and decided it was now legitimate.

<comment> LV-426 > But after running the analysis on the page again, it changed its mind about > that review and decided it was now legitimate.

This doesn't sound very reliable. If nothing had changed on the review, what's the basis for it changing its mind?

<comment><USER19> It probably is a learning system... It got trained more. The review itself doesn't have to change. The system merely adjusted its parameters due to more exposure to new reviews.

<comment> LV-426 Possibly, yeah. I was also under the misapprehension the re-analyse button came up (randomly) immediately after a review, but it seems it appears on old stored reviews.

Anyone who uses this site _really_ needs to do a re-analysis, because on the products I just tested I'd rate its prior ability to detect fakes (reviews and reviewers) at F.

After re-analysis I'll upgrade its ability to C.

TBH, I don't understand the point. I trust my own ability to evaluate reviews more than the unknown logic of a third party website, which may have its own interests/motives influencing its reviews of reviews.

</comment></comment></comment></comment><comment><USER20> I wish this site worked well. Based on my tests it doesn't. This is a real problem without yet a solution.

<comment><USER15> Works well based on mine.

</comment></comment><comment><USER9> I did my MSc in Data Science dissertation on classification of fake reviews - I never looked at Fakespot in depth, but my cursory gut feeling is that ReviewMeta.com is better (their blog is helpful). The problem in general is pervasive across all review based sites, like Yelp and TripAdvisor, but as the top poster points out, Amazon benefits from being a hands-off middleman, so they are not going to bother dealing with any problems until it becomes a PR nightmare.

If anyone is interested, you can find my dissertation at: [https://douglas- fraser.com/datadata/](https://douglas-fraser.com/datadata/)

Only text based features were used, nothing involving other types of signals like reviewer name or spamming groups. I started out with 180 million Amazon reviews from 1997 onwards... but that got too big of a project, so the dataset used was a test set used in a lot of research on this topic.

I will be expanding on the research in my blog, examining other sets of text based features I never got the time to use, and things like PCA as well.

</comment><comment><USER21> Can't a malicious actor use this to check their own reviews before posting, until they pass the test?

</comment><comment><USER22> How do fake reviewers get the "verified purchase" tag on Amazon? This particular analysis[1] explicitly lists "Alex Brown" as a "Unreliable Reviewer", but the Amazon page[2] says his review is a verified purchase.

Did Novopal just buy it for him?

(Also, I wish I could plug my Amazon profile into Fakespot and see what it thinks about me.)

[1]: [https://www.fakespot.com/product/novopal-baby-stroller- organ...](https://www.fakespot.com/product/novopal-baby-stroller-organizer- with-shoulder-strap-universal-fitting-anti-slip-safety-buckle-design)

[2]: [https://smile.amazon.com/dp/B01N5J6LXT#customerReviews](https://smile.amazon.com/dp/B01N5J6LXT#customerReviews)

<comment><USER23> I've heard that one way of doing this is for the seller to give the purchaser a discount coupon for 100% off

<comment><USER24> No, discounted purchases don't get you the "Verified" badge. The way sellers operate today is, they ask you to buy the product "for real" and then reimburse you (and more) via Paypal. This is against Amazon's rules but lots of sellers are doing it.

I rank around 6000th reviewer in FR and yet I constantly receive messages offering me to review products using this method.

</comment></comment></comment><comment><USER25> Is there a way to report/flag a _product_ listing on Amazon? As opposed to flagging individual reviews.

I ran into a software listing that had several hundred reviews, all 5 stars, but in reality is some SEO-driven bug-infested piece of crap. I wish I could take it down somehow, because of its borderline scam nature.

<comment><USER26> There is a "Report incorrect product information" link on each product page, and someone does read what you post there. I've reported a few products where the name didn't match the picture or description, and they were fixed within a few days. Whether they would care about a less obvious problem, I don't know.

</comment></comment><comment><USER16> Does anyone know how well it works to just check to see if reviews have been posted in a short/bursty duration? I feel like most fake ones seem to be ones posted very recently or during a short time period, but I'm not sure if that just means I'm missing the rest of them or not.

</comment><comment><USER15> Fake reviews are clustered per category. Take a look at cream to make you less fat, magic thing to grow your hair back, or iphone accessories, you will find that almost all the reviews are fake. Too much money to be made.

</comment><comment><USER27> Tried on the Amazon echo dot best seller which had 4.5 stars but adjusted review score are trustworthy score 3. Thirty percent low quality reviews according to Fakespot.

</comment><comment><USER24> Doesn't seem to support Amazon European marketplaces.

Review Meta does, and works very well.

<comment><USER15> It works on amazon UK. I think their magic algorithms only support English.

</comment></comment><comment><USER28> I use this site regularly its never let me down.

</comment><comment><USER29> Let's be honest, fake reviews is a fake problem. Reviews were never supposed to work for customers' benefits. I mean how in the world shallow emotional opinions of people with unmet expectations and wasted time or with fanatic love for the brand would help anyone? It's just that when people start ignoring reviews all those corporations don't make as much profits.</comment>


Data to Booty extension - <AUTHOR> https://github.com/quickgiant/data-to-booty/

<comment><USER2> _Chrome extension that replaces occurrences of 'big booty' with 'big booty'_

Shouldn't that first 'big booty' be 'big data'? Also, what is the reason for doing 4 regex replaces with different case? Why not just do one case insensitive replace?</comment>


The State of Sparsity in Neural Networks - <AUTHOR> https://arxiv.org/abs/1902.09574 <comment><USER2> This is an illuminating (and notably rigorous) read for anyone interested in neural network sparsity and compression. But - equally importantly - it's a valuable read for anyone interested in the replicability of neural network research in general. The authors make clear the urgent need to evaluate research (and reevaluate received wisdom) on networks of the scale and complexity used in practice. I hope this paper will spark some important conversations in the community about our standards for assessing new ideas (mine included). As this paper makes exceedingly clear, plenty of techniques and behaviors for MNIST and CIFAR10 manifest differently (if at all) in industrial-scale settings.

My biggest question coming out of this work was as follows: which small scale (or - at the very least - inexpensive) benchmarks share enough properties in common with these large scale networks that we should expect results to scale with reasonable fidelity? Resnet50 is still far too slow and expensive to use as a day-to-day research network in academia, let alone transformer. Personally, I've found resnet18 on CIFAR10 to pretty reliably predict behavior on resnet50 on imagenet, but that's anecdotal. For the academics who can't drop hundreds of thousands of dollars (or more) on each paper but still want to contribute to research progress, we should carefully assess (or design) benchmarks with this property in mind.

(With respect to the lottery ticket hypothesis, we have a complimentary ICML submission about its behavior on large-scale networks coming shortly!)

<comment><AUTHOR> I think the goal should be use the smallest dense network possible as the baseline. For MNIST, this might be a LeNet style convnet with [3, 9, 50] instead of the [20, 50, 500] network which is standard (and way overkill).

I haven't explored on CIFAR, but my guess is that using a more efficient architecture like mobilenetv2 would yield more likely to transfer results.

The general theme is that you should be using the smallest dense model you possibly can as a baseline.

</comment></comment><comment><AUTHOR> We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.</comment>


Receive a push message once an HN post reaches 50, 100 or 250 upvotes  - <AUTHOR> http://push.co/a/w4bcXKT0W?m5

<comment><USER2> You've spammed this same message a number of times now, isn't it enough?</comment>


Optimizing Dynamically-Typed Object-Oriented Languages With Polymorphic Inline Caches - <AUTHOR> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.36.6379&rep=rep1&type=pdf

<comment><USER2> This provides good insight into some of the techniques used to make intuitively slow dynamic operations very fast in practice!

</comment><comment><USER3> Good paper, but I wonder how relevant these optimizations are now that modern processors include indirect branch predictors.

</comment><comment> fuzzy-waffle Sounds similar to <http://psyco.sourceforge.net/>

</comment><comment><USER4> It'd be nice to have a [pdf] warning in the title of this one.

<comment><AUTHOR> I don't think I can edit it now. I'd guess it's because it's not a direct link, so it bypassed the pdf logic. Apologies.

<comment><USER2> Perhaps the submission page could have a PDF checkbox with the default state taken from the PDF autodetection function? This way we could correct the autodetection when it fails. Also it would ensure a uniform title "tagging" style.

Now that I think about it we could have radio buttons: Regular, PDF, Movie, Picture.</comment></comment></comment>


How The So-Called Clinton Recovery Really Happened - <AUTHOR> http://www.forbes.com/sites/bruceupbin/2012/09/19/how-the-so-called-clinton-recovery-really-happened/ <comment><USER2> _As for military spending, the US benefitted from that brief synapse in time in between the collapse of the Soviet Union/fall of the Berlin Wall in 1989, and the emergence of another combatant 10 years later whose conflict with the US is almost as costly, and much more diffuse._

This is of course naive, if not disingenuous. It's as if he doesn't recall that our primary military engagement in the last 12 years has been in a country called Iraq. And that conflict did not "emerge" by any stretch of the imagination. It was invented out of whole cloth.</comment>


Jul 15 Deadline: Code for America's Fellowship Application - <AUTHOR> http://codeforamerica.org/geeks/fellowship-apply/

<comment><AUTHOR> I used to work here. And even though I left, I've still got a huge amount of respect for the people in the org. Fellows get access to some incredible speakers (people like Ken Norton, Tim O'Reilly and Aneesh Chopra) and they get to tackle really hard problems -- like Food Stamps and economic development. It's cheesy, but I really feel like the fellows are doing something noble for US cities.</comment>


Show HN: A dedicated website on how to flush the DNS cache on MacOS - <AUTHOR> https://www.flushdnsonmac.com/ <comment><USER2> But why

<comment><AUTHOR> I know it seems quite excessive... I still think it is a good resource that is dedicated and kept up-to-date. And the URL is just awesome, when I thought of it and found the domain, I had to get it.</comment></comment>


The three words that can save your life - <AUTHOR> https://www.bbc.co.uk/news/uk-england-49319760 <comment><USER3> An open-sourced anatidae-based alternative : what3ducks.com

Honestly I'm a bit annoyed by this article because it looks like an ad since it's a private commercial system.

<comment><USER4> Until a few weeks ago I've never heard before of 3words, but they seems to be online from a very long time.

Another similar geo location system, if I'm not wrong created by Google but released with a liberal license (Apache) and not patent encumbered, is called "plus.codes".

Edit: according to Wikipedia they were founded in 2013, don't recall why I thought they were older...

<comment><USER3> That's nice to learn about this alternative.

I haven't worked a lot with geolocation yet. But I'm driven to it because of my geography background, and I would be mad if every service for that kind of work was closed and proprietary.

</comment></comment></comment><comment><USER5> Alternatively, give 911 the power to read out your gps location ([https://en.wikipedia.org/wiki/112_(emergency_telephone_numbe...](https://en.wikipedia.org/wiki/112_\(emergency_telephone_number\)#E112), [https://en.wikipedia.org/wiki/Advanced_Mobile_Location](https://en.wikipedia.org/wiki/Advanced_Mobile_Location))

Works when you aren't able to talk (clearly) or when you can't look at the screen (injury, screen broken).

Disadvantage: location information will be less precise.

</comment><comment><USER6> Previous discussions about what3words

[https://news.ycombinator.com/item?id=8614198](https://news.ycombinator.com/item?id=8614198)

[https://news.ycombinator.com/item?id=17423251](https://news.ycombinator.com/item?id=17423251)

</comment><comment><USER7> if you have internet signal why not just share from google maps to sms?

On second thoughts, how do you even? I know I've done it before, but doing anything in google maps always takes me 15 minutes of searching howtos - <USER2> awful UI.

oh, you have to drop a pin first, and then share that. It seems like 'share my current location', without turning on tracking for an hour or whatever, would be a fairly obvious use-case.

</comment><comment><USER8> So, people have to download an app because the UK emergency system is too broken to receive the GPS data directly from the phone.

+10 to dispatchers for a clever hack. -100 to governments for an emergency dispatch system that fails to actually be an emergency dispatch system.

In addition, what3words is copyrighted and patented and doesn't have a published standard.

<comment><USER5> [https://en.wikipedia.org/wiki/Advanced_Mobile_Location#Histo...](https://en.wikipedia.org/wiki/Advanced_Mobile_Location#History):

 _"AML was developed in the United Kingdom by British Telecom, EE Limited, and HTC as a solution to problematic caller location in emergencies. When a person in distress calls the emergency services with a smart-phone where AML is enabled, the telephone automatically activates its location service to establish its position and sends this information to the emergency services via an SMS.[3] The services uses either a global navigation satellite system or WiFi depending on which one is better at the given moment"_

This works, will shortly be required in the EU, but requires changes to your phone's software and a change in the law (implicitly sharing your location is a privacy invasion)

It also still may be less accurate than you putting a marker on the map (but then, how good are people in distress in placing such a marker?)

</comment></comment><comment><USER9> That's a really cool concept. I like it.

<comment><USER10> However it seems to be completely under commercial/private control. Source: read the previous discussions at HN.</comment></comment>


Writing my own init with Go - <AUTHOR> http://www.mustafaak.in/2016/02/08/writing-my-own-init-with-go.html <comment><USER2> > The reason that I wrote sleep 3 seconds is that I want to see how the kernel > reacts when init dies.

That was my favorite line in the entire post. I love seeing this kind of fearless experimentation.

When I first learned to program, I desperately wanted everything to work and would try very hard not to break things. As I've matured into senior roles, I spend of a lot of effort creating "safe" development and testing environments like the author did here, just so that I can break things and make observations and then throw the environment away when I've learned what I needed to learn. Tools like vagrant and cheap cloud computing have made this easier to do than ever.

<comment><USER3> it should be called acquiring information, not 'breaking' things :D

<comment><USER4> "Actively exploring modes of failure"

<comment><USER5> Or maybe hacking ;)

</comment></comment></comment></comment><comment><USER6> NOTE: You can't write an init without threads in Go, as you can in C. (I tried awhile back)

At least you can't with its portable interfaces such as exec.Command() and cmd.Wait():

[http://www.mustafaak.in/2016/02/09/forking-process-in- myinit...](http://www.mustafaak.in/2016/02/09/forking-process-in-myinit- go.html)

You end burning a thread _per process_ , which is pretty lame for an init. The wait() call blocks a goroutine AND OS thread, so the go runtime will have to start a new thread for the next wait().

Whereas in C you can do the entire thing from a simple thread (I call it "async processes"). You set SIGCHLD handler and get async exit notifications, and then process those in a single-threaded loop. I'm pretty sure there is a problem doing this in Go (besides resorting to the syscall module)... I think it has to do with the fact that Go turns signals into messages on a channel, but I don't remember right now...

EDIT: Now I remember. The problem with using the syscall module is that Go does not export Fork and Exec. It exports ForkExec, because it needs to ensure they run on the same thread.

[https://golang.org/pkg/syscall/#ForkExec](https://golang.org/pkg/syscall/#ForkExec)

This severely limits the usefulness of your init program, because in Unix there are all sorts of important things that happen in between. It's how you set the child process state. You basically can't do anything with file descriptors or containers at all with this model.

<comment><USER7> Signal handlers are pretty much impossible to write directly in a managed language (at least without heavy contortions from the runtime), because you could get a signal during a non-GC-safe point and then deadlock or corrupt your program. For example, malloc often takes locks internally; if you get a signal during malloc with that lock held and then try to malloc in your signal handler you can deadlock. Managed languages like Go treat allocation as an implementation detail rather than something the programmer has to explicitly invoke, making them basically incompatible with signal handlers.

(BTW, this is mostly a problem with signal handlers as a concept, not with Go.)

> The problem with using the syscall module is that Go does not export Fork > and Exec. It exports ForkExec, because it needs to ensure they run on the > same thread.

And because you can't safely allocate after a fork in managed languages, because fork drops your other threads, leading to the same problems above. :)

<comment><USER6> So the way Python does it is that it doesn't call back into the interpreter loop from the signal handler. That would be a really bad idea as you note, because the interpreter can call malloc and do all sorts of other stuff.

It queues your Python signal handling function, returns from the signal handler, and then runs it on the main loop at the next tick.

As far as I know this works fine. I wrote the same program in Python and it seems to work just fine, and have more functionality than the Go version. I guess one problem is that if you do a long blocking syscall on a tick in the main loop, your signal handler can be delayed indefinitely. There might be some hacks around this in the Python interpreter, but I don't recall offhand.

<comment><USER7> Yeah, that's basically having the runtime advance to the next GC safe point before running the handler. It's not direct control over signal handlers, but it's fairly low overhead. I guess Go could do this too if it wanted to. I don't think it works for fork though.

<comment><USER6> Yeah the problem with Go and "async processes" doesn't have to do with GC -- it has to do with M:N threading. I'm glad Rust abandoned M:N threading.

Though it is definitely a killer feature of Go to write in a blocking style rather than in a state machine style... but I feel that there might have been a way to do it without baking it so hard into the runtime (?). Maybe like coroutines in Python, which run on a single thread and are optional.

M:N threading causes a lot of other complications too, like calling from C to Go, etc.

</comment></comment></comment></comment></comment><comment><USER8> The author might look at how a minimal (yet fully functional) init works:

[http://git.suckless.org/sinit/tree/](http://git.suckless.org/sinit/tree/) (look at config.def.h and sinit.c)

and build up from there. Given how minimal sinit is, it's a great place to start your functionality from, and see the power (and limitations) of a basic SystemV init system is.

<comment><AUTHOR> Wow, this is great, thanks!

</comment></comment><comment><USER9> Nice project! Even if you believe that systemd is too complicated, it's a great example on how to build an asynchronous, modern init system. Upstart uses some highly questionable tricks (ptrace!) to track processes across forks whereas systemd uses cgroups.

<comment><USER10> Not only that, it is a nice example of using Go in a role that many would use C for, so it might convince a few to try their hands to safer programming languages.

Next step, bare metal runtime! :)

<comment><USER11> Is there even any concurrency in this case? Wouldn't e.g. OCaml be a better choice?

<comment><USER12> What benefit would adding the overhead of learning OCaml be to this project? It's not like a functional or immutable language would add any value here, would it?!

<comment><USER11> Even at this 3-line stage, the stateful log.SetOutput construct is ugly and error-prone. So I'd say a functional/immutable language would already be adding value, and that would only increase as the amount of code grows. At a minimum an init is going to involve parsing config and something similar to a state machine, both good use cases for functional languages.

<comment><USER13> Init's job is to start the init scripts and then hang around to foster orphan processes. Ideally, it should not be parsing configs, but should spawn another process to do that sort of work. A good PID 1 is very, very minimal.

<comment><USER11> This project isn't just a PID1 though, it's a full system. (Well at the moment it's a hello world, but that's the intent)

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Part two is now up on his website:

[http://www.mustafaak.in/2016/02/09/forking-process-in- myinit...](http://www.mustafaak.in/2016/02/09/forking-process-in-myinit- go.html)

</comment><comment><USER15> Writing init in Go probably have size issue compare to typical C implementation. Last golang helloworld I built was 1.4MB. The /sbin/init in my ubuntu dist is only 194k. Busybox version of init optimized for size/features can be much smaller.

Tough to use it in OpenWRT, DDWrt type of embedded system where system might only have a few megabytes of RAM/ROM.

<comment><AUTHOR> Yeah, you are right. Go is too heavy weight. But the nice thing in Go 1.4+, you can install go std and dynamically link your applications. I will do it in the next blog posts to save some space, but yeah, I would still need some huge lib.

</comment></comment><comment><USER16> @CSDude: Thanks for sharing this very interesting project!

From a technical point of view, what are the reasons you chose Go over alternatives, such as C, C++, OCaml or Rust?

(Just curious, as all those languages have compilers that produce fast, optimized, self-contained binaries.)

<comment><AUTHOR> Thank you. I found it easier to develop in Go and I use Go in the work environment heavily. But compared to C/C++, I think having a garbage collector is a must for modern init, and I will write all the other utilities in Go as well for better maintainability.

<comment><USER7> Garbage collection isn't a must for a modern init.

Memory safety may arguably be, but I see no reasonable argument that garbage collection (a particular strategy for memory management) is.

<comment><AUTHOR> Well I would say memory safety, not necessarily garbage collection. It was also a bit overstatement but I would really love memory safety for many core utilities.

</comment></comment></comment></comment><comment><USER17> Although very different than a traditional Linux distribution, the init system for RancherOS might be worth taking a look at. It's written entirely in Go as well. [https://github.com/rancher/os/tree/master/init](https://github.com/rancher/os/tree/master/init)

</comment><comment><USER18> Using Go even earlier in the boot stage in an initramfs (as a replacement for tools like dracut) crossed my mind recently. Go's statically linked binaries and fast builds make this particularly appealing to me. With (a lot of) work you could forego busybox entirely.

<comment><USER19> Good point about static linking. Though Go's big fat binaries count against it as a busybox replacement in areas where storage space is a premium.

</comment></comment><comment><USER20> > Linux is mostly perfect as it is

Your critical thinking has not been engaged. Linux is a useful bit of code but "perfect" is not what it is known as everywhere.

My side of the room like this little ditty

Linux, by amateurs, for amateurs -- Dave Presotto [1]

Or

i've wondered whether Linux sysfs should be called syphilis -- Charles Forsyth [2]

Or if you like your detractors with a bit more fame

Unix has retarded OS research by 10 years and linux has retarded it by 20. -- Dennis Ritchie as quoted by by Boyd Roberts in 9fans.

I won't go on

[1] [http://research.google.com/pubs/author4927.html](http://research.google.com/pubs/author4927.html)

[2] [http://www.terzarima.net/](http://www.terzarima.net/)

<comment><USER21> So what does an OS purist like yourself run? BeOS? Plan9? All OS's suck. They're just horrible. But so is everything else. It's all flawed in one way or another. Windows is flawed, OS X is flawed, our tools are flawed, our languages, our editors - everything. If you want perfect, you have to throw away your computer and somehow live inside Knuth's books.

<comment><USER20> Try reading what I said.

I said "Linux isn't perfect, like you think it is"

<comment><USER22> You are being a dick, offering no reasons other than the opinions of people you presumable agree with.

<comment><USER20> If you could just tell my why it's perfect then, give me something to work with.

<comment><USER22> I can try... you scorned the commenter for not reading your comment literally, and inferring things instead. When I read your comment literally, I see you quoting people and saying it's not perfect.

Truly an empty comment, as it adds nothing except makes you look smarter than someone else. You don't offer anything insightful, you are trying to be insightful by proximity. So anyway, you reinforced your complex when you said "Try reading what I wrote". So I called you a dick.

Now take your own advice and "try to read what _I_ wrote". I only called you a dick. Why are you trying to make me defend a position I never took about perfection?

<comment><USER20> > I only called you a dick.

oh we're going down that route. Well if it's literalism we're wasting time with then try reading _your_ own comment

> You are being a dick, offering no reasons other than the opinions of people > you presumable agree with.

That isn't just "calling me a dick".

but lets stop here

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> @CSDude, I recommend putting a link to part 2 on the part 1 page.

<comment><AUTHOR> Done! Thanks.

</comment></comment><comment><USER24> If you improve the perf/functionalities of init using Go, I will give this language a go right after. It seems promising tho.

<comment><USER24> Is that possible to recode init in any other language using your technique? For instance can I try it in Ruby?

<comment><USER25> Since init is the first process that runs, you can't use an interpreted language like ruby (since the interpreter would have to run first, which is impossible if your program itself is init). Maybe there's a Ruby-compiler out there somewhere, but as-is Ruby would be impossible.

<comment><USER26> > Since init is the first process that runs, you can't use an interpreted > language like ruby

Sure you can, you can even use a shell script: [https://wiki.gentoo.org/wiki/Custom_Initramfs#Init](https://wiki.gentoo.org/wiki/Custom_Initramfs#Init)

</comment></comment></comment></comment><comment><USER27> The kernel already supports acquiring a DHCP lease via the "ip=dhcp" boot option. Why does this need to be a feature of init?

<comment><USER10> Because init is common to several UNIXes and ip=dhcp is a Linux specific feature?

<comment><USER27> > Because init is common to several UNIXes and ip=dhcp is a Linux specific > feature?

And yet the author seems to be comfortable using those linux-specific features:

"Also, the kernel paremeters must have the rw flag or you have to remount the /dev/sda1 as rw, but changing kernel parameters are much more easier." (from the follow-up post).</comment></comment></comment>


Citi to Refund $330M to Credit Card Customers It Overcharged - <AUTHOR> https://www.nytimes.com/2018/02/23/business/citigroup-credit-card-refunds.html <comment><USER2> > it caught the error in a routine internal review mandated under federal law

Ladies and gentlemen, this is what regulations are all about. If not for a federal law forcing Citi to audit for this sort of thing they would either never have caught that they stole hundreds of millions of dollars, or more likely just never have reported it to anyone.

The next time some politician tells you regulations are always bad, only lead to more expenses for customers, remember this.

<comment><USER3> What has been the cost of this regulation as a whole on the entire banking industry?

Lets get that number and compare.

<comment><USER4> wait, so suppose regulation costs financial firms too much, you'd rather have no recourse to getting reimbursed as an individual?

<comment><USER5> I'm guessing the argument is that if the cost that the customer bears for such regulated audits is greater than the refund they get from it then the customer actually lost money.

<comment><USER6> That sounds kind of like me complaining about the cost of my car insurance because for 10 years I didn't have any accidents.

Seems difficult to know the benefit of such regulation without hindsight.

Secondly, businesses face market pressures to provide low cost to customers (isn't that the whole argument for the free market?) therefore passing the cost to the customer should be weighed against other potential inefficiencies -- they might even have to take a smaller cut of profit!

Funny how so many people can never justify the cost of doing business when it comes to protecting consumers.

</comment></comment></comment></comment></comment><comment><USER7> I find it interesting how this show up the day after I had to call the customer service to clarify an interest fee on my account that din't make sense. So here is what happened, I got hit with a $12.xx monthly interest on a $2000ish purchase which I made less than 30 days ago and hasn't shown up on my previous statement. For clarification, my current APR on the card is 5.99%.

It took me two reps and two accounts managers to learn how these things are calculated:

1) Interests are daily compounded.

2) Interests are not calculated on actual balance but on the average daily balance.

3) If you don't pay your statement balance in full by the end of the month, you lose your grace period for future purchases and interest would start accruing as soon as the transactions goes through. To reset this it would require you to make two full statement payments in a row.

4) If you purchase something and return it and the return takes a few days to process, you would still pay interest on for those days.

The rep and the first account manager that spoke to weirdly dint have the basic understanding of interest rates. She argued that the $12 interest was on the $2000 for 4 days at the rate of 5.99%. Funny enough, she still din't find anything odd when I pointed out that it would mean, by the end of the year, I would pay $1095 ie. > 50% of the principal in interest!

Anyways the second rep was knowledgeable enough to explain it.

<comment><USER8> Is this for a US based credits card? I have never heard of needing pay off for two full months. Or for the average daily balance used in calculation and I'm pretty sure neither is the case for any of my cards. But I will need to check.

</comment></comment><comment><USER9> I suppose one good thing in this is that Citi caught this error on its own and not as a result of a class action suit. So customers will receive the full value of their overpayment, rather than a puny voucher dictated by a court where a large chunk goes to the lawyers bringing suit.

</comment><comment><USER10> So say that a customer was overcharged $100 in interest in Jan 2017 and they carried a balance all year. By Dec 2017 they carried an extra $100 on their credit card balance needlessly so they paid >$20 in additional interest (APR 20%) due to this overcharge.

The article only mentions returning the original amount AFAIK so it looks like Citi might still be profiting from this error.

</comment><comment><USER11> servers right. well said mabbo >The next time some politician tells you regulations are always bad, only lead to more expenses for customers, remember this.

</comment><comment><USER12> I always suspected their credit card bills. It was very confusing and very hard to verify. May be they did it intentionally? who knows?</comment>