Reshaping New York - <AUTHOR> http://nytimes.com/newsgraphics/2013/08/18/reshaping-new-york/ <comment><USER2> Pretty good work. My thoughts on the visualization:

Criticisms: 1) The moving around over the city skyline is slick and fun, but I doubt it conveys much useful information. I'd rather see the growth of new building over time. In visualization, motion should convey some meaning. Here it doesn't seem to. 2) It's important for visualizations to convey both anecdotal information (achieved through the before/after snapshots). But it's also important to convey quantitative information, magnitude and directions of changes. I don't detect anything like that here, even though there is quantitative information embedded in the text (e.g. "Population is up 2,000, but 600 poor people have left"). Numbers are good, visualizations that artfully convey the magnitude and direction of relevant numbers are even better. It's rare to have news narrative that can't be improved by conveying the quantities involved.

Things they did right: 1) The before/after snapshots are useful and pack a punch. 2) Inclusive. The visualization includes a a variety of changes and doesn't get sucked down a narrowly focused rabbit hole. 3) Simplicity. It's not a data dump. The visualization does a decent job of conveying single themes without including unnecessary extra. It may go too far with this though (see criticism #2 above).

Overall, I look forward to more visualizations like this from NYT and other media outlets.

<comment><USER3> I'm with you that it was mostly flashy without useful information. I actually hadn't really noticed just how much new construction had been going on in my neighborhood. I would have loved if the NYT provided something that I could actually interact with and form some conclusions of my own. It seems like good journalism has taken a back seat to flashy visualizations -- with most of these stories there hasn't really been much of a story.

If they provided something like that, I would actually actively look at their site again, rather than follow the occasional linkbait there.

<comment><USER4> How do you qualify "good journalism"? Data visualization is a powerful tool for story telling and the NYT has been at the forefront of this revolution in news. "Good journalism" is telling a story accurately and in a format that people can understand. Your claim that "good journalism has taken a back seat to flashy visualizations" seems to denigrate the work of data scientists such as Mike Bostock who are transforming news in a very meaningful way.

How does data visualization qualify as "link bait" in the derogatory sense of the word?

[1] [http://bost.ocks.org/mike/](http://bost.ocks.org/mike/)

<comment><USER3> I don't mean "link bait" in quite the derogatory sense it's usually used (though slightly).

I mean it in the sense that other than links from aggregator sites with provocative titles, there is absolutely no reason that I would ever be on the NY Times' site. They've done some pretty serious damage to their reputation in recent years and newspapers are on the decline anyway.

While I'm sure these stories are a big hit for them, I don't really feel like I gained anything from my time spent on the site. It looks cool and the work done in data visualization is _very_ cool, but I want to look at some of the data. I want to dig down and look at the parts of data that are interesting to me.

So yes, data scientists are transforming news, but they're still failing to deliver a product that I want..and something that I think a lot of people want now. People used to look to newspapers as their go-to information source for current events.

Oh the hours of my life that I've spent in a library looking at microfiche. Thankfully we don't have to do that anymore, but instead of going to the NY Times still, most of us go to wikipedia and follow links from there.

The NY Times is actually in a position to provide something that would give them some relevance and make people want to use their resources to find out things.

This site, however, was little more than entertainment. They could even just present a platform to crowdsource information collection about the locations (pictures of the locations over the years, places to put links to building and tax records, etc) and I would be totally pleased with it.

Just give me some interaction with the data.

<comment><USER5> _Just give me some interaction with the data._

Thing is, most people _don 't_ want that. They want something that is the result of digesting that data and pulling out relevant conclusions, presented in a clear and easy to comprehend narrative.

<comment><USER6> If you design things on the basis of what you think most people want, you end up with nothing special. People probably won't complain much, as it is what they expected, but it is unlikely to excite or interest them very much.

</comment></comment></comment></comment></comment></comment><comment><USER7> Another great web piece by the New York Times. The other two I know of are Snow Fall [1] and the Riding the New Silk Road [2].

1 - [http://www.nytimes.com/projects/2012/snow- fall/](http://www.nytimes.com/projects/2012/snow-fall/)

2 - [http://www.nytimes.com/newsgraphics/2013/07/21/silk- road/](http://www.nytimes.com/newsgraphics/2013/07/21/silk-road/)

<comment><USER8> About the making of Snow Fall:

[http://source.mozillaopennews.org/en-US/articles/how-we- made...](http://source.mozillaopennews.org/en-US/articles/how-we-made-snow- fall/)

See also: [http://www.interactivenarratives.org](http://www.interactivenarratives.org)

</comment></comment><comment><USER9> New York Times has been killing it with their visualizations. It would have been nice to have 2013 pictures overlaid on the 2003 pictures though, to really understand what the changes were. In some pictures, like Times Square, I had to flip back and forth a couple of times and still wasn't sure what exactly the changes were (can't see any obvious bike lane in that image, which is supposedly what it's representing).

<comment><USER10> A lot of the photos are aerial, I'd imagine it'd be nigh impossible to get them lined up so perfectly to overlay each other, but it would be very nice indeed.

On the Times Square picture: the entirety of Broadway (the slanted street on the far right) has been closed to traffic and is now entirely a pedestrian mall. It's not just Times Square, large stretches of Broadway has been reconfigured for bikes and pedestrians.

Interesting sidenote: they've been digging up Broadway in Times Square lately to put actual sidewalk in its place. My guess is that Bloomberg is doing this to cement his legacy and throw up additional barriers in case future mayors want to re-automobile-ize Times Square.

<comment><USER9> Fair point. Perhaps some labels on the images, at least, would make it clear what they were trying to point out. The Times Square picture in particular was confusing because I saw the pedestrian mall, but thought that the point was about bike lanes which a pedestrian mall isn't.

</comment></comment></comment><comment><USER11> What are residents of NYC opinion on Bloomberg? Is this reshaping good or worse for the city?

It seems that he won his most recent re-election but barely, from what i know he is a socially liberal, environmentally friendly, fiscal conservative.

<comment><USER12> As an NYC resident, I feel that Bloomberg has made the city a remarkably better place. Probably the most transformative effect of Bloomberg's initiatives has been making the outer boroughs appealing. Young people who might have otherwise lived elsewhere because of ridiculous Manhattan rents (~$2-$3k for a studio in downtown Manhattan!) are now flocking to neighborhoods like Bushwick and Bedford-Stuyvesant. These young people with disposable income create demand for more great restaurants, shops, and venues like the Barclay's center, all of which make the city a much more fun place to be.

It isn't all roses. The poorer residents in Bushwick and Bedford-Stuyvesant are being forced to move elsewhere as their neighborhood gentrifies. However, this is part of the typical cyclic trend: artists / poor people move into a cheap neighborhood, make it trendy, then get forced out when the rent goes up. The starving artists move to another more affordable neighborhood, and the cycle starts all over again. Trying to stop the cycle by preventing development is just delaying the inevitable. Bloomberg has simply accelerated the trend of creating more upscale neighborhoods. I can understand how someone in a less fortunate socioeconomic situation might feel differently, but it's hard to argue that the changes haven't been beneficial from an outsider's perspective.

<comment><USER13> This is what always confuses me about gentrification. The core idea is that improving the neighborhood is a bad thing that should be prevented.

I mean, I understand the arguments. I just have a difficult time saying, "Yes, let's _prevent_ these neighborhoods from being improved".

<comment><USER3> But the neighborhood improves without its residents. The economic situation of those living there doesn't really change and they're forced to relocate to other neighborhoods with similar locations to where they were but much further away.

Bascially "we're going to make this a nice place to live, please gtfo".

It's not an argument against improving a space. It's one against improving a space at the expense of its residents.

<comment><USER14> This assumes everyone is renting which while probably true in NYC is far from universal. In other places people on fixed income have been forced out though propery taxes but it's closer to here is 100,000$ GTFO simply from the difference in propery values.

</comment></comment></comment></comment></comment><comment><USER15> It's interesting to contrast the evolution of NYC with the (lack of) evolution of SF.

This article cites a net increase of 170,000 housing units from 2000 to 2010. I can't find an exact comparison for SF, but it looks like SF has less than 25k net increase from 2000 to 2012.

<comment><USER16> Well, given that NYC is ~10x the population of SF, SF is doing pretty well!

<comment><USER17> SF is 1/10th the population of NYC in part _because_ of those restrictions, so the comparison is more apt than it seems...

<comment><USER18> NYC is also 10 times bigger land-wise, 470 sq miles compared to 47.

<comment><USER19> SF is also twice the size of manhattan

<comment><USER20> I'm trying to figure out where this discussion is going, haha.

Let me simplify it - NYC continues to build up, SF doesn't.

<comment><USER21> San Francisco is currently in a building boom. Over the next 25 years, there is going to be approximately 92,000 more housing units. The population of San Francisco should reach 1 million by then. Here are some links:

[http://www.theatlanticcities.com/jobs-and- economy/2012/12/sa...](http://www.theatlanticcities.com/jobs-and- economy/2012/12/san-franciscos-surprising-building-boom/4213/)

[http://www.spur.org/publications/library/article/san- francis...](http://www.spur.org/publications/library/article/san-francisco- boom-back)

[http://sf.streetsblog.org/2013/07/19/plan-bay-area-passes- in...](http://sf.streetsblog.org/2013/07/19/plan-bay-area-passes-in-a-room- full-of-paranoid-conservative-activists/)

<comment><USER22> It's important to point out that 92k units is the amount _needed_ to accommodate future growth. The creation of those units is entirely dependent on local politics, which is even more polarized here than in NYC.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> It's so obvious he is preparing and advertising himself to run for president. And it's about time.

He has something few other presidents have. Success in both Private and Public office.

<comment><USER24> Does he need to go through a stint in the senate/congress first, or could he run directly for nomination?

I don't mean "need" as in legal, but just as a path to it?

Has anyone ever done that before? Run for president without being a governor or a senator/congressman? (Asking as a Canadian who isn't super familiar with historical presidential politics in the US).

<comment><USER25> According to a Wikipedia article, it would be likely be in his best interest to start somewhere bigger:

Every President of the United States has served as either:

Vice President of the United States a U.S. Senator a Congressman a Governor of a state a Cabinet Secretary a General of the United States Army

<comment><USER26> Mayor of NYC is probably a better executive experience that many (most?) state governorships. Bloomberg isn't my cup of tea, politically, but he's certainly qualified to run if he decides to do it.

<comment><USER27> For example, as mayor of NYC, Bloomberg governs about three times as many people as Bill Clinton did as governor of Arkansas.

<comment><USER28> The governor of Arkansas deals with both rural and urban issues and has a good cross section of what makes up this country.

<comment><USER27> The biggest city in Arkansas is less than 3/4ths of a million people. To say that Arkansas cities gave Clinton experience with urban issues in America is roughly like saying that the sum total of NYC's parks and gardens give Bloomberg experience with rural issues in America.

<comment><USER28> That's more than a bit of hyperbole. He had to deal with the issues of an urban area typical for US residents. He didn't deal with the super metroplexes, but that is not the typical problems faced by most. NYC is not a cross section of what makes up the US.

<comment><USER29> You're using a terrible example. There is exactly one city in Arkansas with greater than 100k people, and that is Little Rock with 193k. That state has zero claim to being "a cross section of what makes up the US".

<comment><USER28> From a government systems point of view, it is a cross section. The White House isn't about demographics, it's about process.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER30> Woud be interesting to see the backstory on how they got access to the data/subsamples. The visuals are well done, but the story is sort of interesting in terms of how its edited. It neglects any mention, for example, of the role of Finance in the building boom. And by extension, glosses over the Financial crisis spawned (under his tenure, in new york) by real-estate investment instruments and the general bubble-icous tendencies that follow from "free money". NYC basically skims the profits of wall street both directly (retail&real estate sails subsidized by incomes from the financial sector) and indirectly through double taxation of corporation (financial lagest sector) and city income taxes. So, there are a lot of grandmother's withs withered pensions and decimated savings outside of New York that are basically subsidizing all of this. Just worth thinking about.

</comment><comment><USER31> Anyone else find this really buggy/broken + A horribly non-intuitive awkward way of presenting information even if it worked?

<comment><USER32> The UX of this site is pretty bad. There's nothing to tell you what you should click on or what clicking on those elements do. I found myself clicking on random things, ending up in places that weren't related to what I clicked on, and not sure how to get back.

<comment><USER31> My first reaction was to scroll down the page, as the scroll bar was indicating there was more below. Apparently I was not supposed to do this as it basically flips out.

This is your typical 'fancy flash website' syndrome

<comment><USER33> Scrolling worked just fine for me. It moved between the pieces of the visualization quite smoothly.

<comment><USER31> On websites it is not customary or to be expected for scrolling to bring you to essentially another page.

This thing is a slideshow fitting badly into webpage format.

<comment><USER33> ...... I don't contest that ......

You said it "flipped out", which I took to mean that it didn't function at all. I replied saying that it worked for me. That's it.

<comment><USER31> Yeah, it also flipped out on me (I had a load of green lines instead of text) But I'm more concerned about the intended design. I think the act of scrolling triggering a full-window video is shocking bad.

</comment></comment></comment></comment></comment></comment></comment><comment><USER34> I am neither from nor do I live in New York so I can't talk about validity and range but that was a very enjoyable presentation of the city.

</comment><comment><USER8> OT: Kudos to the interactive team. I think it would be an interesting job to work for the NYTimes interactive/mobile/media lab just based on the work they have been producing. I hope that they will continue to push the envelope and start to open source more or blog more about the process (which is the interesting part) of creating.

<comment><USER35> Good news, they do [http://chartsnthings.tumblr.com/](http://chartsnthings.tumblr.com/), though tends to come in bursts and looks like they haven't put up much since July.

</comment></comment><comment><USER36> That sure didn't like my 1200x1920 monitor.

It also appears to depend on a codec that chromium on ubuntu doesn't have.

</comment><comment><USER37> Why does the "Time Square, 2013" picture have markers for Hello Kitty, Elmo, Mario and Spider-man?

</comment><comment><USER38> Ironically, the mobile view is a MUCH more usable and navigable look at the data. The technical restrictions of mobile reined in a lot of the excess flashiness.

</comment><comment><USER39> The central park remain the same!

That's a reservation de-facto.

</comment><comment><USER40> Can anyone count how many times "Mr. Bloomberg" is mentioned across all the slides? I lost count...

<comment><USER41> This shouldn't be down voted. It's significant that it's not "Mayor Bloomberg". "Title Lastname" is the format that it used to refer to _candidates_.

<comment><AUTHOR> No, NYT style is to use Mr. and Ms. on second reference when using last names. This applies to even the President. It's a policy that makes sense when you consider the variety of titles (even just political) and the lack of corresponding well known abbreviations

[http://www.nytimes.com/2013/08/18/world/middleeast/pressure-...](http://www.nytimes.com/2013/08/18/world/middleeast/pressure- by-us-failed-to-sway-egypts-leaders.html?hp)

</comment></comment></comment><comment><USER42> How is the camera panning done? It works on Windows Chrome but not Ubuntu or Mint Chromium.

<comment><USER43> It uses video tag and .mp4

</comment></comment><comment> o0-0o Seriously? Including the Freedom tower in this map is D-I-S-G-R-A-C-E-F-U-L. Shame on the NY Slimes.

</comment><comment><USER44> Bloomberg bloomberg bloomberg bloomberg.

Sorry, but most of these changes, are the result of private businesses and taxpayers.

Saying "you can now build a skyscraper in area X" is hardly an amazing achievement.

<comment><USER45> Negotiating the rezoning process was largely a political triumph. Same for the bike lanes, parks, and redevelopment of public land. Bloomberg has had a very visible impact on the city - but even more importantly most of his reforms are a ratchet. Plenty of politicians implement a program for a time, they can later be undone. Bloomberg has done a bunch things that can't be undone. There is no way to put the bike share program or more park land back in the bottle now.</comment></comment>


Why You Should Explain Your Android App's Requested Permissions - <AUTHOR> http://www.22ideastreet.com/blog/2014/05/30/analyzing-an-app-permission-exemplar/

<comment><USER2> In this vein I am personally boycotting apps that request access to my contacts when they clearly have no need. The latest instance was an update to the BoA app that suddenly wants access to my contacts. Hell no. If they want to add a 'share with a friend' feature (my guess) then Android has this feature baked in via intents and in such a way the app has no access to my contacts.

</comment><comment><USER3> When I get any android phone/tablet, I always root it and install XPrivacy, so I can get total control on the permissions. I doesn't like the All-or-none style permission assignment on android.</comment>


Why do we count starting from zero? - <AUTHOR> https://cseducators.stackexchange.com/questions/5023/why-do-we-count-starting-from-zero <comment><USER3> Address lines. So much boils down to address lines, and how they are binary.

Address 0000 is the lowest one, first one, initial address, right?

Well, either we start, given 4 address lines:

0000 = 0

, or

0000 = 1.

Which is it?

Either we have to talk to people about zero, address lines, various indexing and pointer math

, or

We have to start all those conversations with some reason why 0000 = 1.

The former seems to make the most basic sense, and is rooted deep in the hardware.

I have found explaining address lines, and the powers of two very useful. This is how I was introduced to all of this back in the 80's, when confronting computers, memory, hex, pages, and all sorts of basic things.

It made immediate sense. Later, moving to other higher order, and or more complex environments, there are sometimes details needed, but those basics really do not change much, leaving the real problem connecting whatever higher order tools one might be using with the lower level nature of the computer itself.

When I've gone through this, I typically use a 16 bit environment, and 8 bit CPU example. It's 6502, 6809, Z-80 like. Does not matter much to the newbies. We are never going to go there, unless they have an interest.

But, that tiny space, and encouragement to just memorize the first 16 powers of two make for a great foundation!

They can easily do hex and binary with that info. ASCII, and other core things, put into tabular listings of various kinds, all sort out pretty easily. There are lots of examples, like 8 bit color, 24 bit color, those weird "color numbers that have letters in them." Just one of many simple things people can be shown, or talked to about and have it all make some sort of sense.

Why is $000000 black? And why is $FFFFFF white? And just why are there colors from 0 to 255, or levels of red, green, blue?

Decimal conversions are not that difficult either. Just add up all the ones and you are done. Division works the other way. Some people won't need or want those, so no worries. Skip them. They can return later.

Time spent on all that stuff is kind of fun time, and it pays off big. All one needs is a white board, the calculators found in whatever teaching environment is available, and or their computers.

Invite questions. Many people will have seen stuff, and if they ask anything at all, that's your opportunity to connect the dots in a way that will be useful to them and endure easily.

While there, and say it's a school setting, or the people are headed somewhere it may be needed, we've somehow introduced this "metric" or "SI Unit Standard" for Kilobyte as being 1000. I won't argue it, just as I would not for Imperial vs Standard International units, as found in the USA. Not a good hill to die on.

The better thing is to just recognize the units and know what they mean. There are just units, they get used, and the most important thing is to know what those units actually mean. Preferences, politics on them are fun, but secondary.

Kilobyte - 1024, because of address lines. The other unit can make sense for disks, packets, and other things not necessarily tied so fundamentally to the powers of two. It's just enough that they get the differences.

Pages? While those vary considerably, the fundamental idea is each of those powers of two can be a page. On old computers, pages were 256 bytes, 0 - <USER2> inclusive, for example. They were 8 bits, and that's 8 address lines worth, which is...

Pages in modern machines and languages vary widely, but 4K seems to come up a lot. Again, not too important, just link that concept to the address lines, chunks of memory, etc...

And then finally some math. Bytes, words, longs, and so forth. Nibbles too. 4 bits, 8 bits, 16 bits, 32 bits.

When they do the conversions, a whole lot will make sense, and the most basic thing, say 4 bits being 0 - 15, will make sense, as does the "page", and the number 16 being a power of two, and how all that adds together to make most efficient use of address lines. The colors are one super easy way to talk about this. And they can all bring up some program and type them in, get some sense of things that way.

Truth is, with address lines, there will be X number of them, and they have two states. (technical issues, like floating states aside) Why do anything other than use them? It's a waste to do otherwise, and again technical things aside.

Having taken newbies (and I've personally done middle-school and high school student sessions on request many times) through this stuff, they come away with a very basic understanding about the nature of the computer.

If that's all they get, it's fine. Specific numbers, kilobyte, megabyte, megabit, all have some basic meaning, and the unit issue stands a fair chance of being understood too.

Other questions like, "How come it's always 8GB, or 32Kb?", or "How come we never see a 29GB USB drive?" Have easy and obvious, if not always technically correct, answers they will find make sense.

That all leads straight to, "Count starting from 0" having good reasons, and it can make some basic sense. That's all people need. Should they get to a point where it's a real debate, it's a very nice problem for them to have, if you ask me.

Those who advance and want to do programming will build on these ideas and have little trouble with counting from zero, and when and why it is done, or not done.

 __Didn 't put this on Stack Exchange because I'm not on a machine with those credentials, so here it is for anyone who might need it as an introduction for someone new. It works well --even for very non-technical people. And BTW, the non-technical ones will often just get it, then accept it with little more than, "so that is why computer numbers are weird!

Great, that's the point. Just enough to build on.

And again, way back in the day, all that made so darn much sense to my peer group. We grokked it, and literally got better at understanding a whole bunch of things without even feeling it very much.

Finally, a CPU 8 bits, registers, and all that stuff may not be appropriate. Depending on the audience, address lines, and the on-off state is enough. Use houses, blinkenlights, whatever they can identify with and just link it to how the computer can access things. Talk through 4 bits worth, and they will get it.

Doing that is enough for understanding and for those interested to make good progress with somewhat less stumbling.</comment>


Ask HN:why do people prefer github(paid) over gitorious(free)? - <AUTHOR>

<comment><USER2> 1\. I don't need private repos yet, so GitHub is free for me.

2\. If I do need private repos, I can afford GitHub.

3\. GitHub has better UX and design, and its design sense has been contagious to its users. I think part of the reason people make their code/docs super- presentable on GitHub is to fit in with the look and feel of the site, but that's pure speculation and could be wrong.

4\. My main use case is code discovery, not project hosting. GitHub got there first in building a community, so there's more to discover - in addition to it usually being easier to look at as per (3).

</comment><comment><USER3> A few reasons:

1\. People like the sense of security that comes with a paid (and perceived well managed) service. Indeed, GitHub is well managed and the streamlined process makes it easy to use.

2\. It has become both the cool and the expected thing for developers to use. It has just about reached de facto status.

3\. It is well maintained by a very professional, known staff.

4\. While it is not completely free (it has a limited free account), it is not prohibitively expensive. And developers have a few extra dollars to pay for this service that actually helps their careers and their profiles.

5\. It has become social to the extent that it is the place for developers to be.

</comment><comment><USER4> For me it boils down to:

1\. Trust. I feel like I can trust Github. While that may not be true, the fact that the majority of the folks I know use github vs gitorious makes me feel trust, whether it's earned/valid or not.

2\. Github's cost, for me, is negligible so I don't really bat any eyelashes at it. The $7/mo plan is perfectly fine for me.

3\. Their UX is splendid. Every thing feels fairly intuitive me.

4\. The social/community aspect I quite enjoy, though I do not participate as much as I should.

</comment><comment><USER5> I have a github account for my personal code, and we host a gitorious server internally at work for proprietary stuff. Having used both they are interchangeable on the mechanical level (duh) but the user experience for me on github is far better. In addition the extra tools are richer, such as network/commit graphs, issue tracker, wiki, etc. I won't touch upon the social aspects of github since I'm a digital hermit.

</comment><comment><USER6> 1\. Excellent UX and branding

2\. Services like github pages which allow me to host my blog and sites for my projects (if I had any) without dealing with hosting nonsense (and its free)

3\. Everyone else is there

4\. The price is entirely irrelevant. $7/month is less than I spend on lunch most days. (Yes, I should make an effort to spend less an lunch.)

</comment><comment><USER7> I prefer viewing and searching for code on github to any other site I've found, so I got a micro account to support the site (and keep a few repos private until I am ready to make them public).

Plus github hosts my blog for free and without headaches, so that's worth the couple bucks a month on its own.

</comment><comment><USER8> Github is free for public projects.

</comment><comment><USER9> Github would be a networking service for developers through the software code. Gitorious seems not to have such community, I guess. A lot of open source projects on github attracts those people.

</comment><comment><USER10> 1\. Never heard of gitorious

2\. Github is so inexpensive such that cost is irrelevant.

3\. Free is rarely a sustainable business model, and we are hesitant to put our source in the hands of an entity that may not be there tomorrow, even though git is distributed.</comment>


Hundreds of Google accounts suspended for using emoji in live chat - <AUTHOR> https://mobile.twitter.com/markiplier/status/1193015864364126208 <comment><USER2> I wonder what the difference is between this particular livechat and the livechat for say the league of legends world finals. There was definitely twitch style livechat emote spamming in that chat.. i guess Youtubes just keeping an eye on that one while this one was caught with some automated ruleset that identified this as abusive behavior?</comment>


Show HN: Divvy – Expirable GitHub code review requests through Slack - <AUTHOR> https://divvycodereviews.com <comment><USER2> FYI there is already a "Divvy" by Mizage. A window management app that's pretty impressive:

[http://mizage.com/divvy/](http://mizage.com/divvy/)</comment>


Hooking Up With Tinder – An Older Man's Story - <AUTHOR> http://www.mensjournal.com/adventure/outdoor/hooking-up-with-tinder-20140424

<comment><USER2> 34 is an older man now? Geez, I guess I should go check myself into the nursing home now and beat the rush.

</comment><comment><USER3> Older man? He's 34. Jeez.</comment>


TechFounders Talk – Interview with Olivier Grinda - <AUTHOR> https://www.byteant.com/blog/techfounderstalk-1-interview-with-olivier-grinda-co-founder-and-ceo-at-home-61/ <comment><AUTHOR> Explore a fresh interview with Home61 founder, where he shares his story and his opinion on what is coming next.</comment>


DarkUniverse – the mysterious APT framework #27 - <AUTHOR> https://securelist.com/darkuniverse-the-mysterious-apt-framework-27/94897/ <comment><USER2> good article with background

[https://arstechnica.com/information- technology/2019/11/shado...](https://arstechnica.com/information- technology/2019/11/shadow-brokers-leak-of-nsa-code-leads-to-discovery-of-new- apt-hacking-group/)</comment>


Tips for VC Negotiations - 7 startup founders & execs share their strategies - <AUTHOR> http://talktech.tv/2008/11/21/top-tips-for-vc-negotiations-7-startup-founders-execs-share-their-strategies/

<comment><AUTHOR> You really have to be on top of your game to raise $$ in this economy, so I thought the advice in this video montage might help. Let me know what you think.</comment>


FAA Bans Turkish Airlines from Flying to the United States - <AUTHOR> https://skift.com/2016/07/16/faa-bans-turkish-airlines-from-flying-to-the-united-states/ <comment><USER2> If you ever want a directory of every sketchy unsafe third world airline on the planet, here it is:

"List of airlines banned from EU airspace"

[http://ec.europa.eu/transport/modes/air/safety/air- ban/doc/l...](http://ec.europa.eu/transport/modes/air/safety/air- ban/doc/list_en.pdf)</comment>


Google aims to begin drone package deliveries in 2017 - <AUTHOR> http://onvb.co/HyVUB6C <comment><USER2> So many submissions, no discussion at all:

[https://news.ycombinator.com/item?id=10497383](https://news.ycombinator.com/item?id=10497383) (reuters.com)

[https://news.ycombinator.com/item?id=10496461](https://news.ycombinator.com/item?id=10496461) (bbc.com)

[https://news.ycombinator.com/item?id=10496076](https://news.ycombinator.com/item?id=10496076) (reuters.com)

[https://news.ycombinator.com/item?id=10495731](https://news.ycombinator.com/item?id=10495731) (thenextweb.com)

[https://news.ycombinator.com/item?id=10493772](https://news.ycombinator.com/item?id=10493772) (yahoo.com)</comment>


What Engineers Found When They Tore Apart Tesla's Model 3 [video] - <AUTHOR> https://www.youtube.com/watch?v=Lj1a8rdX6DU <comment><USER2> Bloomberg loves to recycle Tesla news with less information. The original announcement by Munro on Autoline has already been posted here.</comment>


Comparing Message Queue Architectures on AWS - <AUTHOR> http://tech.forter.com/comparing-message-queue-architectures-on-aws/

<comment><USER2> Have you also evaluated Kafka? as it is a common choice feeding Storm .

<comment><USER3> This is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment.

[https://aphyr.com/posts/293-call-me-maybe- kafka](https://aphyr.com/posts/293-call-me-maybe-kafka)

<comment><USER2> For that matter, RabbbitMQ isn't either. see [https://aphyr.com/posts/315-call-me-maybe- rabbitmq](https://aphyr.com/posts/315-call-me-maybe-rabbitmq) .

</comment></comment></comment><comment><USER4> The first architecture using two ELBs has a long list of cons, most of which are solved by using HAProxy as your internal load balancer. May want to considering adding that as another option on the matrix.

<comment><AUTHOR> Interesting. Could you please elaborate?

<comment><USER4> Sure! Here's your list of cons and how Haproxy would solve it:

> Some API requests need to get a higher priority over other API requests, and > that is not taken care of. This was one of our main problems with this > architecture, especially with a mix of real-time clients and clients that > send batch jobs.

Haproxy would let you assign pools to different requests, each with their own priority and queue. At reddit, we had it broken down roughly into four quartiles based on 95th percentile response speeds of each API call.

> This architecture assumes that there is enough Processing Servers to handle > all requests (peak throughput). If there isn't the Processing Server applies > back pressure on the API server (error or timeout), which in turn returns an > error to the API user which in turn re-tries the API request (applying more > pressure). To avoid this, the number of running processing servers needs to > be enough to handle peak traffic.

Using haproxy in the middle, that tier will queue the requests, so all the back pressure builds up at that second load balancer. Whether that is good or not is questionable, but at least you won't return errors to the clients right away. You'll still have to have a pretty long time out depending on how long it takes for resources to come online, but then you could go back to the first part and have longer or shorter timeouts based on the api call as is appropriate for your application.

> ELB was not designed to handle huge traffic spikes since it takes a few > minutes to internally scale. You should contact AWS support to warm your ELB > if you have a planned traffic spike. We at Forter are in the eCommerce > market where traffic spikes are rare.

This is still true, and there is no easy way around it unless you also make haproxy your front end load balancer. If you make it your frontend as well, you can have a hot spare on standby and spin up new ones pretty quickly. That being said, I believe the ELB team is making improvements in this area in 2015.

> API Server needs to handle retries. Not provided by the ELB itself. > Processing Server must respond within the http timeout (configurable between > 1 to 3600 seconds). Otherwise the protocol would need two phases which adds > more complexity.

Still true, although it will have to retry less because of the queues in the middle layer. Again though if you use haproxy as the front end you can solve this issue as well by sending the request to a "retry pool".

</comment></comment></comment><comment><USER5> Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures."

FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments ([https://github.com/tylertreat/Flotilla](https://github.com/tylertreat/Flotilla)). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they compare.

<comment><AUTHOR> Do you have a document showing the results of the Flotilla tests ... how beanstalkd compares with RabbitMQ for example?

<comment><USER5> Not yet. The blog post linked in the readme there provides some more background on the motivation behind the project.

I'm hoping to do some in-depth analysis of several brokers at some point, but I want to get the benchmark instrumentation right first.

</comment></comment></comment><comment><AUTHOR> OP here... I would be happy to discuss any comments you have.

<comment><USER6> For RabbitMQ you mention:

      No message delivery guarantee in face of RabbitMQ server failure.

Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?

<comment><USER7> Another very minor detail, but you mention that RabbitMQ can do priority based on multiple queues. While that's certainly a fine way to do it, it's worth noting that AMQP also supports per-message priorities within one queue:

[http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basi...](http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basic)

> The server MUST implement at least 2 priority levels for basic messages, > where priorities 0-4 and 5-9 are treated as two distinct levels.

Depending on your client, it may be difficult to prioritize consumption of one queue over another, so this solution could be preferred.

<comment><AUTHOR> Thanks! I'll update the post. A quick search found this RabbitMQ plugin [https://github.com/rabbitmq/rabbitmq-priority- queue](https://github.com/rabbitmq/rabbitmq-priority-queue) , which specifies: "In contrast to the AMQP spec, RabbitMQ queues by default do not support priorities. When creating priority queues using this plugin, you can specify as many priority levels as you like."

</comment></comment></comment></comment><comment><USER8> Out of curiosity did you look at gearman? I really like the ability to coalesce identical jobs.

<comment><AUTHOR> Not yet. gearman is a job service which sits on top of a queue (much like python celery). We did not need all that, given the fact we are using Storm.

</comment></comment><comment><USER9> You give Redis a yellow rating on prioritization, noting that this can be partially achieved using multiple lists. Wouldn't it be much better to take advantage of Redis's sorted set type?

<comment><AUTHOR> Sorted Set does not have properties of a queue. For example, it does not allow duplicates (different messages with the same priority)

<comment><USER9> Stick a UUID on your queued items as part of the priority queue implementation.

</comment></comment></comment><comment><USER10> Surprised they didn't mention IronMQ: [http://www.iron.io/mq](http://www.iron.io/mq)

It's not perfect but pretty damn close. And it run on AWS et al.

<comment><AUTHOR> You are right about IronMQ, they are said to have a very strong product. For the record, we haven't actually checked redislabs and cloudamqp either, but since they are based on opensource that is mentioned in the blog post, I added them to the notes section. I couldn't say anything intelligent about IronMQ since it is not based on an opensource offering, and so other than actually trying it, I cannot tell its pros/cons.

</comment></comment><comment><USER11> I'm surprised that SNS wasn't mentioned here, being an AWS queueing product.

<comment><AUTHOR> Simple Queuing Service was mentioned (SQS) Simple Notification Service (SNS) was not. Did you mean SNS or SQS ?

<comment><USER11> Given that I was surprised that SNS wasn't mentioned, I think I mean SNS, given they are both queuing services.

<comment><AUTHOR> SNS is a push notification service, that focuses on various consumers outside the cloud. It can be used with web hooks (http endpoints) to publish the same message to various web servers. In that case it would guarantees at least once semantics (which means the message is delivered, but sometimes twice), but it does not guarantee FIFO.

<comment><USER11> SQS also doesn't guarantee FIFO.

SNS guarantees delivery with back-off to HTTP/S endpoints (handling HTTP status codes properly) with very low latency in an asynchronous manner.</comment></comment></comment></comment></comment>


Taking a long look at QUIC [pdf] - <AUTHOR> https://conferences.sigcomm.org/imc/2017/papers/imc17-final39.pdf <comment><AUTHOR> In-depth analysis of QUIC, but to me the most important take-aways were this:

1\. "QUIC now represents a substantial fraction (estimatedat 7%) of all Internet traffic"

2\. "We found that when competing with TCP flows, QUIC is unfair to TCP by consuming more than twice its fair share of the bottleneck bandwidth."

The paper is 2 years old, so the amount of traffic has probably increased. Unless they've fixed fairness wrt TCP, Google is invisibly grabbing more bandwidth than their fair share.</comment>


Proposal: Non-cooperative goroutine preemption - <AUTHOR> https://github.com/golang/proposal/blob/master/design/24543-non-cooperative-preemption.md <comment><USER2> The way I understand this, and that I think some other people in this thread misunderstand, is that this is not a change to the Go language in any way.

The way Go works is that you can spawn green threads, and the Go runtime will magically run them concurrently, in such a way that you do not have to worry about many problems usually associated with concurrent programming.

This proposal seeks to modify the way the "magically" part works. Right now, it suggests that what Go does is to 'cooperatively' preempt at function prologues. This change would instead allow the runtime to preempt at a choice of "safe points" the compiler suggests throughout your Go code.

Note that this is a little bit in the gray area between cooperative and non- cooperative. It's not like the runtime will just randomly preempt, it still requires cooperation from the code to have these safe points. But maybe I'm misunderstanding the details.

Anyway, not much changes for the Go programmer. The Go programmer can expect his code to be a little more predictable, at the cost of having his code be a little less performant, this performance cost I think will not be significant, if not fully compensated by the increased parallelism. All the promises the Go runtime makes towards the programmer will still hold.

<comment><USER3> > This proposal seeks to modify the way the "magically" part works. Right now, > it suggests that what Go does is to 'cooperatively' preempt at function > prologues. This change would instead allow the runtime to preempt at a > choice of "safe points" the compiler suggests throughout your Go code.

That's how it already works, it's just that safepoints only get introduced at function prologues. This is an issue as CPU-heavy functions without any non- inlined function calls mean very few safe points which is problematic for GC and timely scheduling and may even lock up the entire system.

Previous attempts were made to add loop preemption but there are issues with that approach.

The approach in TFA is to make safe points "opt-out", all Go code would be considered safe by default with some "unsafe" regions being excluded. As the author states, this would make Go preemptive rather than cooperative ("I propose that we implement fully non-cooperative preemption").

> Anyway, not much changes for the Go programmer. The Go programmer can expect > his code to be a little more predictable

Go programmers can expect their code to be a little _less_ predictable, not more: currently your code is guaranteed to run uninterrupted between two function calls, that will not be the case anymore.

In theory it should not make any difference, in practice it will probably uncover concurrency bug you're currently shielded from by the runtime's behaviour (though likely not that many as you can already have multiple routines running in parallel).

<comment><USER4> Not just function prologues, but also channel operations. Might sound like a minor nit but it's a pretty important nit.

<comment><USER3> Fair enough, though I'd assume that to be considered IO and would very much expect to be a place which always yields.

</comment></comment></comment></comment><comment><USER5> Interesting old thread comparing actors to software transactional memory (STM):

[https://www.reddit.com/r/haskell/comments/175tjj/stm_vs_acto...](https://www.reddit.com/r/haskell/comments/175tjj/stm_vs_actors/)

Seems like pre-emptive actor-based concurrency is simple to implement using STM, but much harder to implement STM with actors.

</comment><comment><USER6> I'm really excited about this both for the improved long tail latency as well as (what I believe is) a new way of handling this problem.

I'm not aware of another M:N threading model that handles preemption this way with signals. Erlang accomplishes it with reduction budgets and reduced performance.

</comment><comment><USER7> Pre-emptive runtimes are hard. Inserting yield points judiciously in loops isn't that incredibly difficult, and neither is reducing the cost of a yield point. It's far harder to make sure every machine instruction in every part of the runtime is prepared to be interrupted and have all it assumptions violated by the time it resumes.

<comment><USER8> > Inserting yield points judiciously in loops isn't that incredibly difficult, > and neither is reducing the cost of a yield point.

The Go team has done both over the past few years but we haven't been happy enough with the performance. Hence Austin's proposal.

</comment></comment><comment><USER9> I'd love to see this implemented, especially that I already enjoy this feature a lot in Elixir/Erlang/BEAM.

<comment><USER10> The problem with that approach in Go is that BEAM embraces the shared-nothing approach, and the Go runtime is quite promiscuous about sharing mutable data. I am afraid that introducing non-deterministic preemptive stops will just make Go concurrency a mess similar to the mutex/conditionals mess they've been trying to avoid.

<comment><USER11> "I am afraid that introducing non-deterministic preemptive stops will just make Go concurrency a mess similar to the mutex/conditionals mess they've been trying to avoid."

They already took a few steps inside of Go to try to _ensure_ that concurrency is quite non-deterministic. Even very simple, naive code like

        func main() {
            for i := 0; i < 2; i++ {
                go func(j int) {
                    fmt.Println("First", j)
                    fmt.Println("Second", j)
                }(i)
            }
            time.Sleep(100*time.Millisecond)
        }

will very frequently produce different results; I just ran that four times and got four different results. This is good, because it tends to bring bugs to the forefront more quickly, rather than having a almost-but-not-quite- deterministic model as has been the case in many other environments.

(Yes, I know sleep is not production-quality here, just trying to keep it simpler than several more lines for a proper waitgroup usage.)

I doubt there's a lot of code written at the Go level that would explode if you suddenly ran it with a pre-emptive runtime that isn't already exploding. As the proposal discusses, there's a lot of issues down at the runtime level, but at the layer of abstraction presented by the language itself I wouldn't expect a lot of problems. Non-zero, and even then, it would still be things that are technically already bugs. And Go code is already running _concurrently_ all the time in the real world. I'm not even sure how I'd construct an example that runs correctly in the current runtime but fails with true pre-emption.

<comment><USER12> > They already took a few steps inside of Go to try to ensure that concurrency > is quite non-deterministic

I would not be surprised as they are already proactively ensuring things like map ordering are randomised so that people just can't rely on some deterministic ordering: run the example code here[0] multiple times and it produces not just a seemingly random result (as would be for a typical hash table) but a truly different result _for each run_.

[0]: [https://nathanleclaire.com/blog/2014/04/27/a-surprising- feat...](https://nathanleclaire.com/blog/2014/04/27/a-surprising-feature-of- golang-that-colored-me-impressed/)

<comment><USER13> Yes, map iteration is explicitly randomized for just this thing.

</comment></comment></comment></comment></comment><comment><USER14> interesting, is the current situation one of the reason Go debugging experience is so poor? Delve debugger is constantly crashing for me, making step debugging extremely hard, especially when debugging tests. If there is one thing that is inferior to Java and C# with Go it's definitely the debugging experience.

<comment><USER6> I've actually never run into a problem debugging go binaries with gdb. Which is weird because everyone's always told me it doesn't work. I'm sure there are edge cases but over the last 5 years or so my experience has been pretty good.

emacs + gud mode is a great experience for go debugging. I've even used it to debug low-level issues like plugin loading and verification.

<comment><USER15> On Mac, I find lldb is terribly low level. IIRC Linux gcc isn't great either. Even delve will frequently seem to have missing/optimized variables that cannot be printed.

<comment><USER16> I frequently hit the problem with variables that are <optimized out> when debugging C code using gdb, and it's especially infuriating when I can look at the disassembly and see that the variable is _right there_ in the register file.

Does anyone know whether dwarf simply doesn't support the necessary debug information or whether compilers just don't retain it through the optimization process?

<comment><USER17> The latter. With quality register allocation, the same register can be used for multiple variables (or unnamed intermediate results) within the span of a few instructions. The inverse is also true: the same variable could find its value in multiple registers.

DWARF appears to support mapping symbol locations to registers, but I've never seen a language environment/debugger that takes advantage of it for optimized binaries.

</comment></comment></comment></comment></comment><comment><USER18> Can someone make an informed guess on how much that would complicate the implementation? And/or how much of a performance impact it would have (for better or for worse)?

</comment><comment><USER19> So, preemptive green-threads? This seems like a counter-productive and highly complicated re-implementation of what the OS thread model does, but inside the Go runtime.

I would very much like for goroutines to remain cooperative. It is much simpler, naturally more performant, and quite easy to reason about. A local implementation of preemptive execution seems like a very high price to pay to only handle a few corner-cases.

That is not to say that preemptive scheduling does not have their place: They make a _lot_ of sense at an OS level where independent processes are scheduled, but little sense as an internal green-thread implementation.

<comment><AUTHOR> > So, preemptive green-threads? This seems like a counter-productive and > highly complicated re-implementation of what the OS thread model does, but > inside the Go runtime.

You can't easily spawn 10.000 OS threads.

> I would very much like for goroutines to remain cooperative. It is much > simpler, naturally more performant, and quite easy to reason about.

Cooperative is less simple to reason about, as the proposal explains. Preemption fully delivers what the developer already expects of goroutines.

<comment><USER20> > You can't easily spawn 10.000 OS threads.

I just spawned 10.000 OS threads running separate mRuby instances in each thread on my laptop, to see if it'd cause any problems (because I happened to have an app sitting around that spawns separate threads; note: these are not Ruby threads, mRuby does not have built in threading). None at all. So yes, you can. I didn't check how much memory it'd take, however, or attempt to measure overheads in any way, so I'm not saying there aren't potential issues with it.

<comment><USER21> Simply using 1,000+ OS threads is often very close to as efficient as using coroutines or an event loop: [https://www.slideshare.net/e456/tyma- paulmultithreaded1](https://www.slideshare.net/e456/tyma-paulmultithreaded1)

<comment><USER22> If your event loop is as slow as 1000+ coroutines or OS threads, you are doing it very very wrong.

<comment><USER19> [Citation needed]

<comment><USER21> [https://github.com/golang/go/issues/12061](https://github.com/golang/go/issues/12061)

"Mark termination does not have to rescan stacks that haven't executed since the last scan"

With pre-emptive scheduling of OS threads, you potentially dirty a ton more stacks, and have to scan them all for pointers to heap memory. I'm not sure how VMs using lots of OS threads deal with this.

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> They should just make it possible to add safeties manually

<comment><USER24> isn't that what runtime.Gosched does?</comment></comment>


Ask HN: What is a free blog platform with custom domain support? - <AUTHOR> One thing I would like is to start writing a bit more but in the beginning, I don't want to pay for my own domain? Is there any alternative to medium which supports custom domain? <comment><USER2> Easy: wordpress.com . If you get more serious buy one of their very very affordable plans or export your blog and setup your own wordpress instance.

<comment><AUTHOR> Their free tier doesn't have custom domain.

</comment></comment><comment><USER3> Your best match would be Blogger by Google. go to blogger.com</comment>