The Math Behind the Rolling Shutter Phenomenon (2014) - <AUTHOR> https://petapixel.com/2014/10/13/math-behind-rolling-shutter-phenomenon/ <comment><USER2> Original submission (with 41 comments) from 6 years ago, with the original URL: [https://news.ycombinator.com/item?id=8446932](https://news.ycombinator.com/item?id=8446932)

Also, people should notice the "update" mention and shoutout to another blog post by Daniel Walsh, which went further into the math (and so parts were taken from Daniel's post to build the update section).

[http://danielwalsh.tumblr.com/post/54400376441/playing- detec...](http://danielwalsh.tumblr.com/post/54400376441/playing-detective- with-rolling-shutter-photos)

</comment><comment><USER3> CMOS cameras almost always have a rolling shutter, but not always. At the Very High End of things, there are models that don't. e.g. the Sony PMW-F55 with a CMOS global shutter [1], around USD$30k

In the struggling camera industry, I bet a consumer-priced global shutter (or excellent computational correction for a mechanical/electronic roller shutter) could be a boost. Smaller, solid state, etc. The next logical step after mirrorless?

[1] [https://pro.sony/ue_US/products/digital-cinema- cameras/pmw-f...](https://pro.sony/ue_US/products/digital-cinema- cameras/pmw-f55)

</comment><comment><USER4> So here's a fun little hack you can do at home. If you have a signal (up to ~20 khz) and you feed it into an LED and look at the light through your camera phone, you can see the signal as bright and dark rows in the image. The bars are always in the same orientation (dependent on how your camera reads the data) and the spacing depends only on the frequency. One idea I've been toying with is using these properties to quickly isolate the parts of an image that are near to a transmitting LED in order to rig a camera into a 3d input device with very minimal image processing.

<comment><USER5> > If you have a signal (up to ~20 khz) and you feed it into an LED and look at > the light through your camera phone, you can see the signal as bright and > dark rows in the image.

Can you share an example photo?

<comment><USER4> Sure. Here's a direct photo of an LED blinking at ~1Khz [https://imgur.com/a/Kmx9Qhe](https://imgur.com/a/Kmx9Qhe) and here's one at 10Khz (you need to zoom in a bit) [https://imgur.com/a/Q7oqs9J](https://imgur.com/a/Q7oqs9J)

When the light is passive (looking at the wall instead of the LED itself) the pattern is less obvious to the naked eye. Possibly invisible if there's enough ambient light. Around 20Khz the separation between bars is 1 pixel (on my camera) so you need to zoom all the way in to see it.

BTW check you phone camera settings. Most phones intentionally correct for the bars around 50/60 Hz because that is the frequency of AC power so florescent lamps will cause the artifact. The effect is usually unwanted.

<comment><USER6> Thats why my philips hues make that pattern. Thank for the info

</comment></comment></comment></comment><comment><USER7> One of the engineering teams I work with is building an image pipeline to correct rolling shutter distortion from a robot camera. I still have no idea how they're generalizing a solution to different types of movement and angular velocities, but at least now I understand the problem and a taste of the math behind it.

Thanks for sharing!

</comment><comment><USER8> I've wondered if rolling shutter could be (ab)used as a poor man's one- dimensional high speed camera. Depending on the framerate, resolution and exposure time, wouldn't you be able to get hundreds of thousands of frames per second, at the cost of having no vertical resolution.

I'm not sure what it'd still be useful for with those limitations. Maybe estimating the speed of fast moving objects, or global phenomena like vibrations.

<comment><USER9> Definitely. There are programs for the Raspberry Pi cameras which do just that. Up to 1000fps IIRC

<comment><USER10> I don't think they really make use of rolling shutter, they just configure the camera differently? (which is basically limited by the rate of pixels transferred - if you reduce the resolution, more images per second fit down the data transfer channel)

<comment><USER9> I think they make use of the fact that individual lines are updated on the camera by the light and can subsequently be read out. If it were a camera with global shutter, chances are it could not be configured to update lines as fast indivually. A global shutter design copies each pixelsite or line into an indiviual buffer when the "shutter" signal is triggered. This happens in parallell.

I'd say these high framerates on Raspi cameras definitely is an artefact of the rolling shutter design in these cameras.

Maybe one could get even higher framerates by hooking the analog part (or very close thereafter) of the rolling shutter camera up to something else, but I don't see how.

</comment></comment></comment></comment><comment><USER11> OT: Ugh. Another site that hijacks the back button to display more articles rather than letting me go back. (Chrome iOS) When did this start?

Edit: Video: [https://share.icloud.com/photos/0qLMMePqiSOETXWt2Z77Ae8jA](https://share.icloud.com/photos/0qLMMePqiSOETXWt2Z77Ae8jA)

</comment><comment><USER12> Certain types of mechanical shutters exhibit the same phenomenon at high speeds.

See: Jacques Henri Lartigue's picture of the Grand Prix of the Automobile Club of France, taken in 1912. You may have seen the photo before. Notice how the car leans forward and the spectators lean backwards.

[https://www.holdenluntz.com/artists/jacques-henri- lartigue/](https://www.holdenluntz.com/artists/jacques-henri-lartigue/)

<comment><USER13> > Certain types of mechanical shutters exhibit the same phenomenon at high > speeds.

Pretty much all focal-plane shutters do it beyond 1/60 to 1/250 of a second (identical to the flash sync speed), because the finite movement speed of the upper and lower curtains would result in an uneven exposure otherwise.

</comment></comment><comment><USER14> A related video from Matt Parker, aka Stand-up Maths, on this phenomenon [https://www.youtube.com/watch?v=nP1elMR5qjc](https://www.youtube.com/watch?v=nP1elMR5qjc)

</comment><comment><USER15> I suppose it's coincidence that some of the examples look like relativistic stuff round black holes.

</comment><comment><USER16> Eh, this is from 2014.

<comment><USER17> Added the year now. Thanks!

Older posts are fine.</comment></comment>


Evernote Adds Twitter Integration - <AUTHOR> http://blog.evernote.com/2009/04/14/evernote_twitter/

<comment><USER2> I already use IFTTT.com to record all links I send and all of the ones I star to Evernote. Works pretty good, but with a fifteen minute lag.</comment>


Apple terminated developer account for unstated fraud after minor app update - <AUTHOR> https://twitter.com/_zacwood/status/1298053471019634688 <comment><USER2> > UPDATE: Attics has been approved and my account will no longer be terminated > -- apparently my app had been copied across the App Store many times and it > wasn't caught that mine was the original. Thank you everyone for spreading > this and helping me keep my account!

[https://twitter.com/_zacwood/status/1298105327699791878?s=20](https://twitter.com/_zacwood/status/1298105327699791878?s=20)

<comment><USER3> I mean as an iOS developer I hope Apple is banning developers who steal/copy apps like in this case. Obviously they terminated the wrong dev, but they also fixed theor mistake quickly.

<comment><USER4> Yikes... I mean it's not that hard to look at the first uploaded date. Unfortunately they don't really have any pressure to do better in this area not like anyone can go to another app store.

</comment></comment></comment><comment><AUTHOR> From the developer: Submitted a minor update for @AtticsApp after nearly two years on the App Store and @Apple has decided out of nowhere that I have been engaging in "a pattern of manipulative or misleading behavior" and has permanently terminated my account. No evidence or specific reasons given.

This is a completely free "hobby" app by the developer, so the allegation of fraud (with no details) by Apple is quite bizarre.

<comment><AUTHOR> Before anyone starts speculating on what the developer did "wrong", keep in mind that the Mac developer Charlie Monroe was mistakenly flagged and then reinstated by Apple just a few weeks ago: [http://blog.charliemonroe.net/a-day-without- business/](http://blog.charliemonroe.net/a-day-without-business/)

<comment><USER5> His app opens fine for me: [https://apps.apple.com/us/app/attics/id1434981632](https://apps.apple.com/us/app/attics/id1434981632)

Does it normally disappear if a dev account is terminated?

<comment><USER6> Not necessarily.

<comment><USER5> You would think a developer accused of something as serious as _fraud_ would have their apps at least removed from the app store... this whole thing is a little odd.

<comment><USER6> Being considered by Apple to have defrauded Apple does not necessarily indicate that Apple sees a risk of user harm stemming from that fraud.

</comment></comment></comment></comment></comment></comment><comment><USER7> The developer in a tweeted reply: _" 3.2(f) of the agreement seems to be centered around App Store fraud and deceptive business practice. Given it's free w/o IAP, and I've never advertised the app, not sure where they're getting that from"_</comment>


Newzbin Prepares for Litigation - <AUTHOR> http://www.slyck.com/story1782_Newzbin_Prepares_for_Litigation

<comment><AUTHOR> As a user of Newzbin, this makes me sad. It's the easiest/fastest way to download TV shows. I hope their legal team knows what they're doing.</comment>


Charts of alcohol consumption, obesity, and physical activity in Europe - <AUTHOR> http://www.washingtonpost.com/blogs/worldviews/wp/2012/12/04/which-europeans-are-fattest-laziest-and-drink-most-in-charts/?tid=socialss

<comment><USER2> There is a flaw in the obesity chart. If you will note, the highest obesity countries are actually based on measurements; the rest of them as self- reported. If you ask a person (especially a man) if they're obese, quite a few will say "no"[1]. Mixing different measurement techniques is a no-no, and makes for flawed conclusions.

[1] <http://en.wikipedia.org/wiki/Illusory_superiority>

<comment><USER3> I interpreted this as the country self-reporting, not the individuals.

</comment></comment><comment><USER4> As an American I found the update quite interesting. Americans have a 35.9% obesity rate versus 28.5% for the most obese European country.

I'd like to understand why that is, but would guess that the US obsession with cars, parking lots and suburbs is one reason. Another might be related to the fact that Americans tend to, on average, do more poorly, on standardized education tests, which could imply that we have more people here who are ill equipped to understand how things like nutrition, calories and physiology work.

<comment><USER5> There are a lot of demographic factors in play too. The US has a much higher proportion of blacks and Hispanics than any European country. These groups are 51% and 21% more likely than whites to be categorized as obese using the usual methods. The US also has a far higher percentage of people livif in rural and suburban areas. In my own experience, my equillibrium weight when I was living in the suburbs of virginia was notably higher than when I was living in New York or Chicago.

The same is true for educational metrics, by the way. The so-called standardized testing gap between the US and European countries almost disappears when you adjust for demographics.

<comment><USER6> Could those groups higher (if they are higher, I don't know) obesity be attributed to poverty?

In general, poor Americans, are way worse off than poor people in most EU countries.

<comment><USER7> Ever visited a poor country? There aren't many obese people in poor countries for one very obvious reason.

<comment><USER6> Povertry in first world countries are different from povertry in, for example, africa. There are poor people in europe and the US, they are not poor in the sense that a african can be poor but that is not the same as they are not poor and that their behavior are not influenced by their poverty.

<comment><USER7> Let's not conflate the issue. If poverty was a contributing factor for obesity, then poor people worldwide would show higher rates of poverty. People don't get fat because they are poor, they get fat because they don't care that they're getting fat. If anything, being poor would be a disincentive for obesity - why spend money on something that you don't have to when money's in short supply? By basic logic, the poorer you are, the less obese you should tend to be. Notwithstanding the occasional actual medical condition which leads to obesity, being obese is a conscious, cultural decision, and blaming it on poverty seems ridiculous.

<comment><USER6> In first world countries there is a link between obesity and poverty. In general if you are poor you a re more likely to be fat. For example see <http://www.ncbi.nlm.nih.gov/pubmed/14684391>

You also seem to imply that it's cheaper not to be fat, in many first world countries the unhealthy food seem to be the cheapest!

<comment><USER7> I know that there is a link between obesity and poverty, but correlation does not necessarily imply causation, and in this case, anyone who buys and cooks their own food will immediately recognize the causation claim to be ridiculous.

The unhealthy food being the cheapest food argument is bunkum - just go to reddit.com/r/budgetfood and try to tell them that processed food is cheaper. There is nothing cheaper than chicken thighs, rice, beans, noodles, and various veggies. Every week, I make a crock-pot of something tasty and healthy from scratch. This week, I made a wonderful soup with fresh carrots, potatoes, celery, onions, and pork loin. The total cost was about $15, and I got 7 big portions. Substitute something less expensive than pork loin, and it's even cheaper. Last week, I made a huge pot of spaghetti sauce, complete with italian sausage, fresh mushrooms, onions, garlic, and a few extra cans of whole tomatoes for about $10, and I wasn't even really trying to do that one cheaply. I got about ten portions out of that one. The week before, I made a quiche type-thing with a dozen eggs, some breakfast sausage, a bag of hashbrown potatoes, onions, tomatoes, and a bag of cheese. Total cost: about $10. Not the healthiest thing, but better than processed junk, and again, fills you up quite nicely 7-8 times. I could go on, but you get the picture, I'm sure, and this is just with a crock pot - if I had a functional kitchen, my cooking options would expand considerably.

People who eat unhealthy food don't eat it because it's cheaper than healthy food, they eat it because they don't want to cook.

But if you really want to know why poor people tend to be obese, watch what they drink: sugar. All day, every day, it's sugary drinks, from kool-aid to sweet tea to sodas - I don't know why, but for some reason poor people tend to drink a lot of sugary trash.

Which, again, is more expensive than the healthier option: water.

Leftist apologists like to trot out the 'healthy food is more expensive' argument because suggesting that poor people are obese because of their lifestyle decisions rather than their supposed status as victims of the system runs contrary to the left's general line of bullshit, which is that if there is a problem, it stems from systemic failure, and that a government policy should be implemented to alleviate that issue. Of course, not all problems can be fixed with government programs, this being one of them. People are not obese because they are victims, they are obese because they value stuffing their face with unhealthy food more than they value being fit. It really is that simple, and unfortunately, that is a social problem that no government policy can fix.

<comment><USER6> My original argument was that there was a causality between poverty and obesity which I thought you argued against but I am glad that we agree.

The underlying reasons for this link I don't think we will agree on.

<comment><USER7> I think you may be a little confused about the difference between causation and correlation. There is a correlation between obesity and poverty, but whether poverty is a cause of obesity or not is far from certain, and it is my stance that it is not a cause at all - and especially not because 'processed foods are cheaper than healthy foods.' Anyone who believes that has not spent nearly enough time in grocery stores, and anyone who believes that poor people are fat primarily because of an inability to afford healthy foods has not spent enough time around poor people.

<comment><USER6> No I am not confused. If you read my original post I only talk about correlation, not cause. I think you do the argument far to black and white but that was not what was my point.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> What I find most striking is that most European countries have a bigger alcohol problem than America. I might not have guessed that right away. I wonder if it's mainly due to the stronger Christian influence in America.

<comment><USER9> Not really Christian, but puritan. Alchohol is actually a big part of Christianity. It was only the Puritans that were against it. That is probably why the Puritans were kicked off to America.

But I suppose alchohol is just treated differently in Europe. It is considered almost like food. A beer or wine is part of your dinner, it is not something that you only drink on a special occasion or just to get drunk.

<comment><USER10> The UK has a severe problem with alcohol.

There are a variety of drinking cultures. There's the well known youth culture of daily drinking, and drinking to excess at weekends. This is drinking to get drunk.

This problem drinking is encouraged by 'VDEs' (Vertical Drinking Establishments) which use tactics to get people drinking more. There is a lack of seating and tables. That fits more people in, but also you're always holding your drink. This makes you drink faster. Music is loud, so you're not talking as much, which also makes you drink more.

But there's the other less known problem drinking among adult professionals who drink a couple of glasses of wine every night.

([https://commons.wikimedia.org/wiki/File:Units_of_alcohol_cha...](https://commons.wikimedia.org/wiki/File:Units_of_alcohol_chart.svg))

One 200 ml glass of wine at 11% is 2.2 units. 5 of those is 11 units, which is close to sensible limits of women. 10 of those is 22 units, which is over sensible limits for women, and close to sensible limits for men.

This isn't alcohol with a meal, and it's not traditional binge drinking, but it's linked to the rise in younger people with cirrhosis.

<comment><USER11> (Full disclosure: I'm a teetotaler)

Everything you described also applies to Spain. Here, it's usual to have a beer while eating (wine is less common for young people); but, usually, "drinking to get drunk" is the norm in weekends and evenings. Many people actually do it on Thursdays, even if they still have to work or have classes the following day.

Here, most people wouldn't call it a problem.</comment></comment></comment></comment>


Reddit Enhancement Suite - <AUTHOR> https://redditenhancementsuite.com/ <comment><USER2> Reddit w/o RES is like McDonalds without the Big Mac. Been using it for years.

Shame that most of its features are broken with the pathetic new redesign though[1].

[1] - [https://www.reddit.com/r/Enhancement/comments/7urcrp/res_des...](https://www.reddit.com/r/Enhancement/comments/7urcrp/res_desktop_redesign_status/)</comment>


Mind-blowing 3D projection on German building (video) - <AUTHOR> http://freshome.com/2009/07/25/mind-blowing-3d-projection-on-german-building-created-by-urbanscreen/ http://www.engadget.com/2009/07/27/video-life-altering-3d-projection-splashed-on-german-building/ <comment><USER2> Life altering???

Honestly, if anyone knows why this is supposed to be life altering, please tell me. I don't get it at all. Yes, it's cool, but it's a loooong way from life altering.

<comment><USER3> I think they mean "life altering" in the "altering the real world" sense.

</comment></comment><comment><USER4> Pretty clever & cool.

Any ideas on the software used for creating the images?

</comment><comment><USER5> I'm pretty sure some of that audio track was taken from the same audio track that Roller Coaster Tycoon used. Its definitely fake, it loops.</comment>


The Implications of Hitting the Hard 0% Interest Rate Floor - <AUTHOR> https://www.linkedin.com/pulse/implications-hitting-hard-0-interest-rate-floor-ray-dalio/ <comment><USER2> Runaway inflation is coming. Mass unemployment is coming. We wasted the decade of growth on pumping up the equity markets to make the rich richer and ignored schools, hospitals and infrastructure. All this was an inevitable consequence of giving away unlimited free money to the wealthy.

<comment><USER3> The bigger risk is probably a huge deflationary spiral given the huge decline in demand caused by consumers not engaging in the economy.

</comment></comment><comment><USER3> I don't understand finance and economics as well as I'd like but it is always refreshing to read Ray Dalio's views mostly because of his focus on the long term but also because of his proven track record on these things.</comment>


Screw it, I quit - <AUTHOR> http://abstractgeneratorfactory.blogspot.com/2008/06/screw-it-i-quit.html

<comment><USER2> "I have two backup plans (in case the first doesn't work), and I have no fewer than three exit-strategies (how to get back into boring corporate land)."

Just be careful that those don't screw you. Nothing is more motivating than the "oh my god I need to pay my rent and buy dinner but I have no money!" feeling. Having backup plans and exit strategies can sabotage that.

But, as a fellow recent quitter myself, I have to admit that I also have my own backup plan for when the money runs out... grad school. ;)

Congrats to all those that have quit recently, and to all those considering it, just walk in and do it. I guarantee you'll be singing and skipping on the way out.

I simply saved up six months expenses and gave my notice at a convenient time in the project schedule at work (end of one project, before being put on another one). No bridges to burn (I didn't have a lot of complaints about the job, and the coworkers were great) so I have a standing offer for employment in January (when the money runs out) should I need it. I just knew I'd never be happy doing maintenance and rewrites in cubicle land.

</comment><comment><USER3> Funny reading this. I just resigned today. No kidding.

If I had seen this post before resigning, perhaps I would have sent it instead of the one I sent.

<comment><USER4> Congrats :)

</comment></comment><comment><USER5> If not now, when? That was the question I asked myself over and over again. I submitted my 3 month notice, and I'm going to following my dreams of creating a solution to a business problem. Why 3 months? I was actually going to quit sooner, but I just wanted to be nice and give my boss plenty of time to find a replacement. However, don't get me wrong, I'm itching to leave and startup! Life isn't about money folks! It's about following your passion!

</comment><comment><USER6> Short and to the point.

How did others quit? Please share.

<comment><USER7> First, I realized that there are two types of happiness relevant to a job. There is happiness through satisfaction in what you have, and happiness through greater achievement. I suspect these are actually personality traits. Some people are good at being happy with what they've got. I'm not.

Then I realized I would never have happiness through satisfaction in my current job. Even if I succeeded wildly and became a VP or CEO one day. At a very deep level, I hate the feeling of disdain from entrepreneurs. Larry and Sergey won't respect you in the morning, indeed.

Then I convinced myself that my full-time employment was not on a trajectory that would bring me full happiness ever. And life is short.

Thus, the logical conclusion. Change is necessary.

I don't have cofounders, so I'll strike out on my own for a while and see what I can build. I'll go to grad school and find cofounders.

Now, details. I talked my way into a good school long after application deadlines. This satisfies important stakeholders such as my parents.

I gave myself enough unemployed time before school to be able to hack something and test out how I work when I am my boss.

I saved a year's living expenses while employed. I moved to 50% less expensive housing and cut other expenses.

I was diplomatic at work about my departure. I didn't burn bridges. I resisted the temptation to vent about everything I thought was wrong about the company and the job. If everything fails, I have a safety net that is only slightly worse than if I didn't take the plunge. I gave my manager notice. I told him my end date was flexible so he had a chance to deal with staffing issues and I could wrap up work. It ended up being a month's notice. Not so long that I waste a lot of time checked out, but long enough that it is a happy parting.

My last day is Friday.

<comment><USER4> I had a few major events in my life that woke me up to these same concepts.

" full-time employment was not on a trajectory that would bring me full happiness ever. And life is short." - exactly.

Change is also very important to keep me going - and being pigeonholed in a small role at work always frustrated me too. Glad to see it isn't just me!

</comment></comment></comment><comment><USER8> I hope I can say it too .... not now. I still have excuses.

<comment><USER7> Let us help you eliminate the excuses.

<comment><USER9> I have excuses also.. but I'd really rather not eliminate this excuse (getting married, and need the money to pay for it) :)

</comment></comment></comment><comment><USER10> It's a pretty big lifestyle change, and what if ...you fail?!

You do need to prepare and then be able to live without the security that comes with a full-time job.

It's not easy, but it is worthwhile.

</comment><comment><USER11> Well I didn't exactly resign, but the startup I work for is not going to get it's 2nd round of financing...

<comment><USER10> ouch! I hope you're looking around ...

</comment></comment><comment><USER12> i love it. ;)</comment>


Apple loses $64B in stock value as Wall Street is in 'full panic mode' - <AUTHOR> https://www.cnbc.com/2018/04/24/apple-loses-64-billion-in-value-as-wall-street-is-in-full-panic-mode.html <comment><USER2> On the theory that iPhone demand is getting lower, I'd have to agree. I used to upgrade every 2 years because there was some cool feature or battery life improvement I wanted, but my 6S is definitely going to be sticking around until the next announce (so 3 years now). I really can't see why my parents would upgrade their phones in the next 5 years unless their batteries stop working.

Whats driving new phone sales nowadays? Games and apps have sort of plateaued (though lucrative for some), streaming services are pretty much matured and are going to work on any current device for a while. I can't think of what the "next" thing is going to be. I felt that health was their best shot, but it seems like the tech just isn't there to give exceptionally useful data, plus you probably need a watch.

The only thing I can think of that keeps it going with non-tech is the status symbol aspect. Resolution is past the point of really mattering. Battery life could be improved, but its finally alright. Bezels? Who really cares about a few MM of screen space, except for those who want to signal they paid $1000 for a phone?

<comment><USER3> Batteries aren't replaceable anymore so people get a new phone just for a fresh battery.

The battery on my Nexus 6p is unbearable now after 2.5 years, it hardly lasts two hours.

<comment><USER4> Batteries can be replaced for peanuts thanks to Apple's new battery replacement program. (free or $29 for out of warranty if it's a newer model, [https://support.apple.com/iphone/repair/battery- power](https://support.apple.com/iphone/repair/battery-power))

</comment></comment></comment><comment><USER5> 8% drop, stock still higher than it was 1 year ago. "full panic" ?

<comment><USER6> It could go up 20,000% one day and down 50% the next and the headlines would read "STOCK CRATERS".

</comment></comment><comment><USER7> Article says main cause is weakness in Apple partner Taiwan Semiconductor, but could compliance in Ireland on back taxes also be a factor?

<comment><USER8> Ireland was baked into the price long ago.

</comment></comment><comment><USER9> TSMC customers are diversifying into ASICs on "7nm". Shelf life of chips is about to get a lot longer.</comment>


Net Neutrality Will Be a Campaign Issue in 2018 - shaki-dora https://www.wired.com/story/why-net-neutrality-will-be-a-campaign-issue-in-2018/ <comment><AUTHOR> Yeah, maybe... but if that's the case, nobody on the Internet will be allowed to know about it. </sarcasm></comment>


Getting Started with Apache Avro - <AUTHOR> http://idarlington.github.io/2017/Getting-started-avro/ <comment><USER2> Yeah... you're only going to save money if you're an absolutely enormous scale. I'd also really question that point at all when you can compress JSON as it's being transmitted (GZIP, baby).

Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON. Almost all serialization libraries are written in C for MAXIMUM PERFORMANCE regardless of the language you're using, or some sort of hand-tuned-magical bytecode fuckery which has the same effect.

I honestly can't imagine this being useful except for obscenely high throughput service to service communication or as an internal serialization protocol (for something like FoundationDB where you're storing the Avro encoded instance to disk). Truly, JSON and REST are good enough for 99.999% of services and the overhead of using something non-standard will likely never pay off due to the increased developer overhead and maintenance.

This just smells like tech debt and an architecture that can't be debugged.

<comment><USER3> I'm toying with Avro right now for a project and our common JSON payload went from 1400 to 70 bytes, which makes it much more suitable for use over UDP

Also using JSON in a statically-typed language is tedious and error-prone.

<comment><USER4> How much of your data is repeated as part of a cyclical graph? You may want to try jsog + compression.

[1] [https://github.com/jsog/jsog](https://github.com/jsog/jsog)

</comment></comment></comment><comment><USER5> If the author is around:

> Translates to less money saved on provisioning hardware.

I read that as it will cost more. Saving less money is bad.

I assume that is a typo?

</comment><comment><USER6> Okay, so the article claims (without much evidence) those pieces where Avro shines.

I'm more interested where it _fails_. It's the places where a library falls down that's likely to burn me when I'm 24 hours from shipping a product.

<comment><USER7> We've used Avro in production for a number of years now. We use it to define the schema for Kafka messages passing between parts of the system and it has worked well - but you want some downsides, so..

Compatibility: Avro deals with this well, but there are gotchas especially when you need/want to have "forward compatibility" as well as "backward compatibility" in your messages. Adding a new type to a union? Breaking change. Adding a new element to an enum? Breaking change. Want to add a new member to a type? Best give it a default, or.. breaking change. We use the Confluent schema registry to manage our schemas on kafka and we find ourselves having to do carefully managed rollouts (stop all producers, put schema-reg into compat mode, upgrade consumers, upgrade and start producers) every now and again

Performance: we frequently see Avro as the "hot" code in our apps - mainly I suspect because we've got a fairly large schema with lots of union types etc, and our messages are up to 2kb or so in size. This results in our applications topping out at low tens of thousands of messages per second, though it scales linearly with number of cores. The most recent java version has a "fast read" flag that promises to speed up our use-case but we haven't tried it yet

<comment><USER3> > Adding a new type to a union? Breaking change. Adding a new element to an > enum? Breaking change.

This is incorrect. I've personally gone from type X to union {X,Y} in C++, you just need to use the resolving decoder that takes both schemas. Enums can also be resolved based on names. Check out the schema evolution rules here:

See [https://avro.apache.org/docs/current/spec.html#Schema+Resolu...](https://avro.apache.org/docs/current/spec.html#Schema+Resolution)

Of all the binary serialization formats of this ilk (proto, thrift, flatbuffers etc), Avro definitely has the most schema flexibility.

</comment></comment></comment><comment><USER3> The JSON schema format tedious to write and difficult to read. I recommend anyone using Avro to use the IDL language[0] and use the Java tool to convert to the JSON schema format.

java -jar avro-tools-1.9.2.jar idl yourschema.avdl

[0] [https://avro.apache.org/docs/current/idl.html](https://avro.apache.org/docs/current/idl.html)

</comment><comment><USER8> When managing millions of transactions through Kafka that have to be serialized to s3, read by hive natively, translated to parquet by spark, support nullable complex types (map, struct, array), allow schema evolution forward and back, then Avro is a good choice for your base data format.</comment>


Fixing Weak Wi-Fi Router Security - <AUTHOR> https://www.nytimes.com/2018/06/13/technology/personaltech/wi-fi-router-security.html <comment><USER2> there is a linux distribution for wifi-routers:

[https://openwrt.org](https://openwrt.org)

table of hardware: [https://openwrt.org/toh/start](https://openwrt.org/toh/start)

\- current master runs kernel 4.14 / 4.9 for most targets, flow offloading, performance fixes, wireguard in base, lua-based ui called uci.

\- security fixes land after a few hours/days in master, a few days/weeks for a new stable release

\- pretty much only non-commercial and volunteer effort, so be kind and friendly and help - check the wiki and the forum first.

<comment><USER3> > security fixes land after a few hours/days in master, a few days/weeks for a > new stable release

The latest stable release seems to be ~8 months old, though, unless I'm looking in the wrong place: [https://downloads.openwrt.org/releases/](https://downloads.openwrt.org/releases/)

<comment><USER4> Releases are a fixed point in which packages are updated over top, as I understand it

Similar to installing say, Debian 6.1 and then running apt-get to update packages

<comment><USER2> unfortunatly not really - due to the small flash on most devices the rootfs is compressed into a squashfs - but you can checkout the latest stable branch from git and build images with up2date kernel.

</comment></comment></comment></comment><comment><USER5> > Replace your router every few years

How about instead of this, use open source software on your router? It will keep being updated, and with the manufacturer's proprietary software on the device you can't really trust it anyway.

<comment><USER6> That's a good solution for geeks, not so much for everyone else. Regular people don't even update their routers, much less flash 3rd party software on them. I don't think most people even know updating your router is even a possibility.

I use Google Wifi and it updates itself. In the future I might put in a PFSense, but wifi solutions like Google Wifi/Eero/etc are the way to go if you're not a computer person.

<comment><USER7> That may be behind the Eero move to go subscription only. Could be good if implemented properly but much more expensive that just buying a high end router.

<comment><USER8> Plume just announced that you'll need to buy an annual cloud license for their new stuff as well. I just dont see this taking off as a business model. How large is the intersection of people who know give a high enough importance to router security with the people who couldn't roll their own solution?

</comment></comment></comment></comment><comment><USER9> Great resource: [https://routersecurity.org/](https://routersecurity.org/)

<comment><USER10> thanks for this. i'm contemplating an upgrade of my venerable linksys wrt54g running tomato to something more modern. i'll have to check out their recommended peplink surf router:

[https://routersecurity.org/pepwavesurfsofo.php](https://routersecurity.org/pepwavesurfsofo.php)

</comment></comment><comment> amiga-workbench Grab a decent microtik router and a few Ubiquiti Unifi AP's, setup automatic updates, and never touch them again.

<comment><USER11> So... two of the routers affected by the recent VPNFilter malware? Interesting choice.

<comment><USER12> > So... two of the routers affected by the recent VPNFilter malware? > Interesting choice.

If you're looking for a router that's never had a documented security flaw, you're probably going to buy a no-name brand that's full of them (because no one's looked yet, so it has a "clean" record).

The factors that you really need to look for are 1) good engineering practices for security, and 2) prompt and effective response to flaws. 1) can hard to verify completely, but you can get a sense of 2) based on patch cycles.

I have a Mikrotik router at home, and I chose it because their products are inexpensive and aimed at professionals, which means the software support is _much better_ than consumer routers. Mine is quite old, but it _still_ gets patches.

<comment><USER13> I wasn't aware that the Unifi stuff was vulnerable to the latest VPN stuff. I own a few ER-Xs and a Unifi AP. They're reasonable kit, but I wouldn't recommend them at all as a set it and forget it system.

\- Ubiquiti has a track record of GPL violations (e.g. u-boot which dovetails nicely with a security vuln)

\- The Unifi AP is tolerable for a simple home env but not much else.

\- Ubiquiti support is non-existent. They basically slapped a slick GUI on Vyatta and resold it. It's nice, but they don't have much in the way of developers. So, for instance, they still haven't fixed the hardware acceleration bugs in the ER-X or the WPA2 enterprise issues in the Unifi AP.

\- Ubiquiti hardware itself is hit and miss. The ER-L, for instance, is known to overheat and cook itself to death. There was a mixup with some of the PoE stuff (UBNT historically used non-standard PoE) meaning you're not entirely sure what's in the box.

UBNT hardware cheap and you can hack on it, so that's nice. But, being aimed at professionals and actually suitable for professionals are two separate issues.

<comment><USER14> I'm looking for something to update to. If not Unifi, then what brand would you recommend that would be suitable for home use by a professional, that can be updated and has good support?

<comment><USER15> Get an apu2 [0] from pcengines and slap OpenBSD on it (or Linux, if you prefer).

</comment></comment></comment></comment></comment></comment><comment><USER16> The article concludes that you should spend $200 on Eero or Google WiFi. Is this an advertisement?

Also note that $200 in "value" is mostly covering the massive marketing budget of Eero and Google. You do not need to pay $200 for a secure WiFi solution.

<comment><USER17> Possibly. The NYT bought The Wirecutter, which did (pretty good) reviews of "best in class" products and then made money from the Amazon affiliate commissions.

[https://www.recode.net/2016/10/24/13381002/new-york-times- wi...](https://www.recode.net/2016/10/24/13381002/new-york-times-wirecutter- purchase-30-million-briam-lam-consumer-guide)

I'd still say the point stands that 99% of the routers out there have awful security and it's just a matter of time (often months) because your router is overtaken by a botnet, especially if you don't update to the latest firmware (whenever/if that may arrive).

Open source software helps, but if the firmware for your particular router is updated less often than every 12 months, I think you'll also become just as exposed.

<comment><USER18> Is there any reason to believe Wirecutter or NYT has compromised the integrity of their reviews? That seems an unfair allegation to throw around absent any evidence beyond it _possibly_ being in their financial interest.

<comment><USER19> Here is one example allegation:

[http://www.evodesk.com/wirecutter-standing-desk-review- pay-t...](http://www.evodesk.com/wirecutter-standing-desk-review-pay-to-play- model)

With most conflicts you won't know the whole truth, but it's a data point.

The Wirecutter also doesn't give the 'best' in everything. They do have implict budgets.

For example, they won't recommend $5000 stereo speaker pairs, even though they would be better than the KEF Q150s they recommend currently. They don't recommend full frame / medium format cameras either, since they are probably too high end and expensive for their target markets.

I don't really fault them although for having some cost limits although. They would probably make the valid argument that if your buying the full frame cameras and $2500 loudspeakers that you probably know what your doing and don't need the wirecutter. I do wish they pointed it out a bit more in some of their guides although.

<comment><USER18> Fair enough, but you should do the courtesy of posting their response [https://thewirecutter.com/our-response-to- nextdesk/](https://thewirecutter.com/our-response-to-nextdesk/)

</comment></comment></comment></comment></comment><comment><USER20> This should be renamed to "Fixing _router_ security". Nothing in the article is actually specific to Wi-Fi. They don't even mention disabling WPS which is the #1 vulnerability in consumer-grade Wi-Fi networks.

<comment><USER21> Thanks, we've added a "Router".</comment></comment>


Obama's 2011 Budget Proposal, Department by Department (Infographic) - <AUTHOR> http://www.nytimes.com/interactive/2010/02/01/us/budget.html?hp <comment><USER2> One thing that should immediately pop out at even the casual observer: the traditional bugaboos of the right (social services for poor people, welfare, etc.) are a tiny, tiny minority of the budget. You can cut them down to an expenditure of 0, and you won't really change the bottom line.

If we're going to come to terms with the national deficit, we're going to have to accept the fact that military, social security and medicare expenditures are going to have to shrink. The problem is not poor people: it's old people and wars.

<comment><USER3> Isn't the 560 billion for "income security" a social services thing? Isn't it giving money to people who don't have jobs, or make very little money? So that's quite a lot of money in the welfare/social services category.

And personally I'd count social security and medicare in the same sort of category. It's the government spending a ton of money inefficiently providing services that should be provided by the free market, and saying this helps people.

It's good to help people, but it's not the right role for Government, which isn't very good at it.

On the other hand, the military spending may be a lot of money, but at least it's in one of the areas the free market has the hardest time with -- it's spending money in an area where the Government actually has some expertise and some business doing anything.

<comment><USER4> Social security provides money, which is about as efficient as it gets. Are you positing that private pension benefits are somehow spent more efficiently?

As far as whether it helps people, it unquestionably does. Maybe it hurts other people (those who pay taxes), but that income transfer would not likely happen under private plans. So it is not an apples-to-apples comparison.

<comment><USER3> Yes, private pensions have different rules and regulations about when to put money in, take it out, etc. For one thing, social security forces you to contribute. If you would rather spend your money on something else (in general, or just this month), you can't. Sometimes that hurts people who are trying to make rent. Sometimes it's inefficient b/c they could have used the same money for something better. Private pensions, if they get nothing else right, at least are voluntary.

</comment></comment></comment></comment><comment><USER5> $251 billion (+33%) in interest. That is a staggering figure.

</comment><comment><USER6> <http://wiki.lesswrong.com/wiki/Politics_is_the_Mind-Killer>

</comment><comment><USER7> Unfortunately the colour spectrum is unfriendly to those of us with red-green colour blindness: it's hard to differentiate between positive and negative percentage changes without looking at the tooltips. Aside from that, nice treemap.

</comment><comment><USER8> Some Observations, I always find it interesting how the break it up.

1\. Normally the FBI & DOE are not considered national defense

2\. Why is Medicare separate from health?

3\. Why are veterans benefits & Military Retirement not considered national defense

This diagram is very misleading as criminal investigations is squirreled away under Administration. From this breakdown you can't actually tell how much is being spent on each top level category, which defeats the purpose of this type of graph.

<comment><USER9> 1\. The Department of Energy appears in at least three places in the chart: once under "National defense", once under "General science", and once under "Energy". I assume the amount under national defense consists of the allocations for nuclear weapons stockpile stewardship and nuclear reactor production for the U.S. military.

I will agree that it is strange that the Federal Bureau of Investigation has been categorized as "National defense"; the rest of the Department of Justice seems to have been catergorized as "Administration".

2\. One could categorize Medicare with Health, but one can already do that with the chart as shown by considering those two rectangles together. It is interesting to see that Medicare, by itself, has a larger allocation than everything else in the Department of Health and Human Services (and there are pieces in the Health rectangle that probably belong in the Medicare rectangle, e.g. "Grants to states for Medicaid" and "DoD Medicare-eligible retiree health care fund").

3\. The Department of Veterans Affairs is a Cabinet-level department independent from the Department of Defense.

4\. It looks like the Department of Justice and the Department of Treasury have been lumped together under "Administration".

I will agree that some aspects of this breakdown are strange, but I don't think it defeats the purpose of this chart.

<comment><USER8> Excellent feedback, I didn't know that VA wasn't DOD.

Is the chart only about the executive branch? Does it include congressional spending & the federal judiciary?

<comment><USER9> I believe that the budget proposal on which this chart is based is only for the executive branch.

</comment></comment></comment></comment><comment><USER10> I'm going to start a business to sell services to the US government. Even if I only capture 1% of the market that's $36 billion per year!

<comment><USER9> You're hardly the first person with this idea. There are many corporations already working in this sector--they're called government contractors.

<comment><USER11> I think he was making a dot-com (v1) boom joke. Wherein the overly simplistic business model math involving capturing just X tiny percent of a huge market was obvious justification for the profitability of a web venture. Turns out, there's no reason to assume that anyone can capture 1%, 0.1% or even 0.0001% of any market, merely because it's a small value doesn't mean it's easy to achieve.

</comment></comment></comment><comment><USER12> Next book on my reading list is The Wealth of Nations by Adam Smith, so that I can finally understand all of this Economic mess the world is in. Of course it won't give me a full understand, but I have been told it will send me in the right direction.

Recently, I have been trying to understand what exactly a dollar is. From what I can tell, its pretty useless.

I've determined that money != wealth.

<comment><USER13> Have you discovered that if you have something that someone else wants, and they have something you want, and you swap, then you can both end up happier?

Have you discovered that it's tricky to carry with you for exchange purposes every conceivable desire of the guy that owns the grocery store, and thought that it might be handy to introduce an abstraction layer?

</comment></comment><comment><USER14> Related: <http://www.wallstats.com/deathandtaxes/>

I always thought this was a pretty good visual representation of budget distribution. Note this reflects the 2010 budget, not the proposed 2011 budget.

<comment><USER15> It's not that good really. The site you linked to only displays the discretionary budget. The NYTimes infographic is much better, because it also includes all mandatory spending. Mandatory spending is over 50% of the entire federal budget, and includes pretty much all of the 'welfare state': social security, medicare, medicaid, unemployment and other income supplements, veterans' benefits, student loans, and on down the list.

Displaying only discretionary spending is a rather tendentious way of representing the federal budget, because it makes defense spending look like the largest expenditure. Such graphs are usually linked to by people decrying U.S. militarism or imperialism or the military industrial complex. This is not to say that the U.S. does or does not spend too much on the military, but simply to point out that the visual effect of cutting out over 50% of the budget biases people's perceptions and provides undue support for politicized arguments.

</comment></comment><comment><USER16> This is the expenses side. It'd be nice to see an infographic on the revenue side. (Taxes, etc.)

</comment><comment><USER17> Mass transit is being cut, as is R&D, healthcare, and so forth. Joy of joys.

<comment><USER5> Looks like mass transit is actually staying the same at 8.38 billion.

Edit: Looks like there are two different "mass transit" boxes with differing figures.

</comment></comment><comment><USER18> If only we could change "National Defense" by "Startup Fund".

</comment><comment><USER19> What, no giant red box marked "Socialism"? And they're underfunding the death panels!

</comment><comment><USER20> They should cut military spending and focus more on nuclear type weapons, sure they will no longer have the same power in tactical ground warfare but it would be basically saying, sure you may be able to out maneuver us now but we can defiantly blow you off the face of the earth if you try.

<comment><USER20> Guess my message was taken the wrong way? I'm not advocating nuclear weapons but rather scaling back military spending and using them as a deterrent to other countries so that the current US defense force size doesn't need to be maintained.

<comment><USER21> Your message is insane. I don't even know where to start. Why would we prefer to have nuclear weapons as a deterrent instead of having a conventional military as a deterrent? Which is more favorable, defending one of our allies by deploying nuclear weapons, or defending them by deploying a few ten thousand soldiers, planes, and warships? Why would we want to put money into weapons we can never use without tremendous backlash and moral misgivings, instead of ones we deploy regularly to serve our interests?

The current U.S. defense force size _doesn't_ need to be maintained; we haven't used a significant portion of it to defend us or our direct allies from a military attack since World War II. We use our military proactively, as a tool for enacting our will. We cannot hammer a nail if our only tool is a bulldozer.</comment></comment></comment>


MongoDB's lead developer: Foursquare outage post mortem - <AUTHOR> http://groups.google.com/group/mongodb-user/browse_thread/thread/528a94f287e9d77e <comment><USER2> so, in short, a company relying entirely on cloud computing machines for storing its data, which is presumably being billed according to the memory usage of those machines, ran out of memory on them, and suffered a large amount of downtime as a result. mongodb had little to do with the problem, other than maybe it took longer than expected to migrate data to a third server.

i'm baffled at how there could be no monitoring or reporting in place to catch that days or weeks ahead of time, let alone just 12 hours needed according to the mongodb developer, to fix the problem without downtime. it's such a fundamental thing to keep track of for a system designed entirely around storing a bunch of data in the memory of those 2(!) machines. i have more servers than foursquare and none of them do even a small fraction of the amount of processing that theirs do, and yet i have real-time bandwidth, memory, cpu, and other stats being collected, logged, and displayed, as well as nightly jobs that email me various pieces of information. nobody at foursquare ever even logged into those systems and periodically checked the memory usage manually?

worse still, during all of this, the initial outage reports were blaming mongodb or saying the problem was unknown. even at that point nobody at foursquare realized that the servers were just out of memory?

how did the developers come up with 66 gigabytes of ram to use for these instances in the first place? was there some kind of capacity planning to come up with that number or is it just a hard limit of EC2 and the foursquare developers maxed out the configuration?

<comment><USER3> Exactly. How the hell does a fast growing well funded 24 employee startup NOT have load monitoring on their database servers! Pay the 10c the hour for a micro EC2 instance and run Zabbix or one of the half dozen other awesome monitoring packages out there.

<comment><USER4> If it's anything like here, any one engineer's task list is essentially a weighted list of fires to put out and technical debt to pay down, in addition to the features to be delivered next day / week.

Guess more extensive monitoring just got bumped to the top :)

<comment><USER3> If it is anything like any start up I have ever heard of :P

My point is that it is stupidly important to have good monitoring in place and I am surprised that one of the best known startups had a gaping hole in theirs.

</comment></comment></comment></comment><comment><USER5> The "give a crap" factor at 10gen is nothing short of amazing. The company I work at has been using Mongo for a while, and anytime we've had an issue, Eliot has been right there to help us. Mongo is a great product, but like anything, it has its limits. Learning those limits and taking the time to plan your infrastructure is a mandatory part of adopting any technology, and Mongo is no exception.

On a more technical note, it would be nice to have a way to compact indexes online without having to resort to doing so on a slave, but Mongo is a minimum two server product to begin with, so it's not the end of the world. Overall it's a great datastore, and it's only going to get better in the next few years.

<comment><USER6> > Mongo is a minimum two server product to begin with

are you sure? i thought it runs well on one server and even in some sort of run-locally setup.

<comment><USER5> Yes, if you care about your data, it's a two server product. Single server redundancy isn't being added until 1.8. Until then, if the server terminates abnormally, there's potential for data corruption. The solution right now is to run a slave.

<http://www.mongodb.org/display/DOCS/Durability+and+Repair>

<comment><USER7> This solution strikes me as best-case-scenario error. What happens if the connection between my slave and master goes down then my master is unplugged? What happens if both slave and master are unplugged, can both databases be corrupted? What am I missing?

<comment><USER5> If the connection goes away and your master is unplugged, you will need to run a repair on the master and then bring the slave back into sync. If both are unplugged, you will have to run the repair on both. This obviously isn't ideal, and that's probably why single server durability is the biggest priority for the next release.

</comment></comment></comment></comment></comment><comment><USER8> _In essence, although we had moved 5% of the data from shard0 to the new third shard, the data files, in their fragmented state, still needed the same amount of RAM. This can be explained by the fact that Foursquare check-in documents are small (around 300 bytes each), so many of them can fit on a 4KB page. Removing 5% of these just made each page a little more sparse, rather than removing pages altogether._

Interestingly, this is one of the reasons antirez gives as to why redis will not be using the built-in OS paging system, but instead will use one custom- written for redis' needs.

<comment><USER9> I don't think those are the same problem -- could you provide a link?

The problem isn't paging, per se -- the paging system is doing exactly what it should be doing, paging in blocks off of the disk and into memory as they become hot, which for this use case is always.

The problem is that you get fragmentation in your pages. If you allocate three records in a row that are 300 bytes, and then need to rewrite the first one to make it 400 bytes, or delete it altogether, you end up creating a hole there.

The typical strategy for dealing with those holes it to maintain a list of free blocks which can be used and hope that the distribution of incoming allocations neatly fits into unused chunks.

However, as they note, if you have a fully compacted 64 GB active data set and remove 5% of it you just end up with address space that looks like swiss cheese; it's not set up to elegantly shrink, but to recycle space as it grows.

There are a couple of things that I find a bit odd here though: they should have seen this coming before migrating data; it's pretty obvious from the architecture. Second is that they mention the solution being auto-compacting, which wouldn't have actually helped them.

Auto-compaction is in fact useful, but all that it does is, well, compact stuff. It means they would have hit the limits later, but once the threshold was crossed, they'd have the exact same problem. Auto-compaction is either an offline process that runs in the background or a side-effect of smarter allocation algorithms. Both of those things need time once you remove data from an instance to reclaim the holes in the address / memory / disk space ... which is exactly what they did manually.

The only really sane ways to handle something like this are notifications at appropriate levels, or block-aware data removal -- e.g. "give me stuff from the end of the file". I don't know if mongo uses continuation records and stuff like that enough to know how difficult that would be for them.

(Note: Directed Edge's graph database uses a similar IO scheme, so I'm doing some projecting of our architecture onto theirs, but I assume that the problems are very similar.)

<comment><USER8> _I don't think those are the same problem -- could you provide a link?_

You are correct, the actual problem is paging out LRU keys as opposed to memory holes. The issue is related but not the same.

From [http://antirez.com/post/what-is-wrong- with-2006-programming....](http://antirez.com/post/what-is-wrong- with-2006-programming.html)

 _Multiply this for all the keys you have in memory and try visualizing it in your mind: These are a lot of small objects. What happens is simple to explain, every single page of 4k will have a mix of many different values. For a page to be swapped on disk by the OS it requires that all contained objects should belong to rarely used keys. In practical terms the OS will not be able to swap a single page at all even if just 10% of the dataset is used._

<comment><USER9> What antirez is getting at there is actually a much harder (and more interesting) problem that could be generalized as something like "efficient data locality for mixed latency access".

However, assuming that all data must actually be in memory (as stated in the posted email), you don't actually solve the Mongo problem with more efficient organization of the data set, though compacting could be considered a sub- problem of the one that antirez describes.

But as I noted in my earlier comment, compacting wouldn't have actually solved their problems, it just would have delayed them. It's reasonable to ask if all of their data truly needs to be hot, but even there, you'd eventually hit diminishing returns as you approached the threshold where your active set couldn't fit in memory, and there smarter data organization wouldn't actually fix things once you started pulling chunks out for sharding; you'd still need to recompact.

<comment><USER8> I doubt every last bit of their data needs to be hot (i.e. in memory) at any given time, but without specialized paging along the lines of what antirez has discussed for redis, _enough_ of their data probably needs to be hot that, from a paging perspective, all of the vm pages of their data need to be hot.

</comment></comment></comment></comment></comment><comment><USER10> I like the way MongoDB describes the flaws in their own system, but places the blame (four) squarely on the customer, where it belongs.

It wasn't a random failure or a sudden spike that caused the crash -- it was completely predictable growth. Foursquare had already experienced the problem once, and they had solved it. All they needed to do was monitor their growth and iterate that solution.

Sure, Foursquare could have had a better sharding algorithm, but that would only have put off the crash a bit longer. This is a very basic failure -- not monitoring a system that you know is steadily growing.

<comment><USER11> I think they mongodb guys are correct - it is an issue with app architecture and app monitoring failure.

</comment></comment><comment><USER12> The main thing I took from this incident is how much being open and honest about issues improves a company's image. Foursquare and 10gen could have very easily played the blame game, or kept their cards close to their chests, and both would have come off poorly. Instead, they described the problem, owned up to their role in it, and laid out a framework for how to avoid the problem in the future. After reading about the issue, I come away with respect for both groups. Sure, mistakes were made, but they treated their users like adults and took responsibility for what they did wrong.

Good job, guys. If only the likes of Apple followed this example.

<comment><USER13> What are you talking about? How can you see how the users see foursquare after this? Sure, in the hacker community this detail is appreciated, but how do you know how many foursquare users have left because of these outages? How many people are upset with the company, and are now taking facebook locations or the other companies more seriously?

<comment><USER14> Left over this? Foursquare isn't a bank. Foursquare is a game. An achievement system for work lunches.

Of the people I know who use four square the response was "Hm, foursquare isn't working" _puts phone away_.

</comment></comment></comment><comment><USER15> Building sharded systems isn't as simple as throwing consistent hashing into the code and calling it a day. You have to think carefully about what happens when nodes exceed capacity. There is good work in academia on distribution algorithms that gracefully handle reaching capacity (along with data center structure such as rack awareness) [1]. Alternately, if your algorithm doesn't handle a shard reaching capacity you need to have the monitoring and processes in place to ensure you always add more capacity and rebalance before running out.

Also, this validates Redis's VM position about 4KB pages being to large to properly manage data swapping in web application storage.

[1]: <http://www.ssrc.ucsc.edu/Papers/weil-sc06.pdf>

<comment><USER16> I'm wondering if the decision to shard based on users was taken with data (ie, at that point did each user have a roughly similar number of check-ins); if not, hashing by that seems kind of fail for that kind of app.

<comment><USER17> From the email thread, it sounds like the decision to shard on UID was made mostly to increase locality of data, so that you didn't have to query more than one node to get a single user's data.

There's no silver bullet here. Hashing on insertion order would basically guarantee that writes would favor one node over another, which random hashes would force you to aggregate results from all available nodes for each query.

<comment><USER15> This stuff can be very counter intuitive. Locality may not be what you want.

For example, last I heard google's search index was sharded by document rather than by term.

That sounds odd, since if it was sharded by term, then a given search would only need to go to a handful of servers (one for each term) and then the intermediate result combined. But with it sharded by document, every query has to go to all the nodes in each replica/cluster.

It ends up that's not as bad as it seems. Since everything is in ram, they can answer "no matches" on a given node extremely quickly (using bloom filters mostly in on processor cache or the like). They also only send back truly matching results, rather than intermediates that might be discarded later, saving cluster bandwidth. Lastly it means their search processing is independent of the cluster communication, giving them a lot of flexibility to tweak the code without structural changes to their network, etc.

Does that mean everyone doing search should shard the same way? Probably not. You have to design this stuff carefully and mind the details. Using any given data store is not a silver bullet.

<comment><USER18> Sharding a search system by documents gives you several advantages. You can scale horizontally by adding additional indexes for new documents. You can tweak a group of documents more easily. E.g. rank wikipedia higher. Assign better hardware (if necessary), higher priority, etc. Performance is also more uniform. Easier to index content at different frequencies. It's also easier for re-indexing content after tweaking algorithms.

If you shard a search index by term instead, you will end up with duplicate documents stored in each index that contain the same term. And for a large index, you need to scale up your hardware to handle the term or shard by document within that term anyway.

</comment></comment></comment></comment></comment><comment><USER19> Is it acceptable/preferred to store your entire db in RAM? I have little idea about large systems but feel like this may be hard to scale if your db grows to hundreds of TB. I'm intrigued to learn more! Anyone know how fb organizes its massive db storage?

<comment><USER20> > Is it acceptable/preferred to store your entire db in RAM?

This is actually one of the big long term challenges we're going to have to deal with @ foursquare. Right now we calculate whether you should be awarded a badge when you check in by examining your entire checkin history (which means it needs to be in ram so we can load it fast). While this works now, as we continue to grow it will become more and more of a problem so we'll have to switch to another method of calculating how badges are awarded. Several different options here, each with pluses and minuses.

<comment><USER21> Why not keep the algorithm and process in batches? You don't need to have EVERY users checkin history in RAM at any moment. Throw the checkin on a queue, have a few processes that query a db for the checkin history and you're fine. Heck, if you delay the awards it's also a good excuse to throw the user an alert to come back to the site/app.

<comment><USER20> That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.

<comment><USER22> It also depends on your algorithms. Some algorithms are amenable to "running tallies," so a third possible approach would be to store and update various values based only on the aggregate past data plus the incremental data, instead of looking back through the entire history and recomputing when a new piece of data comes in. This of course depends on whether or not this is even theoretically possible with what you're computing.

<comment><USER23> This is probably feasible for some badges, but would be hard to do for all of them.

This approach also increases the complexity of adding new badges, which is undesirable for product and business reasons.

It could certainly help in some case though, and it's something we're considering.

</comment></comment></comment></comment></comment></comment><comment><USER24> _For example, if we had notifications in place to alert us 12 hours earlier that we needed more capacity, we could have added a third shard, migrated data, and then compacted the slaves._

Where did Foursquare find their engineers? I hope no one lost their job here but this is pretty elementary stuff.

<comment><USER20> It's true that this is elementary in and of itself, but looking at things with a bit of a wider lens shows the complexity. We're a small engineering team (10 people) working on a product that is growing extremely fast both in terms of usage and feature set. Meanwhile we're also pretty much constantly re- architecting things to keep up with growth and also doing the immense work of growing the company up from 3 people to 33 and beyond (this has turned out to be WAY HARDER than I would have guessed going in).

Further, there are lots and lots of different things that we need to be monitoring at any different time to make sure that everything is going ok and we aren't about to run into a wall. Automated tools can help a lot with this, but these tools still need to be properly set up and maintained.

I'm not saying we didn't screw up. We had 17 hours of downtime over two days. We screwed up bad, and we feel horrible about it, and are doing a lot to make sure that we don't screw up the same way again.

But it's not because we're morons that never thought about the fact that we should be monitoring memory usage. We just got overwhelmed with the complexity of all that we're doing at once.

-harryh, foursquare eng lead

<comment><USER25> I agree completely about the complexity of monitoring solutions for small teams with fluctuating applications. However, something as simple as htop running on an extra monitor would have alerted you of this issue long before downtime resulted.

<comment><USER26> Awww come on, it's easy to say from an outside perspective. Regardless, the problem was handled well in the end, and we all get the benefit of understanding these limitations better. I think us tech people got the good side (information) out of this ordeal :)

<comment><USER25> I work on a team of 2 where I'm responsible for a handful of servers. I chimed in because I'm in a similar position, I've been looking for a monitoring solution for a while now. Things like nagios and zenoss are over kill, but lack of time has prevented me from finding an ideal solution. That said, I keep htop open and running at all times, and its saved my ass on more than one occasion. I say htop because of the color coding it provides, if things start going red it attracts my attention.

<comment><USER27> Nagios is pretty nice. It's dead easy to write custom monitors and clients are everywhere, there's even a Firefox extension. It requires a bit of learning to get going with but it's not so bad and the pay-off is big.

That said I'm looking at monit too. I hear it's quite nice and has less of a learning curve.

<comment><USER3> I'll put a vote in for Zabbix as a good option, it has saved us more times than I can count.

<comment><USER28> zabbix user here, and it works nicely. But man, it is painful to use.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER29> Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ?

Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM.

Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recently active 14 million users (4 gig / 300 bytes) will still enjoy instantaneous response.

Can someone help me out with the math here?

<comment><USER30> Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory.

Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

<comment><USER29> I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.

<comment> brown9-2 I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally?

harryh's answer above about querying the user's entire history for each checkin answers this question though: <http://news.ycombinator.com/item?id=1769909>

<comment><USER31> IF your system is getting pounded with queries, and you suddenly have to page to serve those queries, the ability of your system to respond to queries and other operations goes WAY down, and the queries are just going to pile up, causing further swapping and load - it's a vicious cycle.

</comment></comment></comment></comment></comment><comment><USER8> It sounds like even if the check-ins _had_ grown evenly across the two shards, that would have only saved them for about another month. Without close monitoring of the growth in memory usage, foursquare was still due for an outage.

<comment><USER20> This is 100% true. We were always planning on moving to more shards, we just thought we had more time to do it than we did.

</comment></comment><comment><USER32> One thing that I admire about Foursquare is that they don't only use the technology (MongoDB, Lift Framework, etc..) but also invest in it. If it wasn't for them I think that Lift wouldn't be such an advanced framework as it is now.

<comment><USER20> Thanks! We do what we can (we're very busy these days!), and only hope to be able to do more of this sort of thing in the future.

</comment></comment><comment><USER33> I find it interesting that with all this talk of scaling, and the specialized tools that go along with it, 4sq's service was essentially running on 2 servers.

Maybe I only need 1 server to be profitable. Hmm...

</comment><comment><USER34> 117 GB of data and 300 B for a checkin means that Foursquare has had 408,944,640 checkins.

<comment><USER35> There are numerous indexes on the data - so actual data is about half of that.

</comment></comment><comment><USER36> While the details are very interesting, there are still many questions to be answered (on both sides):

\- how difficult would be to bring up read-only replicas? (hopefully that should take much less than 11 hours + 6hours) \- why the 3rd shard could accommodate only 5% of the data? \- how can you plan capacity when using the "wrong" sharding? (basically leading to unpredictable distributions)

I have posted the rest of the questions here: [http://nosql.mypopescu.com/post/1265191137/foursquare- mongod...](http://nosql.mypopescu.com/post/1265191137/foursquare-mongodb- outage-post-mortem) as I hope to get some more answers.

<comment><USER20> > how difficult would be to bring up read-only replicas? > (hopefully that > should take much less than 11 hours > 6hours)

Bringing up read only replicas would have been easy, but our appservers are not currently designed to read data from multiple replicas so it wouldn't have helped. We hope to make architectural changes to allow for this sort of thing in the future but aren't there yet.

> \- why the 3rd shard could accommodate only 5% of the data?

The issue wasn't the amount of data the 3rd shard could accomodate, but the rate at which data could be transfered off the 1st (overloaded) shard onto the 3rd shard.

> how can you plan capacity when using the "wrong" > sharding? (basically > leading to unpredictable > distributions)

We weren't really using the "wrong" sharding. And even the uneven distribution we saw (about 60%/40%) wasn't totally horrible. Not 100% understanding your question here.

<comment><USER37> What exactly is the rate like for transferring data across MongoDB instances on EBS? Was the overhead of going across EBS volumes the major factor in the length of the downtime?

The 60%/40% sharding distribution doesn't seem too bad in the big picture; are there any plans to be evening that out?

</comment></comment></comment><comment><USER38> Looks like there needs to be some data engineering to be done to back memory up with disk storage. I think you guys, by now, have enough information about users and checkins to create algorithms that can determine "hot" checkins vs. "cold" checkins as and when they come in. Because not every user is the same, a more user-centric view can definitely help you come up with a more scalable (or graciously failing) system. For example, I don't checkin as much as some of my friends who checkin 10-15 times a day. So, there is no point in giving my checkins as much priority as those busy folks.

</comment><comment><USER29> Can we clear up one issue? Sharding on userid -- bad or fine? I say fine. If your algorithm is "if userid between A and M then server0, else (N-Z) server1", then you could get unbalanced. But if your algorithm is "if random 50/50 hash of userid = 0 then server0, else (random 50/50 hash = 1) server1", then you are going to stay almost exactly balanced (law of large numbers). You would never, ever, get anywhere near a 60/40 imbalance. So the choice to use userid, if implemented correctly, is completely fine.

</comment><comment><USER39> How much can using SSD's for paging help with this? Or to just serve the reads, since they're excellent at random access - surely typical of Foursquare's usage?

</comment><comment><USER40> Why they have only two database servers running (with their database in memory, no less) with 200 million check-ins, is completely beyond me.

<comment><USER41> They are dealing with web scale sharded NoSQL realtime geo scala. Old rules don't apply when 80% of the words describing your company didn't exist two years ago.

<comment><USER42> I honestly cannot tell whether or not that comment was serious.

<comment><USER43> Any comment with the words "web scale" should be considered a joke until proven otherwise. :)

</comment></comment></comment></comment><comment><USER44> This may be a little naive seeing as I haven't used MongoDB or know anything about foursquare architecture, but wouldn't a better result have been just to drop requests that were hitting the disk? Sure some people would have been getting a sub optimal experience but arguably it's a lot better than not having your service up at all.

<comment><USER45> From my understanding, that 'routing' decision would have to take place inside of Mongo -- null routing DDoS packets on a network is one thing, dropping certain database requests on the fly inside a database server sounds a _littttle_ bit tricker.

<comment><USER31> Tunable concurrency control for the number of incoming requests would have been good - before they even hit mongo.

Then you can just dial things down a bit (making customers wait a bit) and let the system recover, and tune things back to the point of optimal behaviour, rather than letting things overload. (You set your concurrency limits at a point where you know the system is just beginning to slow down but is still performing satisfactorily)

</comment></comment></comment><comment><USER46> While I understand this is foremost a monitoring and architecture (mongo and 4sq side...) issue, but I'm curious how dangerous it is to run such IO dependent systems on EC2. Would the prolonged downtime (due to shard migration, etc) been as severe if they were running on hardware? What if these two servers were on SSD RAID 0?

<comment><USER37> The duration of the downtime seems to have been dependent on not just disk throughput, but also network and CPU throughput. Eliot did mention that a large portion of the downtime was partly caused by the slowness of EBS. Making a rough estimate, I think the downtime would probably be about 2/3 of how long it was if it had been on an SSD RAID. However, then you run into the issue of having to maintain your own servers; which, judging by the fact that they are using EC2 and EBS so heavily, Foursquare does not have any desire whatsoever to manage their own infrastructure.

<comment><USER46> It's an interesting angle if that's the reason they are using EC2. Dedicated hosting might be more expensive, but hardware managed is pretty efficient at this point.

Also, given the vast number of EC2 instances they could afford, it seems counter intuitive to be running only two mongrel shards. If you are that stingy about spreading data around, you might as well be using dedicated.

<comment><USER37> It doesn't seem that it was necessarily stinginess of neglect, but rather stinginess imposed by their situation. Like the article said, it took hours to create a new shard, downtime that Foursquare definitely did not want.

</comment></comment></comment></comment><comment><USER47> So some people still use built-in memory allocation for large-scale memory- intense projects like mongodb eh? Pity that, they should have looked at why exactly memcached (and any other sane piece of memory-write-erase intensive application) does slab allocations.

</comment><comment><USER48> I wonder if there ought to be a move to allow for sharded systems to lump groups/pages nicely.. kind of like MySQL's partitioning.. where you won't run into the issue of sparsing out pages when you migrate data.. instead you'd effectively migrate pages.

<comment><USER35> Yes - something that keeps data organized my shard key is being planned.

</comment></comment><comment><USER30> A typical way to deal with uneven distribution in sharding is to do Hash(key) Mod P, where Hash is a cryptographic hash and P is the number of partitions. This way the hash function would randomize the distribution of key.

</comment><comment><USER49> I am not trying to start a debate of sql vs nosql, but i have an honest question. What if foursquare was using a standard RDBMS like Oracle? Would they have run into the same problem?

<comment><USER50> In my experience with things like postgres, yes.

If your working set (the data that you are touching regularly) grows past available RAM, you very rapidly switch from a system where disk access plays no part in most queries, to a system where it is the limiting factor.

Because disks are so much slower, and having queries take only ten times as long can cause them to rapidly queue up, you end up with an unresponsive system. The appropriate response is almost always to turn off features to get the size of your working set back down, which buys you time to either alter the app to need less data, or get more RAM into the box.

</comment></comment><comment><USER51> I don't use NoSQL (yet) but I'm curious if there would be a better key to shard by (instead of user_id)?

</comment><comment><USER52> To fix Google Group's unnecessary use of monospaced fonts so you can read this:

<http://lab.arc90.com/experiments/readability/>

</comment><comment><USER53> for those who don't want to login to see the post:

<http://gist.github.com/616108>

</comment><comment><USER54> That was one of the most well-written, to-the-point explanations I have read in a long time.

</comment><comment><USER55> and out comes mongo's dirty little secret - you have to have enough ram in your boxes to hold not just all the data in ram, but all the indexes too, or it completely shits the bed.

putting hundreds of gigs of ram in a box isn't cheap.

are the foursquare folks considering rewriting with a traditional datastore like postgres and some memcached in front of it?

<comment><USER56> They migrated to MongoDB from Postgres. Specifically because they needed distributed sharding which Postgres didn't have.

Jorge Ortiz, one of the developers at Foursquare said it best on twitter yesterday:

"Baffled by the Mongo haters. If foursquare had stayed on Postgres, we would've had to write our own distributed sharding/balancing/indexing."

"Two problems: 1) Our job is to build foursquare, not a database. 2) These things are hard. Odds are we would have had even more downtime."

<http://twitter.com/#!/jorgeortiz85/status/26563381834> <http://twitter.com/#!/jorgeortiz85/status/26563387808>

Which sums it up incredibly well.

Your comment shows a scary lack of depth - this is the "dirty little secret" of _ANY_ application that needs serious scalable data performance. You need to put your data in RAM where it is quickly accessible.

As soon as you go to disk especially on a low I/O cloud box you are going to be in 'extremely slow' territory. This is why, even before the current NoSQL movement, everyone was using Memcached all over the place. Not because it was a fad but because they needed speed.

SQL Databases have RAM caches as well which give you a pretty similar behavior. And big surprise - if you throw more memory at them - they run faster! Because they cache things in memory! In this case I believe that what Eliot was highlighting was that Foursquare's performance tolerance requirements were such that going to disk was not an optimal situation for them.

There are plenty of applications out there using MongoDB without keeping it all in RAM; I've deployed and maintain several. The more memory you have certainly gives you better performance but it tends to be about a most frequently used cache.

It never "Shit the bed" here to use your sophomoric vernacular. As the post states, it started having to go to the disk after it surpassed the memory threshold which slowed things down... nothing "crashed" as far as the description states. But a read/write queue backlog on any database is likely to exhibit the same behavior.

Your postgres + memcached solution fixes what exactly? They would still need the same slabs of RAM to solve the problem, lest they go to disk on postgres and slow to a crawl the same way they did with MongoDB.

</comment></comment><comment><USER57> Net impact analysis of outage: A bunch of socially awkward technophiles couldn't tell their friends that they were eating lunch.

<comment><USER58> I wonder if burglaries also went down during the outage, since the bad guys couldn't tell if their targets were at home or busy being the Mayor at the local bowling alley.

</comment></comment><comment><USER59> MongoDB is web scale.

<comment><USER59> You guys are lame - you obviously haven't seen: <http://www.xtranormal.com/watch/6995033/>

</comment></comment><comment><USER60> Well Foursquare's Scala/Lift front-end sure has been holding up nicely despite all the FUD about Lift's stateful architecture. Funny that Mongo DB, which gets all the scalability hype, is the first thing to have trouble scaling.

<comment><USER56> The described problem - imbalance in shard allocation - is not one that's specific to MongoDB.

In fact one of the huge choices one has to make when deploying say, Cassandra, is the partitioning algorithm. Sequential partitioning (saying say, users a-l go on Partition 1 and m-z go on #2) gives you certain capabilities for range queries but also risks you overloading if one particular user is significantly larger than others.

Cassandra recommends random partitioning as a way of better balancing your data across shards.

You're going to run into the same problem on just about any sharded setup (With any software) - how do you make sure that you are distributing new chunks/documents/rows in a way that doesn't overload any given server.

<comment><USER61> Or your datastore can take care of doing that for you, c.f. BigTable. This is one of my main gripes with most of the current set of NoSQL offerings — they leave too many decisions in the hands of developers.

Whilst it would definitely be advantageous for all developers to understand the intricacies of various CPU and OS scheduling algorithms, it's not an issue that most developers have to deal with directly. The App Engine datastore, in particular, proves that it is possible to create NoSQL datastores which don't force developers to think about issues like load balancing.

<comment><USER56> Well - in many cases they leave the decisions in the hands of developers because they have made a conscious choice to let the developers decide.

BigTable is more than anything a filesystem - it is NOT a database. It lacks true user configurable indexes, custom sorting, querying etc. These features and the data storage requirements to support them are a very different prospect from what a filesystem (even a distributed one) needs.

</comment></comment></comment></comment><comment><USER62> Right, kick ass. Well, don't want to sound like a dick or nothin', but, ah... it says on your chart that you're fucked up. Ah, you talk like a fag, and your shit's all retarded. What I'd do, is just like... like... you know, like, you know what I mean, like...

<comment><USER62> Geez. No sense of humor. It's a line from Idiocracy. Hilarious.</comment></comment>


For Clarity's Sake, Please Don't Say "Licensed under GNU GPL 2" (2017) - <AUTHOR> https://www.gnu.org/licenses/identify-licenses-clearly.html <comment><USER2> I really do not trust GNU enough to say "or later". There's no reason to think the next GPL's terms will be what I want.

Putting "or later" at the end licenses your software under unknown terms. It could become CC0.

<comment><USER3> Well, yeah; it depends on how closely the FSF's goals align with your own. Personally, mine are very close to the FSF's, so I'm comfortable releasing code under GPL v3 or later. Yours may not be.

It's worth noting however that it's much easier to go from "v3 or later" to "v3 only" than the other way around.

<comment><USER4> > It's worth noting however that it's much easier to go from "v3 or later" to > "v3 only" than the other way around.

Why so? I always wondered why say 'or later' rather than just adopt v4 (et al.) as dual (...) licences as and when, so I can believe it, I just don't understand it.

<comment><USER3> If you don't like the new version of the license, you can start releasing your newer code under v3-only. This doesn't stop someone from forking the old version and releasing it under v4, but for the vast majority of projects this isn't an issue; most are small enough that it's uncommon for forks to overtake the original while the latter is still in development.

</comment></comment></comment></comment><comment><USER5> > As a consequence, code released under GPL version 2 only can't be merged > with code released under GPL version 3 only.

So:

1\. GPL3 introduced an incompatible, breaking change, knowing there was important GPL2 only code.

2\. GPL3 did not include any way to grandfather in existing GPL2 code.

Being idealistic is great, but sometimes it has ugly consequences.

<comment><USER6> You can't grandfather in code under a different license. The recommendation when GPLv2 was published was to include the "or any future version" to allow relicensing within the same series of licenses. A lot of people didnt do that, some for lack of understanding and some fully aware.

</comment></comment><comment><USER7> This is dishonest. They're implying that

"Licensed under GPL 2.0"

Is ambiguous about whether later versions are applicable. They're not. Only GPL 2.0 applies because that's the listed license.

<comment><USER8> The text of the GPL 2.0 states

> Each version is given a distinguishing version number. If the Program > specifies a version number of this License which applies to it and "any > later version", you have the option of following the terms and conditions > either of that version or of any later version published by the Free > Software Foundation. If the Program does not specify a version number of > this License, you may choose any version ever published by the Free Software > Foundation.

The license itself admits that it may cover multiple versions. Being clear, as the FSF is recommending on this page, seems like the best choice to avoid confusion for your end users.

(in particular, this page seems to be about deprecating the existing ambiguous SPDX license indicators and replacing them with ones that have clear 'only' or 'later' suffixes. This seems like a good thing.)

<comment><USER7> >> If the Program specifies a version number of this License which applies to it and "any later version", you have the option...

> The license itself admits that it may cover multiple versions.

... along with the conditions under which this expansive licensing applies. While it's good to be clear, it's not respectful of author's wishes to presume a later license when they didn't explicitly allow that. The article suggests that this case is ambiguous, when it is absolutely not and I find that to be irresponsible

</comment></comment></comment><comment><USER9> I wonder if perhaps RMS is annoyed by the lack of GPL v3 adoption.

<comment><USER10> GPL v3 is toxic. It forces companies to hand out their private certificates. Huge no-no for any hardware manufacturer.

<comment><USER11> > GPL v3 is toxic.

DRM is toxic.

<comment><USER12> Agreed, but no content company will license their shows if anyone can record it in full HD with an app on their TV.

<comment><USER13> > no content company will license their shows

That's their problem. Refusing to sell your product is probably not the best long-term business strategy. If they want revenue, the will eventually sell. The government already distorts the market in their favor by granting them an exclusive _copyright_. Those rights do not include being free from the market forces like supply & demand.

> anyone can record it in full HD with an app on their TV

Why would anybody do that, when its easier to just download a torrent? DRM doesn't stop piracy, because "stopping piracy" has always been the PR-excuse. DRM is an attempt to usurp _property rights_ , with a manufacturer retaining control over your device _after the first sale_.

edit: grammar

<comment><USER12> Companies been very successful at catching and suing people torrenting. Plenty of ISPs/universities block torrent entirely.

If you do it from your TV, its much harder to catch.

</comment></comment></comment></comment></comment></comment><comment><USER14> Url changed from [https://www.fsf.org/blogs/rms/rms-article-for-claritys- sake-...](https://www.fsf.org/blogs/rms/rms-article-for-claritys-sake-please- dont-say-licensed-under-gnu-gpl-2), which points to this.</comment>


Of Course Citizens Should Be Allowed to Kick Robots - <AUTHOR> https://www.wired.com/story/citizens-should-be-allowed-to-kick-robots/ <comment><USER2> >you get into ethical questions very quickly where it's not always so clear what the answer is

Bullshit. The line is very clear. Nobody has the right to initiate the use of violence. You have a right to defend yourself if someone is trying to punch you. You do not have a right to attack someone for being a Nazi, or a Commie, or a BLM-er, or a Trump supporter, or a slut walker.</comment>


Should I work for free? - <AUTHOR> http://www.aphotoeditor.com/wp-content/uploads/2011/01/workforfree.jpg

<comment><USER2> No.</comment>


Training Videos For Web Developers - <AUTHOR> http://learn.lincolnloop.com/

<comment><USER2> Call me crazy, but I wouldn't consider 20-30 minute videos at $18 a piece to be "fairly priced," as the site alleges. There are plenty of subscription services out there which run much cheaper for a greater quantity of high- quality content, with a known reputation behind them.

<comment><USER3> Lincoln Loop has a known and respected reputation in the Django community.

</comment></comment><comment><USER4> The icons on the page (Pricing etc) are misplaced in FF 17 and 18. Slightly too low/in the text.

<comment><USER5> Thanks! We'll look into it.</comment></comment>