Tell HN: Groff needs your help - <AUTHOR> GNU Groff (which is probably doing the behind-the-scenes formatting work when you use man pages on GNU/Linux) currently has no maintainer, and the most recent stable release was in 2013. Who would like to step up and help? <comment><USER2> OpenBSD replaced groff with mandoc[1] which is a much simpler program, once you known that it is just for manpages not a typesetting system.

[1] [http://undeadly.org/cgi?action=article&sid=20110314142734](http://undeadly.org/cgi?action=article&sid=20110314142734)

<comment><AUTHOR> > not a typesetting system.

I _like_ that groff is a typesetting system, it means that I can generate pretty postscript versions of manpages from the command-line and print them off, all in one go. It's really handy that way.

<comment><USER3> You can do that with Mandoc too, although the quality of the PDF output is not as nice. What makes Mandoc less of a typesetting system than Groff is that it supports fewer macro sets (mdoc, man, eqn, and tbl, the macro sets commonly used in manpages); Groff supports more (ms, me, mom…) because it's a more general typesetting system.

</comment></comment></comment><comment><USER4> I'm going to be frank: why? I've recently looked into making my own manpage and it's a pretty old looking system. The docs are not really clear but using some examples and trial and error I got there. My point is though, why does it need a maintainer? The system feels old enough to get deprecated instead of keeping it alive, let alone bring out new releases.

I haven't spent more than a day working with groff though, perhaps many people still use it (every website with info just looked about as old as your average man page), so I may be completely missing the point here. If so, tell me!

<comment><USER5> Replacing man pages because 'they look old' is a terribly stupid idea.

How about we replace them because there is something better? Or because they no longer fill a need?

Things that are old and work are not to be messed with. Frankly, if you don't get that, I don't want you, or anyone else who thinks like you, making decisions about any UNIX I might work with.

I lament the poor documentation on Linux, and OSX, and I lament the FSF's obsession with info pages. There's nothing wrong with man pages. Maybe we don't need groff to prepare them, but we need man pages.

<comment><USER4> > Replacing man pages because 'they look old' is a terribly stupid idea.

I fully agree, and it's not what I meant to say. What I meant is writing new pages in a newer language (insert random lightweight text based format, preferably one where the 'see also' part is linkable) and groff is simply deprecated and supported as well. Since there is currently support, why do we need a maintainer? That is my question.

> Things that are old and work are not to be messed with.

Agreed, so what's the maintainer going to do?

> Maybe we don't need groff to prepare them, but we need man pages.

Once again, I agree. Sorry if I sounded like man pages are unnecessary, that is not what I meant.

<comment><USER3> > What I meant is writing new pages in a newer language (insert random > lightweight text based format, preferably one where the 'see also' part is > linkable)

A little bit of manpage history:

Roff has been around since the beginning of Unix (in fact, the group at Bell Labs who developed Unix got funding by convincing managers they could come up with a good typesetting system). Roff supports a variety of macro sets; for a long time, the most common one for manpages was the "man" macros.

In the early 1990s, BSD came up with the "mdoc" macros, which are a significant improvement over the original "man" macros. mdoc is inherently semantic, and allows easy searching and conversion to other formats, including HTML. You can search for based based on function return type or argument type, program authors, include files and environment variables used, and many more. mdoc pages natively support hyperlinking, including links to other manpages, links within the same manpage, and external hyperlinks.

Mdoc is a very pleasant language, and since it's used in roff you can combine it with other macro sets like tbl (for tables) and eqn (for mathematics). It supports UTF‐8, it can easily be converted to PDF and/or semantic HTML, and provides great searchability. It's well‐documented and widely supported (mdoc pages are supported out of the box on any system using mandoc or groff for manpages, meaning Linux, OpenBSD, FreeBSD, NetBSD, Mac OS X, Illumos, Minix…).

<comment><USER4> Okay, I didn't know all that. Perhaps groff is a better language than I was aware of and there is sure something to say for keeping it available.

But is it really all used? I have never heard of searching man pages by e.g. return type (in section 2 or 3 I assume this would be), nor does hyperlinking work (perhaps due to the pager, but still). If only 1% of the people use it, then either it's up to them to maintain it or we just deprecate it in favor of a new system.

And by the way, the new system doesn't have to be only one language, it can be some generic language that other languages can "compile" to if you have the right packages (just like markdown can be parsed to the current man page language).

<comment><USER3> > But is it really all used? I have never heard of searching man pages by e.g. > return type (in section 2 or 3 I assume this would be)

For example, here's a search for "functions beginning with 'str' and with return type size_t": [http://www.openbsd.org/cgi- bin/man.cgi?query=Ft%3Dsize_t+-a+...](http://www.openbsd.org/cgi- bin/man.cgi?query=Ft%3Dsize_t+-a+Fn~^str&sec=3&apropos=1)

On OpenBSD you can do the same from a terminal:

$ apropos -s 3 Ft=size_t -a Nm~^str

As for hyperlinks, this of course depends on your output format. less(1) in a terminal doesn't do hyperlinks. HTML output will, such as in this page: [http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD- current/man8/...](http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD- current/man8/afterboot.8) And a distribution could, for example, configure man(1) to trigger Lynx (or even Firefox) looking at Mandoc's HTML output.

These toolchains are still being actively developed and improved (semantic search, for example, has only been around for a couple years despite the format theoretically supporting it since the beginning). I try to do my part by contributing manpages to projects that don't have one, converting to mdoc macros when practical, and explaining the great featureset available. The best part is that it is so widely supported—at _worst_ , mdoc falls back to the manpage infrastructure we have now; deployment of a new toolsuite is not a problem compared to converting to some brand new format. At best, it supports all these great new features, and it does so today!

</comment></comment></comment></comment></comment></comment><comment><USER6> I think the obsession with "being maintained" is somewhat unhealthy phenomenon in the FOSS world. If the code works and does what you want then why would it need constant fiddling and consistent stream of releases?

<comment><USER7> The code may need to be adapted to evolving environments. Security bugs might be discovered. In this case I guess "being maintained" just means that someone will react to such events if they happen.

<comment><USER4> Couldn't programmers from major GNU/Linux distributions be signed up for this duty? They compile new OSes and if they want man pages (heck, who _doesn 't_ want man pages?) then it's also up to them to make it work in the new OS. Security bugs can be forwarded to those people by Gnu (assuming they support and host the project).

This may not work for everything, they're not going to maintain any old system, but things that we _want_ to keep around but for which there are not necessarily dedicated maintainers to keep it up...

</comment></comment></comment><comment><USER8> it doesn't look abandoned: [http://git.savannah.gnu.org/cgit/groff.git](http://git.savannah.gnu.org/cgit/groff.git)

</comment><comment><USER9> The Groff Mission Statement looks interesting.

[http://www.gnu.org/software/groff/groff-mission- statement.ht...](http://www.gnu.org/software/groff/groff-mission- statement.html)

<comment><USER10> The question is - do we really need the typesetting part? I had a troff phase (bought some books) ten years ago. But the community of users is small, its capabilities are eclipsed by TeX, and the language is quite arcane.

One could even go a step further and argue that writing man pages would be much easier (and presumably more man pages would be written) if the markup language was switched to e.g. Markdown by default. For my own software, I am already using Pandoc to generate nroff input from Markdown.

<comment><AUTHOR> > do we really need the typesetting part?

I think so; producing high-quality printed manpages is useful, at least to me.

> the community of users is small

So tell your friends to use groff! :-D

> its capabilities are eclipsed by TeX

In what way? Yes, TeX can do complex mathematics better, but most documents that people want to publish don't terribly need complex math _that_ much.

> and the language is quite arcane.

What specifically is 'arcane' about it, versus something like TeX or markdown?

> One could...argue that writing man pages would be much easier...if the > markup language was switched to e.g. Markdown by default.

But then someone would have to rewrite the existing corpus of manpages, and as others have pointed out, using markdown loses some semantics.

I think keeping that functionality in pandoc is the right way to go.

</comment></comment></comment><comment><USER11> I am the maintainer of a man page that turns into a 260+ page PDF document in letter size.

I haven't hacked on groff, but I recently I did a whole bunch of work on the man2html program from the man tools. That code is extremely hacky, like you wouldn't believe!

[http://www.kylheku.com/cgit/man/](http://www.kylheku.com/cgit/man/)

I have it so that a man page can detect whether it's being compiled by groff or by man2html and re-target some of its macros.

That aforementioned large man page is here; the macros are upfront: [http://www.kylheku.com/cgit/txr/tree/txr.1](http://www.kylheku.com/cgit/txr/tree/txr.1)

HTML and PDF here: [http://sourceforge.net/projects/txr/files/txr-104/](http://sourceforge.net/projects/txr/files/txr-104/)

(The index and hyperlinks in the HTML are due to a post-processing pass, implemented in the "genman.txr" script.)

<comment><USER3> Mandoc, which is what the BSDs and some other systems use for formatting manpages, has very good HTML output. In fact, the program used to be named "mdocml" because it was written to be a mdoc‐to‐HTML converter.

Unfortunately the -man macros (as opposed to the modern -mdoc macros) aren't that good for conversion to other formats like HTML, because they're by nature presentation‐focused. All major troff implementations support the -mdoc macros, though, and -mdoc is much better suited. It's what I write all my manpages in these days (and it's a drop‐in replacement—replace foo.1 written in -man with foo.1 written in -mdoc and groff, man, etc will handle it instantly). I also like to convert manpages from -man to -mdoc, or write new pages for programs that don't have one. It gets a little exhausting to convert long pages like the one you linked, though.

Here's some documentation on the format of -mdoc pages: [http://mdocml.bsd.lv/man/mdoc.7.html](http://mdocml.bsd.lv/man/mdoc.7.html)

<comment><USER11> I wonder how good is mandoc's handling of the troff language as such.

The long page that I wrote, though it is based on the old -man macros, is actually to a large extent based on its own macros which are retargettable.

As I started to polish the document for better PDF output, I needed to reach into more of the power of groff, while maintaining compatibility with man2html. That's when I started hacking on man2html to handle more of the troff language. I found that loops didn't work very well and there were issues with nested if/else and such.

</comment></comment></comment><comment><USER12> Does it currently have any important bugs?

Because the troff format was designed with the limitations on early 70's computers in mind and seem to be used exclusively for man pages. Why not upgrade to something that better correspond to how we actually code now (ie markdown)?

<comment><USER13> If nobody is maintaining it, how can you tell it doesn't currently have any important bugs?

Remember that the worst bugs in the last couple of years have been in the code that nobody was looking at.

<comment><USER14> Some people must have reported bugs since 2013. If not, then either it's unused or bug free?

<comment><USER15> For one example: [https://bugs.debian.org/cgi- bin/pkgreport.cgi?dist=unstable;...](https://bugs.debian.org/cgi- bin/pkgreport.cgi?dist=unstable;package=groff)

I haven't compared this with upstream though.

</comment></comment></comment></comment><comment><USER16> GNU Groff is terrible and the only reason it hasn't been replaced with a superior implementation is that all the better versions are politically incompatible with GNU. It's no surprise they're having a hard time finding someone interested in effectively code-laundering and slapping a GPL on it.

<comment><USER17> Really? What are these better versions?

<comment><USER3> There are a couple of other troff implementations in use today.

Mandoc is very new and focuses on providing a complete solution for a system that uses manpages: it renders manpages in the terminal or to HTML on the fly, provides a database for semantic search of manuals, and is probably the second most common manpage renderer in common use (after Groff). It's the default renderer on OpenBSD, FreeBSD, DragonFly BSD, NetBSD, Illumos, and Minix. Unlike the others, it's not a complete troff implementation; it focuses on manpages ("mdoc", "man", "eqn", and "tbl" macro sets).

Heirloom troff is descended from Sun's troff, and focuses on nice typesetting supporting various PDF and OpenType features.

Plan 9 troff is, well, Plan 9's troff, and is used by plan9port to render its own manpages.

There are a few other troff implementations (Neatroff, etc.), but these are the most widely distributed ones these days. I personally use Mandoc for everything except PDF output, for which I use Groff.

<comment><AUTHOR> I've looked at the kind of PDFs heirloom troff can make, on an iPad, and my eyes, just, wow. There aren't words. If Debian would put heirloom troff in its base-install (along with, say, E.B Garamond or Junicode) instead of groff, there'd probably be more people (1) making beautiful man pages, and (2) viewing man pages graphically (with evince or another PDF viewer) than ever before. It's not like there's a licensing problem, AFAICT.

I have not seen a PDF made by mandoc that made me react that way. :-|

<comment><USER3> Yes, PDF output is one place where mandoc is not as good as groff/heirloom yet. (The other major one being support for generic preprocessors and other macro sets.) Mandoc works very well for terminal output, HTML output, and semantic searching though.

</comment></comment></comment></comment></comment><comment><USER13> In which the fundamental problem with not paying for your open source software is revealed.

<comment><USER18> This is a fundamental problem with open source software.

<comment><USER13> I'd limit it to FOSS that people don't pay for (directly or indirectly).

When money is changing hands with the expectation that software is maintained in good working order, you won't see abandonment like this. But when nobody is paying or getting paid, maintenance stops, regardless of whether the software is a key component of crypto on the internet or a fundamental part of the documentation of most Linux installations.

<comment><USER19> oh you mean things like windows XP which people have paid for? I rest my case.

<comment><USER20> You're talking about the same Windows XP which people are currently paying for and receiving support for?

[http://www.theguardian.com/technology/2014/apr/07/uk- governm...](http://www.theguardian.com/technology/2014/apr/07/uk-government- microsoft-windows-xp-public-sector)</comment></comment></comment></comment></comment>


Features that Google Chrome should steal from other browsers - <AUTHOR> http://www.extremetech.com/computing/138675-features-that-google-chrome-should-steal-from-other-browsers <comment><USER2> Something I've wanted from Chrome for a long, long time is default download locations for certain filetypes. I mean, why would I ever want .mp3s in my last selected folder if that was full of .doc files? <1st world problem> I have to spend up to 30 seconds to navigate my hard drive, ugh. </fwp> I heard Opera has that feature, but I like Chrome too much to switch.</comment>


Amazon will stop sales of used first-party Nintendo products starting November 1 - <AUTHOR> https://arstechnica.com/gaming/2019/10/amazon-no-more-third-party-sales-of-nintendo-products-without-approval/ <comment><USER2> The sentiment here seems to be fairly negative, which is surprising, because Amazon is doing exactly what we've been asking them to do for ages -- policing their marketplace against counterfeit products.

They're starting with the easiest things to police, but at least they are making an effort. It really sucks for the legit third party sellers, so hopefully they make the approval process easy for them so they can continue to sell their legit hardware.

<comment><USER3> What people were asking was "make it obvious who I'm buying from (i.e. no commingling), and punish sellers of counterfeits". Just punishing sellers of counterfeits goods might have been enough (reportedly Amazon does have item level tracking despite commingling).

What Amazon is doing sounds more like stopping to be an open marketplace, depending on how hard it is to get approved. That's a solution, but at that point they are just another Target or Wal-Mart.

<comment><USER4> As an Amazon seller, I can confirm that they have tracking linked to each seller. Their system does this via their FNSKU (fullfillment network stock keeping unit) and that is unique to each seller for each product. So if I want to sell a product, that product has an ASIN that defines it specifically and then any seller in the Amazon system that wants to sell that product needs their own FNSKU that applies only to that ASIN. Those FNSKUs are tagged on the items. If you've ever bought third party stuff on Amazon, the numbers are also 10 digits long just like ASINs but they usually begin with an X. FNSKUs can also match the ASIN and that is usually given to the seller that creates the listing so that seller has FNSKU=ASIN. So if you buy private label stuff that is their own branded stuff that no one else sells, it just looks like the ASIN and you probably ignored it. But if you buy a big brand name product like Nintendo, Star Wars, or anything with multiple sellers that sell via FBA/prime, you'll probably see the FNSKU beginning with X.

EDIT - in case it wasn't obvious, Amazon uses the FNSKU barcode for fullfilling their orders. They might have 10 sellers with the same ASIN in that specific warehouse, so the Amazon employees fullfill via FNSKU in those cases, which is why they would require the FBA/prime items in their warehouse to be tagged with the FNSKU to identify which seller owns that exact item vs all the others that also have stock in that warehouse and perhaps even in that exact same shelf/bin.

<comment><USER5> Apologies if I'm misunderstanding, but are you implying that amazon doesn't commingle products? Allegedly this is the reason that you can order a "sold by amazon" product but still receive a counterfeit item (likely provides by a 3rd party seller).

<comment><USER3> No, when they get the product shipped in they stick a seller-specific label on it, then commingle it. When I order an Xbox an employee goes to the Xbox bin, picks any of them and scans it out of the system. Amazon now knows which one I got and where it came from, because the barcode identifies it as originating from seller X. If I complain to Amazon or send it back they can now look up what they shipped me and see if some seller consistently ships them bad product.

In essence commingling prevents you from efficiently grabbing something from a specific seller, but it doesn't prevent you from knowing which one you grabbed.

<comment><USER6> Is this a new change? Because previously, you could label your own commingled products and the label was simply the UPC code.

<comment><USER7> I think all FBA items have had an FNSKU for at least 7 years.

<comment><USER8> But optional, right?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> >. Included in this move are DVDs and Apple products, which now require similar approval processes to move forward.

This is incorrect. You can get approval in DVDs pretty easily if you have invoices from a distributor. It's effectively impossible to get approval for Apple unless you're one of a dozen or so authorized sellers that spends 8-9 figures with Apple yearly. (The official requirements to get approval for refurbished Apple on Amazon is 2.5 million in invoices over 90 days).

It's not clear how the approval process for Nintendo will work, but if this is like most brands, all you'll need is invoices from a distributor. Many brands have this kind of restriction. It's nothing at all like the Apple deal, for which you can't get approval just with invoices, and for which there was a special announcement from Amazon; the Nintendo email isn't special, it's standard (google phrases from it and you'll find many other brands with similar emails from Amazon). E.g. [https://sellercentral.amazon.com/forums/t/marvel- warning/407...](https://sellercentral.amazon.com/forums/t/marvel- warning/407578), [https://sellercentral.amazon.com/forums/t/dc-comic- restricti...](https://sellercentral.amazon.com/forums/t/dc-comic- restriction/329913)

Also, the email makes no mention of used vs new. All Nintendo are now restricted regardless of condition.

Also, this is seller-specific. Many sellers can still sell; most likely Amazon targeted this towards the sellers their algo determined were more suspicious for whatever reason. See e.g. [https://www.amazon.com/gp/offer- listing/B07VGRJDFY/ref=olp_f...](https://www.amazon.com/gp/offer- listing/B07VGRJDFY/ref=olp_f_used?ie=UTF8&f_usedAcceptable=true&f_usedGood=true&f_used=true&f_usedLikeNew=true&f_usedVeryGood=true)

<comment><USER10> I didn't find it "easy" to get approved for DVDs. I had over 20k a month in used media sales for over 12 months and still got rejected. The worst part is that you can buy the media, ask for approval, and then get stuck with the inventory. On top of that what you buy from a distributor will never make you any money. It's all in the used items so you hope to make it up by selling DVDs you buy for a dollar at library sales, etc.

<comment><USER11> Have you tried eBay?

<comment><USER10> It just doesn't have the ease of FBA. We were primarily used book sellers and occasional video games. Unfortunately, while I was incarcerated my partner blew up the business.

</comment></comment></comment></comment><comment><USER12> There is a problem with pirated Nintendo games showing up in the unpoliced Amazon third party seller marketplace. I don't know how big the problem is, but it's there (as I can personally attest, and as the article describes). Unlike the source cited the article, the bootleg I purchased for my son didn't "play fine."

<comment><USER13> It isn't just pirated games/getting fake carts, I've ordered things like a "genuine" GBA, GBC, and gotten fully counterfeit hardware instead of a used old device. Nearly a 100% counterfeit rate on certain lines of product (one example is [https://www.youtube.com/watch?v=7W39bZDu6zA](https://www.youtube.com/watch?v=7W39bZDu6zA))

<comment><USER14> And out of region stuff. I ordered a 3ds game brand new, I wanted to try to get a club code so I ordered one that looked like it would be new old stock, ended up getting a 3ds game intended for sale in Saudi Arabia.

Returned it and just bought it locally instead. Amazon is pretty much a wasteland for collectors. Ebay is far better and more reliable overall.

<comment><USER15> > _Returned it and just bought it locally instead. Amazon is pretty much a > wasteland for collectors. Ebay is far better and more reliable overall._

And also, somehow, much lower-priced for the most part.

For out-of-print but popular and relatively recent toys and books, I've often found things for half the price or less on eBay than what Amazon is pushing. Now that Buy It Now is common and I don't have to mess with the auction system, it's my first choice for stuff like that.

</comment></comment></comment></comment><comment><USER16> How is this going to play out, legally? First-sale doctrine has been rather firmly upheld by the courts: [https://en.wikipedia.org/wiki/Kirtsaeng_v._John_Wiley_%26_So...](https://en.wikipedia.org/wiki/Kirtsaeng_v._John_Wiley_%26_Sons,_Inc).

<comment><USER17> I don't see what first-sale doctrine has to do with this: Amazon is a private (publicly-held) company and is free to disallow certain products from being sold on its site.

First-sale doctrine means you're free to take your used Nintendo cartridges and set up a table on the sidewalk (subject to local laws about street sellers of course) and sell them to passers-by. Amazon doesn't have to provide you a venue for selling them.

Personally, I don't understand why anyone would want to sell older Nintendo gear on Amazon anyway. This is exactly what eBay is for. eBay's fees are probably a little lower too.

<comment><USER18> > Personally, I don't understand why anyone would want to sell older Nintendo > gear on Amazon anyway. This is exactly what eBay is for.

Because the customers are on Amazon, not eBay.

<comment><USER19> FYI buyers on are eBay too, esp. for vintage games.

<comment><USER18> I am confident that the reason I gave is an accurate answer to the question I gave it in response to, for at least a very large share of sellers. Note that you can do something for a given reason regardless of whether that reason is true.

However, the fact that it's possible to run a business doing nothing but selling things on Amazon that you buy on eBay when an order comes in strongly suggests that the customers really are on Amazon and not on eBay.

<comment><USER20> Interesting. I've long suspected the dropship arbitrageurs had existed on Amazon for a while, but I thought they had long since become unprofitable, or at least this Amazon to eBay channel had. Could you go into more detail about this? I'm sorta curious about how big the market is.

<comment><USER19> It's fairly common. I sell a lot on eBay and get maybe an order a week with a message of "do not include invoice, item is a gift" which is dumb since we all know they'r drop shipping. I love it, never have a problem with those buyers. But I do include thank you cards in them with my eBay stores.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> > This decision seems to imply that either Nintendo or Amazon would rather > limit all used sales of Nintendo products than build a more focused list of > scrutinized software and products.

Seems like probably both, no?

> Users on the used-Nintendo forum thread have postulated that counterfeit > retro Nintendo cartridges have been an open secret among Amazon sellers for > some time.

I'm guessing Nintendo isn't too happy about this (in general they seem very concerned about their brand) and Amazon is likely happy to stem some of the negative backlash about counterfeit goods on their platform.

I'm not surprised to see that Apple products are likely to fall under this umbrella - another company very concerned about their brand.

<comment><USER22> I wonder if Apple's special deal is related to this lawsuit

[https://www.theverge.com/2016/10/20/13343682/fake-apple- char...](https://www.theverge.com/2016/10/20/13343682/fake-apple-chargers- amazon-lawsuit)

(note: they sued the 3rd party sellers, not Amazon)

<comment><USER9> Nope. Plenty of brands have sued sellers, but few get that kind of treatment. Otterbox, Skullcandy, Standard Process, Popsocket, ecobee, The Noco Company, Prevagen, Rainbow Light have all sued Amazon sellers.

</comment></comment></comment><comment><USER23> Amazon now says the email sent to the third-party dealers was in error

[https://www.pcmag.com/news/371708/amazon-dealers-need- approv...](https://www.pcmag.com/news/371708/amazon-dealers-need-approval-to- sell-used-nintendo-games)

</comment><comment><USER24> This is the sort of chilling effect that increased corporatization of society can have - but doesn't necessarily have to. Forbidding the sale of legal items, is a form of increased censorship, one society seems to be okay with.

<comment><USER25> This type of rhetoric is very frustrating to me. It seems the word censorship has lost all meaning.

<comment><USER26> It is a tactic. This conflation allows easier confusion of corporate interests and societal ones. And by rendering it so meaningless, it makes identifying and protesting against actual censorship harder and recognizing "we're mad they won't platform our garbage" also harder.

At the moment it is still intentional and usually malicious. The danger comes when it is inculcated and we lose the real understanding entirely.

</comment></comment></comment><comment><USER27> Alternate title: Amazon destroys the livelihood of thousands of merchants without process or appeal.

<comment><USER28> I see this time and time again here on HN. If your livelihood is dependent on another persons platform you are in a very dangerous position. At any moment the terms of use could be changed or they could decide they want to sell the same business you are selling and you are out of luck. In this instance this is a crappy deal and I don't like it and people will buy from other markets.

<comment><USER9> That's what happens when Amazon has a monopoly. Many customers will only look on Amazon and won't buy anywhere else.

<comment><USER29> Back in 2003, people were saying the same thing about walmart. This article talks about how the demand for 'lower prices' almost drove a pickle company out of business:

[https://www.fastcompany.com/47593/wal-mart-you-dont- know](https://www.fastcompany.com/47593/wal-mart-you-dont-know)

<comment><USER9> Yes, Amazon is doing a lot of the same things that Walmart did back in the day.

</comment></comment></comment></comment></comment><comment><USER30> alternatively headline: Amazon takes drastic measures to stop sale of counterfeit Nintendo products

</comment><comment><USER31> Can the headline be fixed? It's currently stating Amazon will stop the sale of Nintendo products. Absolutely not at all what the actual headline is saying.

> Amazon: No more third-party sales of "Nintendo products" without approval

</comment><comment><USER32> It seems like this is pretty obviously what Nintendo would prefer, and can probably put it in their terms of service. It would shock me if this weren't being driven by Nintendo, much more so than Amazon.

<comment><USER33> This makes sense. Another issue with counterfeits is that many consumers are unable to recognize them. Instead they'll think it's just a defective product and waste Nintendo customer service's time.

</comment></comment><comment><USER34> Actual headline is:

Amazon: No more third-party sales of "Nintendo products" without approval

</comment><comment><USER35> What I don't understand is why would Nintendo/Amazon care if people sold things Nintendo were no longer manufacturing? It doesn't seem logical to prevent counterfeiting by preventing all items from being sold on the platform. I'm talking about out of print products only. So much for the "everything store"...

<comment><USER9> It takes effort to segregate products like that, and Amazon doesn't want to put in that effort. For most brands the approval decision is across the entire brand, not individual items. On a handful of items they do have item specific approvals but those are more expensive for Amazon to enforce.

<comment><USER35> How hard is it to request a list from Nintendo on what products they are currently manufacturing? They can create a policy were they only gate products those companies are currently selling. We are talking about millions of dollars for this brand only. If this continues to happen across the platform it can amount to billions of dollars of lost sales. Also Amazon would much rather take a cut from a third party seller (regardless if that seller makes money or not) versus sell it themselves.

</comment></comment></comment><comment><USER9> [https://www.theverge.com/2019/11/1/20943772/amazon- marketpla...](https://www.theverge.com/2019/11/1/20943772/amazon-marketplace- nintendo-deal-used-retro-games-consoles-counterfeit)

Turns out it was all a mistake.

</comment><comment><USER36> Amazon did this for canon equipment some time back. I used to sell my older cameras and lenses on Amazon as used items; I'm not allowed to any more. It's too bad, but it just pushed me back to eBay, which has been perfectly fine, for the limited number of sales I've done.

</comment><comment><USER37> IMO this is ok. Ebay has long been a better platform for these kinds of transactions.

</comment><comment><USER38> I misread the headline and thought that they were no longer selling first- party nintendo products and that it was a horrible decision.

No, this is a superb decision and I wish that they would do it with other products.

</comment><comment><USER39> Ebay is still great for used goods, most rare and lucky finds seem to come from random listings made by normal people. Amazon never seemed friendly for small first time sellers.

</comment><comment><USER40> Why is the headline changed from "third-party" to "first-party"? It fundamentally changes the story...

</comment><comment><USER41> Amazon needs to do this for everything. The quality control problem with Amazon is out of control.

</comment><comment><USER42> Ebay baby... I'm coming home...

</comment><comment><USER43> I've noticed I can't buy many Kirkland brand products on Amazon either.

</comment><comment><USER44> Nice. One step towards the right direction.

</comment><comment><USER45> What about amazon warehouse?

<comment><USER46> Warehouse Deals is just returned / damaged products. If someone returns a replica game cart to Amazon, it will likely end up being resold as authentic.

Unfortunately there's no real control here, but it's the same as any physical store. Cashiers are not counterfeiting specialists and scammers are crafty. Some will even shrink wrap and return an empty case. Ideally you would be able to track issues with the sale of returned products back to a single "customer", but not before several people are forced to return fake/nonexistent products.

</comment></comment><comment><USER47> We changed the URL from [https://reddit.com/r/gaming/comments/dprlaw/amazon_will_stop...](https://reddit.com/r/gaming/comments/dprlaw/amazon_will_stop_allowing_the_sale_of_used/) to an article which adds a bit more background.</comment>


Tell Your Lawmakers: "Anti-Counterfeiting" Treaty Is a Sham - <AUTHOR> https://secure.eff.org/site/Advocacy?cmd=display&page=UserAction&id=451 <comment><USER2> Do form letters like this actually work? I see these types of things all the time, usually in a "Corporation X wants our lawmakers to pass Bill Y, and it's bad for people!" context.

Doesn't the EFF have lobbyists? I think history has shown enough times that lobby money is far more powerful than anything that's good for the citizens.

<comment><USER3> Form letters are, essentially, "half-read once and then weighed weekly to produce a report for $POLITICIAN", according to my brother, who used to be in constituent services.

Themes which come up frequently in non-form phone calls and letters, on the other hand, have a surprising amount of impact. The Republicans, for example, got a _huge_ amount of negative feedback regarding immigration policy during the last election cycle, and more than one politician mentioned the depth and intensity of that (not always in a positive way) in signaling changes in their stance on it.

P.S. If you ever have a severe problem with the government and don't know where to turn, document _everything_ , then call your Congressman's office.

</comment></comment><comment><USER4> I will do that when the EFF stops suing people for violating the GNU and requiring them to:

1) open up all their source 2) hire a "compliance" officer to make sure they aren't violating the GNU license in the future.

More info about this can be found here: <http://gpl-violations.org/>

With all their talk about freedom, it sure sounds like they are just as restricting as the RIAA/MPAA

<comment><USER5> GNU is an operating system. I believe that you're reaching for the GPL. Also, the EFF is not suing people; gpl-violations.org is. To the best of my knowledge (and Wikipedia's), they aren't associated. Perhaps you could explain the connection between the two?

<comment><USER4> "GNU is an operating system."

It's pretty easy to confuse the two when you slap GNU in front of everything (including operating systems and licenses). Stallman even tried rename Linux a few years back to: GNU/linux

<http://www.gnu.org/licenses/gpl.html>

"the EFF is not suing people; gpl-violations.org"

you're right, I meant the FSF, which convinces many people to assign them the copyrights of their apps and then sues violators in court.

<http://www.gnu.org/licenses/why-assign.html></comment></comment></comment>


The Most Popular Keys of All Music on Spotify (2015) - <AUTHOR> https://insights.spotify.com/us/2015/05/06/most-popular-keys-on-spotify/ <comment><AUTHOR> TLDR: G Major because it's convinient to play both for Keyboard & Guitar.

Sorry if it's old news for you. Not sure about the rules of the forum.

<comment><USER2> And the pattern continues. C, G, D and A major are the easiest to play in because their key signature has respectively 0, 1, 2 and 3 sharps.

Further, C# and F major are probably right behind those four because most people playing the guitar will just use a capo and play in C and E major respectively.</comment></comment>


Ask HN: Do you use Wolfram Alpha? - <AUTHOR>

I'm curious about the usage of Wolfram Alpha. How frequently do you search with it?
I think I'll use Wolfram Alfa
a) once a month b) once a week c) dayly d) hourly!!
[have you found a killer application of the engine???]
(also have you found a use to it?) <comment><AUTHOR> I think they have some interesting concepts implemented in their engine, but without users it will be difficult to maintain it running. Thats why I was asking. I also ask here because the marketing campaign hit us plenty in HN.

by the way I use it at most once a week, generally when Im reading news.

</comment><comment><USER2> I tried it and was disappointed. For example search Wolfram Alpha for "Abraham Lincon's Height" and it won't be able to figure it out. Search Google, and it will be parsed out nicely at the top of the page, no wading through search results necessary.

<comment><USER3> Same here. Based on the demo video, I thought a natural language query about some oft repeated numbers - 'Web browser market share by year' would be doable. It wasn't.

</comment></comment><comment><USER4> Good question.

<http://news.ycombinator.com/newpoll>

(Dunno if it still has a karma threshold)

<comment><USER5> It's 20 karma to create a poll, you can see when your logged out and try to create one.

And I never use wolfram alpha, maybe when I start classes again.

</comment></comment><comment><USER6> I probably use it daily or just about daily. Usually to do some basic physics calculations, just for fun. I also use it to check my answers to homework.

</comment><comment><USER7> e) never</comment>


How Much Traffic is Too Much Traffic For CloudFlare? - <AUTHOR> http://phoboslab.org/log/2013/02/how-much-traffic-is-too-much-traffic-for-cloudflare <comment><USER3> This was flagged to my attention and I've reviewed all the interactions between the author and our team. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection.

The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network. While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation.

To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.

Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)

<comment><USER4> You keep calling it an attack. This was normal (growing) traffic for the site.

From the comments on the site: "I didn't notice any "attack" when CloudFlare began to route all traffic directly to us. It looked like normal web traffic - much of it, but no more than usual."

<comment><USER5> He said; she said. A Layer 7 attack is not necessarily something one might "notice." The very nature of such an attack is "normal" looking. I think it's impossible for us to say who is right -- OP or CloudFlare -- without substantial hard-data. Ultimately, it's not the basis of the article, and the OP is looking for a different service -- unmanaged, limited-downlink bandwidth -- than what CloudFlare is looking to provide -- managed, edge network.

<comment><USER6> Irregardless if it's an attack or not, CloudFlare's personell did not handle this well.

Yes, I know they did offer one hell of a starter/"sweet lolipop to sucker you in" pack - but that's still not what's being discussed.

It has been re-iterated many times in this thread - but CloudFlare had a sane person on the other end that was willing to open his wallet - that's something one should act on quickly.

</comment></comment></comment></comment><comment><USER7> On the one hand, I think it's insane that you hit 100TB/mo of usage on a "free" service and expected it to keep going. Maybe you figured you wouldn't rock the boat as long as it was free - I guess I can relate. But still, really???

On the other hand: CloudFlare comes off as terrifyingly incompetent here - which more or less matches with my experiences trying them out on a site serving maybe 100GB worth of traffic at most in a month. They seem to have missed dozens of great opportunities to upsell you on their paid service, and when someone finally noticed how much bandwidth you were using, they completely lost the plot.

What should have happened, IMO, is something like this: A panicked CloudFlare admin realizes your site is using 100TB/mo. Their first step is to send you a sternly worded email, explaining that for this usage level you need plan X, and if you don't upgrade within... let's say 5, maybe 7 days? They'll be throttling or limiting your service. Then you don't feel pressured to solve the problem _right away_ , and they are trying to retain a customer that (presumably) they value.

Instead, they haphazardly change your settings behind your back (???) and then later take various steps to reduce your bandwidth usage before finally deciding that you need to pay them, without even figuring out how much money they want. Ridiculous.

To me this just says that CloudFlare is running on what may be a fundamentally unsound business model, and that by claiming their free tier doesn't charge by bandwidth (and not listing any limits) they're dramatically increasing the odds that customers will suddenly discover there are limits after all, and leave. If they were more up front about what the actual pricing structure is, it'd probably be more likely that people would start paying for the value that CloudFlare gives them.

<comment><USER8> The author stated a couple of times that they would happily pay for the service. The problem is it was free until suddenly it was $3000/month and it took two weeks with an outage in service for that duration to learn that fact.

<comment><USER9> Most reasonable people can be made ok with most things given adequate communication. Expectation management is one of the most important aspects to human relationships. As a business, what was a sales opportunity became a PR issue.

</comment></comment></comment><comment><USER10> I had quite a similar experience where our site (approx 10TB / month) was taken "off cloudflare" and all traffic routed direct to our servers at peak load (we were on the Pro plan @ $10/month after many months of free). They cited network issues in the control panel and a "Layer 7 attack" in a support ticket. We quickly upgraded to a Business plan ($200 / month) and traffic was back through cloudflare within 10 minutes.

Also, similarly to OP, we are regularly (fortnightly) automatically put in "I'm Under Attack" mode without any prior warning or consent which is quite annoying as it tends to happen overnight so I am not alerted until someone checks the live site in the morning (it still returns a 200 so current checks don't pick it up)

<comment><USER11> Sounds like the $200/mo is really worth it...?

<comment><USER10> We use it mainly for the CDN, we have estimated a similar service using CloudFront would cost over $1500 / month (750% more - I acknowledge AWS is are not the cheapest)

</comment></comment></comment><comment><USER12> The TOS seems a bit too illusory:

"SECTION 10: LIMITATION ON NON-HTML CACHING

You acknowledge that CloudFlare's Service is offered as a platform to cache and serve web pages and websites and is not offered for other purposes, such as remote storage. Accordingly, you understand and agree to use the Service solely for the purpose of hosting and serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. CloudFlare's Service is also a shared web caching service, which means a number of customers' websites are cached from the same server. To ensure that CloudFlare's Service is reliable and available for the greatest number of users, a customer's usage cannot adversely affect the performance of other customers' sites. Additionally, the purpose of CloudFlare's Service is to proxy web content, not store data. Using an account _primarily_ as an online storage space, including the storage or caching of a _disproportionate_ percentage of pictures, movies, audio files, or other non-HTML content, is prohibited. You further agree that if, _at CloudFlare's sole discretion_, you are _deemed_ to have violated this section, or if CloudFlare, _in its sole discretion_, deems it necessary due to excessive burden or potential adverse impact on CloudFlare's systems, potential adverse impact on other users, server processing power, server memory, abuse controls, _or other reasons_, CloudFlare may suspend or terminate your account without notice to or liability to you."

In other words, you can't host non-HTML, but you can if it isn't disproportionate, but if it is disproportionate, they can deem you to be a problem and cut off your service, without notice. That's not a contract at all. In legal parlance, that's an illusory contract -- when one side can modify their performance in any way at any time.

I use Cloudflare's $20 a month option and it worries me now that I might be _deemed_ to be using a _disproportionate_ about of space or bandwidth caching images, and then be cut off without notice.

EDIT: I love the cloudflare service and I'm not complaining. I just think their legal department needs to clarify this and the tech side of the house needs to be able to warn users when they are exceeding the bounds of what is acceptable.

<comment><USER13> My interpretation of this paragraph goes something like:

> _If and when_ your bandwidth usage gets high enough that one of our customer > service people gets pinged about upgrading you, they'll also have a look on > the dashboard to see what mixture of filetypes you're serving. If it's all > static content, you're in trouble.

I'm not quite sure how else that could be phrased into legalese, than what they already have there.

</comment></comment><comment><USER14> Based on what I have read on many HN articles, what Cloudflare offers and what you receive are two very different products.

From being shut off when you have a incoming DDoS of some arbitrary size to actually loading webpages slower than your server does vanilla, it seems the benefits of Cloudflare are mostly hype.

</comment><comment><USER15> While I agree CloudFlare handled this poorly, there is this reply on the blog from Matthew Prince (CEO of CloudFlare):

Both enabling "I'm Under Attack Mode" or routing the traffic direct are both supposed to generate an automated message to the customer letting them know what happened. We've reviewed the logs and don't see a message having been sent. I'm investigating why that didn't happen since I agree it is not acceptable.

</comment><comment><USER16> Very interesting article, thanks!

 _At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget._

An interesting proposition - If we take it at face value, $3000 for 100TB works out to be $0.03 per GB. That's pretty high these days. If you are buying downmarket (which is cloudfare like traffic quality) you can get a cdn deal for maybe $0.01 on a 6 month term with these kind of levels, and somewhere around $0.005 for an xc in the us or eu no commitment. Cloudfare should be buying at substantially better rates than these (or at least, they seem to imply it - calling bandwidth free) so it's it seems they have a similar problem as many freemium models - when most of your customers aren't paying you have to really hit the ones that do.

<comment><USER17> Even at $0.01/GBT, you're still looking at $1,000 a month. His conclusion of "two servers and 100 TBT for $200" seems unlikely to last long either.

> _somewhere around $0.005 for an xc in the us or eu no commitment_

From whom? Or is this reverse engineered from a fully saturated link?

<comment><USER16> Not extrapolated from a fully saturated link, no. But I didn't mean to imply that was usage billing.

I was thinking of fdc in amsterdam in the eu on a 10gb unmetered no bw sla, but I also think they offer a similar (slightly higher) deal in denver. I was thinking of somebody else for the US - in SLC - but the name escapes me. If you actually could use the name I'll find it.

I shouldn't have said xc as really they expect you to buy power and space from them. Clearly not what I'd try to run an upmarket video cdn off of, but i would be surprised if people like fdc aren't who CF buys from.

<comment><USER17> > _unmetered no bw sla_

Those kinds of offers are generally unusable _if the bandwidth matters_. This goes back to the oversubscription model, selling the same resource to multiple customers and letting the customers jockey for use, hoping most will never use it enough to catch on.

See this thread for more:

— [http://www.webhostingtalk.com/showthread.php?t=1159276&p...](http://www.webhostingtalk.com/showthread.php?t=1159276&page=2)

Note this analysis:

 _"The FDC offer is a shared 10 Gbps and i believe in another topic was explained, that you're supposed to stay at 1 to 1.5 Gbps usage. Even a pure 10Gbps peering port will cost more then $500,- / month, so a true 10 Gbps for anywhere close to $500,- cannot be expected from any provider worldwide."_

— [http://www.webhostingtalk.com/showpost.php?p=8153934&pos...](http://www.webhostingtalk.com/showpost.php?p=8153934&postcount=18)

And about the actual usable bandwidth:

 _"I always found this to be a problem with all FDC locations. So many places have such horrible speeds that no matter what speed they offer, I have a hard time making a good use of it."_

— [http://www.webhostingtalk.com/showpost.php?p=8258933&pos...](http://www.webhostingtalk.com/showpost.php?p=8258933&postcount=27)

Another comment:

 _"The server is great. The single-thread transfer speed isn't as much, but it's reasonably passable most of the time, considering the server cost. But I've been seeing some severe routing problems, with 5-10% of the net simply being unroutable much of the time, as well as intermittent packet loss. Due to this, the overall fail rate of this server is about 10x that of other servers I have in NL."_

If you're running an ad supported viral image host and can fit your popular content in RAM, this kind of thing may be acceptable, though in the long run users will tend to migrate to image hosts that serve their memes quickly.

TL;DR: Not all bandwidth is equal. You get what you pay for and what your provider pays for.

</comment></comment></comment></comment><comment><USER18> Slightly OT but related to the Cloudflare business/enterprise offering: They "guarantee" 100% uptime, does anyone know details of this? Obviously there is no way to actually guarantee such a thing, but what kind of compensation do I get when they do have downtime? I can't seem to find it mentioned anywhere.

"Service Level Agreement (SLA) - 100% uptime

Industry standard SLAs often feature 99.999% uptime, also known as the five 9s. At five 9's your website could be offline for as long as 5 minutes and 26 seconds each year. All CloudFlare Business and Enterprise plans offer guaranteed 100% uptime because we know that anything less than 100% is an impediment to your organization's success."

<comment><USER19> Usually these guarantees mean credits toward future billing. Cloudflare's enterprise plan offers 2500% uptime guarantee, which means if they are down 10% of the time, customers are credited 2.5 months.

It sounds a lot nicer than it is, much like "we never charge for bandwidth".

</comment></comment><comment><USER20> It's not a directly comparable service, but serving 100TB/month over S3 would cost you more than $9,000 a month. On CloudFront, depending on object size and distribution area, it would cost more than $8,000

<http://calculator.s3.amazonaws.com/calc5.html>

<comment><USER14> <https://google.com/search?q=100tb>

There are many cheaper options. The ones you mention seem to be the most expensive.

<comment><USER21> S3 and Cloudfront are targeted at the dabbler, who will likely only be serving a GB or two per month, in which case S3 and Cloudfront would be pennies with no long term commitment. Once you move beyond that and are storing and serving more than 1 TB, you'd be better off with a monthly commitment.

<comment><USER22> Just FYI, Cloudfront does heavily discount committed pricing.

<comment><USER21> True, but that starts at 10 TB/mo

</comment></comment></comment></comment></comment><comment><USER23> You get what you pay for. When I'm paying for a $16/megabit @95th on a 10 Gig port, and I get hit with a DDOS, it's my providers problem, because I'm not paying $16/megabit for DDOS traffic, so they need to stop that stuff before it gets to me (ideally without impacting my customers)

Contrawise ,if I'm paying $2-$3/megabit @95th on a 1 Gig Port, the amount of support I can expect during a DDOS is pretty minimal, so I end up having to take the hit - but my damage is limited to $3000/month so I don't really care.

Any time I see a "We don't charge for Bandwidth" service, I interpret it to mean one of (A) We'll throttle you once you exceed our unspoken limit, or (B) We'll discontinue your service. (Drop your port from 1 Gig down to 100 Megabits, or slower, traffic shape you, etc...) once you breach that limit.

There is no sustainable third option for those who provision reasonably high quality transit, and those who believe there is will one day wake up with their internet property offline, or seriously degraded.

</comment><comment><USER24> This kind of whining and the comments on the page piss me off greatly.

A customer is a person who pays for a service. Someone who doesn't pay for a service (yet) is a _lead_. Not all leads are good business. 100TB of traffic does not sound like a good lead to me, not even at the $200 level.

Looking at the site, I see an IMGUR clone which was running for free off of CloudFlare's cache servers. I really don't understand the nonsensical comments on the article. WTF is wrong with people these days thinking that everything is supposed to be free? Are you all 16 and on an weekly allowance?

Commenter Matt had a very valid point that some sort of optimization of the stored (cached) files would have been a smart option for yourselves (less local storage) as well as CF (less to cache, less bandwidth). I'd recommend <http://www.jpegmini.com/server> (Oh wait, it's not free, now what... cry me a river)

I have a question for the Phobos peeps. Were you making money? Seems to me like you were... since you can afford the LeaseWeb servers. Instead of bitching publicly, perhaps you should have reached out to the company when you noticed your traffic levels were reaching antisocial magnitudes.

Grumpy Gramps (who used to pay UUNET $6k+ a month in 1997 dollars for the privilege of hosting a basic database-driven e-commerce site for a luxury watch brand on a guaranteed T1 connection)

<comment><USER25> They weren't making money, and they did contact the company. Comments like yours piss me off greatly.

<comment><USER24> They contacted the company _after_ being shut down. Not before, when it should have been pretty clear to them how many resources they were using. There's a simple indicator in the dashboard. "BANDWIDTH SAVED".

This line in the article "I tried to monetize it through ads some time ago, but failed. Advertising-Networks don't want us as a customer, but I'm fine with that." was not conclusive that monetization failed via other means. Not a user of the site, but at first glance some choice links to pr0n affiliate sites would do the trick.

Throwaway ey... hmmm.

<comment><USER25> Notlisted uh... right.

<comment><USER24> Four years bro. Tosser.

</comment></comment></comment></comment></comment><comment><USER26> Why is it that you'll always get the worst posible review by a non-paying customer?

This is a common observation - less the user pay the worse and less objective the feedback is.

<comment><USER27> There are a handful of contributing causes, just off the top of my head:

Once you get past the first few tiers of paid service, providers tend to reign in the bullshiting "technically true by widely misleading" descriptions and are upfront about what they provide. Properly managed expectations don't lead to as strong of emotions when a customer finds out what they're getting doesn't work for them.

Commercial users will save face by A) avoiding discussing anything that could make them look bad, ditching a vendor is a tacit admission that you went wrong when you chose their service. Additionally, if their service is built on top of the ditched service, it casts quality concerns on their product too. B) Avoiding disparaging anyone, ever, is pretty common play-nice-save-face.

Why should I give my competitor helpful advice? I got caught up in a web, I don't need to help the competition avoid my mistakes.

In a multiple person organization, the party angry about the service and the party who decided to buy/use it are different individuals and airing grievances/venting is an internal process that they don't reiterated in public.

In a free service, a failure to deliver is the only downside. If I paid for a service, I'm going to be too preoccupied beating myself up for making a stupid purchase decision and looking for an better alternative than bothering to share my findings.

These causes and more are all often balled up and interpreted as free users acting "entitled". Sure that exists, but its a classic case of reading too much into imprecise metrics when someone assumes the cause.

Furthermore, the feedback from free users isn't inherently worse, and is just as subjective as that from paying users, it just tends to be more negative. For all we know, any given product could objectively be shit and because only rubes will buy it, the free users are the only ones that can let you know.

</comment></comment><comment><USER28> Interesting, I have been using their services for a while now but most of my traffic (probably 99%) actually goes straight to my web servers as i'm only really interested in the free DNS hosting service, however looks like i'm breaking the ToS as well, may be time to rethink :S

</comment><comment><USER29> Could someone explain the reply from "Matthew Prince" to me?

They turned it off due to an attack and would turn it on again if upgraded to business ($200/mo).

Does this mean turn it on while under attack would cost, and after the attack would be free again? Or it would cost the upgrade to turn it on either way?

<comment><USER30> They claim that other customers were affected, so they stopped handling requests.

The process would have been automatic if he was a paying customer. Because he wasn't, the company felt justified in not trying harder to bring their resources back up.

</comment></comment><comment><USER31> Wow.

Imagine the origin server would have been Amazon S3. The webmaster would have incurred an Amazon bill of $400 USD _per day_ after the switch.

</comment><comment><USER32> One thing I'm yet to puzzle out is whether CloudFlare handles apex domains. They say they do but DNSimple says in reality they just route it straight to your servers so that it's pretty useless.

<comment><USER33> CloudFlare does handle apex domains; when you turn the service on, it changes the A records to point at CloudFlare. I don't know where DNSimple is getting their information, you can see CloudFlare working on a naked domain just by trying to resolve the hostname. It'll return two (or more) CloudFlare IP addresses, not the address of your server.

</comment></comment><comment><USER34> And why is 4chan still hosted at CloudFlare?

<comment><USER5> 4chan (and its users) probably want a fast downlink. The author touts a cheaper provider, but the service they are receiving is an order or more of magnitude slower service from my anecdotal test through wget(s) of images on both OP's and 4chan's sites. That is, a pair of dedicated and unmanaged servers is not, AT ALL, comparable to a CDN's offerings.

4chan probably also expects to have more than an email (and timely) endpoint with which to correspond with the provider of their most core site-service -- image serving.

</comment></comment><comment><USER25> Thanks a lot for this article.

Were you already using http cache of 1 year for all the images when this happened? Do you think it could have been avoided by setting it to 1 week, 1 day, or even 1 hour?

It's funny that I have to ask this to you instead of asking Cloudflare. They have really messed up on this one.

</comment><comment><USER35> Where are their sellers? This is just a bad example of how they do not recognize their customers.

</comment><comment><USER36> You couldn't monetize it yet you can spare $200/month?

<comment><USER37> I've rented servers for high-traffic websites with no income for years. If it's something you love to run, and you can afford it, then it's just the cost of your hobby. Not everything is a business.

<comment><USER36> If he is as really rich as you suggest it seems odd to me that he would complain about Cloudflare and not have rented a 100 TB server sooner. They are not exactly difficult to find.

Whether he can tell us if he is rich or found another way to monetize adult or potentially offensive content, it is up to him to share.

<comment><USER38> $200/mo is not "rich," it is what you would spend on a hobby (think a membership to a climbing gym or buying a new gaming computer once a year). Being able to spare $3000+/mo for CloudFlare enterprise is not even close to a comparison.

<comment><USER36> Now this might just be my opinion, but considering many of the people I know in the technology field have a disposable income of around $1k/month, I don't think there's very many people that would throw away 20% of it on an image dump. If you make more than that, then you are rich by my standards.

<comment><AUTHOR> See, for me it's not an "image dump", but a nice hobby project. Managing a community as well as the technical challenges this site presents is interesting. I can spare $200 but not $3000.

Also, I learned a great deal of JavaScript while building that page and I just now learned a bit about load balancing and setting up Varnish - <USER2> which may come handy for godknowswhat. I never imagined making my living by writing a JS game engine either.</comment></comment></comment></comment></comment></comment>


Academia as an 'anxiety machine' - <AUTHOR> http://lemire.me/blog/archives/2014/12/05/academia-as-an-anxiety-machine/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+daniel-lemire%2Fatom+%28Daniel+Lemire%27s+blog%29 <comment><USER2> My girlfriend is getting her PhD. We are planning a vacation right now. She works in a lab as a researcher, which is not a "real job" since she's getting her PhD. There are no specific regulations for vacation time, or anything for that matter. However, she's unsure how long she can take for vacation, and unsure whether she should ask her P.I. This is an example of the irony of academia.

On one hand, the lack of clear policies and regulations governing academia is one of its greatest advantages, because it gives your mind space to explore, and expands creative freedom. If you want to work from 5pm - 3am, nobody will stop you. Ultimately you work for your own advancement, set your own goals, and plan your own schedule.

On the other hand, when other people are involved, especially those with "seniority" like P.I.'s and advisors, it's no longer obvious what decisions you should make. You need to consider more than just yourself, but the lack of regulation creates ambiguity in the "politics" of academia. How do you balance the expectations of those with some power over you, with the expectations you have of yourself? I imagine this question is a source of pressure for many academics, especially those who are extrinsically motivated.

<comment><USER3> Not to pick on your girlfriend but I've seen a variant of this phrase so often that I'm kind of baffled:

 _" the lack of clear policies and regulations governing academia is one of its greatest advantages, because it gives your mind space to explore, and expands creative freedom."_

Are the folks in academia so creatively stunted that they can't find freedom outside of the ivory tower?

It seems like such a silly argument.

<comment><USER2> Surely there are opportunities outside of academia that do not restrict you to the 9-5 grind or subject you solely to the whims of "the man." But academia is one of the only options where that kind of freedom comes safely packaged and institutionalized. You get the advantages of prestige without the risk of something like entrepreneurship, which may be another option for finding creative freedom.

<comment><AUTHOR> Agreed. Academics put up with a lot of crap because they're extremely risk adverse.

<comment><USER4> Very true. The twisted irony is that the long term risks of academia are devastating. We have a lot more short term job volatility in software, but unless we work for some outlier asshole who torches our reputations, we can usually get new jobs. Academics almost never get fired except at legible points (e.g. tenure review) but when they do, it ruins their lives in a way that most of us have never experienced.

</comment></comment></comment></comment></comment><comment><USER5> I left in the middle of a PhD for mostly this reason. I was introduced to politics of the department very quickly because I learned my advisor was not popular with most of the rest of the faculty. Only two of them would collaborate with her. She had a big fight with one of the more established researchers over patent rights a few years back.

I had met her as an undergraduate five years prior to when I became her grad student. She was pleasant, albeit a little pushy, but nothing offputting. She had changed in the interim. She would yell at me (and her other students) in her office for the slightest mistakes. During a meeting with other researchers and my colleagues she accused me of not completing my work. It took all my strength not to tell her to fuck off and walk out. Mind you, this woman gave me a rare A+ in her course just a few years before because I did stellar work.

She's certainly infected by the anxiety of academia and she passes it on to her students and subordinates. When I saw how being a university researcher transformed her I decided to give up my chosen route. It didn't seem worth it to me if there was a significant risk of becoming a bitter asshole.

I became a software engineer. So I probably make double what she does, I have a fraction of the anxiety and I'm not a bitter asshole.

</comment><comment><USER6> The worst consequence of this insecure, anxious, hypercompetitive culture is the near-impossibility of sincere collaboration. I speak from my experience as a very successful grad-student and postdoc in synthetic biology and neuroscience. Unlike equity in a startup, the kudos credit for publishing a high-impact paper is really only split between the first author and the lab PI, removing any incentive for trainees and labs to work together on larger projects. Of course, a great deal of noise is made in grants and PR about academic collaborations, but they are always paper affairs of convenience in securing funding, never truly incentivized joint missions, at least not in the biological sciences. I can't count the number of attempted collaborations that I've seen collapse in heated acrimony. As a result, most academic science is a very lonely affair, and the ambition of what can be achieved today in experimental biology is completely constrained by what one poorly-trained young researcher can do in a few years of hard labor with only a modicum of outside help and advice.

<comment><USER7> This is one of the things that struck me most about moving out of academia and into various other things: how _collegial_ in the best sense of the word most of the engineering teams I've worked on have been.

In academia, if you're any good, you're a threat to other good people. You try to be nice to each other, but when it's you and a few dozen of your closest colleagues in pursuit of the one or two tenure-track openings in your field this decade... there are limits on how collaborative you can be.

In the business world, if you're any good, you're a resource and people want to work with you. It's pretty awesome, actually, although there are obviously bad situations as well. They're just fewer and further between.

<comment><USER8> _> In academia, if you're any good, you're a threat to other good people. You try to be nice to each other, but when it's you and a few dozen of your closest colleagues in pursuit of the one or two tenure-track openings in your field this decade... there are limits on how collaborative you can be._

There are many "academias", but this is not my personal experience of it. Yes, I think most of us PhDs realize the bleak situation we are in, but still, we usually work together at the university as well as across universities.

Whenever I see a PhD student that has a better track record or just more talent, I am very happy for him -- after all, he's got a bigger chance than me for making it in academia.

There might be some politics and bickering among researchers/employees within a university, but that's something that happens in every community.

</comment></comment></comment><comment><USER9> How to get the best out of people is a millennial long question. Egyptians used whips and slaves, we have been using money and propaganda lately, but everyone uses something that may or may not be the most optimal method. The issue is that people are individual and respond differently each time you ask them. My condolences on the loss of life (I too have felt like the world was asking far too much of me and I had no way out) but suicide is rarely the answer. Quitting your job, going to live in Montana for a while, having far too many beers, considering that everyone around you is an asshole, those are good options. And yes, I know it is real hard to climb out of those dark mental holes and realize this, but damnit you have to try.

If there was 2 things I wish the whole world could know, truly know deep down, they would be that you personally are worth all the stars in the heavens, all the grain in the fields, all the water in the oceans, and that everyone else is worth exactly the same amount; that we are all priceless.

<comment><USER10> "Best" is a pretty hairy topic. Does it mean "Most fulfilling existance", does it mean "Most benefit to humanity", does it mean "Most benefit to this country/children/etc..." There's no right answer. I submit that "Letting people expressing their freedom in a way that maximally lets others express the same." is more useful/enlightening/profitable than "Getting the best". But that's pedantic and hair splittery, like the rest of my comment. People aren't priceless. If we were industry would grind to a halt. There has to be a reasonable price on human life or there would be no calculating the impact of 1, 10, or 100 deaths, and then to upper limit on the $ spent on safety. The value of a human life is around 50-100K$/year to 100$K-10$Million Total, depending on the application.

<comment><USER11> Continuing this pedantry, one could argue s/he isn't claiming that what people _do_ be worth all the stars in the heaven, etc. This guy "getting a house in Montana" is him specifically rejecting that his worth is only his salary or position or career...even his pride. His sanity at least was worth it somewhat for him to take a sabbath to regain it.

I guess the next question then is what is it we value in ourselves and how do wee determine it? That's a harder question.

<comment><USER12> The question is wrongly phrased: it's not that we value things _in_ people. People are terminally valuable.

</comment></comment></comment></comment><comment><USER13> "There is no trace of evidence that you can get the best out of people at high-level tasks through pressure and competition. The opposite is true. Worried people get dumber. They may be faster at carrying rocks… but they do not get smarter."

That last sentence is a great turn of phrase.

<comment><USER14> You reminded me of a fantastic video about positive and negative motivation, especially with how they relate to blue-collar vs white-collar work:

Drive: The surprising truth about what motivates us ([https://www.youtube.com/watch?v=u6XAPnuFjJc](https://www.youtube.com/watch?v=u6XAPnuFjJc))

</comment></comment><comment><USER15> Going into academia seems like the dumbest thing only smartest people do.

Jump into a job market where you compete against the bright and ambitious individuals who will do everything to out-work you to fight for a small prize pool.

The ending will most likely not be a happy one.

<comment><USER16> "Jump into a job market where you compete against the bright and ambitious individuals who will do everything to out-work you to fight for a small prize pool."

Isn't that sort of just defining finding a job in the general market? You're competing against other folks who are either more or less qualified than you are for a small prize pool.

That's the same thing.

</comment></comment><comment><USER12> > This is not, I shouldn't have to say, how academia works. Peter Higgs, of > Higgs Boson fame, said that there was 'no Eureka moment' to his work, and he > only has 4 papers listed on Google Scholar: but what papers! Science rarely > has a Eureka moment: it's rather a series of careful, thoughtful > developments of work done by one's forebears and peers. A management which > demands a Eureka a day is one which doesn't just not 'get' academia, it's a > management which contradicts the academic method and it's one which has > forgotten that it's meant to serve the needs of science, the arts, students > and researchers, not the insatiable maw of attention seeking 'Leaders' > (that's the word they use now) and the PR office. It's also a management > that kills.

 _slow applause_

</comment><comment><USER17> Life is an 'anxiety machine.'

Claiming academia killed this specific man is in extremely bad taste. And if we're speaking in generalities, I'd need to see evidence that academic positions correlate with higher rates of suicide.

<comment><USER18> Come now! Surely you can defend your sacred cow more vigorously than that. You're not starting to entertain second thoughts of your own, are you?

</comment></comment><comment><USER19> "One of the reasons academic infighting is so vicious is that the stakes are so small" \- Kissinger

<comment><USER20> He's not wrong, but he's not right either. The stakes aren't so much small as they're impossible to wrap your head around. They tend to be something like, "understanding something we don't even know exists yet". Compared to something like global politics, where a stake is often at least quantifiable ("1 country", "500 million children", etc.), it does help explain the viciousness: it's not clear what's lost when there's a casualty.

<comment><USER19> I've seen academics fight harder for a new filing cabinet than bankers for a 500K bonus.

</comment></comment></comment><comment><USER21> I didn't stay in formal education beyond high school so I can't really comment, but from that article:

"Unfortunately, some of his colleagues felt that he did not secure sufficiently large research grants. So he was to be fired."

However, Imperial College' statement on his passing claims otherwise:

"Contrary to claims appearing on the internet, Professor Grimm's work was not under formal review nor had he been given any notice of dismissal."

[http://www3.imperial.ac.uk/newsandeventspggrp/imperialcolleg...](http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/newssummary/news_4-12-2014-18-0-17)

<comment><USER22> More fully:

"Contrary to claims appearing on the internet, Professor Grimm's work was not under formal review nor had he been given any notice of dismissal. It is standard practice at Imperial to conduct both informal and formal performance management. Professor Grimm's line manager met with him on a number of occasions to see how the College could help him to develop more competitive grant applications, for example through internal peer-review, collaborations and letters of support. Discussions included talking about the best place for him to do his science, both inside Imperial and outside, and, with Professor Grimm's permission, his line manager made enquiries about opportunities on his behalf."

So this may still be a situation of pressure being brought to bear, just in some manner other than via formal review.

<comment><USER23> A line manager for academics. We're not in Kansas anymore.

</comment></comment></comment><comment><USER24> Professor Grimm's death is a horrible tragedy and a major blow to the community. He will be greatly missed by those who were fortunate enough to know him.

Perhaps institutions that issue grants should require that the receiving organization ban "quotas" from their internal policies.

Condolences to Prof. Grimm's family and friends.

</comment><comment><USER25> [https://news.ycombinator.com/item?id=8687714](https://news.ycombinator.com/item?id=8687714)</comment>


In Twitter's early days, only one celebrity could tweet at a time - <AUTHOR> https://theoutline.com/post/4147/in-twitters-early-days-only-one-celebrity-could-tweet-at-a-time <comment><USER2> If anybody is interested in random Twitter internal stuff I might bang up a medium post one day. I was neck deep in the infra side of things for years and have all sorts of funny stories.

Our managed hosting provider wouldn't let us use VPNs or anything that allowed direct access to the managed network they provided, but we wanted to make internal only services that were not on the internet so I setup a simple little system that used DNS to point to private space in the office and a SSH tunnel to forward the ports to the right places. Worked great, but over time the internal stuff grew up, and our IT team refused to let me have a server in the office so it was all running of a pair of mac mini's. We called them the "load bearing mac minis" since basically 90% of the production management traffic went over the SSH tunnels they hosted. =)

<comment><USER3> Posting that here is like showing a big juicy steak to a pit full of hungry lions: of course we're interested!

</comment></comment><comment><USER4> 2010: "At any moment, Justin Bieber uses 3% of our infrastructure. Racks of servers are dedicated to him"

[https://gizmodo.com/5632095/justin-bieber-has-dedicated- serv...](https://gizmodo.com/5632095/justin-bieber-has-dedicated-servers-at- twitter)

<comment><USER5> But, but, but...why does Twitter have so many engineers? I could write Twitter in a weekend!

\--95% of anti-TWTR posters circa 2010-2016.

<comment><USER6> Before bring acquired, WhatsApp had what, 30 employees?

How did they do it? I know they used custom BSD servers so that a single box could keep close to 1M TCP connections open. I'm sure with a fixed target to aim for and all scope known upfront a small crack team of devs could do something similar for Twitter.

<comment><USER7> One-to-one vs. many-to-many messaging. The amount of work you need to do to deliver a WhatsApp message is constant and small -- just route the message to a single recipient's mailbox. The amount of work Twitter has to do to deliver a message grows as a function of followers. One celebrity tweeting another celebrity means you have to deliver the message to the mailboxes of the followers of both -- millions of times more work than WhatsApp per message. In addition, Twitter persists all the messages while WhatsApp doesn't.

<comment><USER8> This! Everyone keep's saying it's BSD Erlang and jumping on the Erlang train. Fine tools btw, but WhatsApp is super simple compared to Twitter. _rolls eyes_

</comment></comment></comment></comment></comment><comment><USER9> If you want to read more about activity feeds there are a ton of papers listed here: [https://github.com/tschellenbach/stream- framework](https://github.com/tschellenbach/stream-framework) I've been working on this stuff for years. Recently I've also enjoyed reading Linkedin's posts about their feed tech. Three are a few different posts but here's one of them: [https://engineering.linkedin.com/blog/2016/03/followfeed-- li...](https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s- feed-made-faster-and-smarter)

Scaling a social network is just inherently a very hard problem. Especially if you have a large userbase with a few very popular users. Stackshare recently did a nice blogpost about how we at Stream solve this for 300 million users with Go, RocksDB and Raft: [https://stackshare.io/stream/stream-and-go-news- feeds-for-ov...](https://stackshare.io/stream/stream-and-go-news-feeds-for- over-300-million-end-users)

I think the most important part is using a combination of push and pull. So you keep the most popular users in memory and for the other users you use the traditional fanout on-write approach. The other thing that helped us scale was using Go+RocksDB. The throughput is just so much higher compared to traditional databases like Cassandra.

It's also interesting to note how other companies solved it. Instagram used a fanout on write approach with Redis, later on Cassandra and eventually a flavor of Cassandra based on RocksDB. They managed to use a full fanout approach using a combination of great optimization, a relatively lower posting volume (compared to Twitter at least) and a ton of VC money.

Friendster and Hyves are two stories of companies that didn't really manage to solve this and went out of business. (there were probably other factors as well, but still.) I also heard one investor mention how Tumblr struggled with technical debt related to their feed. A more recent example is Vero that basically collapsed under scaling issues.

<comment><USER10> I used to work at Hyves. Hyves overcame it's scalability issues, but went out of business for other reasons. Hyves used MySQL and Memcache similar to facebook at that time.

By the way, RocksDB which is now Facebook's main database (afaik) is written on top of LevelDB. So both Google and Facebook run on software written by Jeff Dean...

<comment><USER11> Its a leveldb fork with substantial changes. For example, a non binary search file format option: [https://github.com/facebook/rocksdb/wiki/PlainTable- Format](https://github.com/facebook/rocksdb/wiki/PlainTable-Format)

Pull down the code some time. Everything and the kitchen sink is in there somewhere. It's a crazy project.

</comment></comment></comment><comment><USER12> I haven't seen anyone touch on this, but I remember reading about this in Data Intensive Applications[1]. The way that they solved the celebrity feed issue was to decouple users with high amounts of followers from normal users.

Here is a quick excerpt, this book is filled to the brim with these gems.

> The final twist of the Twitter anecdote: now that approach 2 is robustly > implemented,Twitter is moving to a hybrid of both approaches. Most users' > tweets continue to be fanned out to home timelines at the time when they are > posted, but a small number of users with a very large number of followers > (i.e., celebrities) are excepted from this fan-out. Tweets from any > celebrities that a user may follow are fetched separately and merged with > that user's home timeline when it is read, like in approach 1. This hybrid > approach is able to deliver consistently good performance.

Approach 1 is a global collection of tweets, the tweets are discovered and merged in that order.

Approach 2 involves posting a tweet from each user into each follower's timeline, with a cache similar to how a mailbox would work.

[1] [https://www.amazon.com/Designing-Data-Intensive- Applications...](https://www.amazon.com/Designing-Data-Intensive-Applications- Reliable- Maintainable/dp/1449373321/ref=sr_1_1?ie=UTF8&qid=1527213498&sr=8-1&keywords=data+intensive+application&dpID=51PjhtI9VRL&preST=_SX218_BO1,204,203,200_QL40_&dpSrc=srch)

<comment><USER13> It's an oft overlooked inequality in these systems. People get so wrapped up in some whiz bang thing that they don't stop to think if they should.

At the end of the day, one of the most important aspects of your information architecture is how many times is each write to the system _actually_ observed? That answer can dictate a lot about your best processing strategies.

<comment><USER14> This. It took me a while to learn to look at this and eventually focusing less on caching on some items that don't enough read in it to justify the caching or pre calculation.

</comment></comment></comment><comment><USER15> This isn't shocking - Twitter was notorious for being held together with Scotch tape technically.

Honestly this hands-on approach is an impressive example of doing things that don't scale.

<comment><USER16> I found it amusing that Twitter was Rails' biggest advertisement. Everyone wanted to use Rails but Twitter turned into a franken app with different stacks to keep it running

<comment><USER17> Twitter was Rails' worst advertisement. They used Rails as a scapegoat to hide their bad tech. I still hear things like "Rails can't scale; remember Twitter?"

<comment><USER18> At least 3 large companies I know my friends developers work at, over the last 5 years switched from Rails to PHP. All told me same story "after twitter, noone wants to work or touch rails anymore"

<comment><USER19> Meanwhile all 3 large companies probably host their code on Github :-)

<comment><USER20> And even if they don't, many if not most of their dependencies do.

</comment></comment></comment></comment></comment></comment><comment><USER21> There was a fun high-scalability article around their 'fan-out' approaches to disseminating tweets by popular users etc : [http://highscalability.com/blog/2013/7/8/the-architecture- tw...](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses- to-deal-with-150m-active-users.html) .

When I was working on something with similar technical requirements I also came across this paper ([http://jeffterrace.com/docs/feeding-frenzy- sigmod10-web.pdf](http://jeffterrace.com/docs/feeding-frenzy- sigmod10-web.pdf)) that outlined the approach in a more 'formal' manner.

</comment><comment><USER22> Ah, I read that Twitter thread a few days (weeks?) ago and it was much longer. As far as I remember, it started with someone asking Twitter ops people, former and current, to share some stories about things that went spectacularly wrong.

It contained a lot of Twitter ops battle stories, some very interesting. I was pretty impressed to read Twitter internals in the given level of detail, but now it seems that the thread that held them all together is protected (probably didn't expect it to be so popular, or just wanted to continue more privately).

<comment><USER2> And yet I bet nobody mentioned the "fire rain" in our first data center, the load bearing mac mini, or the surprise Snoop Dog visit/party! =)

<comment><USER22> Hah, I don't think anybody has, but I'd love to hear those stories!

</comment></comment></comment><comment><USER23> In the early days of Twitter the "fail whale" was so common it got assimilated into culture as a term to use for anytime any site gets overloaded. Nowadays it seems like that term is "hugged to death"

[https://www.theatlantic.com/technology/archive/2015/01/the-s...](https://www.theatlantic.com/technology/archive/2015/01/the- story-behind-twitters-fail-whale/384313/)

<comment><USER2> Everybody knows about the "fail whale".. nobody knows about the "moan cone".. The image is lost to history but it was captured by this account ages ago: [https://twitter.com/moan_cone](https://twitter.com/moan_cone)

IT was thrown when the system was failing at the rails layer rather than the apache layer. I believe that Ryan King and I were the last people to ever see the moan cone in production. =)

<comment><AUTHOR> Do you remember the fixit kitten?

<comment><USER2> That got nuked right about the time that I started! I never saw it live.

</comment></comment></comment></comment><comment><AUTHOR> Missed that this went to the front page! I will answer questions if I can.

I am now CEO @ [https://fauna.com/](https://fauna.com/), making whales and fails a thing of the past for everybody. We are hiring, if you want to work on distributed databases.

<comment><USER2> Long time no see man! =)

<comment><AUTHOR> You too! Hope Zookeeper isn't still keeping you up.

<comment><USER2> Funny thing is that the place I am at now uses Zookeeper.. luckily it has not been trouble for us though. =)

<comment><AUTHOR> It's saving it up for you...remember deleting thousands of Nagios notifications on the bus? Those were the days.

<comment><USER2> I remember all those iPhone suckers having to download special apps to remove conversations in the early days since it only allowed you to delete them one at a time. =)

I also remember getting on a bend to make all fake alerting go away after I started only to find that like 50% of the configured alerts couldn't ever succeed and had been paging every 10 minutes for over a year. We got that stuff sorted out pretty quick. Once we started getting good visibility it helped us make much better decisions about where the problems actually were.

The first six months felt like everything was just on fire constantly and nobody know what was going on, but then things started falling in place and it felt like everything was still on fire, but some people actually had fire extinguishers for a change. =)

<comment><USER24> The best monitoring system was the TV in the Ops area scrolling a constantly updating search for #failwhale.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Fun fact, Twitter's feed is still kinda broken. If you visit the site after being gone for a week or so it tells your timeline is empty. It recovers after a few minutes, but its still a pretty poor user experience.

<comment><USER25> I always get a message saying the request took too long to execute, if I click a link leading to Twitter. Have to reload the page a couple of times to make it work.

Usually don't click on Twitter links for this reason.

<comment><USER26> Yes, I get "rate limited" on a single request on mobile web quite often. It's strange.

</comment></comment></comment><comment><USER27> Say what you will, Twitter managed to make a $25.23B company out of printing

       char whatever[140];

<comment><USER28> should be 141 if you want to print out a string that would be 140 characters, NULL termination is a thing.

</comment></comment><comment><USER29> What frameworks would you use to handle such an steep growth curve? Most startups I know of start of with rails or the like - and obviously they couldn't handle the strain. So what would you use?

<comment><USER9> The web part of it is pretty easy to scale. You simply add more web servers. The problem is in the storage/db layer. I imagine the feed was the primary challenge scaling Twitter.

Other components such as the search were probably also quite tricky. One thing i've never figured out is how Facebook handles search on your timeline. That a seriously complex problem.

Linkedin Recently published a bunch of papers about their feed tech:

[https://engineering.linkedin.com/blog/2016/03/followfeed-- li...](https://engineering.linkedin.com/blog/2016/03/followfeed--linkedin-s- feed-made-faster-and-smarter)

And Stream's stackshare is also interesting: [https://stackshare.io/stream/stream-and-go-news-feeds-for- ov...](https://stackshare.io/stream/stream-and-go-news-feeds-for- over-300-million-end-users)

<comment><USER30> >One thing i've never figured out is how Facebook handles search on your timeline. That a seriously complex problem.

From a user's point of view, they don't. I search for things a lot (or at least, I used to before I realized how bad it is) and things that I _know_ I saw yesterday don't come up - even if I put in a very specific search. Sometimes it'll say "no results found", but I'll find it in a tab I hadn't closed yet.

</comment></comment></comment><comment><USER31> This made me laugh and feel better. Being in a startup is tough!

</comment><comment><USER32> It would be fascinating if they returned to this capability. One celebrity gets the podium at a time.

</comment><comment><USER33> > Jason Goldman, who served as Vice President of Product at Twitter between > 2007 and 2010, responded to Weaver's tweets with the observation that early > Twitter was "held together by sheer force of will."

I would dispute that, I don't think they can take that much credit. Regardless of their "sheer force of will" the site was down very, very frequently.

<comment><USER2> Much of that was just bad design decisions in the early days creating a momentum that was unstoppable and unreplacable at the speed we were growing.

In the early days they implemented a Kitama system for memcache. It worked great so long as nothing failed since a node coming in and going out could lead to bad results. In the old days they would just flush caches when that happened. Later, when memcache was the only way we could serve the loads we had it became imperative that memcache never be restarted. A single memcache restarting would overload the mysql backend so bad that the site would be down for hours. Adding memcache had more or less the same issues though it was a little easier to prewarm things. We wrote a kernel module that allowed us to change the ulimits of a running process so we could increase the file descriptor limits for memcache without restarting it.. Replacing it completely was damn near impossible given the growth rate and inability to get the data out of mysql quickly enough.

Us reliability guys worked 16 hour days 7 days a week for years trying to keep things working well enough to not fail completely. Some of the crazy hacks we did just to survive were fantastic and impressive and I am still proud and disgusted by them to this day. =)

<comment><AUTHOR> Indeed. "It was amazing that it worked, and terrible that we had to do it."

<comment><USER2> I believe it was James May that said: "You have come up with a clever solution to a problem you shouldn't have had in the first place!" =)</comment></comment></comment></comment>


A steampunk morse code Twitter client - <AUTHOR> http://getshacked.com/work/teletweet/

<comment><USER2> So, Google Tap is real and it's on iOS, not Android.</comment>