A fablab burned down in France by anarchists: Makery - <AUTHOR> http://www.makery.info/en/2017/11/28/apres-lincendie-de-la-casemate-la-communaute-des-fablabs-reagit/ <comment><USER2> Not sure why they're labeled anarchists though. If you burn something because it doesn't align with your own personal view of the world, you're a thug with authoritarian / violent tendencies and that's about it, no need to embellish idiots' actions with fancy ideology.

<comment><USER3> Arsonists are simply agitators with incendiary ideas.

</comment></comment><comment><USER4> For anyone that was confused while reading it, by fablab they mean a general workshop to build things, not a place to produce semiconductors.

<comment><USER2> Indeed. In that context, it's fab as in fabrication, which is the French word for the process of making / building something.

</comment></comment><comment><USER5> Further within the article is the message from the alleged arsonists:

"City managers cater to start-ups greedy for money and the fashionably geek masses by opening fablabs in trendy neighborhoods. These extremely diverse measures on the surface all aim to accelerate the general social acceptance and usage of the technologies of our sinister era. (…) And we couldn't care less whether these fablabs came out of the stale imagination of a revered hacker, which isn't the case, or whether they're participating in fruitful scientific collaborations with one of the temples of technocracy, MIT (Massachusetts Institute of Technology), which is the case—because they are a nuisance, we destroyed one"

</comment><comment><USER6> Seems like a French thing, create a deconstruction space. Unmaker? Razer Space!

</comment><comment><USER7> Instead of burning things down, how about building things instead?

<comment><USER8> Because they are opposed to technology and ever increasing systemic stability/tyranny

<comment><USER9> They're childish cowards. Childish because like young children they're not able to express themselves reasonably, so they resort to throwing tantrums. Cowards because they do it anonymously. Instead of opposing "systemic stability", they're encouraging it to harden.

</comment></comment></comment><comment><USER10> anarchists, or insurance fraud

<comment><USER11> If you read the article, it makes it difficult to imagine it was fraud. While it's possible there could have been a "cover-up" motive, it looks unlikely, but at least more likely than outright insurance fraud.</comment></comment>


Seth's Blog: The power of an algorithm - <AUTHOR> http://sethgodin.typepad.com/seths_blog/2009/02/the-power-of-an-algorithm.html <comment><USER2> I'm no Computer Science major, but I've easily wasted a lot of my time rewriting algorithms for n.

<http://www.cs.iupui.edu/~jschleut/n341/labs/#proto>

It took awhile for me to understand what _O_ -notation really represented until I found this primer: <http://www.csc.liv.ac.uk/~ped/teachadmin/algor/algor.html> ; these 5 bullet cover a lot of ground if you want to approach algorithms head-on, that is given you don't have a Masters in CSCI, but being of the self-taught taxonomy I went ahead and tested a few out on my own. . .in my browser. . .where _n_ is 27. . .using 2^ _n_ , which is essentially what a bubble-sort measures out at.

This is a bad algorithm, but I was curious how something like this would look in a browser. 2^27 = 134 217 728, which is roughly 130 MB. Firefox is willing and capable of running this while it could even run 2^28 = 268 MB. Anyways, don't delve too much into the topic, after awhile you'll recurse anything just to see what it looks like. . .worse than heroin.

<comment><USER3> It sounds like you're really very confused ...

      I'm no Computer Science major, but I've easily
      wasted a lot of my time rewriting algorithms for n.

I'm not even sure what that means.

      ... tested a few out on my own ... in my browser ...
      where n is 27. . .using 2^n, which is essentially
      what a bubble-sort measures out at.

No, BubbleSort wuns in n^2. The difference is significant.

      This is a bad algorithm, ...

No, it's only bad when the input gets beyond 10 or so, where 10 is not necessarily equal to 10. This is why O(n) notation is so important, so understand where the trade-offs occur.

      2^27 = 134 217 728, which is roughly 130 MB.

No, MB is a measure of memory, and you're talking about measuring time. Further, the exponential example on your web page is not BubbleSort, it's Fibonacci. That _is_ exponential, but the base isn't 2.

      Firefox is willing and capable of running this while
      it could even run 2^28 = 268 MB.

This is what convinces me that you're very confused about what is happening, and why. This stuff doesn't need a masters in CSCI or anything else. It's really very simple. It just sounds to me like you've read some bad tutorials, and done some irrelevant (although perhaps interesting) exercises, from which you've learned the wrong lessons.

</comment></comment><comment><USER4> Is it just me or did that blog post not really say much? Maybe I'm just missing something.</comment>


HP EliteBook: A friend to the User - <AUTHOR> https://ifixit.org/blog/9641/hp-elitebook-a-friend-to-the-user/ <comment><USER2> This kind of repairability has been a general feature of "enterprise" friendly laptops like ThinkPads and Elitebooks for a while.

I've had an Elitebook 840 for 4 years now, great laptop, very solid, still working well. The repairability is handy for me in two ways.

1) If I want to upgrade/replace the storage I can do that easily, so I was able to upgrade to a 512GB SSD without any problems.

2) As the laptop gets older and the battery inevitably starts to wear down, I can just buy a replacement battery.

All this means a much longer lifespan for my laptop, which is good for me as I like it as an overall device.

<comment><USER3> Repairability is a good thing. That being said, the 5 year old macbook pro I replaced had way better battery life than the then-new HP 840 G3 I am now using day to day.

</comment></comment><comment><USER4> Two models I'd like to add to this as examples that this is actually possible with all form factors:

First the The HP Elite X2, a Surface Pro-class tablet. Quite analogous to the EliteBook - sadly the RAM is soldered (IIRC one of their first actually still had a SODIMM slot):

[https://ifixit.org/blog/8135/hats-off-to-hps-repairable- tabl...](https://ifixit.org/blog/8135/hats-off-to-hps-repairable-tablet/)

[https://ifixit.org/blog/9309/hp- elite-x2-1012-g2/](https://ifixit.org/blog/9309/hp-elite-x2-1012-g2/)

And second, the Fujitsu Lifebook U93x. smaller and lighter than most Business- Ultrabooks - almost MacBook 12 size/weight, but still KabyLake U, >10h battery and everything socketed+lots of ports (including USB-A, HDMI, even full size Ethernet!):

[https://www.notebookcheck.net/Fujitsu- LifeBook-U937-Core-i5-...](https://www.notebookcheck.net/Fujitsu- LifeBook-U937-Core-i5-Full-HD-Laptop-Review.240080.0.html)

<comment><USER5> > The HP Elite X2, a Surface Pro-class tablet

This tablet is absolutely fascinating; it turns the "cost of repairability" of a pro-tablet into a material measurement : 100g on 700g (14%).

I wonder, all things being equal (which of course, it's not realistic), what's the impact on the attractiveness to the general (not HN) public. It would be significant even it if would conquer a "solid nieche".

To make an interesting comparison, a 12.3" Elite X2 weighs as a 15" Surface Book 2 (though the latter has a different form factor, while stil being in the pro-tablet category).

(~700g is the standard weight of a "pro-tablet": 12.x inches, without (cover) keyboard)

<comment><USER4> Indeed, this is something I've wondered myself, since in my experience even 100g does make a big difference in a tablet. But I've come to the conclusion - it's hard to really pinpoint the "cost of repairability".

Taking the numbers:

        IPad Pro 12,9: 692g
        Surface Book 2 13: 719g
        Surface Pro 2017: 770g
        Elite X2 1012 G2: 800g
        Surface Book 2 15: 817g

Of course, the IPad pro shines, and apple certainly are famous for "shaving off every possible gram" \- but it's just so different a platform, esp. thermals, that it's hard to make conclusions towards "cost of repairability". The Surface Book 2 tablet modules _are_ light for their size, but I'd say this is primarily bought due to a significantly smaller battery compared to the SP and the Elite (23Wh vs 46/47Wh) - batteries are _heavy_.

The only devices here realistically comparable IMO would be the Surface Pro and the Elite X2.

But IMO the more interesting device I've posted for this kind of comparison (even though I wouldn't consider +/-100g that critical here as I would with tablets) is the Fujitsu U937, because it's one of the lightest x86 notebooks overall on the market, while still offering competitive specs in all metrics (CPU, RAM, ports, battery size, display size, etc.) and still being easy to repair/upgrade. (Too bad that keyboard and touchpad are rather mediocre :( )

</comment></comment></comment><comment><USER6> Does anyone know what the story is for running Linux on EliteBooks?

For context, I'm in the market for a slightly older laptop for my daughter. Something that I could maybe pick up on ebay (in the UK) and upgrade some components would be ideal. Would prefer one I can put Linux on.

The Chromebook route turned into a pain when I tried it so I'd like a machine I have some control over.

<comment><USER7> Thinkpads might be a good option. Linux support is fairly good, especially the older models and the T-Series does have some models with upgradable RAM and battery. (Not the slim T4x0s)

<comment><USER6> I did have a look at Thinkpads but was somewhat overwhelmed by the number of models. Will have a look at the T series to see what's around. Thanks for the tip.

<comment><USER3> The W series are workstations. Solid, fast, heavy, expensive.

<comment><USER8> I think the workstations start with "P" these days. Good machines.

</comment></comment></comment></comment></comment><comment><USER9> I don't see the Dell XPS 15 9560 on their site but I was pleasantly surprised when I dismantled mine the other day (to re-do the thermal compound, which I can highly recommend btw).

It looks very similar to the HP inside and everything seems to be upgradeable / swappable - M2 SSD, MiniPCIE wifi / bluetooth module, proper socketed RAM, battery, fans etc. etc.

I was genuinely surprised considering it's a fairly sleek and compact machine for the spec.

That said, I didn't take apart the screen so I can't speak for the webcam or LCD panel.

</comment><comment><USER10> HP is not jumping to my mind at all when I am thinking about user friendly companies. Are they changing their image?

<comment><USER11> HP's EliteBooks and ZBooks (and Z-series workstations) have always been incredibly solid computers that are trivial to repair and and upgrade. However for some unfathomable reason HP has always insisted on primarily marketing its low end consumer line of laptops, which tend to be complete dog shit.

<comment><USER12> I'd say that's common though. A lot of pc makers have widely different quality standards for their consumer and business lines.

</comment></comment></comment><comment><USER13> On the flip side, the % of space taken up by the battery is tiny compared to macbooks.

One could argue that of all the components in a laptop, the battery wearing down is what most people will face, so having custom components + a larger battery might be better than having modular components.

<comment><USER14> Or just make the battery replaceable.

</comment></comment><comment><USER15> That's how all business line laptops are - Latitudes, ThinkPads, Elitebooks. That's how they've been for decades.

<comment><USER16> Unfortunately Levono has ditched this approach with the x280.

<comment><USER15> Not even remotely true: the keyboard is again a simple replace, unlike the in the X240-X270 line. Yes, the RAM is soldered, but that's all.

</comment></comment></comment><comment><USER17> Another counterexample to the "Apple make the best hardware" meme that is so often repeated. I find the whole concept of a disposable 1000+ dollar machine with a glued-in battery rather abhorrent.

<comment><USER7> The glued-in battery makes fixing it at home harder, but comes with significant weight and space savings. It's a trade off, like so many things and many people (including me) are willing to accept it. Apple is not the only manufacturer to do this, my Thinkpad has a non-user-serviceable battery as well. It's not like the battery cannot be replaced, and I'm willing to have my machine serviced at an authorized repair shop when the battery needs changing. In exchange I get a lighter and slimmer laptop that I lug around every day. The cost for a battery change on a MacBook is just slightly higher than the battery for the older models.

This may not be a trade off you're willing to make, but it does not make the machine disposable.

<comment><USER17> My Thinkpad Carbon X1 has a user-replaceable battery and is practically the same weight and size as a MacBook Air. So it is certainly not a compromise I agree with. The "disposable" label was a bit harsh, but there are other compromises in the MacBook Pro, for example the SSD, which is more prone to failure than other components, is soldered on to the logic board.

<comment><USER7> We may have a different notion of "user-replaceable". I'd only consider a battery that can be swapped without opening up the laptop user-replaceable. Soldering the components to the board is the same kind of trade-off: It saves weight, space and increases battery lifetime. You also compare the relatively new X1 with an outdated (though still sold) Macbook that was designed and built at a time when no laptop even got close to the size, weight and battery life of a MacBook Air, especially the 11" version. There were other laptops just as small, but they'd compromise on battery life, others that had the same battery life, but weighted more. Granted, it's certainly an extreme compromise, but I know more than a handful of people that would never trade their 11" Air for anything else. You can, btw., have the SSD changed, it's not soldered to the board for Airs. It's just an uncommon form factor that few vendors sell. See [https://everymac.com/systems/apple/macbook-air/macbook- air-f...](https://everymac.com/systems/apple/macbook-air/macbook-air- faq/macbook-air-mid-2013-how-to-replace-upgrade-ssd-storage.html) for example

<comment><USER17> On the X1 Carbon, you will have to remove some screws, but Lenovo provide a guide to the process: [https://support.lenovo.com/gb/en/solutions/pd103715](https://support.lenovo.com/gb/en/solutions/pd103715)

I believe the latest X1 Carbon weighs significantly less than a MacBook Pro.

<comment><USER7> You'd be surprised: 1.39kg for the carbon, 1.37kg for the MacBook (13") The latest Pro models are comparable to the Airs in weight.

</comment></comment></comment></comment></comment></comment><comment><USER18> My refurbished Zbook had problems with the screen (it looked like). I called HP, engineer came to my flat few hours later, replaced the screen for a new one. He tested it and realized that it's still flickering, so he replaced the GPU. Things worked well, so he left. All of this took ~ 45 minutes.

Ah, and it happened in CEE, not in the USA, so it's even more surprising.

<comment><USER19> What kind of special warranty did you get for them to do that for you?

<comment><USER18> Next business day onsite. It's a normal option for business laptops.

</comment></comment></comment><comment><USER20> I'm glad companies are finally coming back to this. Looks very similar to my Inspiron I bought a few years ago. Once you open up the back plate everything is laid nicely in front of you ready to service. I changed my RAM/HDD/Wifi NIC in a matter of minutes. In case you're wondering the model is Dell Inspiron 15-7537 from 2013.

<comment><USER21> I had a 2010 Inspiron, it was much worse back then. I remember that it overheated and I wanted to open it to remove dust and apply fresh thermal paste... I literally had to remove every single piece of it, they were all spread all over my room :D

<comment><USER22> That's strange, I have 2009 Inspiron (1545 iirc), where this is easy. There's a bunch of Youtube videos for that.

</comment></comment></comment><comment><USER23> Isn't that because their primary target audience is Enterprises? All enterprise computers specially the most sold of them all- thinkpads are amazingly easy to repair

</comment><comment><USER24> I really dislike labeling repairability as user-friendliness.

I never ever bought a device with intentions of repairing it, I always hope that I will never have to repair it and just use it.

Yes, it is nice to be able to repair it but if it fails, next time I'm buying a laptop I will avoid it and stick with the brand that did not fail and sits in my drawer because it's too slow by modern standards.

Repairability is probably useful for enterprises but enterprises are not users despite that sometimes they can be described as such(they are customers).

<comment><USER25> But it's not just repairing, it's being able to upgrade.

I have a 6 year-old Lenovo and it's still performing amazingly. Since buying it I have upgraded the HDD twice (to an SSD and then to a slightly larger and faster SSD), upgraded the RAM once and even upgraded the screen (to a higher resolution IPS display that was backwards compatible).

If I had bought an un-upgradable laptop I'd probably already have had to buy a new one (no matter how well it is built).

<comment><USER26> Yup. It's also surprisingly hard to find a decent new laptop these days.

About a week ago I started at a new company, and was asked to find a laptop that I wanted. The cheapest one with a 1TB SSD was ~$2000, and the cheapest one that would be decent out of the box was a $1300 Dell.

Instead I ended up getting a $600 gaming laptoo with a decent CPU, and installed the RAM and SSD myself for $450, ending up with a far better laptop that was still hundreds cheaper than that Dell.

Try doing that with a Mac.

<comment><USER27> The only gaming laptops I've seen that weren't _vastly_ inferior to both every Mac currently sold and also the EliteBook line from the OP in terms of battery life and portability costs between 3.5-5 times what you said you paid. Mind sharing what make and model you got? Thanks.

<comment><USER28> Many developer laptops probably live on a desk most of the time, so these two things are easy to sacrifice for many.</comment></comment></comment></comment></comment>


Stencila – Spreadsheet-like live reactive programming environment - <AUTHOR> https://stenci.la/stencila/blog/introducing-sheets/spreadsheets-are-dead-long-live-reactive-programming-environments- <comment><USER2> Hey, Stencila developer here. Thanks to the original poster for sharing the link and for all the interest. Unfortunately, the site is not handling all that interest too well (R session hosting instances filling up, timeouts,...). I'm working on it but please bear with me while I try to stabilize things.

<comment><USER3> Hey great work. I like that people are re-thinking spreadsheets as users get more technically inclined to use more sophisticated tools.

What do you think about fixing R notation to not describe cells, but the named array? i.e. instead of Average(A2:B2) it's Average(this_array) where this_array is the name for the row/column data and when the Average cell is clicked, highlight cells A2:B2?

There are many cases around detecting the proper array name or the user not inputing column or row names, but due to the ability to quickly read and understand the code as it relates to the sheet may make certain trade-offs more acceptable (i.e. forcing he user to enter array names).

<comment><USER2> Thanks for the feedback.

Your suggestion about named arrays is important and relates the discussion here around the approach taken in the OSX Numbers app and the now dead Lotus Improv. Its connected to a planned feature in Stecnila sheets that I call cell "mapping" \- effectively projecting an R/Python object onto grid cells. There is some more about that here: [https://github.com/stencila/stencila/issues/118](https://github.com/stencila/stencila/issues/118).

Also related is an proposed improvement in how sheet expressions are translated into R code for arrays: [https://github.com/stencila/stencila/issues/157](https://github.com/stencila/stencila/issues/157)

</comment></comment></comment><comment><USER4> I've been shifting a bunch of data analysis out of spreadsheets (and some adhoc SQL) to Jupyter/Pandas, and we've found some unexpected tradeoffs on both sides.

The lack of testability and version control (and really, long-term maintainability) is what drove us out of spreadsheets into Jupyter. We've found though that the workflow in Jupyter, even with Pandas, is not great for exploring and getting a feel for data --- we end up missing the very quick "poking around" you can do in excel or in a good sql client.

I'd have to use Stencila more to know if it strikes the right balance for the kind of analysis work I do, but I'm glad to see such a thoughtful attempt to try a new balance.

<comment><USER5> Try R and RStudio. It excels at 'poking around' data.

In addition, you can show any data frame as a spreadsheet of data, import/export CSV files, etc... Knitr, Shiny, Plotly and other technologies also make producing documents, graphs and whatnot super easy.

This Stencila also looks cool though. Spreadsheet + R....

</comment></comment><comment><USER6> Lotus 1-2-3 got people to use computers. One could argue that spreadsheets are the single biggest business innovation of the late 20th century. I don't think they're going away that easily.

I know people who type whole letters in Excel. I know an accountant in particular who I showed how to use MS Word for writing letters, but he prefers Excel. He writes full documents in Excel, prints them out and mails them.

Laugh if you want but spreadsheets are not going away in my lifetime.

<comment><USER7> Sooner or later, all project management ends up in Excel. Doesn't matter what ticket or tracking tool you've paid for.

<comment><USER8> My experience has been that it always ends up in Google Docs. Collaboration in Docs is just so stupidly easy compared to emailing spreadsheets around. And yeah I've been through several cycles of this:

        start tracking in google docs, we'll get a system eventually.
        -> OK we have bought a new PM system, let's start using it.
        -> Everyone please start using the new system. Move your google docs there today.
        -> I ... why is this still in Google Docs?
        -> Ok we've paid a bunch more money to get all of the requested features in our new system. Please please please start using it
        -> Sigh. Just share me on the Google Doc.

<comment><USER9> Purpose-built software systems allow you to ask a lot of important questions, but they don't allow you to ask all of the important questions. It's useful to have a tool that you can fall back on that gives you a basic representation of the data that you can manipulate with simple tools. For most people that will be a spreadsheet. You can also do it with *nix tools but they're really unapproachable for the uninitiated.

</comment></comment></comment></comment><comment><USER10> I think the more usual form of that phrase is "Spreadsheets are dead, long live spreadsheets!". And that captures the truth of the situation better.

Despite all the known weaknesses of spreadsheets, it is sheer hubris to believe that they will be supplanted by reactive programming (or databases, or anything else for that matter). It seems that users will give up their spreadsheets only when they are pried from their cold, dead hands.

European Spreadsheet Risks Interest Group has some great links to articles about the pros/cons/whattodo of spreadsheets: [http://www.eusprig.org/](http://www.eusprig.org/)

Jocelyn Ireson-Paine has done lots of work with spreadsheets and their problems. Lots of links to spreadsheet sites from there: [http://www.j-paine.org/](http://www.j-paine.org/)

<comment><USER11> Sorry, I can't resist indulging in a moment of pedantry here.

The phrase "The king is dead, long live the king!", which this title and many others play off of, is referring to two separate individuals, the now-dead king in the first clause and his successor in the second. The phrase is representative of a theory of monarchy in which, upon the death of the sovereign, their authority passes instantaneously to their heir, without the need for a coronation or a formal investiture, in order to foreclose on the possibility of pretenders to the throne arising during an interregnum.

So the correct (and I know you said 'usual and not 'correct', but bear with me) form of the phrase would be 'spreadsheets are dead, long live [whatever is succeeding spreadsheets]'. Of course, historically speaking, it was not at all uncommon for dead royals to suddenly reappear and reassert their claim to the throne....

<comment><USER12> Spreadsheet software is improving here and there. Excel today is not your old lotus-1-2-3. Although you'd be hard pressed to notice.

<comment><USER13> Indeed. After all the phrase is not: "The king is dead. Long live the republic"

</comment></comment></comment></comment><comment><USER14> Spreadsheets can be a pain, but be careful what you ask for to replace them...

Having contracted at two big american banks ( _cough jp, cough boa_ ) that decided to build their own proprietary technology sinking ship solutions ( _cough athena, quartz_ ), it is a horrible experience.

The business get sold on the idea of the "benefits" of this new "solution" so they pay lots for it. Lots and lots. Then all tech projects are guided towards it, otherwise it becomes a "political" problem if you don't use it, even if you can present a technical case.

Basically, it works, just about, if you are building the equivalent of a simple option spreadsheet pricer. Even then, the effort required should ring alarm bells. Still, job for life for those that maintain it...

</comment><comment><USER15> I think notebook environments like iPython provide greater flexibility in terms of programming. I don't think they can replace spreadsheet nor spreadsheet would develop functionality to replace notebook environments.

The missing link is a seamless experience between spreadsheet and reproducible data analysis (programming).

My team is working on [https://nxsheet.com](https://nxsheet.com). We look to provide this seamless experience. For example: Generate normal distribution - [https://nxsheet.com/sheets/56e845da4030182e337c6c2b](https://nxsheet.com/sheets/56e845da4030182e337c6c2b)

Stencila looks interesting. Great work!

<comment><USER2> Hey, Stencila developer here. Thanks, nxsheet looks really nice - great work too!

I agree, somehow getting spreadsheets into the workflow of reproducible data analysis is important. They are a tool that I for one had ignored (focussing instead of reproducibility of text + code) - but there are too many people that use them to ignore them!

</comment></comment><comment><USER16> Iteration on spreadsheets is great, and there's opportunity for tons of innovation in this space. But this suffers from the same problems that most spreadsheets have. Errors like not averaging over the right set of data (which you pointed out) don't come up because the code can't be diffed in git. They come up because there's not enough clarity in whether a set of cells are consistent or not. Making this text-based doesn't add clarity because ultimately there will be too much text to read.

Here's a test for the most common kind of error. If I add data in A5 and B5, will that data be represented in the chart? How about in the averages? Will I be able to even see that?

Here's the pivot: Break your data into regions, where regions have repeated elements. Something like this:

        A1 = 'Height
        B1 = 'Width
        C1 = 'Area
        A2:C2 = Region {
          C1 = =A1*B1
        }{
          A1 = 2
          B1 = 3
          A2 = 4
          B2 = 5
        }
        B3 = 'Average
        C3 = =sum(C2)/count(C2)

This could be used to generate:

        Height  Width   Area
        2       3        6
        4       5       20
                Average 13

The dataset associated with that sub-block can be clearly annotated to an editing user as such, with simple tools for adding a row to that dataset, or sorting it without effecting the rest of the sheet. Within the dataset, there's no corruption of the formulas (the third row's C cant be different than the second's), you've still got your diffing (probably better because it's clear that data changed but formulas different) and it's extremely hard to make the calculated average come out wrong.

Yeah, that exact representation isn't great. Maybe a "view" metaphor, so that headers and footers can be attached to the dataset instead of floating outside it. But once you've gone this direction, there's all sorts of amazing things possible by sharing datasets, linking them, transforming them, etc.

<comment><USER17> The Numbers app on OSX does something quite like this. It's highly underrated, but having a number of smaller, floating spread sheets in one big sandbox is waaaaay better for many things than a single sheet. I think the concept should be pushed even further.

<comment><USER18> The UI is clunkier than that of Numbers, but you can do something similar in Excel with tables (blocks in a sheet that you somewhat enforce some structure on. See [https://support.office.com/en-gb/article/Overview-of- Excel-t...](https://support.office.com/en-gb/article/Overview-of-Excel- tables-7ab0bb7d-3a9e-4b56-a3c9-6c94334e492c)).

And of course, Excel can flag cells that, according to its heuristics, seem to use inconsistent formulas, such as using empty cells in a summation, or having one cell in a column use a different formula.

</comment></comment></comment><comment><USER19> This article reminded me of an interesting video by Chris Granger, co-author of the Light Table code editor, titled "In Search of Tomorrow: What does programming look like in 10 years?" He's designing a visual/code block hybrid editor.

[https://www.youtube.com/watch?v=VZQoAKJPbh8](https://www.youtube.com/watch?v=VZQoAKJPbh8)

<comment><USER20> Thanks for the link! I've been working on the same problem as well (reinventing computing)—vastly different solution though.

</comment></comment><comment><USER21> Spreadsheets somehow hit a vein with the public, so that tells you something. I've been kind of obsessed with the Tup build system, and in trying to distill what it does into one sentence, I find it helpful to say, It's like a spreadsheet for files, where the formulas are shell commands. Even though I can grok FRP on its own, I still find that analogy helpful.

</comment><comment><USER22> I am personally a fan of Calca, not only does it do live coding, it can also do some powerful math:

[http://calca.io/](http://calca.io/)

<comment><USER23> I agree, and variable names (with spaces!) are much more descriptive than cell ids. I do wish for an open source version through.

</comment></comment><comment><USER24> Exciting stuff. I'm a financial analyst and I spend most of my time in Excel because I'm not about to send an R script to our executive team. Everybody understands spreadsheets.

But spreadsheets are terrible at everything except their original use case (small bore financial modelling).

Right now I'm transitioning a lot of my workflow to MSFT's Power BI tools, but I'd love to have a justification for using more R at work. Stencila could be that.

</comment><comment><USER25> Has anyone here ever used Lotus Improv? Does this program solve some of the flaws that this article highlights? I keep on hearing how Improv was one of the great programs that never took off, and I'm curious to now try it.

For those of you that don't know (or weren't even born yet), Lotus Improv was a spreadsheet alternative that was released for the NeXT computer in the early '90s that was well-reviewed but never sold well. It was eventually abandoned when IBM bought lotus in the mid-90's.

<comment><USER2> Hi, article author here. I have never used Improv but while researching for this work I did read about it and yes, it did seem to have introduced several innovations to address flaws in the spreadsheet approach. The OSX Numbers app seems to have picked up some of these ideas. And, I'm hoping to apply some of them to Stencila Sheets too.

</comment></comment><comment><USER26> This looks very sexy, and I don't say that often about technology. This is something I've wanted to see for a long time - the ability to seamlessly move from a spreadsheet to an application

Edit: Quick feedback

Cell names appear to be case sensitive. While this makes sense from a programming standpoint, if I can't have a cell named "a1" the application should convert that to "A1" for me.

Not being able to tab between cells is annoying. Was there a decision made not to capture the tab event?

<comment><USER2> @debacle, thanks for the nice feedback.

Regarding cell names. Currently every cell has an "id" e.g. A1 which is represented in the background R session as the variable A1. If you enter a formula like "pi = 3.14" into cell A1 then that cell has an id of A1 as well as a "name" (like an alias) of "pi". You can name a cell "a1" e.g. enter "a1 = 42" into cell A1 - but that could be confusing.

Yes, not being able to tab between cells is annoying - no decision was made on that - we just haven't got round to it. Just created an issue: [https://github.com/stencila/stencila/issues/162](https://github.com/stencila/stencila/issues/162). Thanks for the prod!

<comment><USER26> It might be good user experience to either make variable names case- insensitive (could be done any time before R), or default a1/A1 as the same pointer.

You'll really want to appeal to less technical folk, and they have trouble with case-sensitive things.

</comment></comment></comment><comment><USER27> It seems, perhaps, Resolver One's time has finally come. Alas, Resolver One (and Resolver Systems) died a slow death and ceased to be in 2012. Perhaps this will fare better.

I always thought Resolver was a brilliant idea, but they never gained any traction against the heavy weights (Excel).

Edit- Add link to Resolver One wikipedia article [https://en.wikipedia.org/wiki/Resolver_One](https://en.wikipedia.org/wiki/Resolver_One)

<comment><USER2> Thanks for the link. I was aware of Pyspread (which sound similar) but not of Resolver One - I'll add a link in the article

</comment></comment><comment><USER28> I think another change is needed for spreadsheets to be a more reliable tool.

Right now a sheet is a full document thing. Meaning that it starts with A1 in top left, and spreads out from there.

but what if you could break a "sheet" down into units?

So that you can have say a constants unit, with its own A1 to whatever as needed, and then another unit, perhaps called results, that hold the formula in their own A1 to whatever, each unit movable and extendable on the screen at the same time.

Now if formula A1 wants to refer to something in constants A1, the entry would look something like constants(A1).

This way one would have everything on screen, while avoiding the worry that as the sheet grows and one move things around to make it more readable, the formulas are referencing the wrong cells.

<comment><USER29> This is literally exactly what Numbers.app on OS X does.

<comment><USER28> And its stuck to OSX, yay...

<comment><USER30> There's iWork for iCloud, but I'm not sure what the deal with it is if you're not a Mac user.

</comment></comment></comment></comment><comment><USER31> One project I've recently been wanting play around with is Beaker, which touts itself as a polyglot data science notebook tool. The pitch is that you can use the right tool for the job, no matter what questions you want to ask next to each other. It looks like an iPython notebook on steroids.

[http://beakernotebook.com/](http://beakernotebook.com/) [https://github.com/twosigma/beaker- notebook](https://github.com/twosigma/beaker-notebook)

Disclaimer (or not): no involvement, but looks very handy.

</comment><comment><USER32> This looks very similar to AlphaSheets and the thing I was making over spring break until my friends called it stupid.

Very interesting!

<comment><USER33> > the thing I was making over spring break until my friends called it stupid.

Those sound less like friends, and more like people holding you back from curiosity and creativity, which can bring many direct and indirect fruits.

<comment><USER32> Eh, they'd rather have me work on shady cryptocurrency projects instead.

</comment></comment></comment><comment><USER34> Off-topic: how do people make those animated gifs like the one in the blog post ([https://stenci.la/stencila/blog/introducing- sheets/screencas...](https://stenci.la/stencila/blog/introducing- sheets/screencast.gif))?

<comment><USER2> Hey, author here. I used byzanz for this one: [https://www.maketecheasier.com/record-screen-as-animated- gif...](https://www.maketecheasier.com/record-screen-as-animated-gif-ubuntu/) . For snazzier stuff, Screenflow is good.

</comment></comment><comment><USER35> This kind of makes me want to brush off POI and attach it to Scala to develop a git-able way to create spreadsheets.

Create your spreadsheet, import it to an IDE, and create a way to intelligently create reports that can be connected to business intelligence suites... Someone must have done this.

</comment><comment><USER36> Can you make this in your spreadsheet software:

[http://www.quertime.com/article/arn-2012-08-22-1-25-drawings...](http://www.quertime.com/article/arn-2012-08-22-1-25-drawings- and-games-made-with-microsoft-excel/)

<comment><USER2> Hey, that's really cool, thanks for the link. Right now you can't do that...but give us a few weeks. We are planning to implement conditional cell formatting using CSS i.e. cell CSS styles as a function of R expressions [https://github.com/stencila/stencila/issues/97](https://github.com/stencila/stencila/issues/97). Once that is done, it would be fun to see if we could create these types of pretty picture.

</comment></comment><comment><USER33> very well done! a plaintext spreadsheet. I think it has a lot of appeal, and for all the reasons outlined. Looking at it raw, is fairly understandable. Awesome! Coolest thing i've seen all month on HN.

</comment><comment><USER37> hi - quick question. Is your Dockerfile open source ? We have been building an R powered backend for one of our analytics services and it's been a struggle getting a scalable R-based infrastructure going.

BTW - another killer usecase would be an easy way to build database backed spreadsheets. instead of the TSV, it would be cool to figure out how to interface to postgresql for example.

<comment><USER2> Hey, yeah it's all open. The Dockerfile is here: [https://github.com/stencila/stencila/blob/master/docker/ubun...](https://github.com/stencila/stencila/blob/master/docker/ubuntu-14.04-r-3.2/Dockerfile) I'm no expert, but it seems to be doing the job. Would appreciate any suggestions you might have! I'd like to make it smaller i.e. use a smaller base image.

<comment><USER37> just one comment - your entire js is basically building a reactive workflow through vanilla js. it is a very commendable piece of work!

however, I'm wondering if using Reactjs+Redux would not reduce the amount of js by many orders of magnitude. You would get all your reactive processing for free.

EDIT: your CPP code is quite cool ! But same comment there, you could probably use nodejs and npm packages to get this out of the box for free. for example - git.cpp -> [https://www.npmjs.com/package/git](https://www.npmjs.com/package/git), frame.cpp -> [https://www.npmjs.com/package/dataframe](https://www.npmjs.com/package/dataframe), http-client -> [https://github.com/mzabriskie/axios](https://github.com/mzabriskie/axios) \+ [https://github.com/petkaantonov/bluebird](https://github.com/petkaantonov/bluebird), etc.

</comment></comment></comment><comment><USER38> Do you think named cells and ranges in Excel / Google Sheets gets at this problem?

<comment><USER2> Yes, I think that they certainly help at making spreadsheets more readable. At present Stencila sheets only has named cells but one of the things we want to add soon is mapping of cells onto underlying `data.frames` or matrices. In other words to "project" data objects onto the spreadsheet grid.

</comment></comment><comment><USER39> Awesome.

</comment><comment><USER40> Misspelling Dan Bricklin's name within the first four words gives me low expectations…

<comment><USER2> Hey, thanks for pointing that out. Fixed now. I actually got it wrong in all three places I mentioned him :(

</comment></comment><comment><USER41> OT and just my opinion: it seems to exists an unfortunate trend in the promotion of new ideas where, instead of simply exposing the virtues of the idea on its own, it first needs to knock down the (proven and world adopted) predecessor as something unpolished, unplanned, untested and, in general, a bad idea from the start.

Compare this trend, for instance, with the first Linux announcement by Linus Torvalds [1] below.

It is an unfortunate trend because it ends up polarizing and creating unnecessary division among the early adopters of the new idea and the current userbase of the old one.

 _From: email@example.com (Linus Benedict Torvalds)_

 _Newsgroups: comp.os.minix_

 _Subject: What would you like to see most in minix?_

 _Summary: small poll for my new operating system_

 _Message-ID: <email@example.com>_

 _Date: 25 Aug 91 20:57:08 GMT_

 _Organization: University of Helsinki_

 _Hello everybody out there using minix –_

 _I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like /dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things)._

 _I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and I'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-)_

 _Linus (email@example.com)_

 _PS. Yes – it's free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(._

[1] [http://www.thelinuxdaily.com/2010/04/the-first-linux- announc...](http://www.thelinuxdaily.com/2010/04/the-first-linux-announcement- from-linus-torvalds/)

<comment><USER42> The article itself is much more positive on spreadsheets than the headline -- it suggests improvements to the underlying data / computational model while preserving the interface.

And it's about time somebody said something nice about spreadsheets. Go to your average (non-software) engineer and ask to see an engineering model. Chances are, it's in Excel. It's less buggy than you expect, because your average engineer is going to notice and fix nonsensical results. Unlike the scientists the author is talking about, your average engineer is looking at very similar data from day to day and is attuned to anomalous behavior.

I think there's a lot of room for a tool like the one described by the article that includes built-in validation and is git-friendly. However, given the overwhelming dominance of Excel, it's going to have to integrate with, rather than replace Excel.

<comment><USER2> Thanks for the comments. Actually, I was just working on importing Excel spreadsheets into Stencila sheets last night. Some interesting aspects involved in translating Excel formulas into R expressions e.g. `AVERAGE(A1:A5)` to `mean(A1:A5)`. To do it properly looks like we'll need to build an Excel formula parser. But do-able - and could be fun!

<comment><USER42> Interesting! I imagine there's a more-or-less reasonable subset of Excel formulas you could start with. I look forward to seeing what comes of this!

</comment></comment></comment></comment><comment><USER43> Since that title is baity, we changed it to something more neutral in accordance with the HN guidelines:

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

If anybody suggests a better title, we can change it again.

</comment><comment><USER44> Spreadsheets are optimized for data entry. Try entering CVS data manually using a text editor.

Without Excel there would be no way for my sales/marketing team to give me valuable data to process at the backend.

If you have been doing programming in excel for the past 20 years then you are simply a fool - sorry.

<comment><USER35> You would be surprised how many people have done this. The average office worker doesn't have a development environment on their computer, but they have Excel, and they have a coworker who was a whiz who said, "Did you know you could do this?"

They just kept automating task after task, silently, without the IT department ever knowing. 90% of them never thought of it as coding, and 10% realized they could make more money as an expert who knows how to coerce excel combined with their domain knowledge than they could as a programmer.

<comment><USER16> This.

I've been at top 20 banks in the world that run 14 hour jobs in Excel that import data from web services, run analysis, and FTP result sets elsewhere. They were used to analyze risk on tens of billions of dollars worth of securities.

They were riddled with bugs, but it let the front-line traders iterate quickly and prototype something that the backend team (my team) could turn into a "real" reporting process.

<comment><USER2> Absolutely. I think it is the rapid prototyping aspect of spreadsheets that is perhaps the most powerful. For analysing large amounts of data they are obviously not great. But they excel (pardon the pun) at providing an environment in which to quickly formulate and visualise numerical models - because they are reactive so you can see what happens when you change inputs - without having to recompile, or manually rerun code.</comment></comment></comment></comment>


Coronavirus May Kill Our Fracking Fever Dream - <AUTHOR> https://www.nytimes.com/2020/04/10/opinion/coronavirus-texas-fracking-layoffs.html <comment><USER2> As I understand it, fracking doesn't even make economic sense at $70+ oil... The whole industry uses accounting scams to stay alive.

</comment><comment><USER3> Fracking produces lots of methane and pollutes lots of water while being expensive, we can do away with it.

</comment><comment><USER4> > Coronavirus May Kill Our Fracking Fever Dream

So at a time where Coronavirus has showed us the importance of supply chains not being overseas for national security we are learning to not be energy independent?

It's almost like someone takes a premise "Fracking is bad" and adds words around it to keep people happy with that premise and gets... clicks?

<comment><USER5> I don't think the article is suggesting that the US not be energy independent but rather just that fracking isn't providing that.

<comment><USER6> Energy independence matters when oil is scarce and trades at $100/barrel or more. Not when oil trades at $20/bbl and there are places where it has a negative price.

<comment><USER7> It matters, but not at the cost of polluted groundwater, people getting sick, and triggering earthquakes. There are other ways; nuclear, solar, and wind are among them.

</comment></comment></comment></comment><comment><USER8> Fracking is expensive and dirty and benefits the worst types of people in the US.</comment>


Video calls for Signal out of beta - <AUTHOR> https://whispersystems.org/blog/signal-video-calls/ <comment><USER2> Recently: [https://news.ycombinator.com/item?id=13645980](https://news.ycombinator.com/item?id=13645980)</comment>


Is HN the new Techcrunch? - <AUTHOR>

I see more depth and breadth of quality information coming out of HN than from Techcrunch, which used be the dominant channel in my daily routines.<p>Thoughts? <comment><USER2> For my money the answer is yes! What Techcrunch did well back in the day is that it covered the startup scene -- HN improves upon that by acting as a link toolkit of sorts. Also unlike Techcrunch which was top/down this site is really about the community (i.e. the quality of the comments) so you've captured some of the magic of reddit/digg. What's interesting to me is that this has nothing to do with the technology, but with the curation skills of the community.

</comment><comment><USER3> No. Pass the word. HN is _not_ the new Techcrunch. These aren't the droids you are looking for. Move along. Move along.

</comment><comment><USER4> I'm not convinced. Isn't HN the new Digg, and Digg was the new Slashdot?

These 3 sites (HN, Digg, Slashdot) are all user submitted stories. Techcrunch is just one source of many for those stories.

<comment><USER5> Digg is nothing like HN. Granted, they are popular user submitted story sites with comments, but it's about the community. I visited Digg for a few weeks then stopped going because there was frankly lots of crap and trolls on the site. That is not to say that HN doesn't have some bad cookies here and there, but I got fed up with Digg when I saw lots of ignorant comments that were just meant to be hateful rather than for a positive discussion (particularly in the religious domain). HN for the most part, stays on the topic of startup/tech news, and has reasonable discussions.

<comment><USER4> True, but going back a few years, Digg had a very similar community to HN. As it branched out into other topics it really lost focus and is the site we never visit today :)

Many HNers are former Diggers. It's just been a few years.

<comment><USER5> Ah well I guess I saw the site post focus loss.

</comment></comment></comment></comment><comment><USER6> I would say no. For a while now Apple/Twitter/Facebook have dominated TC. TechCrunch has shifted from startup/funding focused stories to more mainstream tech news.

HN will always have an awesome community that TechCrunch never had. HN is a place to exchange ideas and read about what's going on in the startup industry.

I sure hope this place doesn't become the new TechCrunch...

</comment><comment><USER7> If I end up at TC these days it's because of a link here.

I used to check web20.originalsignal.com multiple times a day to keep up with TC (and the others) but their constant forcing of crunchgear/etc exerts into their RSS feed turned me off that, and their topic-incest turned me off actually going to the site a long time ago.

</comment><comment><USER8> Apples and oranges in my opinion. Techcrunch is not a community driven site but rather a contributer driven site.

<comment><USER9> I think Techcrunch is community driven because you read the comments more then the actual posts themselves.How many times have you ''Skip To comments'' on TC

</comment></comment><comment><USER10> Yes, unfortunately.

HN lately is much more about entrepreneurship and startup PR than it is about crafting software.

Not that it's bad, it's just that the ratio has gone out of hand. I question wether "Hacker News" is an appropriate name for this place anymore.

</comment><comment><USER11> Are there bloggers here who primarily submit to HN for their audience? And if so, if you listed them all, how would HN stack up to the Techcrunch roster?

</comment><comment><USER12> First Jeff Atwood kept posting all the links to HN in his blog. Now people are perceiving this is the new TechCrunch? Wow.

</comment><comment><USER13> my level of agreement could be qualified as "strong" to "quite strong".

</comment><comment><USER14> HN is HN.</comment>


Traditional TV Is in Trouble - <AUTHOR> https://www.nytimes.com/2018/05/13/business/media/television-advertising.html <comment><USER2> I feel like the decline of traditional television is behind a lot of the push to destroy net neutrality.

The strategy being to first introduce some artificial scarcity on internet bandwidth through data caps, and then offer exceptions. Either the cost to the business to be on the exception list will be high, or the cost to the user will be high to override the data caps, with the cost of service increasing as oligopoly in a region allows.

They can use this to steer people to the services that they back, like Hulu, or at least to make their own cable-box video on demand services seem more reasonable.

If one were truly pessimistic, the categorization of content wouldn't stop at simply high bandwidth services like video. A separate limit could be applied to things like html, with websites indexed as to their type and views counted. Which could result in an online equivalent of Sinclair media's control of local television news as certain channels of information are more accessible because they don't count towards a cap. And anything untrackable like VPN use goes into a more expensive version of the existing bandwidth-based quotas.

I believe it will not come to that last part, but less because I don't believe the telcos would attempt it than that I think it would incite people to legislative action. If it's introduced in slow increments, though, we might get closer to this worst-case scenario than many believe.

<comment><USER3> Traditional television is behind the push to destroy the net in general.

Look at us here in Australia - look up "NBN". we had a plan (that was already 10% complete and working great) to install gigabit fibre in 93% of homes. Then Rupert Murdoch handed the election to the Coalition, in exchange for them:

\- Immediately stopping work on the fibre network

\- Replacing the bulk of the remaining rollout with a VDSL system (ancient technology, my parents got it and instead of gigabit fibre their internet went from 13 Mbps to 19 Mbps - and this costs 2400$ per house to install)

\- Using loans and taxpayer money to expand the Fox cable TV\internet network to double the number of homes, and paying Foxtel hundreds of millions of dollars for the privilege, while continuing to grant them exclusive rights to broadcast on it on all frequencies they want, including the ones that are optimal for internet

\- Purchasing and decommissioning the Optus cable TV network which competes with it, replacing it with VDSL service

It's a national tragedy, wasting fifty billion dollars and a decade, just for the sake of protecting a conservative TV monopoly.

<comment><USER4> Do you have any sources on this? I always wondered why NBN was such a disaster.

<comment><USER5> You know not what you ask!

I didn't post what you're replying to, but I'm also an Aussie who's been reading articles about the NBN disaster for its entire duration.

Sources are difficult because the NBN has been on on-going project for over a decade now, so there's an absolute haystack to search through, and all the hay and most of the needles are strongly coloured by political ideologies.

Whirlpool has what looks like a comprehensive timeline / article here: [https://whirlpool.net.au/wiki/nbn](https://whirlpool.net.au/wiki/nbn)

I always found [http://delimiter.com.au/](http://delimiter.com.au/) to be the best source of information around the progress (or otherwise) of the NBN. Here are a couple of semi-targeted articles that demonstrate the depth and density of this political quagmire:

[http://delimiter.com.au/2013/01/17/fact-check-the-nbn- wasnt-...](http://delimiter.com.au/2013/01/17/fact-check-the-nbn-wasnt-a- media-stunt/)

[http://delimiter.com.au/2013/07/19/breaking-the-rules-how- nb...](http://delimiter.com.au/2013/07/19/breaking-the-rules-how-nbn-cos-spat- with-turnbull-breaches-all-convention/)

[http://delimiter.com.au/2016/03/29/qa-panellists-agree- polit...](http://delimiter.com.au/2016/03/29/qa-panellists-agree-politicians- screwed-fibre-nbn/)

The whole issue goes back before NBNCo was created, however, to when Telstra, the government-owned monopoly infrastructure owner, was sold without being broken up into wholesale and retail arms, thus meaning their retail plans directly competed with those of their wholesale customers. The Australian Government at the time didn't see that as being a potential problem in the future, and yet here we are...

<comment><USER6> There is an interesting comparison with what could have been. When I moved to New Zealand in the early 2000s, Internet access was worse than in Australia.

"Broadband" was 256kbps ADSL at best, insanely expensive, with punitive data caps, and the monopoly telco had no interest in investing to increase service or drop prices. There were other ISPs but they had a hard time competing due to underhanded tactics by the monopolist as well as having to build out their own infrastructure entirely whereas the monopoly had inherited all its infrastructure from the taxpayer built state owned company that was privatised for a song.

Around 2007, the government removed this monopoly power and broke up the company into infrastructure and service companies, with the infrastructure company having to provide equal access to any ISPs.

Then the initiative to build fibre to the home started in 2008. Luckily, the government of the day did not compromise on the technical approach and it remained fibre to the home, not the node.

Roll forward to today, and I have gigabit fibre to my home, with no data caps, for a decent price.

There's still a fair bit of build out to happen but the option now exists for people in most urban areas.

<comment><USER5> I've read the occasional article about the New Zealand rollout. Rest-assured, the Aussies who care about the NBN are jealous of what New Zealand now has.

It really pushes the point home about how much politics and special-interests (as opposed to real things like technical limitations) got in the way.

Now I'm angry all over again.

</comment></comment></comment></comment></comment></comment><comment><USER7> I think the perfect example is "The Expanse"

\- Easily the best sci-fi show in 10 years(1)

\- Latest season has a 100% score on rotten-tomatos

\- The visual effects are absolute top of the art (2)

\- The fanbase is franticly close to the show (look at twitter how much they interact w/ the cast etc)

\- Season 3 and we are about to start the big turn of the story arch

BUT… Canceled by Syfy because not enough people catch it at the time they happen to transmit it into the air.

If this show would be on Netflix it could become on of the defining sci-fi shows of our generation. Now we have to hope Netflix or Amazon happen to pick it up before the team disperses. (3)

(1) seriously watch it… the first few episodes are a bit slowish worldbuilding but it gets really good

(2) [https://www.youtube.com/watch?v=6yKn_EqA0ik](https://www.youtube.com/watch?v=6yKn_EqA0ik) (shitty rip unfortunately)

(3) Please sign: [https://www.change.org/p/netflix-or-amazon-please-buy-the- ri...](https://www.change.org/p/netflix-or-amazon-please-buy-the-rights-to- the-expanse-savetheexpanse)

<comment><USER8> While I love RottenTomatoes for Movies, I find it abysmal for TV Shows. Here are some shows with a >90% score:

\- Almost all seasons of Supernatural

\- Seasons 3 & 4 of Veep

\- Seasons 4, 5 & 7 of New Girl

\- Season 2 of The Flash

\- Season 2-4 of Arrow

Now I'm not saying that those shows are all shit (tho they are certainly not in my ballpark), but I doubt anyone thinks of them as 100% shows, in the same sense that Moonlight is a 98% movie.

The problem comes with few reviews (which is also a thing with more niche-y movies), which is probably amplified in later seasons, where reviewers who don't like it stopped watching.

<comment><USER9> To a far greater degree than the vast bulk of movies, TV shows, especially genre ones, spawn pretty fanatical followers. And, as you suggest, rather than ragging on a series for wasting 2 hours of their life, they just stop watching.

Among the series you listed, I see several that, for me, were basically meh superhero shows I might have watched an episode of and then tuned out. There's too much good TV for me to bother with stuff that I suspect I'll be indifferent to in the 5 hours or so if TV I watch in a week.

</comment></comment></comment><comment><USER10> None of this discussed the 'why' traditional TV is in trouble, it just said that younger viewers aren't there. The actual WHY is that it is a terrible product which is being successfully outcompeted by a much better product. Television networks got large because they solved a very valuable problem - distribution. With the Internet, distribution is worthless. A clever 12 year old can do global media distribution on their free time as a hobby. It's not something to build an empire on. And it's certainly not something people are going to pay you $100+/mo just so you can lace it with ads and fill it with garbage content that exists not because an audience is wanting it but because some programming director wanted to make a name for themselves by going in a 'bold new direction'.

Public response to creative works is random. Truly random. Maximum entropy. There are no patterns to discern. So humans do what they always do when they face randomness.. they invent myths and replace reason with bullshit. If you control distribution, you can still force a win by leaning hard on your successes and kicking out your losses as fast as possible. But once the audience has extremely broad choice... you are screwed.

The future for media distributors is death. The future for media production companies is probably either working for Netflix, Amazon, or similar services or being supported by viewers who want their content directly via Patreon or a similar avenue. The creatives will be fine. The suits will be jumping off ledges.

</comment><comment><USER11> What's most fascinating to me is that the 50-64 age group spends vastly more time overall viewing nearly every medium except streaming TV and video games.

My own experience with my parents bears this out; the TV is always on and they're also on their phones as much or more than teens. But it looks like if habits don't change, there will be fewer hours overall spent on "consuming content" as the population ages.

<comment><USER12> I wonder if the real reason advertising companies are worried about millenials is that they just avoid advertising whenever they can. There is a growing awareness about the effects of advertising, and the popular trends are towards reuse and recycling, even if those trends are mostly bullshit.

In Australia we have a popular TV show called Gruen Transfer which does nothing but breakdown ads and explains how they do what they do.

Finally, a biased anecdote from a millenial: Myself and many of my friends avoid ads by using ad-blockers and subscription services and not leaving the TV on when there is nothing to watch. Whereas my parents (about 60) watched far more television and didn't use ad-blockers... although recently they have been turning the TV off and using their ipads instead.

<comment><USER13> _I wonder if the real reason advertising companies are worried about millenials is that they just avoid advertising whenever they can_

I don't think advertising companies are worried about millennials; they can target them and measure results far more effectively with online advertising than they ever could via television, and thanks to smartphone addiction, screen time/ad exposure time is the highest it's ever been.

 _Myself and many of my friends avoid ads by using ad-blockers and subscription services_

That may be true of sophisticated users like yourself and your peers, but I'm fairly sure you're in a small minority. Most people just use Facebook and Instagram and Snapchat and YouTube and read news articles unfiltered, and outside of that, brands can insert their message directly into the content via sponsorships and product placements with online celebrities.

Broadcast TV may be declining as millennials spend their time and dollars elsewhere, but the advertising industry has never been stronger.

<comment><USER14> idk about ad companies. But publisher are kinda worried about ad blockers :), I head numbers as high as 50% of users with ad blocks. normies use ad block now.

<comment><USER15> I expect then publishers love the Facebooks where ad-blockers are impotent.

<comment><USER14> i havent seen an ad on facebook (the website, i do not use apps) for quite some time.

<comment><USER13> Yep but regular folks do much/all of their Facebook/Insta/Snapchat viewing in the apps not the web.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> What I find weird about most of my generation is not that they don't buy overpriced cable subscriptions, but that they don't buy _TV screens_ and instead choose to stream video at home on a 15" laptop screen at best or a 4" phone screen at worst. TVs are obscenely cheap these days -- to the extent that it blows my mind that anyone doesn't have one.

(And then once you have a TV screen you might as well pay another $10 for an HD antenna, which gives you completely free access to unlimited content, most of which is admittedly total garbage. But even if you use it once a year to watch the Super Bowl, Olympics, or live coverage of some local emergency, the tiny cost is easily justified. The existence and free-ness of HD antennas seems more or less unknown to most millennials, who have a "mind blown" reaction when they come over to my apartment to watch the Super Bowl or NBA finals or whatever and find out I'm paying nothing for it.)

<comment><USER17> A big part of the problem is effortlessly getting content from your phone (where it's easiest to find) to your TV (where it's best to watch) through the competing hostile ecosystems of Apple/Google/Amazon/Facebook.

<comment> King-Aaron My HDMI cable would disagree with you on that one.

I understand a lot of people want the convenience of wireless / push to screen functionality, but it always makes me laugh when friends sit there for several minutes trying to get content on a screen, and I just bung a cable into my laptop and it's up and running.

<comment><USER18> In last few years, you can also connect HDMI cable to your phone; MHL has been a thing for a while.

</comment></comment></comment></comment><comment><USER19> One of the reasons traditional TV is in trouble is because it is a pain in the neck to deal with. The episode-at-a-time schedule is frustrating, now that I'm used to binging on Netflix. To get around this, I tend to watch network TV shows only at the end of the season, so I can save up an entire season and binge watch the show (I have a PVR and an antenna). However, you often find that some or all of your season is missing because the networks can't keep to a schedule.

Just last night, Golf on NBC pushed their primetime schedule back by 10 minutes, causing me to loose the last 10 minutes of the series finale of something that I was binge watching. This is ... disappointing.

Similarly, a previous episode of the same show was been 100% interrupted by overzealous local news/weather folks covering phantom "severe weather" which was basically some high wind and lighting.

Except for sports, I'm now planning to just ignore linear TV, and wait for the shows to appear on streaming services, which actually deliver the content..

<comment><USER20> The idea of "channels" seems so archaic now. Why should extended news coverage push back the schedule?

News going long? Spawn a temporary, new channel and the coverage can continue there for those interested while the scheduled content continues on the main channel.

I know why things are the way they are, but we have computers now: we can do anything.

<comment><USER21> > we have computers now: we can do anything.

That's the title of my upcoming self-help book for millennials dealing with their aging baby-boomer parents' technology issues.

<comment><USER20> Will there be a chapter called, "Can't you just...?"

<comment><USER22> Brilliant. This phrase is so loaded with passive-aggression and condescension.

</comment></comment></comment></comment></comment><comment><USER23> > It's hard to keep up with the many devices and apps people now use to watch > shows. And there is a host of material from Silicon Valley that is competing > for viewers' attention, including Google's YouTube, Facebook and Netflix.

It's very hard to keep up with streaming services and knowing what's where too. It's a big mess since the streaming services seem to provide control on what to watch, when and where, but that turns out to be a kind of illusion when one can't figure out how to achieve that with content either moving around services or being produced and restricted to one service. While BitTorrent traffic may have reduced over the years because of streaming services, it still reigns supreme if you want to choose what to watch, when and where.

> Both NBCUniversal and the Fox Networks Group have said they will trim the > total time of commercials shown during some of their shows; Fox has > announced a goal of reducing ad time to two minutes an hour by 2020.

This is one of the worst aspects of traditional TV once you become accustomed to streaming services, especially ones with no ads. Traditional TV literally puts ads for about one third of the air time in many cases, and that's a huge waste of time for those who just want content (and there are many; how many times can one watch the same ad even if it's the best ad ever?).

The writing has been on the wall for a long time coming. Better user experience, with lower prices, will eat into traditional modes of content distribution.

</comment><comment><USER24> I tried watching a Star Wars movie the other day. The play time on TV was 7-10:30pm, 3.5 hours. The actual play time of the movie is 2.25 hours. And they cram most of the commercials towards the end. They do it so much so that I got sick of them cutting to commercial every 5 minutes and pulled out the blu ray and finished watching it. There was still 40 minutes left on the TV after I'd finished watching the blu ray.

Here I am, willing to pay for cable and watch it, and they ruin it by trying to milk every last cent out of it.

<comment><USER25> I thought pay tv in the US had no ads?? Is that not the whole point of it? Are there ad breaks in game of thrones and silicon valley and all the other hbo shows??

<comment><USER26> Regular bundled cable has ads. (What IS the point?) HBO is a premium channel you pay for beyond the bundle. It has no ads except lots of self-promo in between shows.

</comment></comment></comment><comment><USER27> As a millennial, I can proudly say that I never cut the cord, because I never even signed up.

<comment><USER28> Same here, but gen Z instead. Only been out of the house for two years (i'm 20) but I have never had an inkling of a desire to sign up for television service. I'm sure part of that is skewed by torrenting, but I still probably wouldn't sign up if such things weren't possible.

<comment><USER29> Stick to it. In your latter years you'll never wish you had watched more TV, but you might fondly remember that moment you walked on a beach at sunset instead and had an amazing encounter.

Same applies to your phone too though :)

<comment><USER30> Most of the time when you walk on the beach, you don't have an amazing encounter.

There's a long list of television shows I haven't started yet and wish I had. Game of Thrones, Sopranos, The Wire, Mr Robot, and lots of others. It's surprising how often conversations with friends and strangers revolve around our common culture.

I think a lot of people my age credit their love of sci fi while growing up in the 70's and 80's with putting them on their career path.

My advice would be to watch the TV you want to watch, then take your dog out for a walk after. Best of all worlds.

<comment><USER28> Man, this is literally my Plex library. <15 shows that I ever enjoy watching (ATHF, Mr Robot, The Wire, Sopranos, GoT, Squidbillies, House, King of the Hill, Silicon Valley, Buffy, Futurama to be exact.) They still take up a full 1.5TB because I like to get the highest quality rips possible.

Some shows like House I have edited personally to create 'binge versions,' with the intro and outro cut out, with only special episodes that have special endings or intros included. Saves a good chunk of time with House and The Wire when I rewatch them. I wish a lot of other people did this as well. Instead groups stick to following scene rules that leave us with ugly formats, poor management and naming schemes and more. They were created for a different era.

Maybe I'll try to put it out there.

</comment></comment></comment></comment></comment><comment><USER31> The dopamine-per-minute is so much higher with smartphones. Can't compete with the super-instant-gratification.

</comment><comment><USER32> [https://en.wikipedia.org/wiki/Top- rated_United_States_televi...](https://en.wikipedia.org/wiki/Top- rated_United_States_television_programs_by_season)

Going all the way back to 2000, it's pretty clear that the top rated tv shows have always skewed towards older demographics. ER, CSI whatever, Lawyer drama #25.

<comment><USER33> That and the top raised shows have fallen dramatically, in the 50's you had 60 million people watching now it's capping under 12 and steadily dropping. Not to mention how much larger the population has gotten.

<comment><USER34> Maybe the traditional networks would make a decent comeback if they started producing view-worthy shows again. There was so much content actually worth watching and getting excited for one or two decades ago, now at most I'm looking forward to the next episode of Bob's Burgers as far as domestic network shows are concerned. Anything else worth watching to me comes from Netflix or Amazon, or premium networks that are already preparing for the death of cable like HBO.

There just isn't a modern day equivalent to ER, and THAT is why TV is dying (along with absurd price hikes).

<comment><USER35> I'd argue another angle to it, which is there were fewer channels to choose from back then.

Modern shows aren't just competing to win their timeslot; they're competing against the choice to stream whatever you feel like.

<comment><USER34> Maybe I'm just overly picky, but I am constantly at a loss as to what I want to watch. My wife gets more use out of our streaming services than I do, as I have mostly watched or own digital copies of everything I care about - so I get more use out of Hulu, Crunchyroll/VRV and iTunes to keep on top of current shows.

I think the era of strictly scheduled programming may be coming to an end, though there's still people who like to channel surf - but I think the networks could find something in the middle if they tried (especially as somebody who doesn't like the binge model that Netflix operates on).

</comment></comment></comment></comment></comment><comment> oh-kumudo The new generation don't watch TV, period. TV is ancient, forced ads, too few choices.

<comment><USER36> Yeah, I was surprised that live TV was still on the top and tied with smartphones in terms of hours per week for 25-34 year olds. I figure live sports makes up a majority of that.

<comment><USER37> And live sports are slowly moving online. The biggest problem, in my estimation, is the cost to consumers is very high per game (to make up for the large advertising spend on traditional TV broadcasts).

</comment></comment></comment><comment><USER38> "Fox has announced a goal of reducing ad time to two minutes an hour by 2020."

Is this correct? Two minutes of adverts per hour? Or is this a typo from NYT and 'by two minutes'. Two minutes of ads would be 4 slots an hour, they'd have to increase prices by ~7x per slot to make the same amount of money.

</comment><comment><USER39> Just the other day my mother asked me why I don't sign up for a package deal to get TV for essentially $10 a month. Why would I? Why pay $10 a month to be force-fed overly loud advertisements that play for more than 50% of the runtime of the shitty shows that ruined the networks I loved as a child. The vast majority of my curiosity about history comes from the history channel and discovery channel, learning about wars and "science" and things, while my interest in tech can be traced back to Modern Marvels, a show on the History channel. All that was dead by mid 2000's, replaced by "Deadliest Catch" and "Did aliens do history? (probably not but we will lie to you and say they might have" and "Is bigfoot real? (OH COME ON)"

Sometimes I try to watch TV when I visit home, and it is infuriating in every possible way. An ad break every five minutes, sometimes twice in a row. The thirty minutes of ad space is populated by a grand total of two different ads that scream at the top of their lungs to get your attention. Sure the ads may have seemed funny or insightful or unique the once or twice the marketing department for a company viewed it, but I promise you after the tenth time seeing it it in one hour.

TV content used to enrich my mind, at least on certain specific and enjoyable channels. Now I view it as actively being cancerous, and am disgusted every time it's on in my mother's house. I worry for her mental health.

I mean for christ's sake, just look at "The Learning Channel"!

<comment><USER40> > I mean for christ's sake, just look at "The Learning Channel"!

No such channel has existed for 20 years. TLC is not, and does not pretend to be, The Learning Channel, any more than the Paramount Network pretends to be The Nashville Network.

</comment></comment><comment><USER41> Advertising supported media is in trouble in general. The networks still have a PVR problem from since forever.

All this really means in practice is that prime time dramas are in decline and will become more and more marginal. Stuff like sporting events and news is the future of advertising supported real time streaming. The medium is not important. Some of this will still come from TV transmitters.

<comment><USER42> What is PVR?

<comment><USER43> I think it's Programmable Video Recorder?

</comment></comment></comment><comment><USER44> Millennials still watch sports and it's still pretty hard to watch sports without having your parent's cable service logins..

<comment><USER45> Pretty hard to watch them with their logins too. The Comcast stream for game 1 of the Eastern Conference Finals was completely borked yesterday with a weird mixed-content warning in the dev console, and I'm fairly certain it wasn't just my account affected.

</comment></comment><comment><USER46> I remember watching MASH, Sanford and Son, Carol Burnette and Flip Wilson with my mother when those were first run shows. 5 minutes per half hour for ads and credits. But the shows were good enough to tolerate the garbage.

That has not been true for a long time now.

Television is a medium, so called because it is neither rare nor well-done. -- Ernie Kovacs [0]

[https://en.wikipedia.org/wiki/Ernie_Kovacs](https://en.wikipedia.org/wiki/Ernie_Kovacs)

</comment><comment><USER47> I personally don't have a TV antenna plugged in[0]. I do however have my TV connected to my WiFi directly and an Android TV device (with Chromecast) plugged in.

I'm not the only person like this I imagine.

[0] This is mostly due to the inconvenience of a TV antenna, maybe partially related to where we position TV antennas in older homes and apartments before the TV became so big.

</comment><comment><USER48> Off-topic question: shouldn't this be 'Why is traditional TV in trouble'? I see that there is no question mark at the end of the title too; is that why 'is' is after the subject?

<comment><USER49> It's phrased as a statement, with the usual headline-speak of dropping articles: "[This is] Why Traditional TV is in trouble".

Why they chose that over an interrogative form ("Why is traditional TV in trouble?"), I don't know. Maybe the NYT has a style manual that prefers the first formation.

<comment><USER50> I have noticed US media convention, both print and web, is to prefer statements: "Why <x>", "How <y>", and "<z> did <z'>, and it's <z''>"; the last one even conveniently tells you how you're supposed to feel about the article. I imagine it conveys more authority than questions do.

</comment></comment></comment><comment><USER51> The diarrhea analogy still stands.

Two restaurants:

One serves random food at random times. It's $80/plate and each plate is going to be 30-50% diarrhea. The bill has at least 20% unexplained markup.

The other serves whatever is on the menu all the time. It's $10/plate and each plate is diarrhea free.

<comment> the-pigeon A more accurate analogy would be

One restaurant that serves only one dish a day and there's always a 30 minute line. You have to purchase a monthly pass to eat there.

The other has no wait and a menu that rotates slowly over the year. You have to purchase a monthly pass to eat there.

Each is comparable in quality of food.

</comment></comment><comment><USER52> When u cancel 'The Mick' no wonder

</comment><comment><USER53> I can't fucking wait to get rid of our TV. There is so much better (and worse!) programming on YouTube and Netflix.

</comment><comment><USER54> Some channels do not show commercials, and they show old movies that are not available elsewhere other than buying a DVD.

I used to watch CNN, but don't anymore because it doesn't have news anymore. Tune in at any moment 24/7, and it's an editorial on the topic of "we hate Trump".

</comment><comment><USER55> NY Times writing an article about why TV is in trouble.... in 2018.

Can we just ban this trash website from HN?

<comment><USER13> One flimsy article doesn't warrant banning what's arguably the most prominent news outlet in the world. I can agree that a lot of NYT content is lame, but its stature remains strong and the HN community would justifiably revolt if it were banned altogether.

</comment></comment><comment><USER56> Post- the GDPR adpocalypse maybe TV advertising won't look so bad!

<comment><USER49> The GDPR isn't going to make millenials start watching network television, which is the main issue here.</comment></comment>


Why is e^(pi I) = -1? - <AUTHOR> https://www.math.toronto.edu/mathnet/questionCorner/epii.html <comment><USER2> 3Blue1Brown's video "Understanding e to the i pi in 3.14 minutes" is a nice compliment to this post. [https://www.youtube.com/watch?v=v0YEaeIClKY](https://www.youtube.com/watch?v=v0YEaeIClKY)</comment>


Caching at Reddit - <AUTHOR> https://redditblog.com/2017/1/17/caching-at-reddit/ <comment><USER2> > When you vote, your vote isn't instantly processed—instead, it's placed into > a queue.

I remember looking into this a while ago and was bewildered to find that when I upvoted or downvoted, there was no XHR call to the backend! There was no hidden iframe/image, no silent form post. Absolutely no network activity. Yet when I refreshed, my vote was shown correctly. I thought I was going crazy.

This was long ago so I'm a bit fuzzy on the details but after a bit of digging, I found the most elegant data collection technique I've ever seen. Instead of sending network data when I voted, a local cookie was set with the link id and vote value. Then when I went to another page, my browser naturally sent the cookie to the server, where I believe it was processed, and then a fresh cookie was sent back to my browser. I could vote on 10 links, the local cookie would get large and then on the next page refresh, the backend would receive my batch of votes, process them, and send me a fresh cookie again.

I don't think they do that now and I've never seen anyone do something like this. Even HN just makes an XHR call on voting. After twenty years on the web, it's not often that I am surprised so this was quite a thrill.

<comment><USER3> What if the next page that you go to isn't on Reddit?

<comment><USER4> Then the next time you visit reddit, I guess.

<comment><USER5> Maybe I'm not a 'normal' user so it wouldn't have mattered to them, but I used to open loads of tabs at once, and slowly make my way across all the stories, then more often than not, I'd close the browser where all my cache, cookies and history and whatnot are cleared.

This would mean none of my votes would ever have made it to the server!

<comment><USER6> To an extent it might be worth letting those votes be lost to keep things simple, if only relatively few users browse that way.

There are ways around it if it were a problem though:

* Have a timeout on updates. If a vote has been sat in the local cookie send for some time send an XHR request to push the current queue to the server

* Use onunload event handlers to push any remaining queued votes when a tab closes. You could even try maintain a count of how many tabs are currently open (in the same cookie) and only send that request on the last one closing.

* If the amount of votes hits a certain length, also send a XHR request to register them.

Probably not perfect (I have in the past found onunload events to not be terribly reliable) but it would capture most of the otherwise lost votes you describe. It might be too much work to write/debug/maintain though, if the number of lost votes would be small anyway.

</comment></comment></comment></comment></comment><comment><USER7> > Performance matters.

It took 8.4 seconds to load the Reddit front page on my phone. Hacker News took 1.1 seconds. This feels like advice from the overweight gym teacher on how to do pushups.

The desktop Reddit site took 2.2 seconds over the same connection, by the way. It seems like it would be much more valuable to optimize whatever is taking up >75% of page time on mobile.

<comment><USER8> HN serves the same front page to all users, other than the bar at the top. To make a reddit front page, you have to look up the hottest results of ~100 subreddits and shuffle them all together in the correct order. Literally every time a vote is cast, that could change the front page for every single user, in a different way for each one.

<comment><USER9> False. It cannot since it has too look up the vote status for (logged in users).

<comment><USER10> I dunno man, I think I'm going to trust the former Reddit admin (/u/raldi)

<comment><USER11> Blind trust in such a simply verifiable case is quite naive.

You can easily open up the front page of HN as a logged in user and see that it contains information about which storeies you've voted for and which you've flagged. On top of that, you can click "hide" to hide stories. These will only be hidden for your specific account, not for every person who loads the front page.

What's more, the go-to call-to-action by HN admins during really popular stories is to have people log out, because logged in users don't get cached results. Have a look at the comment by HN admin dang on the Trump winning story. [1] The key part being " _please log out to read HN today, unless you want to comment. Then we can serve you from cache_ ".

[1] [https://news.ycombinator.com/item?id=12907201](https://news.ycombinator.com/item?id=12907201)

<comment><USER12> That's not the same thing as showing different results _per_ user. It's just tacking on the user's operations when it renders out the front page.

They're two completely different operations, with completely different complexities behind them.

The "hide" feature is not analogous to what reddit is doing.

<comment><USER8> Exactly. It's the difference between "Look up the first 100 people in this phone book not named Justin" and "Look up the 100 Justins that come first alphabetically across these 100 phone books"

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> The pain of static slab allocation is real! Changing usage patterns causing problems can be tricky to track down too; mcsauna looks helpful for this. Upgrading to memcached 1.4.25 and running with "slab_reassign,slab_automove,lru_crawler,lru_maintainer" was a huge improvement for our primary memcached cluster at Eventbrite.

</comment><comment><USER14> The slowness of page load mentioned by folks here is the reason why I think caching at the HTTP level (ex: Varnish) is much more efficient than caching at the service level (ex: Memcached), which is much further down the stack and is bound to be latency-sensitive. Because it's much less entangled in your code and deep into your infrastructure (less technical debt). A hybrid approach can work too but only if it's light and unobtrusive.

By the way, and I'm going out on a limb with my shameful plug, I built a Varnish-as-a-Service kind of infrastructure called Cachoid ( [https://www.cachoid.com](https://www.cachoid.com) ). But to my own defense, I'm putting my energy, time, and money where my mouth is.

<comment><USER15> To my understanding Reddit serves a highly customized content to each logged in user. Can you help me understand how HTTP level caching will solve this problem more efficiently than their service level caching does right now?

<comment><USER14> You cache non-logged in users to start with. And then you cache based on sessions (logged in users) because you don't really need to show fresh votes on each visit and right away (admitted to it in the article). Plus there's lots of room for ESI.

<comment><USER16> They already do those things.

Logged out users see a "snapshot" of the page updated every so often.

And I really don't think that caching pages per session would really help with their load all that much. Why not just use HTTP cache headers at that point?

Plus while you don't really need to show votes ASAP, logged in users will want up to date comments.

<comment><USER14> This is where ESI helps. You can portions of the page that don't change much.

</comment></comment></comment></comment></comment><comment><USER17> The cache-perma seems pretty clever to me.

> For example, when new comments are added or votes are changed, we don't > simply invalidate the cache and move on—this happens too frequently and > would make the caching near useless. Instead, we update the backend store > (in Cassandra) as well as the cache. Fallback can always happen to the > backend store if need be, but in practice this rarely happens. In fact, > permacache is one of our best hit rates—over 99%.

They basically have their application state duplicated in both places. Interesting architectural choice.

<comment><USER18> This sounds an awful lot like an old(ish?) technique termed "write-through caching". Building data structures that can take advantage of write-through caching at reddit-comment-scale does sound like it'd be an interesting problem to optimize.

<comment><USER19> That's basically what it is. The data is abstracted away so that when you do a write it just goes to both places as needed.

</comment></comment></comment><comment><USER20> I wonder how much better/worse the site would run if they had their own hardware like StackExchange. And I wonder how StackExchange would run if it were on AWS.

<comment><USER21> In my experience running large sites, dedicated hardware is not only cheaper but orders of magnitude faster because you can finely-tune the hardware to very, very specific use cases, and you can have a fully private, highly optimized network with very small cable runs to effectively eliminate network latency issues.

Most of my current job involves solving problems that are caused by cloud limitations.

<comment><USER22> And if you need throughput, 40gigE / 56gigE is not that hard expensive with your own hardware. You end up with like 4x to 10x servers to handle the same load.

</comment></comment></comment><comment><USER23> Some back of the envelope math for their caching costs:

54 x R3.2xlarge EC2 instances

On demand = $314,571.6/year

w/ 1 year term = $166,860/year

w/ 3 year term = $110,340/year

w/ convertible 3 year term = $150,174/year

Is that a lot? Seems like a lot

<comment><USER24> For a text link site that lets you vote submissions up and down? Yes that's a lot. I'm sure a lot of the decisions at reddit corp were along the lines of, "well, we can't go back and change that now, so let's do whatever we need to fix it and move forward" AKA lots of RAM and caching all over the place.

<comment><USER25> If a company is grossing $20MM/year and stands to improve that by more than 0.75% by having a faster site, then yes, it makes sense to spend 0.75% of your revenue to do so.

</comment></comment></comment><comment><USER26> Thanks for sharing the details. It's impressive to see the memory allocation and pool size for a site handling this much traffic. I would love to get some more information on reddit's platform overall traffic volume as I feel this would complement the discussion nicely.

</comment><comment><USER27> I wonder if switching from memcached to redis would make a bottom line difference in terms of the number of instances needed (cost) and performance?

<comment><USER28> Maybe a difference in performance, but caching comes down to bits-to-be- stored. That likely wouldn't change.

<comment><USER29> redis does have some built-in features that automatically compact data. [0] I'm sure reddit has considered this, but worth noting.

[0] [https://redis.io/topics/memory- optimization](https://redis.io/topics/memory-optimization)

<comment><USER30> client-side compression works out better for a couple of reasons:

1\. You use less bandwidth in/out of the box. This matters in AWS because your bandwidth is (relatively) limited.

2\. Your CPU usage to (de)compress is distributed over a larger fleet. The clients spend the CPU time, not the cache servers.

</comment></comment></comment></comment><comment><USER31> Are those visualizations done in Grafana?

<comment><USER32> Yes, Grafana with Graphite as the backend store.

</comment></comment><comment><USER33> Ironic timing that Reddit is currently undergoing a major outage right now...

<comment><USER34> Not really, there were some routing issues with our CDN affecting an extremely small percentage of traffic.

<comment><USER33> Seemed to be down for lots of people in the bay area: [http://downdetector.com/status/reddit](http://downdetector.com/status/reddit)

Maybe only small because most californians were still sleeping? I was only up due to a sick kid. :D

</comment></comment></comment><comment><USER35> What's the difference between mcrouter vs something like haproxy?

</comment><comment><USER36> Slightly Off Topic, as of 2017, what's the advantage of Memcached over Redis. I thought we are basically in the era of Redis.

<comment><USER37> It's hard to talk in general terms but Memcached is threaded so you can saturate your CPUs without requiring multiple instances, and Redis has more advanced features both from the POV of programmer API facing and operations. But if we want to zoom on Reddit itself, the fact of using data structures and changing caching paradigm to really use Redis the proper way, that is, storing metadata in Redis and not just only in their main DB, I suspect would provide a very big boost to Reddit. For some reason Reddit always has been an "anti Redis" shop. I'm sure they have their good reasons, and btw I love Reddit too much to complain, whatever they do to run it, I don't care as long as they provide such a wonderful service to the community :-)

But... their use case is IMHO one that you can accelerate tremendously by using Redis. I ran the most popular Italian Reddit-alike site for years, and I wrote a simple Reddit clone that uses Redis so I had the opportunity of exploring the problem a bit.

<comment><USER34> Reddit is most definitely not an "anti Redis" shop! If we were to rewrite it today, we might start with Redis. We've just put so much work into making memcached reliable and easy to understand for developers that the possible benefits of switching to Redis for many of these use cases don't outweigh the operational knowledge we have from years of running memcached at this scale.

That said - we use Redis in at least 2 (soon to be 3) capacities across the site for different services (outside of the monolith) and it works really well.

<comment><USER37> Thanks! That's great to know. Please ping me anytime if you want certain features or alike. Thanks for your work on Reddit, I love it.

</comment></comment></comment></comment><comment><USER38> Taking into account how reddit performs, I'll take this as a guide on how not to use cache.

(This is a joke please. Understand it as such. I know reddit has the problems it has because it is severely understaffed)

<comment><USER39> Honestly I do recall periods when reddit wasn't performing particularly well but recently it's seemed fine to me.

<comment><USER38> For me all pages take 3+ seconds to load (generation time), which means reddit is easily the slowest site I visit.

<comment><USER40> Have you tried turning off all adblockers on your browser? Or is your time before all those come into play?

<comment><USER41> FWIW, a GET / (front page) while logged in takes 1-1.5s for me (TTFB). ~3s till load with ads blocked, ~5.5s with ads. Both numbers without caching, but that only seems to shave off about 200ms (most of the time is spent in JS and rendering the site).

<comment><USER40> Okay, thank you.</comment></comment></comment></comment></comment></comment>