Survey of scientists sheds light on reproducibility crisis (2016) - <AUTHOR> https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 <comment><USER2> As a working scientist, I feel both sides of the problem every day. Most papers I come across turn out to be difficult if not impossible to reproduce, and I'm sure some of my own papers have fallen into that group at some points (although given that I'm a theorist, what "reproducible" means can be a bit fuzzy sometimes). At the same time, I'm confused every time I see people wondering about the scale of the problem or what to do about it. There is absolutely no mystery here whatsoever.

Scientists are generally fairly smart people. Put smart people in a hypercompetitive environment and they will quickly identify the traits that are being rewarded and optimize for those. What does academia reward? Large numbers of publications with lots of citations, nothing else matters. So, naturally, scientists flock to hip buzzword-filled topics and churn out papers as quickly as they can. Not only is there absolutely no reward for easy reproducibility, it can actually harm you if someone manages to copy you quickly enough and then beat you to the next incremental publishable discovery in the neverending race. This is absurd and counterproductive to science in general, but such is the state of the academic job market.

Anyone who purports to somehow address the situation without substantially changing the fundamental predicates for hiring and funding in the sciences is just wasting their breath. As long as the incentives are what they are, reproducibility will be a token most scientists pay tribute to in name only.

<comment><USER3> Biology postdoc here, seconded.

But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". This was mentioned in the survey and it conforms to my informal experience of attitudes.

I do not know how you square that circle. Maybe, because of the pressures you mention, we are all supposed to engage in an informal convention of pretending to believe most previously published work is true, was done correctly, and is reproducible even if we know damn well how unlikely that is. I find it hard to do.

One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters. These are not my fields, but if they are as prone to error, bias, irreproducibility, etc, as my own, I'd exercise some humility. It is one thing for we scientists to lie -- errr, project unwarranted confidence -- to each other for the sake of getting grants, but it is quite another to do it directly to the public.

But when the public _does_ figure it out, what do you think will happen to funding? It will get tighter, and make the problem even worse. We need reform from within before the hammer falls, and quickly.

<comment><USER4> _"One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters....I'd exercise some humility."_

You shouldn't be cringing. You should be educating people about how science actually works, and how it simply _doesn't matter very much_ whether any particular paper is reproducible. It's a straw-man argument, because most papers aren't _worth_ reproducing. I don't know many good scientists who take what they read (in any journal) at face value. If you do, you've been mislead somewhere during your training (fwiw, I also have a PhD). At best, papers are sources of ideas. Interesting ideas get tested again. Most get ignored. Even if a few hokum theories become popular for a while, eventually they're revealed for what they are.

The tiny percentage of subjects that rise to the level of public policy discussion end up being so extensively investigated that reproduction of results is essentially guaranteed. And yeah, you hear lots of silly noise from university PR departments, but that stuff is a flash in the pan.

For example, nobody legitimate is in doubt of the broader facts of global climate change or evolution or vaccination, even if 95% of (say) social science reaults turn out to be complete bunk. Yet climate deniers, anti- vaxxers and "intelligent design" trolls absolutely _love_ it when this distinction is ignored, because it allows them to confuse the public on the legitimacy of science _as a process._

<comment><USER5> It's true that science doesn't value every published paper equally, but it's also true that publish or perish is creating ever-growing mountains of worthless papers. This is a real problem, and drags the quality of everything down.

Besides, the fact that there isn't one reputable journal in most fields that remains untarnished by the replication crisis is both a practical problem and a problem of public trust. A lot of this BS science is paid for directly by the public's tax dollars, or else by their student loans. I wouldn't expect the public to be so forgiving if 95% of it is bunk.

<comment><USER4> _"it 's also true that publish or perish is creating ever-growing mountains of worthless papers."_

Is this true? Prove your claim.

 _"This is a real problem, and drags the quality of everything down."_

Let's assume your first assertion is true. Is it automatically true that your second claim follows? Why?

I see no evidence that the individual productivity of scientists has changed much in the last 30 years, nor do I notice much of a change in the aggregate quality of science. Crappy science existed hundreds of years ago, and it continues to exist today. The main difference, as far as I can tell, is that we have a lot _more_ scientists now.

In any case, these are just assertions, not arguments.

<comment><USER5> There have always been weaker scientists, but there hasn't always been the economic incentive to publish in order to maintain a teaching position. This is a relatively recent (few decades) thing and is due to structural factors in academia and society. If you require proof, I'm not really sure what to say to you, as evidence is not hard to find, but if you don't already see it, I'm unlikely to change your mind.

<comment><USER4> If you want to claim that "publish or perish" (which, btw, has been a part of academic life essentially forever) is somehow _recently_ affecting the volume of papers being produced, you should be able to provide evidence of that in a straightforward manner. One obvious test: is the per-capita rate of publication increasing? (my experience says "no", but I'm open to contrary evidence.)

You have a hypothesis of what's going on, but you've provided no evidence for that hypothesis, and when challenged to provide some, you tell other people it's _their_ job to do it for you.

It's not my job to prove your extraordinary claims.

<comment><USER5> The number of publications per year per academic seems to me to have increased over the last 50 years. I don't have a citation.

Regardless, my original claim was that the absolute number of papers is growing, and most of them are trash. I think the sheer volume of trash has consequences that were not so serious 100 years ago, even if the percentage of trash was the same. I strongly suspect the percentage of trash has been going up, as well.

My argument is that "publish or perish" makes less and less sense the more active scientists and researchers there are, even if the average quality and the rate of publication per academic were constant, because the appetite and rate at which research can be assimilated by society is limited, and does not scale with population, while the number of scientists does.

I don't think these claims are extraordinary, and if you do, I'm not going to go looking for extraordinary evidence to try to convince you. I don't think I'm the only that sees these effects, however.

<comment><USER4> _"The number of publications per year per academic seems to me to have increased over the last 50 years. I don 't have a citation."_

Yeah, that's not evidence.

The absolute number of papers _is_ growing - along with the number of working scientists. There's been huge growth in academic science since the 1970s.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> One factor that might increase reproducibility is to enhance negative reputational effects of publishing an un-reproducible study. That is, it should be so rare, so _outré_ , to publish something unreproducible, your professional reputation is severely damaged, and if you do it again, you will leave academia.

It sounds harsh, but severe self-regulation is what prevents the catastrophic failure of any institution that is really only answerable to itself. Professors aren't doing any favors by rubberstamping weak papers, or giving weak students PhD's (or BS's for that matter).

EDIT: down-votes and no replies, as expected. Hey academics, sometimes to save the patient you have to amputate the limb. I care about science and want it to survive, so you guys continuing to be nice to each other even when you publish BS isn't serving that purpose.

<comment><USER7> The problem is not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor, it may be bunk but it may show a worthwhile direction for others to investigate further. The problems come when an non-reproducible study gains some authority, citations in other studies or published in the media without proper vetting.

It's probably worth punishing people that consistently publish UN-reproducible work, but it shouldn't a rare career destroying event like you are suggesting.

<comment><USER6> _> not being reproducible is not necessarily a failure and does not necessarily indicate a bad actor_

I disagree with the first part and agree with the second. I would argue that failure to reproduce is perhaps the worst kind of scientific failure, because the activity cannot be called science anymore. And no, I don't mean anything personal when I say 'failure', you don't have to be malicious to do bad science -- after all 'doing bad science' is the _default_ human condition.

An analogy with classical music: A musician that fails to reproduce the score with his violin consistently, will lose his job quickly. It doesn't mean he's a bad person, but it does mean he's not good enough for the orchestra. The metaphor for the "reproducibility crisis" in an orchestra is what happens if they let bad players stay, the orchestra sounds worse and worse, and finally the audience stops coming. In the apocalyptic scenario, cultural forces cause ALL orchestras to stop honestly evaluating the skill of their players, and all orchestras simultaneously lose their ability to accurately play any but the simplest music.

Standards are painful for those that can't meet them, but they make the world better, overall.

<comment><USER5> You can judge a musician in a 20-minute audition, but you can't tell if research will reproduce without doing the work. However, there are papers that are so vague and poorly written that they cannot possibly be reproducible, and this is becoming the norm in some fields, so I think you're right that standards would help there.

Also, sometimes strange things just happen randomly (like the CERN faster- than-light neutrinos) and there's no shame in publishing results that aren't correct as long as you're honest about it. Some early-stage, low-power studies are going to randomly show results that don't reproduce, and that's also fine. Suppressing these results would hurt progress just as much as making the opposite error.

</comment></comment></comment></comment><comment><USER8> Kinda-OT but: During my studies I've read papers from everything to organisational theory (MBA stuff) to pretty "pure" CS (like compiler construction etc), and have in that process read so many papers in many different fields that just don't provide any insight or are barely comprehensible.

Of course, sometimes it can probably be blamed on the reader (me) for not having enough insight, but many times the papers are just crap.

My naive imagination of academic papers before university was that they were like those I now read in the __top__ journals / conferences in each field: concise understandable writing with actual insights and clear methodology that can be repeated. But papers like these are probably something like less than 1% of everything published. Many papers seem to be almost approaching levels of the Bogdanov thesis's [0].

To relate to the topic: I'm wondering if we'll see a shift in the academic culture and system, where the pressure to publish is lowered (somehow), and where focus is on quality and actually providing insight and reproducibility.

The worst possible thing is that bad science gets influential on false grounds, which needs to be avoided. But increasing the signal-to-noise-ratio is probably not a bad idea either. And I suspect that they will both turn out to be improved by fixing the underlying incentive problems.

[0]: [https://en.wikipedia.org/wiki/Bogdanov_affair](https://en.wikipedia.org/wiki/Bogdanov_affair)

<comment><USER9> The bar for publication (like much of our media) used to be far higher. Particularly with the push to get everyone through college, even at the cost of lowering standards, there's a sort of regression to the mean with respect to publication quality. It happens with any difficult, technical field that initially attracts the more capable in society.

Hell, look at what the internet was back when it was still mostly needs with passion.

<comment><USER10> College students don't produce papers. Published papers and how many students go through college are unrelated.

<comment><USER11> But you have to go through college to do a PhD so I guess they are correlated.

</comment></comment></comment></comment><comment><USER9> I really think there's lost opportunity in the typical American master's program. Typically for a master of science you need to perform original research for your thesis, ostensibly to prepare you for the process of publication. But I think a master's could be a far more useful stepping stone between undergraduate academics and PHD or industry research if it were taken as an chance to have graduates reproduce important research. The grade will be based on the merits of the reproduction, which will be an excellent final introduction to research work, while also hugely benefiting the community by verifying previous research. Right now there simply isn't any incentive to reproduce anything but the biggest results.

</comment><comment><USER12> Worth mentioning that scientific research, even if difficult to reproduce, is still better than, say, superstitious crap. Looking at you, chem trail antivax climate deniers.

<comment><USER13> Is it though? Some ideas are so obviously nonsense to most of us, whereas bad research can have a more robust veneer of legitimacy - which makes that research a potentially useful tool for bad actors. The internet is full of authoritative sounding tripe backed up by "science".

<comment><USER14> Bad research doesn't usually hold up. Even good research doesn't always hold up over time. Was Bohr a bastard for getting the atomic model wrong, or just offering his best interpretation given what was known at the time?

<comment><USER15> _> Bad research doesn't usually hold up._

Your faith in the reliability of the scientific process is touching, but naive. Bad research is driving public policy in many areas.

 _> Even good research doesn't always hold up over time._

In the sense that it gets superseded by better models (as Bohr's original model of the atom did), sure. That's to be expected in a healthy scientific field. But one of the things that's needed to keep a field healthy in that respect is the ability to do controlled experiments with high accuracy. That's how scientists figured out that Bohr's original model of the atom wasn't right --it couldn't match the results of the experiments as they got more accurate.

</comment></comment></comment></comment><comment><USER16> My feeling is that this reproducibility crisis points to deeper stuff just waiting to be collectively understood about how to use math in experimental sciences.

So many researchers rely on assumptions of linear dynamics. So many experiments and studies are designed without consideration of observer effects.

Is it any wonder that a model might fit one day but not the next?

There is so much to be learned from applied topology, dynamical systems theory and (quantum) information theory, but methods from these disciplines are only just barely starting to become more widely accessible.

<comment><USER17> Making the model more complicated doesn't add external validity. It seems like the problem often comes from trying to make do with collecting a minimum amount of data, collected in the most convenient way. (For example, doing experiments on local college students.)

It seems like a practice of at least collecting the data at two different colleges might help? But this would require more coordination.

<comment><USER16> Just because alternative methods aren't widespread, it doesn't necessarily mean they are "more complicated", just that rely on different assumptions.

I'm really excited, for example, about "model-free" time series methods emerging from non-linear topological data analysis. Takens' embedding theorem shows that low dimensional "shadows" of high dimensional attractors can be constructed from a time series alone, and that they can reveal deep, useful facts about system a time series is part of, even if that system defies geometric modeling. These kinds of methods are radically different from conventional statistical methods.

see here: [https://www.youtube.com/watch?v=NrFdIz-D2yM](https://www.youtube.com/watch?v=NrFdIz-D2yM)

Previously, researchers found themselves flailing around with geometry, trying to come up with models that accurately described the dynamical nature of the systems they were studying (e.g. fish populations). A model might work well for a few years, then stop working. These topological methods cut through and get straight to the underlying relationships that drive various observables in a complex system, without relying on fickle geometric assumptions about how exactly those relationships will be expressed.

</comment></comment></comment><comment><USER18> IMHO the simple problems with reproducibility (of statictical studies) can be solved with pre-registration, which should become a standard. The rest really is a system crisis.

Imagine: Scientist A uses (a statistical) method M to assert X. People praise A until scientist B uses method M and cannot assert X (which BTW usually doesn't imply X is wrong). People wait for scientist C to use method M and re- check assertion X. People then take the majority vote (or meta-analysis) for true.

That's the way much of science works today. It becomes a problem only in the following situations:

0\. People hunt for statistically significant results.

1\. X is mostly irrelevant to anyone who does not belong to the in-group of A, B, C, and peers. So nobody else really notices or cares about X.

2\. There is no other way to observe any meaningful consequence of X other than by method M -- which eventually results in #1.

3\. Method M is really expensive and complex, it's an almost impossible undertaking so that B or C most likely won't get a funding.

4\. Everything is ok, actually, but the people who pay for the study don't care about A's methodological fineprint as long as the results play well with their other goals.

The presented list of "factors that build reproducibility" focuses on #0, for which pre-registration is a simple and clean solution. IMHO the list is much to narrow and focused on academic practice, though.

</comment><comment><USER19> Maybe because some of the "science" today is more about spreading some political view rather than anything else.

For example, there are a lot of gender studies that are presented as a science in my country which I believe is complete bullshit and an ideology.

People are pushing shit like that today and I believe we need more of the hard sciences and less of the soft sciences.

</comment><comment><USER20> Another thing I'd like to ask from my fellow colleagues: please at least to some extent detail in your papers the things that you've tried that didn't work. I see this in my field (computer vision / deep learning) from time to time, but very rarely.

There are typically ablation studies which aim to determine how much each of the _successful_ improvements contributes to the result, but there's almost never any mention of things that looked promising on paper but didn't pan out in practice, nor is there any discussion of the reasons why, even though the authors often have a good idea post-facto.

</comment><comment><USER21> Reproducibility is referred to as a "crisis," but I'd like to know if it's really a new thing. What if we tried to replicate the studies that surrounded great developments such as electromagnetism or thermodynamics. Did those advances emerge from an unbroken series of reproducible studies, or from a tangle of good and bad results?

Before looking for root causes that invariably turn a cynical eye towards the motivations of scientists, let's make sure the effects don't precede the causes.

<comment><USER6> I really like this idea of having students (not just advanced ones) reproduce classic experiments (ideally _before_ getting the theory, so the class can attempt to puzzle out what's happening). My sense is that early science was very hands-on and readers of such results were also doers. Plus, doing the experiment before theory puts them in the same spot as the scientist, and might give them a little more respect for an achievement that is really quite magical (and too often is taught in a kind of arrogant hindsight that implies the discovery is/was sneeringly obvious to any human that draws breath)

</comment></comment><comment><USER22> Hmm... 40% of irreproducibility is due to fraud? Who in their right mind would put in all the effort that science requires if they think everyone else's work is commonly malicious bullshit?

Obviously this survey is biased towards people with a certain outlook.

<comment><USER23> Academics thinking little of other academics is not surprising.

</comment></comment><comment><USER24> Is there room for a genre of "failed to reproduce" journals? Contrarianism seems like it might be powerful enough to support a journal whose only purpose is to dispute major findings.

<comment><USER25> The problem is that's it's very easy to fall to reproduce a result.

Maybe even easier than the p-hacking used to produce an irreproducible result.

</comment></comment><comment><USER26> This is why I want to industry after MIT, science is really real when it manifests as a useful technology that someone will pay for. Also, I didn't want to spend so much time writing grants...

</comment><comment><USER27> Is a 30% reproduction rate unusual?

We know 19th-century science was productive. Was it similarly plagued by irreproducibility?

I'm sceptical of this argument. But we need a baseline to render judgement.

</comment><comment><USER28> As a chemistry PhD, this sure looks a lot like the reproducibility is directly proportional to the complexity of the underlying phenomena being studied.

</comment><comment><USER29> As somebody only used to CS research, can any explain to me how much the costs of reproducing is considered into unreproducibility?

</comment><comment><USER30> There is a startup called [http://www.myire.com](http://www.myire.com) that is working on this issue. They have a platform that provides an A-Z approach to publishing research.

Their marketing is probably the worst in class but the CEO is a passionate developer who has worked hard on this.

<comment><USER31> Really doubt that this is an issue that can be solved by one startup. It's a procedural and cultural issue that affects the whole scientific world; not a problem that can be solved by business or technology.

</comment></comment><comment><USER32> From my experience, math/computer-science type research is almost always exactly reproducible.

<comment><USER33> > math/computer-science type research is almost always exactly reproducible.

Math, including computer science, is almost entirely pure logic, not empirical science (despite the name of the latter), so it's not even in the same epistemological domain where reproducibility is conceptually a concern.

</comment></comment><comment><USER34> On the other hand we have the government saying they don't want other countries reproducing our results in AI, cryptography, genetics...

</comment><comment><USER35> How would the state of social science progress if every academic had all Facebooks data available to them?

</comment><comment><USER36> Weird how there can be a reproducibility crisis, but also climate change is 100% unquestionable science.

</comment><comment><USER37> there were a few articles quoting the use of software like git, simple formats, Jupiter notebooks, etc etc

is there anything of the sort that helped a bit ?

very lightly funny to see this when nix and guix (and pure fp) poping at the same time in a very different context.

</comment><comment><USER38> I wish more people would map their understanding of the reproducibility crisis to things like say, the sokal squared hoax. The current pressure to publish regularly in academia is very high, and it results in poor work and journals that don't do their due diligence, in every field.

</comment><comment><USER39> Is there an incentive to make certain experiments harder to reproduce.

<comment><USER40> One explanation that I've heard (often in terms of global south v. global north) is to prevent more well-funded labs to quickly mirror your setup/method and, given their higher means/staff, deplete that line of research leaving your lab with no publications, patents, etc. and therefore no money. Science as a whole may benefit, but individual scientists still usually appreciate keeping their jobs.

<comment><USER14> Big labs don't work any faster than small labs, they just have more people to chase more projects; generally it's the same amount of people per project as a small lab. Chances are if you got scooped it was just because someone got started with the idea before you did. After all, the future directions at the end of the paper are usually active research by the time you are drafting and submitting for review.

Getting 'scooped' isn't so bad in biology at least. If you were thinking about XYZ and someone publishes X and Z, great. Cite that paper, now you don't have to bother so much validating X and Z and can focus on strengthening your Y argument or adding argument W to your paper.

<comment><USER40> > Big labs don't work any faster than small labs

Not true, at least in my field. If $BigLab can throw 100k CPU cores at a simulation it is going to finish way faster than $SmallLab 1k simulation. If $BigLab builds 10 dedicated test rigs they are going to finish that parametric study way waster than $SmallLab with 1 shared test rig.

Getting scooped may not a big problem for papers (except when you start getting rejections because lack of novelty), but it is a big problem for patents, licenses, royalties and such things that allow small labs to survive when their governments have little to no money to spend.

</comment></comment></comment></comment><comment><USER41> Note a spike in Earth science at 30% confidence. Probably reflects a lot of vested interest funding politically beneficial results in this discipline.

</comment><comment><USER42> This is from 2016.

Can we get one of the moderators to add (2016) to the headline?

<comment><USER43> Done, thanks.

</comment></comment><comment><USER44> I guess it's time to evaluate and grade scientific findings?

It would be interesting to see how reproducibility varies by field, university, country, etc. Although I guess scientific reputation would already give a clue over the quality of scientific work?</comment>


Building a Developers Index. Looking for Beta Users - <AUTHOR> http://www.dev-hub.io

<comment><AUTHOR> Hello, been posting about building a index of developers. I keep reiterating over and over, with the main objectives being:

* Raising awareness of the devs in a certain area. * Provide information related to open source contributions to potential employers.

I'm a developer myself ([http://github.com/guiman](http://github.com/guiman)), and I'm tired of getting emails on technologies I had stopped using and no longer intersted. Also, having to keep my profile up to date is time consuming and having to replicate that in every job posting site become annoying.

I'm looking for some feedback and please don't hesitate to contact me with any questions regarding the site email@example.com. Here is a small example of how beta users can use the site [https://youtu.be/Aqs- sfEg7j4](https://youtu.be/Aqs-sfEg7j4)</comment>


Automated home brewing system - <AUTHOR> https://decibel.ni.com/content/projects/the-wobbly-boot-pico-brewery/blog

<comment><USER2> The WilliamsWarn Nanobrewery <http://www.williamswarn.com> is the most elegant solution to home brewing out there. The control system (which was done by a company related to me), had to be complex enough to cope with everything going on, simple enough to operate by anyone and flexible enough to allow the brewer to create. I'm obviously biased - but the nanobrewery is getting great reviews.

<comment><USER3> Do you have any insight into their target market?

I'd be more inclined to mix the hobby of building the machine into the hobby of brewing the beer than I would be to spend $6,000 on something off the shelf. But I realize that's just me.

</comment></comment><comment><USER4> NI had a section of demos like this at their technical conference this August. They were all hobby projects. Things that engineers just thought "were cool" and were stupidly easy to make with this hardware. There was a crane game, a 6-foot-tall aluminum slingshot controller for Angry Birds, and a compactRIO- powered guitar amp.

It's startling to think of how much money/time engineers save with prototyping and graphical system design software like this, and even though NI stuff powers telescopes and water filtration systems and loads of other important stuff, things like this come out and remind everyone that there's fun to be had.

</comment><comment><USER5> I really like "The Electric Brewery". The plans are easily available and you can buy all the parts from them and build it yourself (or, if that's not your thing, you can buy the panel pre-built): <http://theelectricbrewery.com/>

It's a good combination of automation-where-it-hurts and still-hands-on-where- it's-fun - some of the other ones seem too automated and take all the fun out of it (we're homebrewers, not looking to start the next Budweiser).

</comment><comment><USER6> An interesting approach. I wonder how much regularity he gets out of his brews, given that everything's so tightly controlled.

Personally, I think part of the fun of homebrewing is coming up with your own unique take on a recipe. The mishaps that create something weird in your beer make for a good story.

<comment><USER7> Anyone brewing any kind of volume would want successful variances (mishaps or otherwise) to be replicable. Thus they would need to have a highly consistent baseline (or at least be able to see and measure the variances) in order to experiment with single variables.

</comment></comment><comment><USER8> Can I buy one?

<comment><USER9> There are automated home brewing systems available on the market already. Seem to be 5-10 times the cost of a non-automated homebrew setup.

Here's the first I found with some random googling around: <http://brewmation.com/Panels.html>

</comment></comment><comment><USER7> This is amazing.</comment>


Dr J's Guide to Drawing 3D ASCII Pictures (1994) - <AUTHOR> http://www.sunnyspot.org/asciiart/docs/stereogr.html <comment><USER2> I was a little skeptical - after all was it really tricking me into thinking it's 3D? Or is this just an odd visual artifact? Then I read:

      Notice how the shaft now seems tilted to the left?
      (Did you just catch yourself shifting your head to
      the right? Convincing isn't it?!!)

And I realized I had just moved my head to the right to center the shaft without realizing it. I laughed out loud when he caught me on it.

</comment><comment><USER3> Now I want a 3D text editor. Syntax highlighting by depth? or Python indents in the third dimension!

<comment><USER3> Proof of concept for Z-dimension Python indenting (cross your eyes):

      for line in file:          for line in file:
      print line                  print line
      print "all done!"          print "all done!"

<comment><USER4> Actually, I prefer the inner part of the loops negative z-indexed (but you can image how it could flip as you move the caret around):

      for line in file:          for line in file:
      print line                print line
      print "all done!"          print "all done!"

However, staring at an IDE cross-eyed while trying to program sounds like a punishment one might cook up for devs who bundle Browser Toolbar Adware with screensaver packages that my mum likes to install, and not something I'd choose for myself.

In university, I did some research into Multi-Level Displays (essentially a transparent LCD a few millimetres in front of a standard LCD screen), but what held the tech back for consumer applications was that we hadn't yet found a killer app. (In contrast, the gambling industry loved them, and I see them often in Casinos in Vegas).

But it seems like it would be a perfect application for fake 3d IDEs.

<comment><USER5> Maybe if you couple that with an Oculus/hololens it could be a viable display.

I guess you can mix that with color syntax highlighting. Or maybe have the IDE run timing analysis on your code and adjust z-indexing to reflect where most of the time is spent.

<comment><USER4> One of the best things about the MLD was that it is a 'real' 3D (well, 2.5-D) display. The real separation of screens means that there's no need for special eye-wear or projection, and there is no nauseating delay in dealing with head- motion, eye-saccades or any of the other problems that I've heard plague long- term use of the Oculus.

</comment></comment></comment></comment></comment><comment><USER6> This uses the same technique as used for viewing Stereograms. Consider me rather impressed.

</comment><comment><USER7> This is so cool! Could this be made into a little 3D roguelike?

</comment><comment><USER8> Date: 1994.

<comment><AUTHOR> But still not dated.

I don't understand some posters' obsession with "it's not from this week".

<comment><USER9> The convention is to append the year to a title if the article is not from the current year. But that's definitely not to disparage older content. Quite the opposite! Older articles are often more interesting (because time culls the less interesting stuff), and historical material is a lot of the best that appears on HN. We're always hoping people will post more.</comment></comment></comment>


Inception becomes reality: People can teach themselves new skills in dreams - <AUTHOR> http://www.dailymail.co.uk/sciencetech/article-2077185/Inception-reality-People-teach-new-skills-dreams.html#ixzz1hY5hgMRG

<comment><USER2> That's awesome! But I am waiting for the day when we could learn skills while we are awake! Like Trinity learned to fly the chopper or the way Neo learned Kung Fu :). I know its still a dream, but I believe that day is not very far!

Thanks for sharing this post!</comment>


Tip: show "invisible characters" only when you highlight text - <AUTHOR> http://justinhileman.info/articles/coda-tip-show-invisible-characters-when-you-highlight-text <comment><USER2> I personally prefer the "automatically strip unnecessary whitespace from the file when you save" approach, but this is an interesting feature nonetheless.</comment>


Assistly.com becomes Desk.com! - <AUTHOR> http://www.desk.com/ Indeed, it looks like Assistly is completely rebranding: http://www.assistly.com/blog/assistly-deskcom/ <comment><USER2> And also got hit with a giant stick of corporate-inspired fugly.</comment>


Infographic examples, ca. 1920s - <AUTHOR> http://www.creativereview.co.uk/cr-blog/2011/december/painting-by-numbers

<comment><USER2> Just to add some more. Some examples of Soviet era infographics (in Russian):

<http://community.gidlipetsk.ru/blog/view/6786>

</comment><comment><USER3> I went to the London Transport Museum that chronicles the history of the city's mass transit system. It's chock-full of these types of posters from Transport for London, and also delves deeply into the design strategies throughout the different eras. It's worth a visit if you're ever in the area.

</comment><comment><USER4> I particularly enjoyed the last infographic on temperature because it makes me appreciate air conditioning all the more. It says music halls reached 150 degrees and theaters 125 degrees. Can you imagine listening to a concert in that kind of heat?

<comment><USER5> 150 degrees Fahrenheit (65 C) would exceed the highest surface temperature ever recorded on Earth -- by a wide margin. I can't find a good reference at the moment, but I think 150F would be immediately fatal.

I think the infographic is humorously saying that you can find whatever level of excitement you want by taking the underground. Fishing is a boring 5 degrees, football gets you to body temperature, and music halls are a scorching 150.

<comment><USER6> Definitely not fatal. According to Wikipedia[1], the temperature in a sauna is "typically between 70 °C (158 °F) and 100 °C (212 °F)".

[1] <http://en.wikipedia.org/wiki/Sauna>

<comment><USER7> The high end of that scale is a little low. My sauna is regularly at about 105°C (221°F), and 110/230 is the starting temperature in the World Sauna Championsips.

</comment></comment></comment></comment><comment><USER8> The traffic congestion one could easily be an ad from today.

<comment><USER9> It is :)

[http://www.treehugger.com/cars/amount-of-space-required- to-t...](http://www.treehugger.com/cars/amount-of-space-required-to-transport- people-by-car-bus-or-bicycle.html)

</comment></comment><comment><USER10> Does anyone recognize these fonts? In particular the font of the first, as well as the main text of the last graphic?

They are quite beautiful.</comment>


RPC Olympics – The Search for the Perfect RPC Protocol - <AUTHOR> https://www.perimeterx.com/tech-blog/2020/rpc-olympics/ <comment><USER2> I recently spent some time between raw UDP and GRPC and learned a ton. This was for a hardware project so it's different from what web devs typically work on.

The most important thing that I found out while doing my research is it's not the fastest bytes that win. Well, that does matter, but it's not that important. It's reducing performance variance [1]. While my project wasn't optimizing for speed, it was optimizing for zero-copy de/serialization, but this often ends up as a solution for high-speed transfers. SBE, Flattbuffers, Cap'n Proto all had their places, but I ended up not using any of those and just hand-rolling something similar to what SBE would do. If this was a $DAYJOB project I'd probably end up doing something with SBE.

[1]: [https://speice.io/2019/07/high-performance- systems.html](https://speice.io/2019/07/high-performance-systems.html)

<comment><USER3> At my last job, I had a need for high bandwidth low latency messaging over UDP - we started out with raw C structs over a home built reliable UDP library. We ended up laying protobuf on top a year after because it was honestly a headache not having a serialization library as message types got more complex. It didn't end up causing enough slowdown to impact the run speed of the application (all serialization was done on the app thread rather than the very performance sensitive network thread) as we had other bottlenecks, and the developer workflow (and additional message validation) made it worth it. SBE looks neat, though, I didn't know about that one.

</comment></comment><comment><USER4> Apples to oranges.

TCP has no concept of a message, you need to build it on top of TCP.

Which HTTP does, but it doesn't have a standard concept of a message format, you need to build it top of HTTP.

Which gRPC does, but it closes the connection after a successful exchange.

Which is avoided by streaming gRPC - makes sense if you know you'll be talking more over this channel.

None of those protocols are really comparable, and most of the differences boil down to the serialisation protocol (binary/proto or textual, like JSON).

<comment><USER5> Usually grpc connections are left open after a message is sent and received. And since it's based on h2, multiple message streams can be multiplexed over a single connection. That is part of the benefit of grpc, establish a long lived connection and send your messages as needed. Similar to using keep-alive with http.

</comment></comment><comment><USER6> I think it comes to no surprise that sending data through a plain TCP socket is faster than using gRPC. The only downside is that over time requirements will likely start to stack up:

\- Suddenly you want to enable TLS between one or more components, meaning you need to wrap the socket inside a TLS channel.

\- You discover that your client behaves poorly when servers go offline, so you add your own logic for keepalives/pings.

\- At some point you want to add metrics to all of this, so you decide to manually add Prometheus metrics to the client/server.

\- Later on you want to attach OAuth2 tokens to requests as well, so that you can do credential passing.

\- In order to get more insight in your setup, you decide that you want to use this in combination with OpenTracing/Jaeger.

Once all of those features are added to your Redis-like protocol, you discover that you've basically reinvented gRPC... poorly.

<comment><USER7> In the conclusions the author ends up recommending Redis protocol and the plain TCP conclusion is least attractive of all, so I think you and the article are agreement that plain TCP is not the way to go.

</comment></comment><comment><USER8> Cool! I don't know if a comparison of these really makes sense. You'd be conflating transport, serialization, and stream format (unary/stream).

For fullstack dev, I've been immensely happy with grpc/protobuf over http because of the type safety I get communicating between Golang and Typescript. This eliminates a whole class of bugs but is only a serialization benefit.

Generally: use the thing with the best tooling!

<comment><USER9> There is no type safety in grpc or any other protobuf RPC scheme, full stop. The recipient of a message makes an assumption about the meaning of the message and decodes it accordingly. Any encoded protobuf might successfully decode as any other protobuf.

To repeat, grpc has no "type safety" whatsoever.

<comment><USER9> Quick reminder that rebuttals are more valuable than downvotes.

<comment><USER10> Rebuttals take much longer to write.

Assuming that two endpoints are using the same or forward-compatible schemas, there's no "assumption" involved: if you sent a message of type T, it gets decoded as type T. There isn't protection against inauthentic messages, admittedly.

Would you like to cite an RPC scheme that meets your definition of type safety.

<comment><USER9> If you just assume that all the participating programs are correct, then type safety has no meaning. Part of type safety is being able to check, statically or dynamically, that a program isn't going to interpret a value of type t as an incompatible type T. GRPC has no such facilities. It does not pass around or assert anything about the scheme that was used to encode the message such that the decoder can check safety. Nothing prevents you from sending a wire- compatible EmployeeVacationMessage to the ReactorSelfDestructService.

</comment></comment></comment></comment></comment><comment><USER11> Having just looked into this recently, I can also safely say that moving to REST HTTP/2 on its own already provides a significant speed benefit. Closing the gap to GRPC (or even substantially beating it) is possible by switching to a fast serialization format (eg; flatbuffers, msgpack, capnproto).

While GRPC has its place it also comes with headaches like pretty lousy generated interfaces, horrible debuggability, and unpredictable scaling.

<comment><USER12> It's kinda weird to compare REST to RPC protocols though. It's a completely different paradigm. Maybe you're just building a HTTP API without the REST part? (JSON-RPC is a thing)

<comment><USER13> Actually traditional RPC is one of the paradigms considered in Fielding's thesis coining the term. I don't know why people seem to insist there are no other models than REST if you happen to have browser on one end. Sending Javascript down the asynchronous pipe designed for XML was at the very least a step back away from REST toward something more traditional (moving code, not data).

Section 3.5 "Moving code styles" (and 3.4 for RPC):

[https://www.ics.uci.edu/~fielding/pubs/dissertation/net_arch...](https://www.ics.uci.edu/~fielding/pubs/dissertation/net_arch_styles.htm#sec_3_5)

</comment></comment></comment><comment><USER14> Strong recommendation for ZeroMQ as well. It's a thin layer on TCP, is very fast, offers bindings in many languages, and is fairly flexible as well.

[https://zeromq.org/](https://zeromq.org/)

<comment><USER15> especially when you use the router pattern on server side in order to have multiple sessions on the same socket.

<comment><USER16> Got any references on the router pattern? I searched but came up with a lot of IT related articles.

<comment><USER15> [https://netmq.readthedocs.io/en/latest/router- dealer/](https://netmq.readthedocs.io/en/latest/router-dealer/)

</comment></comment></comment></comment><comment><USER17> For speed, NATS.io also has RPC semantics and protocol is fairly straightforward like Redis: [https://docs.nats.io/developing-with- nats/sending/request_re...](https://docs.nats.io/developing-with- nats/sending/request_reply) || [https://docs.nats.io/nats-protocol/nats- protocol](https://docs.nats.io/nats-protocol/nats-protocol)

</comment><comment><USER18> We did a similar analysis in the early days at Elementary after rolling something custom based on ZMQ. We would up creating Atom: [https://github.com/elementary-robotics/atom](https://github.com/elementary- robotics/atom)

Atom is an easy, Redis Streams-based RPC that also emphasizes docker containerization of microservices. We support plug-ang-play serialization with msgpack and Apache Arrow currently supported and more on the roadmap. You can also send raw binary if you please.

Another nice thing about Redis is that if you're running microservices on the same instance you can connect to redis through a linux socket on tmpfs and bypass the TCP stack to get even better performance.

</comment><comment><USER19> This isn't a proper comparison because the writer compares a few things that aren't really the same...

Anyway the problem with comparing rpc protocols is that you need to do a per case benchmark if you really care about performance. Pretty much every decent solution will be better than another equally good solution depending on the use case.

Years ago, one of my customers told me he used to work doing high frequency trading in a bank, and the bank had several tailor made solutions for data serialization and RPC made for very specific cases, and they were just better than any generic solution.

</comment><comment><USER20> I performed the exact opposite migration at $DAYJOB recently - swapping an internal text-based protocol to HTTP (over a local IPC pipe).

The main benefit was we could suddenly reuse all the codegenerated routers/docs/authentication from the HTTP ecosystem. It significantly simplified/standardised our IPC layer and reduced the "weirdness" in the codebase.

<comment><USER21> Yep. Unless you're doing thousands of requests per second+, the right RPC protocol is likely just HTTP due to the massive improvement in mature tooling, debugging, etc.

<comment><USER22> Today's toasters can handle 1k http requests per second. :)

The overhead of Http (especially 2/3) matters so little on modern hardware.

</comment></comment></comment><comment><USER23> Folks might want to checkout RSocket: [https://github.com/rsocket/rsocket- go](https://github.com/rsocket/rsocket-go)

</comment><comment><USER24> Couple of protocols that would be nice to benchmark for comparison:

* AMQP 1.0 - can also be used for RPC without a broker in between client and server. See [https://qpid.apache.org/proton/](https://qpid.apache.org/proton/)

* Aeron - low latency, UDP based, see [https://github.com/real-logic/aeron](https://github.com/real-logic/aeron)

<comment><USER25> Re proton: I tried to use it in point-to-point mode, but haven't been able to figure out how; Javadoc reference is useless for that. There exist only Python examples but Python APIs don't map 1:1 to Java APIs.

<comment><USER24> For Java, have a look at vertx-proton which builds on top of proton-j and is a bit more intuitive (still not great) than proton-j for creating servers and clients. :)

Example "blocking" client [https://github.com/EnMasseProject/enmasse/blob/master/amqp-u...](https://github.com/EnMasseProject/enmasse/blob/master/amqp- utils/src/main/java/io/enmasse/amqp/ProtonRequestClient.java) , but might give an idea of how to set "dynamic source" required for rpc.

In general though I think the Qpid python and c++ examples might be better.

</comment></comment></comment><comment><USER26> I wish capnproto was included in this.

</comment><comment><USER27> No such thing as "RPC Protocol", there's a huge insurmountable gulf between "I want my Python and Javascript services to pass ad-hoc messages between themselves" and "I want to send C++ structs over the network with typesafety and minimal overhead".

</comment><comment><USER28> Damn it, I just learned gRPC and now we're doing something different?

This is just as bad as front-end web dev!

<comment><USER29> For perspective, RPC was invented in the 50s and 60s.

<comment><USER30> So it was invented twice?

<comment> enjoy-your-stay It was invented once in the 50s, but they didn't get the message, so they invented it again in the 60s.

</comment></comment></comment></comment><comment><USER10> The failure to use the same vertical scale for the graphs or provide an overlaid one is infuriating. So all the bar charts _look_ identical and you have to read the scale to pick up subtle factor-of-ten differences?

</comment><comment><USER15> My current rpc model for python is to pickle function name and args/kwargs and publish to redis for any subscribers that might be listening.

If the channel is marked as persisting, I put the message to a hashset with current nanosecond as the timestamp and just send a ping instead.

It is pretty fast with asyncio redis client clocking at an avg 15k rpc/s on a single core with uvloop on a lowly i7

<comment><USER31> Be careful using the pickle format; it's not considered safe to unpickle untrusted data. [https://docs.python.org/3/library/pickle.html](https://docs.python.org/3/library/pickle.html)

<comment><USER15> I did see the big warning labels everywhere. However, there is simply no replacement that is equally fast (protocol 5), easy to use (copyreg) and imports necessary modules when deserializing. So tradeoffs were made.

<comment><USER31> > imports necessary modules when deserializing

This is one of the fundamental security issues.

<comment><USER15> Most of the security issues are mitigated if you are only running the software internally. But it would be interesting to see a hacker who managed to get into the production systems somehow figure out your RPC scheme and try to craft packets to exploit it instead of going directly for the user/password database.

</comment></comment></comment></comment></comment><comment><USER32> If you're on the same machine then pipes, unix sockets and shared memory are even faster options.

<comment><USER33> It wouldn't really be an _R_ PC, then would it?

<comment><USER32> It's not a normal procedure call within the same process.

<comment><USER33> Right. Those are methods for _I_ PC. _I_ PC doesn't have to deal with all the same issues as _R_ PC, and so those techniques are going to be faster as a result. You can't really compare them.

It's like comparing communication between people physically located in the same room and people located on different continents.

<comment><USER32> RDMA shared memory muddies that distinction.

</comment></comment></comment></comment></comment><comment><USER34> No mention of authentication or encryption? I can't believe that everyone is either using HTTPS with either client certificates or plain HTTP authentication, or using IPsec (or some form of IP tunneling) to make encryption transparent to the application.

<comment><USER10> Don't know why this was downvoted - it's one of those things that's a huge pain to retrofit.

</comment></comment><comment><USER35> Curious to hear anyone's experience with RPC over Nameko in Python. We use it extensively on top of RabbitMQ and its been pretty robust.

<comment><USER36> RPC over Rabbit can work (and does) but you do have to remember that it in the end it is a queue and a couple of bad messages can stop your entire system.

RPC over Rabbit is fine if you don't care about the result, or you can guarantee that each message gets processed in a short constant time.

</comment></comment><comment><USER37> Should include smf[0].

[0] [https://github.com/smfrpc/smf](https://github.com/smfrpc/smf)

<comment><USER38> A niche cpp-only lib that is looking for a maintainer.

I would not consider using it for production code either.

</comment></comment><comment><USER39> i think rpc solutions should be able to gain in performance by having host byte order/little endian as the wire format. I mean spending all those cycles on bit swapping is pointless if both endpoints are in host byte order.

</comment><comment><USER40> Wait what ? Why is emulating a redis server (and to whom ?) is even a thing ?</comment>


Archiving web sites - <AUTHOR> https://lwn.net/Articles/766374/ <comment> fake-name Whee, a topic close to my heart!

[https://github.com/fake-name/ReadableWebProxy](https://github.com/fake- name/ReadableWebProxy) is a project of mine that started out as a simple rewriting proxy, but at this point is basically a self-contained archival system for entire websites, complete with preserved historical versions of scraped content. It has a distributed fetching frontend[1], uses chromium[2] to optionally deal with internet breaking bullshit (Helloooo cloudflare! Fuuuuucccckkkkk yyyyooouuuuuuu), supports multiple archival modes (raw, e.g. not rewritten and destyled, and a rewritten format which makes reading internet text content actually nice), and a bunch of other stuff. The links in fetched content are rewritten to point within the archiver, and if content is not already retreived, it's fetched on-the-fly as you browse.

It also has plugin-based content rewriting features, allowing the complete reformatting of content on-the-fly, and functions as a backend to a bunch of other projects (I run a translated light-novel/web-novel tracker site, and it also does the RSS parsing for that).

I've been occatonally meaning to add WARC forwarding to the frontend, and feed that into the internet archive, but the fetching frontend is old, creaky and brittle (it's some old code), and does a lot of esoteric stuff that would be hard to replicate.

[1]: [https://github.com/fake-name/AutoTriever](https://github.com/fake- name/AutoTriever) [2]: [https://github.com/fake- name/ChromeController](https://github.com/fake-name/ChromeController)

</comment><comment><USER2> Things that attempt to rewrite links and inline css and javascript are doomed to fail. Many sites do wierd javascript shenanigans, and without a million special cases, you'll never make it work reliably. Just try archiving your facebook news feed and let me know how it goes.

Instead, archivists should try to record the exact data sent between the server and a real browser, and then save that in a cache. Then, when viewing the archive, use the same browser and replay the same data, and you should see the exact same thing! With small tweaks to make everything deterministic (disallow true randomness in javascript, set the date and time back to the archiving date so SSL certs are still valid), this method can never 'bit rot'.

When technology moves on, and you can no longer run the browser and proxy, you wrap it all up in a virtual machine, and run it like that. Virtual machines have successfully preserved games consoles data nearly perfectly for ~40 years now, which is far better than pretty much any other approach.

<comment><USER3> Communication is dependent on JS events, which are dependent on user's action. There's also localStorage and other such things. Your method might work for some simple JS based websites, but it's no silver bullet.

<comment><USER4> While what you say is true, the above method is the _only_ method to archive arbitrary web pages. Yes it depends on user interactions to some extent, but it's possible to reasonably let a page load until fetches stop, and consider it rendered. Generally speaking, you can only archive some preset interactions with a modern web page. You can't hope to capture it all.

There are tools like WebRecorder[0] that do this to some extent by recording and replaying all requests. It's certainly a step in the right direction and demonstrates that the approach is viable. This was the only approach I tried that worked for archiving stuff like three.js demos. Worth mentioning there's also an Awesome list[1] that covers various web archival technologies.

[0] [https://github.com/webrecorder/webrecorder](https://github.com/webrecorder/webrecorder)

[1] [https://github.com/iipc/awesome-web- archiving](https://github.com/iipc/awesome-web-archiving)

<comment><USER5> What I'm trying to do with Bookmark Archiver is record user activity to create a puppeteer script while the user is visiting the page, then replay that on a headless browser later and record the session to a WARC. That should cover both dynamically requested and interactive stuff that is otherwise being missed by current archiving methods.

I also plan on saving an x86 VM image of the browser used to make the archive every couple months so that sites can be replayed many decades into the future.

</comment></comment></comment></comment><comment><USER6> In the realm of scraping and page archiving, I'd like to note a library I found useful recently, called `freeze-dry` [0][1]. It packages a page into a SINGLE HTML file, inlining relevant styles. The objective is to try and replicate the exact look and structure of the page instead of all the interactive elements. Very useful for building a training dataset for any algorithms that read web pages.

[0]: [https://www.npmjs.com/package/freeze- dry](https://www.npmjs.com/package/freeze-dry)

[1]: [https://github.com/WebMemex/freeze- dry](https://github.com/WebMemex/freeze-dry)

</comment><comment><USER7> I've been using

        wget -E -H -k -K -nd -N -p -P pageslug URL

for some time now and never had any issues with it. I created a .bash_aliases entry so that now I only have to type

        war pageslug URL

to archive some website.

I haven't archived to many websites (I focus more on media files like videos, ebooks and such) that's probably why I haven't run into any issues yet but I'd be interested if somebody has a link that doesn't work with this method just so that I can see what the result would be like.

Here's an explanation of the method I use for anyone interested: [https://gist.github.com/dannguyen/03a10e850656577cfb57](https://gist.github.com/dannguyen/03a10e850656577cfb57)

</comment><comment><USER8> Archiving one's web browsing trail seems to be a common use case. Here are some promising related projects that have been on HN:

* [https://github.com/pirate/bookmark-archiver](https://github.com/pirate/bookmark-archiver)

* [https://getpolarized.io/](https://getpolarized.io/)

</comment><comment><USER9> I've got a set of small repos of small government sites I've used a combination of `wget` and `curl` and other shell commands to snapshot, mostly so I can have a reliable mirror when teaching web scraping: [https://github.com/wgetsnaps](https://github.com/wgetsnaps)

But as the submitted article points out, archiving the Web is much trickier these days, and wget is no longer sufficient for anything relatively modern. I've been impressed with what Internet Archive has seemingly been able to do, and I've been interested whether it's the result of improved techniques on their side, or of certain sites following a standard that happens to make them more archivable.

For example, 538's 2018 election trackers are very JS dependent, yet IA has managed to capture them in a way that not only preserves the look and content, but keeps their widgets and visualizations almost entirely fully functional:

[https://web.archive.org/web/20181102125134/https://projects....](https://web.archive.org/web/20181102125134/https://projects.fivethirtyeight.com/2018-midterm- election-forecast/house/)

However, even the excellent archive of 538's site shows a huge weakness in IA's efforts: IA (quite understandably) aggressively caches a site's dependencies, such as external JS and JSON data files. If you scroll down the 538 example posted above, you'll see that despite being a snapshot on Nov. 2, 2018, many of its widgets only contain data from the last time IA fetched its external dependencies, which appears to be August 16, 2018.

</comment><comment><USER10> This post doesn't mention the Webrecorder Player[1], which is a GUI app that displays WARC files. It's probably the easiest way to view Web ARChives.

For those willing to set up a docker container, check out Warcworker[2].

[1] [https://github.com/webrecorder/webrecorder- player](https://github.com/webrecorder/webrecorder-player)

[2] [https://github.com/peterk/warcworker](https://github.com/peterk/warcworker)

<comment><USER11> I have dietary restrictions against Electron apps which is why I didn't mention Webrecorder player. Besides, last time I suggested anything touching node.js to LWN readers, the answer was a clear "nope" so I tend to thread carefully there as well. ;)

</comment></comment><comment><USER12> Disclaimer: My company works with Teyit and I've built the archiving product. Also: This is a shameless plug.

Teyit.org[0], the biggest fact-checking organization in Turkey, has their own archiving site called teyit.link[1].

It's a non-profit organization and they automatically archive any link that's sent to them via their site, Twitter, Facebook etc.. It's also usable by the public.

It's open source on GitHub[2] and we've actually been developing a new version[3] and have a plan to add `youtube-dl` along with WARC.

[0] [https://teyit.org](https://teyit.org)

[1] [https://www.teyit.link](https://www.teyit.link)

[2] [https://github.com/teyit/teyitlink- web](https://github.com/teyit/teyitlink-web)

[3] [https://github.com/noddigital/teyit.link](https://github.com/noddigital/teyit.link)

</comment><comment><USER13> gwern has a very involved post on archiving as well, [https://www.gwern.net/Archiving-URLs](https://www.gwern.net/Archiving-URLs)

Somewhere on my to-do list is archiving everything I visit on the internet. It's frustrating to know that I've seen something, but be unable to find it again.

<comment><USER14> For this, you could use SingleFile [1], an extension for Chrome and Firefox. It can auto-save pages.

[1] [https://github.com/gildas-lormeau/SingleFile](https://github.com/gildas- lormeau/SingleFile)

<comment><USER15> unfortunately it requires a lot of permissions. I try to minimize exts like this

<comment><USER14> I really did my best to minimize the APIs used by the extension. Note that Chrome 70 allows you to restrict extensions by host [1].

[1] [https://blog.chromium.org/2018/10/trustworthy-chrome- extensi...](https://blog.chromium.org/2018/10/trustworthy-chrome-extensions- by-default.html)

</comment></comment></comment></comment><comment><USER16> I think you guys might like this personal web archival tool I launched about a month ago:

[https://getpolarized.io/](https://getpolarized.io/)

It's basically an offline browser where you can capture full HTML pages locally including the iframes, and tag and annotate the content.

I should have cloud sync support in the next release (1-2 weeks) which will allow you to keep your data in the cloud and sync it between machines. Initially it will just support Firebase but I have plans to support other cloud providers via plugins.

I'd also like to support end to end encryption so that you don't have to worry about people reading your data.

There's a huge Hacker News about Polar here:

[https://news.ycombinator.com/item?id=18219960](https://news.ycombinator.com/item?id=18219960)

A semi-requested feature is full recursive archival of content but I don't think we're going to go in that direction. Instead I think we're going to support pasting or importing a list of URLs.

Many documentation sites have an 'index' of Table of Contents and this way I can just fetch and store all those URLs without over-fetching.

My background is search and I built a petabyte scale search service named Datastreamer ([http://www.datastreamer.io/](http://www.datastreamer.io/)). I'm also one of the inventors of RSS - so I have a lot of ideas on the roadmap here.

It also supports PDFs, text and area highlights, comments, flashcards and sync with Anki.

The initial response after our release has been amazing. The user base is really engaged with thousands of monthly active users and contributors.

Anyway. Take it for a spin. It's free and Open Source.

</comment><comment><USER17> I recently had to do this and after a lot of frustration with wget, httrack and some other commercial ones too, I ended up settling on the results of this free product, WebCopy.

[https://www.cyotek.com/cyotek-webcopy](https://www.cyotek.com/cyotek-webcopy)

Background: We couldn't keep the existing platform running, so had to transition to static html files.

I used the WebCopy scan log to create the apache rewrite rules to preserve the existing link structure.

Where I say WebCopy was better, it was this simple log, but also the file structure it was producing was much cleaner with less junk pages and duplicates. (the site was an absolute inconsistent mess to begin with)

<comment><USER17> I was surprised to not find any product that could create a perfect static clone of the original, as far as maintaining the incoming link structure.

I know there would be a tonne of edge cases and obviously it would need to be targeted to a particular platform, but I think we came pretty close with this simple technique.

</comment></comment><comment><USER18> I am of the opinion that anything you might need to read two-three times over a longer period of time should be copied locally, or to some service, for later retrieval.

I use Pocket a lot, and "share" into it from different devices.

But sometimes just do "wget --mirror -np -k --limit-rate=10k [https://interesting.stuff.here.org/"](https://interesting.stuff.here.org/") on a PC.

I'd actually like a self-hosted variant of Pocket, but have not really researched if those even exist. Anyone with a suggestion?

</comment><comment><USER5> Thanks for the mention of bookmark-archiver! WARC support has been high on my list for a long time, but unfortunately, I have a day job that keeps me super busy.

Also, the author, Antoine Beaupré is also an engineer living in Montreal who works on mesh networking stuff, are we the same person?! I just sent him an email to make sure it doesn't land in my own inbox...

<comment><USER11> Hi Nick! Yes, I am the same person as ... er... myself if that makes any sense at all. :) I'll followup by email, thanks for reaching out!

</comment></comment><comment><USER11> author here. AMA.

since I wrote this, i started experimenting with grab-site:

[https://github.com/ludios/grab-site](https://github.com/ludios/grab-site)

it's a wrapper around (and a fork of) wpull but the main advantage over it is it can do on-the-fly reconfiguration of delay, concurrency, ignore patterns and so on. it also provides a nice web interface. if you're only crawling one site every once in a while, wpull and crawl are fine, but for larger projects, grab-site is a must.

</comment><comment><USER19> I was working on an archiving tool a little while back, though I haven't touched it recently.

It would recursively convert a page into a single URI. Chrome seems to have a limit for URLs, but Firefox doesn't, so far as I can tell.

Copy the contents of [0] into your URL bar, and you'll see not just the page, but the Python script is embedded in it too. (It's a bit long to dump onto a forum page).

[0] [https://shakna.keybase.pub/offlineweb](https://shakna.keybase.pub/offlineweb)

</comment><comment><USER20> Those desiring something simple can check out [https://www.pagedash.com/](https://www.pagedash.com/). Note: requires login and saves pages to the cloud.

(Disclaimer: I am the maker of PageDash)

</comment><comment><USER21> I use a wget invocation similar to the one listed for ps-2.kev009.com. I've recently used HTTRack in a few places where that had issues and was impressed with it as well.

</comment><comment><USER22> Love wget! The waybackmachine is a great tool but I wish there was a more robust/complete service out there. Maybe the government is/will archive the top million sites or something like that.</comment>


The other SEC vs. Tesla settlement - <AUTHOR> https://www.reddit.com/r/teslamotors/comments/9k3qbm/the_other_sec_vs_tesla_settlement/ <comment><AUTHOR> Notice my comments. Being a good CEO/board tweeter is not that difficult.</comment>


China, UK and others' plan to replace the dollar as global reserve currency - <AUTHOR> http://experts.foreignpolicy.com/posts/2009/04/22/reserved_for_china

<comment><USER2> So on one hand, no country should ever take its currency's favor status for granted. But if you've ever spent a fraction of your life looking at the rules and regulations behind currency trading by central banks, you'll see two rules everywhere:

1 - Every central bank will trade its own currency.

2 - Every central bank will trade in USD.

(Even if they hate the USD with a blinding passion.)

The USD is the global standard not because of some diabolical scheme by the US, but rather because it is liquid beyond liquid. EVERYONE IS WILLING TO TRADE IT!!

Can the same be said about the Mexican Peso ? Nope, even some of its neighboring central banks are not willing to trade it.

<comment><USER3> The reason USD is standard is because the US used to be gold backed.

So all the countries in continental europe inflated like crazy and got off the gold standard. That left England and America which became the standard. Then England inflated, so it was down to America. America got off the gold in standard in 1971, but there was no one else to go to. America could then inflate its currency essentially taxing all holders of US Dollars, many of which are over seas.

This is the cause of the current economic problem. By removing all ties to gold the FED was able to inflate the currency causing malinvestment & seeking risky returns to stay ahead. The final play out of this is either a tough recession or hyper-inflation; what's going to happen isn't exactly clear -- it depends on how much money they continue to print.

</comment></comment><comment><USER4> Not quite whats going on. It sounds more like they just want to create some sort of international currency instrument which is a blend of %44 dollar, an the rest being a mix of euro, pound and yen. The idea being (cf normal investing) that a blend of these will be more stable than any single one.

<comment><USER3> None of them will be stable if you want something stable it has to be based on some sort of real asset, so that they can't print as much money as they want.

</comment></comment><comment><USER5> The title is unfortunate: it doesn't reflect the tone of the article; the UK government isn't plotting against the dollar's reserve status; and the phrase "dollar as global reserve currency" implies a monopoly that does not exist (<http://en.wikipedia.org/wiki/Reserve_currency>).

[http://www.economist.com/finance/displaystory.cfm?story_id=1...](http://www.economist.com/finance/displaystory.cfm?story_id=13447239) has more information on what special drawing rights are, and some of the politics involved.

</comment><comment><USER6> Russia, Iraq, Iran and Venezuela are already using as much EUR as USD for their reserves.</comment>


The History of Email - <AUTHOR> https://eager.io/blog/history-of-email/ <comment><USER2> If you're enjoying this series[1] as much as I am, let me highly recommend the book "Where Wizards Stay Up Late - The Origins of The Internet"[2].

1\. [https://news.ycombinator.com/submitted?id=zackbloom](https://news.ycombinator.com/submitted?id=zackbloom)

2\. [http://katiehafner.com/books/where-wizards-stay-up- late/](http://katiehafner.com/books/where-wizards-stay-up-late/)

<comment><USER3> I enjoyed that book. I've been surprised it doesn't get mentioned more often.

</comment></comment><comment><USER4> Obligatory link to "the case of the 500-mile email":

[http://www.ibiblio.org/harris/500milemail.html](http://www.ibiblio.org/harris/500milemail.html)

</comment><comment><USER5> This is all wrong! If you search "Inventor of email" on Google you are shown the one-true email god is VA Shiva Ayyadurai. Stop denying him the title of inventor of email with all this propaganda.

Seriously though... I thought Google was going for accuracy in their search results. It's amazing that this idiots website is listed at the top when searching for this information.

<comment><USER6> I'd argue that Google achieved maximum accuracy with that result. Most people searching for the "Inventor of email" seem to be looking for that person, because the story of his controversy seems to be way more popular than the actual history of email.

</comment></comment><comment><USER7> Alleged "History of email" and no mention of bang-addresses? For years we were all routing our emails via uunet with bang addresses before @ became a thing.

<comment><AUTHOR> I actually left them out only because I had talked about them in the context of URLs: [https://eager.io/blog/the-history-of-the-url-domain-and- prot...](https://eager.io/blog/the-history-of-the-url-domain-and-protocol/)

</comment></comment><comment><USER8> We created a visualization called, A Brief history Of Email Apps : [http://email-apps-timeline.missiveapp.com](http://email-apps- timeline.missiveapp.com)

We highlight:

\- All email client ever built on major platforms on a timeline (GH issues to add more [https://github.com/missive/email-apps- timeline/issues](https://github.com/missive/email-apps-timeline/issues))

\- Acquisition + discontinuation

As you can see last few years have been interesting!

</comment><comment><USER9> We are still using email to this day as a relied-upon means of communication -- easy to spoof, difficult to secure, and commonly exploited for spam.

I don't see this as a case of "if it ain't broke, don't fix it".

<comment><USER10> This makes me a little speechless. If a system has these attributes that was carefully crafted, complex, proprietary, then I would agree it's a bad thing. But email is basically putting text in a certain format and transporting it from A to B. It's simple and open, of course people use it to do bad things as well.

Side note: Do you really still have problems with easy to spoof messages? Or with spam? Especially the last one is a non-issue nowadays, I think. In the past we had like hundreds of spam mails unfiltered in our inboxes. Now I have a really good spam filter and checking my spam box from time to time I don't even see that many messages popping up.

<comment><USER11> Haha, ever tried to set up your own mail server?

Nowadays you are de facto bound to mega providers to get reasonable mail service. It's just not an open platform anymore if you periodically can't send mail to 50% user base silos (yahoo, gmail, live) because of unpredictable spam filtering and no support from them to fix it.

And you need like 1GiB of RAM just to run the spam filtering monster software patchwork of perl spaghetti. Not even mentioning the maintanance work of updating this stuff.

Mail is forever broken and converges to a marked with high entry barriers in which only google and microsoft play.

<comment><USER12> I've got one Barracuda left in production. It has 512 MB of RAM but handles tens of thousands of messages a day.

Hosts running FreeBSD with 1 GB of RAM handle a few times that with ease.

</comment></comment></comment></comment><comment><USER13> The feeling when you overwrite the spool instead of appending to it.</comment>


Find Solutions Faster by Not Looking For Them - <AUTHOR> http://www.infoq.com/presentations/Searching-Without-Objectives <comment><AUTHOR> For some problems, looking for novel outcomes rather than a specific one may actually lead to finding better solutions, more quickly.</comment>