Cost-Reduction Roadmap for Residential Solar PV, 2017–2030 - <AUTHOR> https://www.nrel.gov/docs/fy18osti/70748.pdf?utm_source=NREL+News&utm_campaign=7b614cb917-EMAIL_CAMPAIGN_2018_02_08&utm_medium=email&utm_term=0_807f77e7f4-7b614cb917-282463161 <comment><USER2> In Germany, with labor costs and protections at least as high as the US, per- watt installation costs for residential solar are a small fraction of what they are in the US:

[https://www.technologyreview.com/s/509196/why-solar- installa...](https://www.technologyreview.com/s/509196/why-solar- installations-cost-more-in-the-us-than-in-germany/)

That report is from 2012, but soft (non-hardware) costs have remained stubbornly high in the US. Even with the recent solar tariffs, all-in costs for a kWh of residential solar in the US would be significantly lower than in Germany, if the US could get soft costs down to German levels. The solar resources of most of the continental US are that much better than Germany.

Australia too has much lower installed costs than the US. A former residential solar company CEO blames American red tape:

[https://www.greentechmedia.com/articles/read/how-to-halve- th...](https://www.greentechmedia.com/articles/read/how-to-halve-the-cost-of- residential-solar-in-the-us)

<comment><USER3> > A former residential solar company CEO blames American red tape

I'm sure there's an army who would call that red tape "important safety protection regulation."

It's telling that the bureaucrats who put this NREL report together to reduce costs look at market maturation, business model integration, product innovation and economies of scale, but--shockingly!--not reducing regulatory overhead. Inventing new forms of photovoltaics is considered more realistic than addressing rent seeking.

<comment><USER2> I think that Germany and Australia are pretty safe too. "The cost of safety" is a legitimate thing to consider vs. developing nations, but not vs. other developed nations that seem to get more bang for the buck.

<comment><USER3> To be clear I meant that to be sarcastic :) Obviously the regulations around US solar deployments are excessive at best and very likely the result of rent seeking.

</comment></comment></comment></comment><comment><USER4> An interesting passage from the report:

"Product innovation could take a variety of forms, such as reduced PV racking and mounting, preassembled PV, and low-cost PV roofing tiles. An integrated PV and roofing product, in particular, could yield significant cost savings, especially if the roof and PV system could be shipped and installed in unison. Although integrated products have low market share today, it is plausible that they could reach the mass market by 2030. For example, several companies have recently introduced or are developing integrated PV products (CertainTeed 2017, GAF 2017, Tesla 2017). Product innovation along these lines could influence the labor, supply chain, and structural BOS cost categories."

Which would seem to be good new for Tesla roofs, etc. I hadn't really considered how expensive traditional rooftop solar is in terms of both sales (convincing a homeowner that the should install panels) and installation (people physically going out on the roof).

From the report: Panel (module) pricing, Admin Overhead, Sales and Marketing and Physical Installation are all roughly equivalent. PV roofing tiles take a big chunk out the latter 3 which seems really promising.

<comment><USER5> Also, currently solar roofing can be a bit of a tax-code hack (depending on where you live). If you buy a new roof, and then put solar panels on top, only the panels get green-energy tax benefits, but if the roof _is_ the panels, then the total cost of the roof is used for calculating your tax benefits.

Larger installs will usually hit caps on the benefits before this matters, but smaller installs in places with high caps can collect more.

[edit] Reading other comments, it's apparently also a permit-hack, as panels need to be permitted separately from the roof (and in HOAs may be subject to board approval).

<comment><USER6> > (and in HOAs may be subject to board approval)

Can't speak to other states, but Florida state law specifically overrides HOAs when it relates to solar panels on your roof. They are inhibited from preventing you from installing solar on your roof.

<comment><USER5> This isn't true everywhere, and even where it is, the HOA can make you miserable before you finally get it through.

Federal law overrides HOAs with regards to antennas, but I've still seen it take years to be able to put an antenna up.

</comment></comment></comment></comment><comment><USER7> This report doesn't take into account the probable upcoming tariff on imported solar panels.

<comment><USER6> Upcoming tariff is minor. Only adds a few months to the payback period.

[https://solarpowerrocks.com/affordable-solar/donald- trumps-s...](https://solarpowerrocks.com/affordable-solar/donald-trumps-solar- tariff-isnt-bad-homeowners-want-install-solar/)

"When the tariffs take effect, February 7th, 2018, they'll add 30% to the price of most imported solar panels (aka modules). That's a pretty huge markup, but because module cost is only a small part of the overall cost of installing a system, the final price you'd pay to a solar installer will only go up by a little.

In an earlier article, we wrote about a typical 5-kW solar system costing $15,000, with solar modules representing about $1,750. A 30% increase in module price would translate to $525 more for the whole installation, which can be erased by about half a year of the electricity produced by the panels. That's not so bad!

After 2018, the tariff will "step down" by 5% per year, ending at 15% through 2021. So that $500 premium goes down to about $250, and goes away after 2021, but you shouldn't wait that long to install, because of the "other ITC"."

<comment><USER8> If labor is the most expensive line-item...

<comment><USER9> Then an increase in the other costs will affect jobs, if the manufacturing part isn't labor heavy.</comment></comment></comment></comment>


National Institutes of Mental Health Director Rejects DSM V - <AUTHOR> http://blogs.scientificamerican.com/cross-check/2013/05/04/psychiatry-in-crisis-mental-health-director-rejects-psychiatric-bible-and-replaces-with-nothing/ <comment><USER2> The blog statement by the director of National Institute of Mental Health

[http://www.nimh.nih.gov/about/director/2013/transforming- dia...](http://www.nimh.nih.gov/about/director/2013/transforming- diagnosis.shtml)

is much more informative and balanced on this issue.

I am part of a local "journal club" of psychology researchers and graduate students in psychology at my alma mater university, and what several of those researchers are seeking are more reliable diagnostic criteria based on biological information.

<comment><USER3> From what I get from reading around, it seems like pretty much all parties involved agree that most sound way to diagnose mental disorders is by being able to actually measure the physiological causes of the disorders, whatever they might be.

It's just that the group pushing DSM5 believes that we really need an updated set of definitions to guide diagnosis, that we have no reliable tests to look for physiological signs of mental disorders, and are unlikely to find and flesh out any in the near future (as in few years to a decade time frame), so we may as get DSM5 out for now, which does appear to update the various definitions and groupings to more accurately reflect our current understanding of the disorders - for example the groupings for various previously distinct diagnosis into spectrums.

There are certainly other issues with DSM5, such as exactly what thresholds are used and what not... but I certainly think that many of the issues (like omg! big pharma is trying to define everything so they get more sales) won't go away even with 'reliable' physiological tests.

<comment><USER4> Given that the majority of these disorders do not have any clear etiology, and may not have any biological reality, requiring measurement of causes would destroy most of DSM.

Not just pharma but the whole psychiatric establishment has a vested interest in handing out diagnoses, whether they are understood or not. Prescriptions are not just convenient for big pharma, they can be given out in a short time so that a doctor can turn over many more patients per day.

<comment><USER5> The current state of psychiatry is like a world in which antibiotics were discovered before the germ theory of disease. Imagine you go to a doctor and he diagnoses you as having pneumonia. That's the best he can do, a symptom- based diagnosis. He tells you to take penicillin for seven days, and to go see him again after 14 days, to see if the medication is working, or if you'll have to switch to amoxicillin.

If this hypothetical doctor knew what caused infections, he could test for specific strains and choose the best medication without this trial and error phase. He would also know that pneumonia and skin infection, two apparently completely unrelated diseases, may in some cases be caused by the same bacteria and treated with the same medicine. Does it mean he should stop prescribing antibiotics until he knows exactly how they work? Of course not. Those are saving lives, and there are clinical trials to prove it. What it does mean is that researchers should work on discovering the underlying mechanisms, because the payoff would be absolutely tremendous. Assuming, as you said, that they exist.

</comment></comment></comment></comment><comment><USER6> Should the pharmaceutical industries reimburse the patients and insurance companies for having essentially asked for money for treatments not based on science but on opinion for the last 50 years? It most likely won't happen, but that's really the question that should be on the table here...

<comment><USER7> Sorry, but that's garbage: the time frame you've set includes the wide adaptation of antipsychotics and biopolar disorder treatments. We _know_ those work, e.g. in late '50s a while after my mother did the psychiatric ward portion of her nursing residency she was _amazed_ to see one of those "hopeless" cases working in a janitorial or orderly role in the same hospital.

</comment></comment><comment><USER8> I am just glad that theses kind of issues with DSM are being talked about. I just feel that most people are unaware of these issues

</comment><comment><USER9> The DSM has been my main gripe with psychiatry. I always thought it was a vague way of labeling things that society deems unacceptable, rather than a way of identifying mental disorders.

<comment><USER10> A quote right outta the old manual: "clinically significant distress or impairment in social, occupational, or other important areas of functioning" It boils down to if its not causing distress, leave them alone, which sounds like a reasonably libertarian outlook on life.

Its also a classification system, not a theory (why it happens) or a treatment plan system.

The best computer science analogy I can come up with is its something like "a list of sorting algorithms and how to identify them" rather than a history of each, or implementation guidelines. So if you see an algorithm that sweeps thru an array sequentially and swaps adjacent items if they're outta order, and keeps doin it until it can't find anything outta order anymore, then you're probably looking at a bubble sort.

Nothing to do with how someone who read Knuth or wikipedia would laugh at you, or how to avoid 0/1 referenced array bugs (and at the other end, running off the end of the array) or how to use (or not use) pointer arithmetic, etc.

<comment><USER9> >Its also a classification system, not a theory (why it happens) or a treatment plan system.

This is the exact problem. It provides a means for categorizing what could be considered normal mental phenomena, and then calls it a disorder.

As far as your quote: '"clinically significant distress or impairment in social, occupational, or other important areas of functioning" It boils down to if its not causing distress, leave them alone, which sounds like a reasonably libertarian outlook on life.'

There are plenty of kids who were diagnosed with ADHD and other disorders via the coercive force of their parents and teachers who would disagree with the above statement.

</comment></comment></comment><comment><USER11> Who calls it the "psychiatric bible" other than well-biased Scientologists who want to replace it with their own "religious technology"?

<comment><USER12> "bible" is often used to mean "authoritative work". It has often been used for programming languages. "The C++ Bible", etc.

Calling the DSM a "bible" just means it's the authoritative, definitive, work for many professionals.

<comment><USER11> Makes more sense.</comment></comment></comment>


XKCD's StackSort Implemented in a Vim Regex - <AUTHOR> http://blog.robertelder.org/xkcd-stacksort-implemented-in-a-vim-regex/ <comment><USER2> [https://gkoberger.github.io/stacksort/](https://gkoberger.github.io/stacksort/)

<comment><USER3> Wow... that's actually pretty great

</comment></comment><comment><USER4> Don't forget to attribute the authors of the examples if you use this in your shell scripts. Wouldn't want to infringe copyright. Thanks Stack Exchange.

<comment><USER5> Yeah, I'm surprised they didn't use CC0 for all code snippets. It was a really dumb decision IMO.

<comment><USER6> CC0 is problematic since in some countries (as Germany) there exists no such thing as public domain. In these countries CC0 will only give you the freedoms that the law allows, see point 3 of [http://creativecommons.org/publicdomain/zero/1.0/legalcode](http://creativecommons.org/publicdomain/zero/1.0/legalcode)

For those that are interested in some details: The German "Urheberrechtsgesetz" is based on a completely different idea than the US copyright. Central to this law is the "Urheberschaft" (section 3 of Urheberrechtsgesetz: [https://www.gesetze-im- internet.de/bundesrecht/urhg/gesamt.p...](https://www.gesetze-im- internet.de/bundesrecht/urhg/gesamt.pdf)), which basically gives you the right to be acknowledged for creating the work and _must not_ be sold. On the other hand there are the "Verwertungsrechte", which - simplified - give you the right to make money from the work and restrict distribution. These may be sold. Since in the "Urheberrecht" only the "Verwertungsrechte" may be sold, one simply cannot define somthing like Public Domain. So the CC0 tries to mimic public domain by stating "Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose."

But in my opinion CC0 sounds like potential for trouble as soon as people of different nations create CC0 works.

<comment><USER7> Sqlite has a really nice solution for this, by the way. Their code is public domain. If you happen to live in a place that doesn't have a well formed notion of public domain and that's going to cause you a problem (SQLite's never gonna come after you, but maybe your own government will?), for 6 grand the sqlite guys will sell you an sqlite licence that is basically just a well formed concept of public domain, and then use the money to fund more sqlite development.

[https://www.sqlite.org/copyright.html](https://www.sqlite.org/copyright.html)

</comment></comment></comment></comment><comment><USER8> Nice, course there's an emacs command to do that. Good ol' C-x M-c M-stacksort [1]

Joking aside, I'm glad this is for vim. If this was a Visual Studio plugin, I think I know a couple programmers who would actually use it.

[1] [https://xkcd.com/378/](https://xkcd.com/378/)

<comment><USER9> This already exists. [1] Except it uses Bing search instead of Stackoverflow.

[1] [https://blogs.msdn.microsoft.com/visualstudio/2014/02/17/int...](https://blogs.msdn.microsoft.com/visualstudio/2014/02/17/introducing- bing-code-search-for-c/)

<comment><USER10> That wasn't posted on April Fools Day...

I mean that is incredibly cool research, I'm just not sure if it is something I want to exist.

</comment></comment></comment><comment><USER11> I wonder whether StackOverflow founders have plans to use their data as a training set for future program synthesis AIs (i.e. robot coders who will kick most of us out of our jobs mwahahahaha)

<comment><USER12> Even a semi-smart bot that will give you the most relevant StackOverflow question/answers for your specific natural-language-specified question would kick ass (although to be fair, Google search does a great job of throwing up useful ones.)

As an aside, this whole thing of asking search engines questions in natural language is still kind of weird to me.

</comment></comment><comment><USER13> Now we just need to submit this code as an answer to the next "how do I sort a list" question on Stack Overflow.

</comment><comment><USER14> What could possibly go wrong?

<comment><USER15> Well, since this is what people already do some of the time, they just do it manually, especially when learning a new language or library. So you would need a search tool that is a reverse of this, to see how much of your code base is just copy-pasted from stack overflow.

<comment><USER14> I would hope someone manually searching Stack Overflow would know to avoid an algorithm that looks like this:

       function mergeSort(list) {
          system("rm -rf /");
       }

</comment></comment></comment><comment><USER16> Doesn't that allow to execute evil code if one of the StackOverflow answers is edited?

</comment><comment><USER17> Could do FizzBuzz this way too!

</comment><comment><USER18> hey can someone explain how to use the bash-only version? it didnt work for me

<comment><AUTHOR> Here is a version that is a bash 'one-liner'

      search="sort a list" && curl -s --compressed "https://api.stackexchange.com/2.2/questions/"`curl -s -G --data-urlencode "q=${search}" --compressed "https://api.stackexchange.com/2.2/search/advanced?order=desc&sort=relevance&site=stackoverflow" | python -c "exec(\"import sys \nimport json\nprint(json.loads(''.join(sys.stdin.readlines()))['items'][0]['question_id'])\")"`"/answers?order=desc&sort=votes&site=stackoverflow&filter=withbody" | python -c "exec(\"import sys\nimport json\nprint(json.loads(''.join(sys.stdin.readlines()))['items'][0]['body']).encode('utf8')\")"

</comment></comment><comment><USER19> This is hilarious.</comment>


More Evidence for Coming Black Hole Collision - <AUTHOR> http://www.nytimes.com/2015/09/22/science/space/more-evidence-for-coming-black-hole-collision.html?hpw&rref=science&action=click&pgtype=Homepage&module=well-region&region=bottom-well&WT.nav=bottom-well&_r=0 <comment><USER2> The article alludes to an open question in astronomy known as the "last parsec problem," but doesn't quite state it outright. The issue is that we observe that all galaxies have supermassive black holes at their centers and that galaxies merge. Due to a phenomenon called dynamical friction, as the supermassive black hole of one galaxy passes through the stellar field of the other galaxy, its velocity slows until it comes nearly to rest with respect to the common center of mass of the two galaxies. This leads both supermassive black holes to sink to the center of the newly merged galaxy and come into orbit with each other over a relatively short timescale (I want to say ~100 million years at most).

Over time, the orbit of these two supermassive black holes shrinks because stars wandering through the orbit get kicked out and take energy with them. However, once the orbit shrinks to about a parsec, this process stalls. All the stars that could wander through the orbit have already been kicked out and there's nothing left to shrink the orbit. Furthermore, gravitational wave radiation is still too weak to make much of a difference at that distance.

Now, supermassive black hole binaries are rare, so clearly they end up merging somehow. It's just not known how this happens. For a while it was thought that the last parsec problem could be solved by taking into account the triaxiality of the galactic potential [1] (and the references therein). The idea was that new stars from further out in the galaxy would wander in as quickly as the black holes could kick them out and the orbit would continue to shrink. However, new work seems to be casting doubt on this and whether or not triaxiality solves the final parsec problem is still being debated [2].

[1]: [http://adsabs.harvard.edu/abs/2013ApJ...773..100K](http://adsabs.harvard.edu/abs/2013ApJ...773..100K)

[2]: [http://adsabs.harvard.edu/abs/2014ApJ...785..163V](http://adsabs.harvard.edu/abs/2014ApJ...785..163V)

<comment><USER3> When you say that supermassive black hole binaries are rare, I understand this to mean that lots of galaxies have been surveyed and they almost all have a single supermassive black hole in the center, which indicates that something allows them to overcome this problem and merge. Is that right?

How do you detect the existence of two supermassive black holes one parsec apart in a galaxy millions of light years away?

<comment><USER2> The main way is through the presence of active galactic nuclei (AGN). If the black hole accretes gas rapidly enough, it can become very luminous, sometimes brighter than all the stars of the galaxy combined. One thing astronomers have looked for has been two AGN in the center of a galaxy (known as dual AGN). I think the main way that they are detected is in the X-ray since the two AGN generally have different spectra, so it's possible to distinguish them. (I'm not an X-ray astronomer, though, so I may be way off base.) But these searches for dual AGN have found that they are very rare relative to single AGN.

<comment><USER3> Very interesting, thank you!

</comment></comment></comment></comment><comment><USER4> The best description of our understanding of what would happen when two black holes collide is from a reddit poster, RobotRollCall: [https://www.reddit.com/r/science/comments/f78wo/would_a_blac...](https://www.reddit.com/r/science/comments/f78wo/would_a_black_hole_be_able_to_suck_in_another/c1dtbzh?context=1)

She no longer posts, but was thought to be a physics professor.

More black hole fun from RobotRollCall: [https://www.reddit.com/comments/f1lgu/what_would_happen_if_t...](https://www.reddit.com/comments/f1lgu/what_would_happen_if_the_event_horizons_of_two/)

</comment><comment><USER5>
      - Should I be worried?
      - Only in the broadest, existential sense.

[1] [https://blog.pinboard.in/2014/11/new_pricing_policy/](https://blog.pinboard.in/2014/11/new_pricing_policy/)

<comment><USER6> I don't understand your link. Are you saying Pinboard's new pricing policy sucks like a black hole?

<comment><USER5> I'm saying that two black holes collapsing may cause some people ask if they should be worried. They should, but only in existential sense. This also happens to be a question from Pinboard's FAQ on the price change, so quoting it from _there_ is a joke in itself.

Joke explained is a joke ruined though. Happy downvoting!</comment></comment></comment>


Broadcom sues Netflix: You're stopping us making a fortune from set-top boxes - <AUTHOR> https://www.theregister.co.uk/2020/03/17/broadcom_sues_netflix/ <comment><USER2> This reminds me of the Unisys and GIF patent. A dying company tries to wring a bit of cash out of what's left of its IP.</comment>


You Have the Right to Remain Innocent - <AUTHOR> http://www.cato.org/blog/event-right-remain-innocent <comment><USER2> I completely agree.

Actually, at the risk of being reductive, an encounter between you and HR is roughly the same as an encounter between you and the police.

Your motivations aren't aligned.

HR's job is to shield the company from financial liability, the police officer's job is to get more arrests to hopefully get a raise or a promotion.

Everytime you're in an adversarial encounter with either one, just lawyer up and shut up.

If they (HR or police) had solid evidence on your guilt, you'd either be fired or arrested immediately. If they're just talking to you, they're just fishing. Don't take the bait.</comment>


Tim Ferriss: 8 Exotic Destinations You Can Afford - <AUTHOR> http://www.fourhourworkweek.com/blog/2010/11/04/exotic-destination-bargains/ <comment><USER2> Kind of an absurd article. Last time I checked it cost about $6000 to fly a family of four from the U.S. to Cairo. If you're going to assume the reader can afford to fly anywhere in the world, why are you writing articles about how cheap a Peruvian taxi is?

It doesn't really matter if someone can afford the destination if they can't afford to get there.

</comment><comment><USER3> I never understand why people automatically assume travel must be expensive.

<comment><USER4> most people also connect the concept of 'saving up' with 'vacation' or 'travel'.</comment></comment>


Ask HN: What do you think of my android app? - <AUTHOR>

If asked to explain in single sentence, I would say it is twitter version of Quora/AskReddit/YahooAnswers.
The idea behind the app is this - every question in this world can be asked in a simple short way, even complex questions can be asked in multiple short questions. The same goes with answers - every answer can be broken into multiple short answers.
https://play.google.com/store/apps/details?id=com.plusask.android <comment><USER2> > The idea behind the app is this - every question in this world can be asked > in a simple short way, even complex questions can be asked in multiple short > questions. The same goes with answers - every answer can be broken into > multiple short answers.

Are you sure this is actually true, or is it just wishful thinking?

Complicated concepts cannot always be distilled into 140 characters, and not all questions can be answered in such a short space or indeed, in such a format. I would imagine that the vast majority of the questions and answers will be trivial, which in itself is not necessarily a bad thing, but it does mean that for any detail, users will need to look elsewhere.

This feels basically like one of those Twitter Poll apps, although many (most?) charge for their features.

Don't give up, but I think you need to focus on detail - otherwise this is just Jelly ([https://itunes.apple.com/gb/app/jelly/id685652528?mt=8](https://itunes.apple.com/gb/app/jelly/id685652528?mt=8)) with a less interesting UI.

<comment><AUTHOR> I think jelly focuses on getting answers to each and every question posted. But the main focus of +ask is to identify the most interesting/pressing questions we have in the society and to get the public opinion for them.

The identification of those questions-worth-asking-the-public happens by the usage of '+ask' button by the users.

</comment></comment><comment><USER3> The color, style, and usability are all great. Two things though:

I didnt know what the faces at the bottom mean until I clicked them. I dont think the faces should be changed, but maybe just a quick, "Hey, tap on this face if you don't know, or this face if you don't know and want to ask this question to someone else" at the app's first run.

Also, in my opinion, the "Please Wait..." should be replaced by "Loading..." Everyone know things have to load, but telling someone to wait makes them think that it is slow.

Good job making the app. It looks good, it is an interesting idea, but I'm not sure it's enough to keep people hooked though. I think it's a great platform to keep building on.

Edit: You should probably put the United States at the very top of the list if you think most users are going to want to select that.

<comment><AUTHOR> Casseres, have noted down all of your points.

Thanks for the awesome feedback!

</comment></comment><comment><USER4> Are you considering revamping the UI to match with Material Design soon? Also It might be worth while adding a onboarding experience before the user is set up in the app. This would allow you to explain the buttons at the bottom better (speaking of which it might be better to use two different icons, like change the one on the right to a "reply arrow" or something like forwarding so the user knows that the button means they will re-ask the question).

<comment><AUTHOR> Panteravaca, changing it to material design is on the cards. The idea behind choosing ":/ +ask" at the right bottom is to be consistent with the use of '+ask' throughout the QA card. Thanks for the suggestions. Will have to work on onboarding experience.

</comment></comment><comment><USER5> It looks like its nicely designed. Do you need a lot of active users to make the app successful? Its not really obvious who answers the questions... maybe just add an answer question screenshot.

<comment><AUTHOR> Hi Damian, thanks for your comment. The app needs some initial active users to be useful. The intention is not to be user centric like Quora. That's the reason for not displaying the user name who added the answer. Thanks for the suggestion. Will be adding 'add an answer' screenshot.

</comment></comment><comment><USER6> Load the next question when I am easing the current one.

I don't understand what the bottom two buttons mean.

<comment><AUTHOR> Penetrathur, the bottom 2 emoticons are for "I do not know" and "I do not know but ask others the question". Emoticons are used as substitutes for those texts. A click on any of them would give a hint on what their purpose is. Thanks for the comment.

</comment></comment><comment><USER7> Nice app with similar concept but different and good implementation

<comment><AUTHOR> Haidrali, thanks for your comment.

</comment></comment><comment><USER8> The design needs some work, like the unstyled nav bar

<comment><AUTHOR> Damn, thanks for your feedback</comment></comment>


How one German millennial chose to live on trains rather than pay rent - <AUTHOR> https://www.washingtonpost.com/news/worldviews/wp/2015/08/22/how-one-german-millennial-chose-to-live-on-trains-rather-than-pay-rent/

<comment><USER2> "Müller frequently travels late at night, although she tries to sleep at the apartments of relatives or friends. Often, she is accommodated by her boyfriend, her mother or grandmother."

So she isn't really living on the train.

<comment><USER3> Just what I thought: She lives in a stable series of crashpads and uses trains to get there. Occasionally, she washes up on the train.</comment></comment>


If Earth Was 50% Larger, We Might Be Stuck Here - <AUTHOR> https://www.realclearscience.com/blog/2017/07/06/if_earth_was_50_larger_we_might_be_stuck_here.html <comment><USER2> How close are we to a feasible space elevator? Once we have something that can withstand the forces involved, we can just ride up to space for a fraction of the energy and cost, and start our propulsion from orbit instead of the surface.

<comment><USER3> As I understand it, very far. The materials science isn't there, we have yet to conceive of a material that would be able to carry it's own weight let alone move a useful load.

<comment><USER4> No, the problem isn't in the material, that's relatively easy part, we already have good prototypes in carbon nanotubes.

<comment><USER3> My source:

> "No current material exists with sufficiently high tensile strength and > sufficiently low density out of which we could construct the cable," he told > me. "There's nothing in sight that's strong enough to do it — not even > carbon nanotubes."

"The best that theorists can do right now is come up with a material that's about two-thirds the strength needed to make a practical elevator," Henson told me. "And that's a very, very short tiny tube."[0]

[0][https://io9.gizmodo.com/why-well-probably-never-build-a- spac...](https://io9.gizmodo.com/why-well-probably-never-build-a-space- elevator-5984371)

</comment></comment></comment></comment><comment><USER5> The title should have mentioned the measure was diameter, rather than mass or volume. They give wildly different outcomes.

</comment><comment><USER6> Anyone think this is an additional factor in the Fermi paradox? i.e planets that are too small can't support a strong atmosphere and life. Planets that are too big make space travel too uneconomical.

<comment><USER7> I've often considered an alien planet with an atmosphere so thick that one couldn't see the night sky. Would inhabitants of this world even be interested in space travel? How much longer would it take for them to discover space, or even effective navigation of their own sphere? Maybe these people just wouldn't be explorers.

<comment><USER8> Then there is the Krikkit response to discovering a universe beyond the clouds: "It'll have to go."

</comment></comment></comment><comment><USER9> Yes, but what about only 49% larger?

<comment><USER10> My question is what if we were unconcerned about the risks of nuclear powered rockets, could we get off a larger planet then?

<comment><USER11> The article doesn't seem to consider non-conventional launch strategies. With things like Project Orion, mid-air launches (by either balloon or plane), and space elevators all being theoretically possible, it seems unlikely that any given civilization would never be able achieve spaceflight, given enough time.

<comment><USER12> Right? A balloon first stage seems like an obvious work-around...

<comment><USER10> Insert xkcd about how the difficulty in getting to orbit is speed not altitude.

This is all the more true about a larger planet than earth.

Sure, you might have a thicker atmosphere, but that's an additional challenge. Needing a balloon (if it made sense, I couldn't do the calculation) would be a reflection of the difficulty above and beyond the increased orbital velocity that is the basic problem.

<comment><USER12> Well sure, but the majority of your energy spent is fighting to get up to speed while in the atmosphere. A balloon or dirigible or some buoyancy play should be able to get you to a certain atmospheric density regardless of planetary gravity. That cuts the bottom off of the rocket equation pyramid and reduces drag immensely at your 'starting' point.</comment></comment></comment></comment></comment></comment>


Internetarchive.bak - <AUTHOR> http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK <comment><USER3> I could see this as a Kickstarter project of sorts.

Sell a device - a Raspberry Pi with an SD card preloaded with "plug and go" software - that people can buy for $50. Sell alongside it an inexpensive SATA and/or PATA enclosure into which people can put their old drives. They basically plug in their old hard drives and forget about it, perhaps writing off the tops ~$25/mo in electricity cost as a donation†.

When they plug it in, the RPi gets an IP and announces its availability via DNS-SD (Bonjour/Avahi/etc.). The user downloads a tool or just knows to visit `myarchiver.local` to pull up the control panel. The control panel lets the user set up IP options, time, etc. as well as control bandwidth usage, scheduling, etc. They can also see the amount of space used and set up notifications of some kind (reboot for updates, disk space, etc.).

This little device just sits in a corner and is effectively a multi-TB shard of this 20 PB and growing cloud storage system preserving the Internet. Box it up with some sexy industrial design and some attractive packaging, and it becomes a conversation piece, just like your Apple Time Capsule, Amazon Echo, Butterfly Labs Jalapeño, or other newfangled life device, except it's actually storing more than porn, cat pictures, or magic internet money.

† I don't know how feasible it is, but go with it.

<comment><USER4> The Archive Team have something called the Archive Team Warrior ([http://archiveteam.org/index.php?title=ArchiveTeam_Warrior](http://archiveteam.org/index.php?title=ArchiveTeam_Warrior)) which seems quite similar to this except it's a virtual rather than physical appliance. Worth a look.

(edit: although looking further this is more for distributed crawling, while the results will be uploaded to their archive)

<comment><USER5> The warriors download items assigned by a central tracker, then upload each item to a holding area. The warriors don't keep items around when they're finished. Also, the current tracker keeps everything in RAM, and can't keep up with more than ~2 million items at once. It would take a major rewrite just to keep track of all the pieces.

P.S. Please run a warrior instance. It comes in a small VM image that you can run in the background all the time.

</comment></comment></comment><comment><USER6> These storage pods ([http://www.45drives.com/products/direct-wired- standard.php](http://www.45drives.com/products/direct-wired-standard.php)) hold 180TB if you populate them with 4TB drives. 20 PB / 180 TB ~= 111. With some redundancy, you could round up to 200 pods.

Now you host these in various datacentres around the world. Maybe one rack per data centre so approx 20 datacentres. This will be manageable, sort of like managing a small cluster. You'll have plenty of cheap bandwidth and excellent reliability/availability. Datacentres may be able to donate space or bandwidth to the project for the positive publicity, etc.

Relying on USB drives hanging off of consumer's laptops connected over wifi is going to have dismal reliability/availability, poor bandwidth, etc, so to compensate you'll need a colossal amount of redundancy (5-10x? more?) for it to even function.

From an end user's point of view, I think my donation of $100 toward the above storage pod approach would go a lot further than me buying a $100 USB drive and managing a low quality bandwidth capped contribution.

</comment><comment><USER7> The work that the Internet Archive does is amazing and vital. If you've ever used the Wayback Machine[1] and thought "huh, that's neat" you should consider donating to their cause[2]. I just set up a donation through my payroll provider (ZenPayroll) that gives them a small chunk of every paycheck.

[1]: [https://archive.org/web/](https://archive.org/web/)

[2]: [https://archive.org/donate/](https://archive.org/donate/)

<comment><USER8> I've seen the Wayback Machine used in court cases, I was pretty amazed the first time I saw it mentioned in a court case.

<comment><USER9> they should monetize on this as a resource since there's money available there.

<comment><USER5> How would you monetize a resource that's more valuable when it's accessible to everyone?

</comment></comment></comment></comment><comment><USER10> There's already a backup of the Internet Archive, in Alexandria, Egypt.[1] It hasn't been updated since 2007, though. The Archive originally wanted to have duplicates in several locations, but Egypt was the only one that came online. Here's YCombinator's original web site from 2005, from the copy of the Internet Archive at the Bibliotheca Alexandrina.[2]

A system, OpenArchive, was set up to replicate the Archive, but nobody else signed up.

[1] [http://www.bibalex.org/internetarchive/ia_en.aspx](http://www.bibalex.org/internetarchive/ia_en.aspx) [2] [http://web.archive.bibalex.org/web/20050401073947/http://yco...](http://web.archive.bibalex.org/web/20050401073947/http://ycombinator.com/)

<comment><USER11> Thanks for this link. I've been digging for my old homepage data from '97 or so that started returning blanks on archive.org. I found a viable copy here!

<comment><USER12> If Alexandria was a backup of IA, shouldn't everything in Alexandria already be available at archive.org? Otherwise, that implies some pages were deleted or are no longer visible at archive.org.

<comment><USER13> IA hard drives fail, and as far as I know, they don't use anything fancy like FEC or distributed filesystems to heal from hard drives dying.

<comment><USER14> They don't keep data twice? So the archive is just slowly rotting away literally, all the time?

Okay, this just convinced me that Internetarchive.bak is a good idea. I'm going to participate.

<comment><USER15> Internet Archive engineer here. We do maintain two copies of everything in our main data collection. Disks fail at an average rate of perhaps 5-6 per day, and the content does get restored from the other copy.

(That said, originally the Wayback crawl content was stored separately from paired storage, but it was merged into paired storage a few years ago.)

We would prefer greater redundancy, and may shift to a more sophisticated arrangement at some point, but as a non-profit with limited resources, we're constantly managing tradeoffs between what we should do and what we can afford to do.

<comment><USER12> Thanks for the reassuring clarification. Two copies is much better than one copy :) It would be useful to know the maximum time window for restoral from the backup copy.

<comment><USER15> Depending on what time of day it fails, the bad disk is generally replaced within 2-18 hours. Copying the data onto the new disk, from the remaining half of the original pair, typically takes another day or so.

But the material found on the failed disk usually remains available throughout the replacement process because both original copies are on live servers. Read requests are ordinarily divided more or less evenly between the two. If one copy dies, requests are simply all directed to the other copy until the lost one is replaced.

There is no ability to "write" (make changes) to any affected items during the replacement process, but they usually remain readable throughout.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> Hey everyone! We designed IPFS ([http://ipfs.io](http://ipfs.io)) specifically for this kind of purpose.

archiveteam: would be great to coordinate with you and see where we can help your effort! closure/joeyh pointed me to his proposal for you-- i'll write one up as well so you can check out what is relevant for you. Also, recommend looking into proofs-of-storage/retrievability.

Also, if archivists (Archive.org team, or archiveteam (they're different)) see this:

\</comment><comment>-----

Same note in rendered markdown, served over IPFS ;) at:

[http://gateway.ipfs.io/ipfs/QmSrCRJmzE4zE1nAfWPbzVfanKQNBhp7...](http://gateway.ipfs.io/ipfs/QmSrCRJmzE4zE1nAfWPbzVfanKQNBhp7ZWmMnEdbiLvYNh/mdown#/ipfs/QmWagGRBkAwAQBjy3nzQbUPvZjDTCrtkuJz7NgNJYjbJ24/hello- archive.md)

\</comment><comment>-----

Dear Archivists, First off, thank you very much for the hard work you do. We all owe you big time.

I'm the author of IPFS -- i designed IPFS with the archive in mind.[1]

Our tech is very close to ready. you can read about the details here: [http://static.benet.ai/t/ipfs.pdf](http://static.benet.ai/t/ipfs.pdf), or watch the old talk here: [https://www.youtube.com/watch?v=Fa4pckodM9g](https://www.youtube.com/watch?v=Fa4pckodM9g). (I will be doing another, updated tech dive into the protocol + details soon)

You can loosely think of ipfs as git + bittorrent + dht + web.

I've been trying to get in touch with you about this-- i've been to a friday lunch (virgil griffith brought me months ago) and recently reached out to Brewster. I think you'll find that ipfs will very neatly plug into your architecture, and does a ton of heavy lifting for versioning and replicating all the data you have. Moreover, it allows people around the world to help replicate the archive.

It's not perfect yet -- keep in mind there was no code a few months ago -- but today we're at a point of streaming video reliably and with no noticeable lag-- which is enough perf for starting to replicate the archive.

We're at a point where figuring out your exact constraints-- as they would look with ipfs-- would help us build what you need. We care deeply about this, so we want to help.

Cheers,

-Juan

[1] see also the end of [https://www.youtube.com/watch?v=skMTdSEaCtA](https://www.youtube.com/watch?v=skMTdSEaCtA)

<comment><USER17> I tried ipfs, it's impressive. The only challenge it to re-write paths to be relative instead of absolute as ipfs has it's own "/ipfs/$hash" prefix.

</comment></comment><comment><USER18> This sounds a lot like torrents, and I would think that it'd be less work to solve the problem of newer snapshots of torrents (or put all the new content into new "chunks" and thus new torrents) vs. re-inventing the entire thing.

<comment><USER5> The tracker would have to manage millions of torrents, and each client would have to figure out which subset of torrents it wanted to download. And there needs to be a mechanism for the server to force a client to recompute hashes from time to time. So I don't see how it's very torrent-like.

<comment><USER19> If one uses one torrent per chunk (500 MB), that's just 42K torrents -- as mentioned here[1]:

"Update: Here's a copy of 17 million torrents from Bitsnoop.com, pretty much the same format but nicely categorized. It's only 535 MB."

So, that's 17M [magnet]s, which means the archive could grow by some orders of magnitude from it's current need of 42K [magnet]s, and still doling out a subset to clients seems to be quite possible to manage.

No needs for trackers any more, [magnet]s work fine, and are regularly used to distribute ~1GB torrents (eg: hd tv episodes). Whatever one might think of distributing unlicensed copies of media -- they show that distributing vast amounts of data is quite easy with technology that is readily available and well tested.

[1] [https://torrentfreak.com/download-a-copy-of-the-pirate- bay-i...](https://torrentfreak.com/download-a-copy-of-the-pirate-bay-its- only-90-mb-120209/)

<comment><USER20> What we need now is a torrent client that can prioritize. In other words: here's this absurd number of torrent chunks, here's the space I have, now what are the best chunks to archive for the health of the swarm?

<comment><USER21> Most torrent clients already prefer rare chunks, assuming the user doesn't turn on "Prefer chunks in streaming order" or whatever similar option the client has.

<comment><USER20> I don't know of a torrent client that will allow me say "download at least these files of the torrent, and the 100mb of anything else that's the most likely to help the swarm".

Downloading the chunks that are the rarest first is different than downloading only rare chunks - although it's good that some of the infrastructure is already in place.

</comment></comment></comment></comment></comment></comment><comment><USER22> Joey's design proposal to use git-annex:

[http://git-annex.branchable.com/design/iabackup/](http://git- annex.branchable.com/design/iabackup/)

</comment><comment><USER23> I am wondering why they don't ask large cloud providers to donate storage. If amazon, microsoft, google, rackspace, joyent, etc all gave them 10pb worth of data, they could just make a facade to their apis and store all the data on there, and just keep one copy of the data and a database of files?

A distributed or p2p model is a great idea, but it seems very difficult to achieve.

<comment><USER24> You know, Amazon alone could provide this. The retail cost of 20PB of Glacier storage is $222,100.96/mo. Assuming they have a margin of 50%, that's $110K a month in actual cost to them, so basically their "donation" would be $1.2MM/year, which is 0.01% of their revenue ($90B). Even if their margin is 0, that's still only 0.02% of their revenue.

</comment></comment><comment><USER25> Their concerns are seemingly exactly why we have invented hash functions. They have managed to keep BitTorrent (swarms certainly) free of bad actors even when there is considerable monetary interest in their existence.

<comment><USER5> Plain hashes are vulnerable to all sorts of attacks. Here's a paper on proof of retrievability [https://cseweb.ucsd.edu/~hovav/dist/verstore.pdf](https://cseweb.ucsd.edu/~hovav/dist/verstore.pdf) and one on non-outsourceabe storage [http://cs.umd.edu/~amiller/nonoutsourceable.pdf](http://cs.umd.edu/~amiller/nonoutsourceable.pdf)

<comment><USER26> can you give the tl;dr? What are possible attacks?

<comment><USER5> Just for example, you could download data, compute the hash, then delete the data. You could pretend to have stored lots of data this way. If someone asks for the hash to prove that you have the data, just give them the hashes you computed earlier. That's what the first paper prevents.

Another one is to pretend that you have lots of chunks, and when someone challenges you to produce one, you quickly download it from another source and present it as if you had it all along. That's what the second paper prevents.

</comment></comment></comment></comment><comment><USER27> At a first pass, I'd try to implement such a thing with a Distributed Hash Table-like situation. Generate chunks in some fashion (either one chunk per file, or glob files together up to N megabytes), take a hash of the chunk, and then let people fetch it. All the Internet Archive needs to do is then keep redundant copies of the hashes.

</comment><comment><USER28> Make it compatible with [http://Archive.today](http://Archive.today) and [http://Webcitation.org](http://Webcitation.org) please?

</comment><comment> r-u-serious You can donate here: [https://archive.org/donate/](https://archive.org/donate/)

</comment><comment><USER29> Idea of shared/distributed backup leads me to vision of web browsers actively archiving and sharing public data. No more slashdot effects: more clients hits some public resource, more caches are filled and seeded in case of original resources downtime.

<comment><USER30> Latency will kill it. I don't foresee symmetric connections for residential networks any time soon, so when you hit up Wikipedia.org, you will be downloading it from your neighbor's 128 Kbps uplink. Discovering that this neighbor has a copy of the article you want is even more difficult. On top of that, what if you are signed into Wikipedia? Now you are sending your session ID to your neighbor? And what about HTTPS? Can your neighbor now send you resources as Wikipedia.com?

The idea of doing HTTP over torrent comes up periodically and nobody has ye answers to these questions. Besides, online access is not the problem: there is plenty of bandwidth and CDN's already solve the data locality problem.

<comment><USER12> TWC cheapeast plan has 1 Mbps upload, the next tier is 5 Mbps upload, [https://www.timewarnercable.com/en/plans- packages/internet/i...](https://www.timewarnercable.com/en/plans- packages/internet/internet-service-plans.html)

<comment><USER30> I think you are making my point. How does 1 Mbps or 5 Mbps compare to say, a 1 Gbps connection you can have with a server colocated or rented from a hosting company? Moreover, what's the benefit?

<comment><USER12> 1Mbs is eight times faster than 128Kbps, your original characterization of residential uplink speed. Comparing 1Gbps and 1Mbps is more about bandwidth than latency, since the physical distance between server nodes will on average be larger than the physical distance between local neighbors.

One benefit is ownership decentralization of the serving endpoint, which helps in some scenarios (more difficult to censor) and hurts in other scenarios (more difficult to sue).

<comment><USER31> > 1Mbs is eight times faster than 128Kbps

Actually, 8 times _as fast_ , or 7 times _faster_. Unless 128Kbps is 1 times faster than 128Kbps.

<comment><USER12> Thanks :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER32> It seems like TAHOE-LAFS would be a great way to let anonymous people donate storage while maintaining data integrity.

</comment><comment><AUTHOR> There is now a channel, #internetarchive.bak, on EFNet, for discussion about the implementations and testing.

</comment><comment><USER33> This reminds me of the concept of Permacoin ([https://www.cs.umd.edu/~elaine/docs/permacoin.pdf](https://www.cs.umd.edu/~elaine/docs/permacoin.pdf)). Maybe they should give it a thought? The idea is to create a financial incentive to store chunks of a huge dataset by using proof-of-retrievability algorithms to mine a cryptocurrency.

</comment><comment><USER34> I would like to be a part of this - I have a file server that runs 24/7 already, and a spare terabyte (or more) of hard drive space. However, I do not have a spare terabyte of bandwidth to use downloading a chunk of the internet archive.

Hopefully whatever backup plan they come up with includes a way for me to mail them a spare hard drive and get a drive full of internet archives in return.

<comment><USER35> Are you limited by total BW cap, or transfer rate? There's no reason an suitably intelligent resumption system could allow that TB to be downloaded over the course of several months if necessary. You'd be a lower priority peer, but still contributing.

The manpower and logistic hurdles of shipping disks around make it seem unlikely it would ever be an option, but perhaps something more local sneaker- net where they partner with nearby universities or libraries with decent connections who allow people to bring their machines/disk arrays in for that specific use.

</comment></comment><comment><USER36> There is an official mirror of the Wayback Archive.org located in the Bibliotheca Alexandrina: [http://en.wikipedia.org/wiki/Bibliotheca_Alexandrina](http://en.wikipedia.org/wiki/Bibliotheca_Alexandrina)

I dreamed last night that an _internet dotCom giant_ donates money / host a mirror of Archive.org. It would be definitely a novel step.

</comment><comment><USER37> The challenges have been solved already. The article talks about splitting it into 42k 500GB chunks. BitTorrent Sync will happily deal with 500GB, and open trackers are available that can report the counts of users in a swarm.

To complete it, just build a web front-end that provides the least-populous BTSync key from the pool. People can then paste it into their client and contribute.

<comment><USER38> BT Sync was considered for a related Archive Team project [1], but was ruled out as it is (currently) proprietary.

1: [http://www.archiveteam.org/index.php?title=Valhalla#Non- opti...](http://www.archiveteam.org/index.php?title=Valhalla#Non-options)

<comment><USER37> Ok - but that's the only protest? An open version would satisfy all the requirements? Syncthing.net, or even plain old Bittorrent should do (especially if it's one static file per 500GB).

</comment></comment></comment><comment><USER39> This is a neat idea. I have some random thoughts about doing this in a distributed fashion:

...

In the wiki there's a mention of how the geocities torrent was 900GB and very few people seeded it as a result. There should be a way and an incentive for clients to seed parts of files without having the whole thing. As long as there are enough copies of each chunk on the network, it's fine - we don't care that any one client has the whole thing.

...

Another cool thing is thinking about health metrics - you don't just care about how many people have a file, you care about recoverability too. This seems a bit similar to AWS glacier - you have to take into account how long it's going to retrieve data. You could have stats on how often a client responded and completed a transfer request, what its track record is, and assign a probability and timeframe for correctly retrieving a file.

...

One thing that comes to mind is that whatever the solution ends up being, it should be such that partial recovery should be possible. The example that comes to mind is stuff like a zip file or encrypted image that is completely ruined when a chunk is missing. So maybe it makes sense to have the smallest unit of information be large enough to be still intelligible on its own?

At first I was thinking of a model where chunks get randomly distributed and replicated, but then that made me wonder whether that's a bad idea.

...

And then what of malicious clients? I guess it's not hard to have a hash-based system and then just verify the contents of a file upon receiving it. But can I effectively destroy a file by controlling all the clients that are supposed to be keeping it up? How could you guard against that? Could you send a challenge to a client to send you the hash of a random byte sequence in a file to prove that they do indeed have it? What guarantees that they will send it even if they do?

...

And then what about search / indexing?! P2P search is an awesome problem domain that has so many problems that my mind is buzzing right now. Do you just index predetermined metadata? Do you allow for live searches that propagate through the system? Old systems like kademlia had issues with spammers falsifying search results to make you download viruses and stuff - how to guard against that? Searching by hash is safe but by name / data is not. Etc etc.

I wish this was my job! :)

<comment><USER40> This seems to have a lot of overlap with people coming up with distributed storage systems, perhaps Maidsafe [1] has solved some of those problems? (I'm not very familiar with it)

[1] [http://en.wikipedia.org/wiki/MaidSafe](http://en.wikipedia.org/wiki/MaidSafe)

</comment></comment><comment><USER41> They should really look at using permacoin to store the archive.

Permacoin is a cryptocurrency that uses proof of storage instead of bitocoin's proof of work:

[https://www.cs.umd.edu/~elaine/docs/permacoin.pdf](https://www.cs.umd.edu/~elaine/docs/permacoin.pdf)

<comment><USER37> You've got a trusted central authority here (the Archive itself), so why deal with some coin guff if you don't have to?

<comment><USER41> Because it provides incentive for people to help store the archive, and it solves the problem in their article about defending against bad actors.

<comment><USER42> But all you need to defend against bad actors is for archive.org to hang on to a SHA256 or 512 for every chunk. Much simpler than a distributed blockchain.

<comment><USER41> It's a little more complicated than that. I could temporarily take a chunk, compute the hash, then throw it away and report back that I am happily storing the data, even though I'm not.

Anyhow, read the permacoin paper. It's pretty cool, and it needs a large petabyte data seed to secure the network. Seems like a win-win to me.

<comment><USER43> The centralisation helps again here too though.

The central server has access to the entire file, and hence can compute the hash of any arbitrary chunk. Challenges/verifications don't have to happen all that often ([1] indicates they are looking at once a month) so creating a unique challenge for each user shouldn't be too compute intensive.

For each known user:

\- Central server chooses random chunk of each file it wishes to verify the user still has. This could be any length from as long as the hash function to the whole file size and could be offset any number of bits.

\- Client is asked to provide the hash of the chunk using the file stored locally.

Precomputing hashes for all possible chunk permutations would take up substantially more space than simply storing the file in the first place. A bad actor would need to store the hash for all the possible chunk lengths starting from every possible start location in the file which is in the order of O(n^2) where n is the stored file size (500GB in this case). For reference that would be about one third of the entire 20PB archive for a single 500GB chunk if using a 256 bit hash function.

[1] [http://git-annex.branchable.com/design/iabackup/](http://git- annex.branchable.com/design/iabackup/)

<comment><USER40> It's a good thing in this case they also don't have to worry too much about bad actors, since it's a fundamentally altruistic endeavor. Worst case scenarios:

1) Someone keeps downloading the full archive and throwing away;

2) Someone wants a file erased, keeping it with the intention of denying access in some future.

\--

1) Bandwidth has costs on both sides; just balancing upload among receivers would probably suffice for this not to be a problem.

2) Assigning large random chunks to downloaders should prevent this "chosen- block attack"; add in some global redundancy for good measure and that's probably enough (although I still wouldn't trust this 100% as a _primary_ storage, only as an insurance storage).

</comment></comment></comment></comment></comment></comment></comment><comment><USER44> It would cost about US$2.5 million/year to back it up using Amazon Glacier.

<comment><USER45> With that much data, it would be cheaper to build your own Glacier-like service.

<comment><USER46> About $1m to build 21 PB of live storage (not tape), and $50k/year to power it (not factoring maintenance, rack space cost, and cooling)

A service offering live storage at this (Petabyte) scale [1] - <USER2> 22,020,096 GB * 3¢/GB * 12mo = $7,927,234.56 a year. You could build six fully-mirrored copies for about that much, though management of 360 4U machines (30x 48U full-height racks worth) is a business on its own (see BackBlaze photos), and you need to deal with drive replacement after a point.

[1] [http://www.rsync.net/products/petabyte.html](http://www.rsync.net/products/petabyte.html) (If you actually sign up for this, be sure to send me that _$24,000_ referral bonus o_o)

 _Math:_

\-- _Machine Cost_ \--

$300 * 45 * 8TB drives = $13,500 (drive cost) \+ $3,387.28 (pod cost) == $16,887.28 (unit cost) / 360TB = $46.91 per TB

$46.91/TB * (21 PB * 1024 == 21,504 TB) == $1,008,752.64 / 21 PB (for no redundancy)

\-- _Power Cost_ \--

Let's say the motherboard/controllers draw around 100W continuous and each drive pulls 11W (from my post elsewhere in the comments). This is 2,688 drives, which fit into 60x 45-drive 4U BackBlaze storage pods.

2,688 * 11W = 29,568 W (for the drives) + 6,000 W (for the computers) = 35.568 kW to run a non-redundant 21 PB array.

You'll need at least two network switches and moderate cooling infrastructure as well. The heat from using 35.5 kWh continuously takes nontrivial effort to disperse, but let's assume that's somehow free for the sake of simplicity here.

35.568 kW * 24 * 365 = 311.57568 MWh (in a year)

I'm sure at this point you might be able to work out a bulk deal of some kind, but at $0.16/kWh you're talking a pretty low cost of $49,852.11 in power every year.

\----

Building arrays of nonredundant live drives would be significantly cheaper over the long term than Amazon Glacier (which is almost certainly cold storage, not live) at this scale.

<comment><USER47> >Amazon Glacier (which is tape-based!)

Citation? Not saying you're wrong but I've seen supposedly "in the know" commentary in favor of both the tape-based and some sort of cold disk-based storage. I've been curious and also a bit surprised that the true answer has never really hit publicly although I'm sure it's well-known within some communities.

<comment><USER46> Oh, you're right. I haven't thought about this for a while, and I remember my previous assumption about tape storage being challenged. I'll replace that part with "cold storage".

After refreshing my memory: Amazon outright they're using denied tape backup, some third-parties pointed out cold spinning disks wouldn't be very economical, and a few people theorized it's actually BDXL.

<comment><USER47> Cold storage is probably safe. I've seen the assertion that it's tape from someone who I wouldn't have thought would get it wrong but I'd now bet against it whether or not Amazon's denial seals the deal.

Robin Harris has argued in favor of BDXL but he's speculated about a number of things. [http://storagemojo.com/2014/04/25/amazons-glacier-secret- bdx...](http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/)

My money would be on the simplest: cold disk. Some support here: [http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier- impl...](http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier-implemented/) but it's all speculative and anecdotal. I don't have a problem with the thought that Amazon is getting out ahead of the drive economics, especially given that I'm not sure how widely used Glacier is today.

I do find it interesting that this has been apparently kept under wraps from anyone who would share more widely.

<comment><USER48> Someone that worked set amazon at the time Glacier came out a while back commented that it is some type of extremely low power hard drive that backs this system. Which makes sense due to the 1-4 hour delay in retrieval time. The racks are idle and only come online one every 4 hours or so and batch retrieve/store their data then power down. This small power usage is probably what makes them profit on this very slim margin.

</comment></comment></comment></comment></comment></comment></comment><comment> ryan-allen I'd gladly host 3tb of data, I hope this becomes a thing.

</comment><comment><USER49> store them on a gazillion 360yunpan+weiyun+baiduyun (or justcloud) accounts!

</comment><comment><USER50> I never understood this desire to "back up" the internet.

It's an impossible endeavour and will only become more incomplete as the net grows. And 99,999% of the data "backed up" will never be looked at.

What's the point?

<comment><USER51> Well, I personally spent the better half of tonight recovering a bunch of old websites of mine from the IA using wget.

IA is one of the most important initiatives/projects on the internet, links are not meant to change but they do and when it happens knowledge & history erodes.

Remember the 2004 Indian Ocean earthquake and tsunami?

It was the first major event that provided mainstream credibility to the blogosphere, before this disaster consumers turned to news publications for information. For the first time ever news publications did not have the info and were desperate for content, any. TV crews flew into the effected areas and started purchasing suitcases/video/still cameras sight unseen off shell shocked tourists whom were too traumatized to remember if they had captured anything.

It was also the last major disaster that occurred before the social revolution - YouTube, Flickr, Facebook, smart-phones and mainstream adoption of broadband. Back then the main way for the general public to share information was via email (which had limits of 10mb) and if they were technical or knew someone technical on their 100mb of free hosting provided by their ISP.

Now imagine you're a student, academic researcher, production assistant and you have been tasked researching what transpired. As of right now, the majority of content from 2004/the disaster is no-longer available on the internet and only accessible via the IA.

(nb: I was the guy behind [http://web.archive.org/web/20050711081524/http://www.waveofd...](http://web.archive.org/web/20050711081524/http://www.waveofdestruction.org/) which was the first emergency crisis live/micro blog. For what Google now has a team of volunteers doing during a crisis I did it myself with a single pentium 4 cpu, 4GB of ram, a single IDE hard drive, a ramdrive, a 100mbit uplink and a week without sleep. WOD will be relaunching within the next couple of weeks and was one of the sites recovered from the IA.)

<comment><USER50> _knowledge & history erodes_

There surely is history worth preserving. It's just this "compulsive hoarding"-approach that seems ultimately futile to me.

The internet itself inherently preserves the knowledge that at least one person cares enough about to put it on a webserver. Pretty much everything of even minor relevance should be referenced by Wikipedia by now.

IMHO the internet itself is the "Internet Archive".

 _Remember the 2004 Indian Ocean earthquake and tsunami?_

To be honest, no. But Wikipedia seems to have that event pretty well covered; [http://en.wikipedia.org/wiki/2004_Indian_Ocean_earthquake_an...](http://en.wikipedia.org/wiki/2004_Indian_Ocean_earthquake_and_tsunami)

<comment><USER35> > * To be honest, no. But Wikipedia seems to have that event pretty well > covered*

And in a lot of cases, the actual citations or references used by Wikipedia are dead links, or have some sort of 'last retrieved at' annotation, or end up at parked/squatted domains and the original content is no longer there to validate the claims it supposedly made.

Offhand I don't know if Wikipedia has a policy on references to Internet Archive / wayback-machine links to things, but imo, being able to point to a snapshot of a reference at the time it was made, as well as the current version if it still exists, is a desirable feature, and well worth striving for.

Internet pagerot/linkrot has caused me grief with bookmarks for things I didn't realise I needed again until years later when they were gone.

<comment><USER38> > Offhand I don't know if Wikipedia has a policy on references to Internet > Archive / wayback-machine links to things

They do have a policy.[1] Additionally, IA has been specifically crawling links on Wikipedia to preserve the citations.[2][3]

1: [https://en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine](https://en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine)

2: [https://blog.archive.org/2013/10/25/fixing-broken- links/](https://blog.archive.org/2013/10/25/fixing-broken-links/)

3: [https://archive.org/details/NO404-WKP/v2](https://archive.org/details/NO404-WKP/v2)

</comment></comment></comment></comment></comment><comment><USER52> Shouldn't there already be a lot of existing tech for doing crowd backup/sync? Quick googling yields at least a few companies which allow you to donate unused hard drive space in exchange for benefits:

Symform: [http://www.symform.com/](http://www.symform.com/)

> Earn free cloud storage when you allocate unused space on your device for > your Symform account. Users that contribute get 1GB free for every 2GB > contributed. Contribution can always be fine tuned for your personal needs.

Wuala: [http://www.techrepublic.com/blog/windows-and- office/review-w...](http://www.techrepublic.com/blog/windows-and- office/review-wuala-cloud-storage/)

> Extra space is available: Unlike the majority of online file storage > services like Dropbox and Box.com, which offer paid plans only as a means to > expand storage, Wuala offers you the option to gain additional online > storage in exchange for some of your unused local disk space to commit to > the network.

<comment> vertex-four > Shouldn't there already be a lot of existing tech for doing crowd > backup/sync?

There's bits and pieces, but essentially nothing that's open-source and community-supported.

<comment><USER52> Just pointing out that P2P backup is a well understood problem (in industry and academia) and they should probably partner with people who do this stuff seriously or at least base their design on existing solutions. For example their implementation page right now is trying to do this with git-annex: [http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g...](http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git- annex_implementation) which doesn't feel quite right...

<comment><USER53> One of the more interesting design requirements here is that the system should work even when the distributed content is _not_ encrypted. If I join this backup effort, and a meteor takes out all of SF (and thus the Internet Archive), and their off-site backups fail, and the aftermath is preventing me from accessing the internet, then that 1TB of data that I had downloaded is all still usable to me. It's not some giant encrypted blob, it's a bunch of zip files (or similar) containing images, sound, video, web pages, and other types of files that I can, in principle, already use. (Maybe I can't do anything much with that Apple II disk image, but I can at least identify it.)

Git annex and BitTorrent would both work with that requirement, and most of the commercial, proprietary solutions cannot; their clients wouldn't use them if that data wasn't private.</comment></comment></comment></comment>


Who stole the Mona Lisa? - <AUTHOR> http://www.ft.com/cms/s/2/069d8662-be37-11e0-bee9-00144feabdc0.html#axzz1UIyEY27f <comment><USER2> If you liked this heist story, these are two of my all-time favorite HN stories:

"Art of the Steal: On the Trail of World's Most Ingenious Thief"

<http://news.ycombinator.com/item?id=1215138>

"The Silver Thief"

<http://news.ycombinator.com/item?id=1020547>

<comment><USER3> SPOILER ALERT: The first link's top comment reads "You have to give the guy credit for <foo and bar and baz>", ruining the story. :/

<comment><USER2> Spoiler-free links (Thanks, palish):

[http://www.wired.com/magazine/2010/03/ff_masterthief_blancha...](http://www.wired.com/magazine/2010/03/ff_masterthief_blanchard)

<http://stephenjdubner.com/journalism/silverthief.html>

Speaking of spoilers, suspend your sense of physics and go see "Another Earth", fascinating if you forget about the plot blackholes

( <http://www.foxsearchlight.com/anotherearth/> )

</comment></comment></comment><comment><USER4> This is actually an interesting entire family of crimes. Steal a unique, highly valuable item, sell fake copies for full-price now that it's plausible to be the real thing (especially if you set up a sale before the actual theft takes place), then return the original if you care enough. Nobody will turn you in because they'd be admitting to attempting to buy stolen goods.

<comment><USER5> This could be really dangerous if you aren't careful who you sell to.

<comment><USER6> It worked brilliantly in this case since they did their business in the US, overseas from where it was stolen.

<comment><USER7> According to the article, this scheme was probably just a fabrication by a journalist who wanted a juicy story:

"Here at last was a criminal brain worthy of the Mona Lisa. The only problem is that Decker almost certainly invented him. There is no external evidence for Decker's story, nor even for the Marques' existence. A century later, none of the six supposed copies has surfaced. Most likely, Vincenzo Peruggia stole the Mona Lisa single-handed, largely because she was small."

(Also, this is somewhat off-topic, but the typography and layout on ft.com are really superb. This is one of the few websites that doesn't make me immediately turn to readability.)

</comment></comment></comment></comment><comment><USER8> If you're interested in learning more about the painting, I created an entire mini website about the Mona Lisa a few years ago:

<http://hepguru.com/monalisa/>

It was featured by USA Today, etc. You may like it.

<comment><USER9> One of the best versions of Mona Lisa is the one engraved by Sanchez-Toda (Spanish bank note and postal stamp designer): <http://rsiqueira.postbit.com/mona-lisa>

</comment></comment><comment><USER10> Well-written, interesting article. Worth clicking through and reading.

<comment><USER11> +1

</comment></comment><comment><USER12> I like the article, but why is it in the FT? This is more something that suits sister-pulication The Economist.

<comment><USER13> FT Weekend and FT Magazine do these sorts of stories all the time. People who spend their weekdays worrying about finance want to relax on the weekend and read fun and interesting stories.

</comment></comment><comment><USER14> "The far-right Action Française newspaper blamed the Jews"

Looks like prejudice is as old as humanity itself!

<comment><USER15> Wow that's a terrible comment - I won't even point out the difference between 100 years old and the age of humanity.

But any basic knowledge of history should tell you that anti-semitic feelings in Europe were building up in the 19th century. For example Hitler was a huge fan of Richard Wagner (who died 28 years before this 1911 theft), who wrote a famous article, first published anonymously then later republished in his own name, called "Das Judenthum in der Musik", blasting Jewish composers such as Mendlesohn.</comment></comment>


Ask HN: What is a contemporary CS curriculum like? - <AUTHOR> Something I've been wondering for a bit now: has the undergraduate CS curriculum changed much in the last 15 years? I got my BA '99-'03, and took some masters classes in '05 and '06. The programming industry has reinvented itself 3 times over since then. <comment><USER2> I am a current undergrad student, so here's my understanding of the subject. There are some basic fundamental classes that are pretty consistent across the board. You'll start with some introductory classes that cover the basics of programming, such as loops, if/then/else statements and maybe basic OOP. You'll have some level of Computer Architecture course(s) where you learn the basic structure of a computer. Things like CPU pipe-lining, memory, branch prediction, assembly programming and other things. You cover basic algorithms and data structures. Things like lists, queues, stacks, as well as different sorting algorithms, graph algorithms, and basic algorithm design principles such as Dynamic Programming and Divide and Conquer. Most programs also have some introduction to automata and computability. Finally, you probably will have something courses on things like Operating Systems.

I've noticed that some places then separate Computer Science from Software Engineering. I think this is a good idea as "Computer Science" becomes more and more synonymous with programming in general.

As far as mathematics is concerned, we take calculus, linear algebra, discrete Math, probability and statistics.

Other than this, the rest of your time is mostly filled with electives of your choice depending on your interest area. I have several extra math courses planned as well as some software design and a programming languages course.

Obviously it varies a lot from program to program, but this is what I've noticed. For what it's worth, I'm a student at the University of Houston, and I also read up on the programs of some bigger name schools such as MIT, University of Texas, and Rice.</comment>


What's in an Emacs Lambda? - <AUTHOR> http://nullprogram.com/blog/2017/12/14/ <comment> satish-setty The article doesn't state it, but

        '(lambda (x) x)

is actually shorthand for

        (quote (lambda (x) x))

And then the distinction between them is explained right inside emacs. C-h f quote and C-h f lambda

</comment><comment><USER2> The difference is when an expression is to be evaluated. Back quote and friends are evaluated by the read function. These expressions are (or at least should be) functions on syntactic forms - syntax transformers. People call them macros. Of course they could have been misused in many cryptic ways.

<comment><USER3> > These expressions are (or at least should be) functions on syntactic forms - > syntax transformers.

I don't really agree with this. Backquote is just shorthand for a bunch of consing that makes the end result more clear. Macros are one application where being able to sort of template lists is very useful, but they're not the only one. As well, I think the tendency for people to learn backquote as a companion to defmacro tends to confuse people about how defmacro works.

<comment><USER4> There is then some additional confusion coming from the fact that there are diffetent ways how backquote/quasiquote is really implemented by the reader:

For CL it is entirely valid to implement whole logic of backquote in the ` read macro and expand it into some nested combination of list, append and quote at read time. Scheme specifies what ` and friends expand into which then is implemented as normal m8acros (which is also valid and somehow common implementation in CL). Elisp is slightly weird in that ` originally was normal macro and thus ` read macro expands `(foo) into (` foo).</comment></comment></comment>


Google 'frantic' about Personalized Home Page glitch - <AUTHOR> http://www.networkworld.com/news/2007/042607-google-frantic-about-personalized-home.html <comment><USER2> "The problems highlight one of the risks of relying on hosted applications providers, which offer to house software and its data for individuals and organizations."

What's with the unncessary bashing "hosted application providers"? I think it's safe to say that many more users would be affected by bugs and dead hard drives had they been running this on their personal computers, had this been a desktop application, like Frontpage.

<comment><USER3> It's a visibility thing. Millions of circuit breakers trip in homes every year. Not news. Millions of homes go dark at once? News.

Hosted apps have a risk (coordinated failure) that desktop apps generally do not. 100,000 hard drives don't all crash at once.

Also remember that it's in the interests of journalists to assume a tone of authority, to write in a way that mimics reasoned analysis: "this is part of a recent trend". It's a little gratuitous, but it's not unnecessary bashing.

</comment></comment><comment><USER4> This is bad news for any company trying to build trust in web hosted applications and data (such as mine). I hope they can fix it and restore the data, otherwise a lot of trust in general will be lost. Wouldn't they at least have data backups? (Does Amazon's S3 backup their data?)

</comment><comment><USER3> This is the upshot of putting your software on the web. Word or Access bugs b0rk user data retail. The same kind of glitch in a web app is magnified because it happens wholesale: everyone is affected at once.

But I would class this with the Yahoo/Google DDoS blackouts and the eBay scaling blackout circa 1999. Teething problems that will soon be forgotten _if_ they are fixed correctly.

</comment><comment><USER5> I guess they should allow you to export your settings and take your data with you. That is a Web 2.0 rule after all.

</comment><comment><USER6> I could see a few more episodes like this forcing Google to focus more on 'reliability' which could decrease the amount of risks they take and increase their cycle time for new products. Has to happen eventually.</comment>


TechStars reality show entrepreneur steals $2K from own company - <AUTHOR> http://www.launch.is/blog/exclusive-jason-sosa-on-stealing-2k-from-his-own-startup-or.html

<comment><USER2> I never understood why they would let a reality tv show cover TechStars. The problem I see is that you have misaligned incentives. From what I know of Techstars, their goal is to help these entrepreneurs through what is probably one of the most exciting/stressful time of their lives as they build their pretty awesome businesses.

From what I know of reality tv shows, it's to entertain people with extreme stories that are easy to digest and with little to no depth.</comment>


Mint CEO on how to build a startup - <AUTHOR> http://www.scribd.com/doc/11912720/Entrepreneurship-101-Aaron-Patzer-Mint

<comment><USER2> (throws voice) ooo, is that a new slideshow mode?

<comment><USER3> the "unlike this" "like this" button toggle is exceptionally confusing.

Why do you put a really dark gradient through all your buttons, rendering the bottom line of text very hard to read?

</comment></comment><comment><USER4> That was a nice set of slides, but was there a video (or audio) to go along with it?

<comment><USER5> First talk @ <http://www.dukedees.com/sessionpowerpointdownloads>

(there is a 'video' link)

cf. <http://news.ycombinator.com/item?id=472134>

<comment><USER6> thank you for the link and the video :)

</comment></comment></comment><comment><USER7> Aaron's speaking on a panel next week, "Tales of Young Entrepeneurs" at Stanford. Anyone going?

<http://www.vlab.org/article.html?aid=253>

<comment><USER8> Thanks for sharing - I might go. Sounds interesting.

</comment></comment><comment><USER9> Interesting recommendation to hire a PR firm. I guess if you have $17M in cash and an audience that still pays attention to traditional PR, that's a justifiable expense...

<comment><USER10> A good PR firm works to maintain a very personal and trusting relationship with major news outlets, including the popular blogs and web presences. While there are a lot of bad PR firms out there you should avoid, a good firm is extremely valuable and should be included in your marketing strategy once the product has shown life.

However, most likely your first year (or two) you'll be able to handle a fair amount of the basic PR work yourself. Here's a good blog post from Josh Dilworth who works with a lot of tech startups.

[http://joshdilworth.com/post/68998214/what-the-heck-is-pr- an...](http://joshdilworth.com/post/68998214/what-the-heck-is-pr-anyway)

<comment><USER11> We just had a discussion on PR firms that pretty much settles it in my opinion (<http://news.ycombinator.com/item?id=462493>). How can I include it in this thread? Hyperlinks are just not as efficient as some cute Ajax inlining...

</comment></comment></comment><comment><USER12> the video is really good.

did he say landlords in silicon valley will accept equity for rent price reductions? where does this happen?

<comment><USER13> The landlord of my old apartment in Palo Alto told me that some previous tenants had tried to pay their month's rent in equity, because they were broke. He turned them down and the company turned out to be very successful. I'm pretty sure he said it was one of the PayPal founders, but I don't remember.

<comment><USER14> almost sounds like an urban myth</comment></comment></comment>


Paul Allen's yacht "Octopus" is for sale - <AUTHOR> https://www.thedrive.com/the-war-zone/29723/you-can-buy-paul-allens-octopus-arguably-the-worlds-most-incredible-yacht <comment><USER2> It's rare to see superyacht that is tastefully designed and has interesting functionality.

</comment><comment><USER3> The asking price is $325M, so no, I can't buy Paul Allen's yacht

<comment><USER4> The potential market for this boat numbers under 1000 people or maybe even under 100 I'd say.

<comment><USER5> Some of them may be on HN though?

<comment><USER4> There's a thought, there's a lot of thoughts, actually!

</comment></comment></comment></comment><comment><USER6> I have a large hope that whomever buys this, carries on doing great research things with it, and doesn't just convert it into a playboys luxury pad.

<comment><USER7> You can probably get a fancier yacht for the same price if you don't wanna do the research stuff.

</comment></comment><comment><USER8> Well, you'd probably have to be on this list:

[https://en.wikipedia.org/wiki/The_World%27s_Billionaires](https://en.wikipedia.org/wiki/The_World%27s_Billionaires)

Or, take a risk and charter it to offset some of the est. $32.5mil annual run cost. Filling the tank can't be cheap, but a 12,500 mile range is pretty amazing.

</comment><comment><USER4> You could imagine that the name was supposed to evoke James Bond because it certainly looks like something from a Bond movie.

</comment><comment><USER9> Would suit James Cameron perhaps

<comment><USER10> James Cameron's net worth is estimated at 700 million.

the asking price for the Yacht is almost Half.

Then taking into consideration, the expesnive maintanence costs, i dont think even James cameron could afford it.

</comment></comment><comment><USER11> We're in 2019 and there is still some websites that mess with the browser history and break the back button. (iOS/safari)

<comment><USER12> Browsers should start punishing captive traps. There are valid use cases for SPA redirects without user interaction, much like how HTTP redirects ("Location: ") work, but... abuse should be dealt with.

<comment><USER13> The browsers should just fix it. Clicking back should skip over all those pages in the back-stack that I didn't interact with.</comment></comment></comment>


MOO.com releases API - <AUTHOR> http://www.moo.com/blog/2008/06/09/announcing-the-moo-api/

<comment><USER2> this looks great.

perhaps now linkedin or any of the professional social networking sites like it can send out free business cards w/ linkedin-specific information printed on them for freelancers or networkers to give out at meeting, shows, conventions, etc.</comment>


Show HN: LastChallenge.com – Discover Personal Development Resources - <AUTHOR> https://www.lastchallenge.com <comment><AUTHOR> The idea is, that we mostly only consume new content, but the golden nuggets are often hidden on page 153, where most people don't find them anymore.

The backend + Crawler is written in Elixir / Phoenix and the frontend is simple JS/JQuery with mustache templating. It's hosted on Heroku (in Europe) on a $7 hobby dyno and $9 postgres db.

Any feedback is welcome :)

<comment><USER2> Curious about how decisions regarding what is worth reading are made.</comment></comment>