Archivists Are Trying to Make Sure LibGen Never Goes Down - <AUTHOR> https://www.vice.com/en_us/article/pa7jxb/archivists-are-trying-to-make-sure-a-pirate-bay-of-science-never-goes-down <comment><AUTHOR> This is an extremely important effort. The LibGen archive contains around 32 TBs of books (by far the most common being scientific books and textbooks, with a healthy dose of non-STEM). The SciMag archive, backing up Sci-Hub, clocks in at around 67 TBs [0]. This is invaluable data that should not be lost. If you want to contribute, here's a few ways to do so.

If you wish to donate bandwidth or storage, I personally know of at least a few mirroring efforts. Please get in touch with me over at legatusR(at)protonmail(dot)com and I can help direct you towards those behind this effort.

If you don't have storage or bandwidth available, you can still help. Bookwarrior has requested help [1] in developing an HTTP-based decentralizing mechanism for LibGen's various forks. Those with experience in software may help make sure those invaluable archives are never lost.

Another way of contributing is by donating bitcoin, as both LibGen [2] and The-Eye [3] accept donations.

Lastly, you can always contribute books. If you buy a textbook or book, consider uploading it (and scanning it, should it be a physical book) in case it isn't already present in the database.

In any case, this effort has a noble goal, and I believe people of this community can contribute.

P.S. The "Pirate Bay of Science" is actually LibGen, and I favor a title change (I posted it this way as to comply with HN guidelines).

[0] [http://126.96.36.199/stat.php](http://22.214.171.124/stat.php)

[1] [https://imgur.com/a/gmLB5pm](https://imgur.com/a/gmLB5pm)

[2] bitcoin:12hQANsSHXxyPPgkhoBMSyHpXmzgVbdDGd?label=libgen, as found at [http://18.104.22.168/](http://22.214.171.124/), listed in [https://it.wikipedia.org/wiki/Library_Genesis](https://it.wikipedia.org/wiki/Library_Genesis)

[3] Bitcoin address 3Mem5B2o3Qd2zAWEthJxUH28f7itbRttxM, as found in [https://the-eye.eu/donate/](https://the-eye.eu/donate/). You can also buy merchandising from them at [https://56k.pizza/](https://56k.pizza/).

<comment><USER2> > Lastly, you can always contribute books. If you buy a textbook or book, > consider uploading it (and scanning it, should it be a physical book) in > case it isn't already present in the database.

There's no easy solution for scanning physical books, is there?

<comment><USER3> There are providers [1] that will destructively scan the book for you and return a PDF. If you want to preserve the book, you're stuck using a scanning rig [2]. The Internet Archive will also non-destructively scan as part of Open Library [3], but they only permit one checkout at a time of scanned works, and the latency can be high between sending them a book and it becoming available. FYI, 600 DPI is preferred for archival purposes.

[1] [http://1dollarscan.com/](http://1dollarscan.com/) (no affiliation, just a satisfied customer, can't scan certain textbooks due to publisher threats of litigation)

[2] [https://www.diybookscanner.org/](https://www.diybookscanner.org/)

[3] [https://openlibrary.org/help/faq](https://openlibrary.org/help/faq)

<comment><USER4> A big +1 for 1dollarscan.com. They've scanned many hundreds of books for me. The quality of the resulting PDFs is uniformly excellent, their turnaround time is fast, and their prices are cheap ($1 per 100 pages).

I've visited their office -- located in an inexpensive industrial district of San Jose -- on multiple occasions. They have a convenient process for receiving books in person.

I believe the owners are Japanese and the operation reminds me of the businesses I visited in Tokyo: quiet, neat, and über-efficient.

<comment><USER5> > quiet, neat, and über-efficient

I wish the same could be said for the Tokyo office I work in!

</comment></comment></comment></comment></comment><comment><USER6> The new architecture of pirate sites, what I call the Hydra architecture, seems pretty interesting to me. There isn't a single site hosting the content, but a group of mirrors freely exchanging data between one another. In case some of them go down, the other ones still remain and new ones can appear, copying data from the remaining mirrors. This is like a hydra that grows two heads every time you chop one off. It's absolutely unkillable, as there's no single group or server to sue.

A more advanced version of this architecture is used by pirate addons for the Kodi media center software. Basically, you have a bunch of completely legal and above board services like Imdb that contain video metadata. They provide the search results, the artworks, the plot descriptions, episode lists for TV shows etc. Impossible to sue and shut down, as they're legal. Then, you have a large number of illegal services that, essentially, map IDs from websites like IMDB to links. Those links lead to websites like Openload, which let you host videos. They're in the gray area, if they comply with DMCA requests and are in a reasonably safe jurisdiction, they're unlikely to be shut down. On the Kodi side, you have a bunch of addons. There are the legitimate ones that access IMDB and give you the IDs, the not that legitimate ones that map IDs to URLs, and the half-legitimate ones that can actually play stuff ron those URLS (not an easy taks, as websites usually try to prevent you from playing something without seeing their ads). Those addons are distributed as libraries, and are used as dependencies by user-friendly frontends. Those frontends usually depend on several addons in each category, so, in case one goes down, all the other ones still remain. It's all so decentralized and ownerless that there's no single point of failure. The best you can do is killing the frontend addon, but it's easy to make a new one, and users are used to switching them every few months.

<comment><USER7> > It's absolutely unkillable

Just like any other distributed system, this is vulnerable to organized take downs and scare tactics. There was a whole bunch of mirrors of Pirate Bay, yet once most of Europe's legal systems adopted the "sharing is theft" mindset, it became pretty much impossible to find one.

<comment><USER8> But now the main site seems to be bullet proof. There was a time where weekly there would be a new official link. I'm not sure what changed structurally with hosting tbp

<comment><USER9> They just stopped going after it, and focused resources on stopping streaming websites

</comment></comment></comment></comment><comment><USER10> Yongle Encyclopedia was a similar project of the 15th century China. It was the largest encyclopedia in the world for 600 years until surpassed by Wikipedia.

Alas, Yongle Encyclopedia is almost completely lost now. Archiving is harder than you think.

[https://en.wikipedia.org/wiki/Yongle_Encyclopedia](https://en.wikipedia.org/wiki/Yongle_Encyclopedia)

<comment><USER11> I read the Wikipedia article about it and the sad thing is that the majority of the Yongle Encyclopedia seem to have been destroyed only in quite recent times.

<comment><USER12> > but 90 percent of the 1567 manuscript survived until the Second Opium War in > the Qing dynasty. In 1860, the Anglo-French invasion of Beijing resulted in > extensive burning and looting of the city,[16] with the British and French > soldiers taking large portions of the manuscript as souvenirs.

Preservation is easy if you don't get invaded.

<comment><USER8> It's easy if you anticipate these things. Who put the dead sea scrolls in that cave in the middle of nowhere? Not someone who went in and forgot their scroll one day. Someone who had the foresight that this would be a safe space in the face of who knows what future threat. And it payed off.

<comment><USER13> I doubt it was that intentional. I would wager that "someone forgot about it" is the more likely explanation.

</comment></comment></comment></comment></comment><comment><USER14> Maybe we should print this out on acid-free paper-thin flexible wood-pulp sheets stitched to together to form linear organized aggregations. Each aggregation would contain one or more works and be searchable using a SQL-like database. To make this plan really work there would need to be a collection of geographically distributed long term physical repositories that would receive periodic updates as new material became available.

All joking aside, I do wonder wither digital or analogue formats are better able to survive into the distant future.

* What impact will DRM have on the accessibility of our knowledge to future historians?

* Is anything recoverable from a harddrive or flash media after 500 years in a landfill?

* Will compressed files be more of less recoverable? What about git archives?

* Will the future know the shape of our plastic GI Joes toys but not the content of the GI Joes cartoon?

<comment><USER15> > I do wonder wither digital or analogue formats are better able to survive > into the distant future.

There are 5000 year old clay tablets we can still read.

There are centuries old documents on paper, vellum etc. that we can still read.

I personally have decades-old paper documents I can easily read, and a box of floppies I can't.

It's not just a problem of unreadable physical media, I have a database file on a perfectly readable HD that was generated by an application that is no longer available. I might be able to interrogate it somehow, but it won't be easy.

Digital formats and connectivity make LOCKSS easier, so that's a plus. There's less chance of a fire or flood or space-limited librarian destroying the last known copy. However, without archivists actively transforming content to new formats as required, it might only take a few decades before a lot of content starts to require a massive effort to read.

<comment><USER14> Clay is the plastic of the ancient world.

Let's say the probability that: a single copy of a physical book survives 1,000 years, is found and is understood by an archaeologist _, is pB and the probability that a single copy of a book on an SSD survives 1,000 years is found and understood by an archaeologist_ is pD. Even if pB is far larger than pD it could be the case that there might be so many more copies of single book held on SSDs thus making it more likely the book will survive via an SSD than a physical book. On the other hand the technology to recover data from SSDs might not exist in 1,000 years.

It could also be the case that each generation would copy these books onto new digital media providing an unbroken chain of copies. The oldest copy of the Iliad is Venetus.A which is from 1000AD (1000 years ago) despite the Iliad probably first being written down in 800BC (2800 years ago). It was copied from earlier copies of copies of copies.

I really don't know how this will play out and I've been unable to find research on how long SSD and flash memory based media survives especially if buried in a landfill.

* - If archaeologists exist in the future. The current push from the STEM boosters to defund and de-emphasize the humanities may result in a near-future without archaeologists or funded archaeological projects. Over 1,000 years the entire field could die.

<comment><USER8> Would an SSD even function after 1000 years? Unless sealed, I imagine ambient moisture would do a number inside the drive. The same is true for books of course, but we still have 1000 year old books that have lasted by sitting on a shelf in churches and temples, etc., without any specific care until recent history.

The nice part of a book in an apocalyptic scenario is that you can copy it even if you don't know the language. You don't need a special tool for this, only one capable of marking a surface. It wouldn't be fun or fast, but it's possible and it's what monks did for centuries. Would archeologists 1000 years from now be lucky enough to find a SATA cable too?

<comment><USER16> It doesn't really matter if the SSD as a whole still works, because after 1000 years you'll never recover the data via the normal interface. Modern MLC flash is often specified for less than 1 year data retention, and even SLC is unlikely to make it to 1000 years. Attempting to read it will only make things worse ("read disturb"). The best hope of saving the data is with some future nanotech that directly probes each floating gate transistor and counts the electrons, and reverse engineering all the error correction and wear leveling.

<comment><USER14> I would assume they would read the SSD not by powering it on and plugging it into to a computer but by disassembling it and physically imaging the physical structure. This would also bypass the all the write leveling infrastructure allowing them to recover deleted data. It reminds me of the current techniques of using x-rays to read writing on the odd scraps of paper used to bind a book [0].

[0]: "X-rays reveal 1,300-year-old writings inside later bookbindings" [https://www.theguardian.com/books/2016/jun/04/x-rays- reveal-...](https://www.theguardian.com/books/2016/jun/04/x-rays-reveal- medieval-manuscripts)

</comment></comment></comment></comment></comment></comment><comment><USER17> Libgen is one of the greatest contributors to scientific productivity worldwide, possibly beaten only by Sci-Hub. Just about everybody in academia knows about it. If it ever vanished, some of us could probably still get by trading files from person to person, but nothing could be as perfect as what we got now.

<comment><USER18> > possibly beaten only by Sci-Hub

Today I learned that Library Genesis is actually "powered by Sci-Hub" as its primary source.

So I guess they're sister projects by similarly minded people (who seem to be mostly/originally based in Slavic countries, which I find interesting culturally - perhaps it's due to a looser legal environment + activist academics?).

> Just about everybody in academia knows about it.

That really says something about the state of society, this tension between copyright laws (and the motivations behind them) and the intellectual ideal of free and open access to knowledge.

<comment><USER19> I am not an expert on the topic, but I believe that in the former Soviet Union it was common between mathematicians to pass around preprints (a la arXiv). These then perculated through to the West. I think it had to do with the USSR and their restrictive (if we are being euphemistic) policies towards academics.

<comment><USER20> "the USSR and their restrictive (if we are being euphemistic) policies towards academics."

What do you mean?

<comment><USER21> Their policies were for more than "restrictive" is how I'm reading it

See [1]

[1] [https://en.wikipedia.org/wiki/Suppressed_research_in_the_Sov...](https://en.wikipedia.org/wiki/Suppressed_research_in_the_Soviet_Union)

<comment><USER19> Yes. I'm saying restrictive to describe the effect on academic papers. The effect on (oppression of) the academics themselves was much worse.

<comment><USER20> No, that's BS.

There are well known cases of genetics and cybernetics being banned for ideological reasons during Stalin's time. Scientific books and articles of convicted 'enemies of the state' were dangerous to possess in that time too. Some scientists used ideological 'arguments' in scientific debates which were dangerous to argue against.

But all that, AFAIK, ended after Stalin's death in 1953.

Moreover, I've never heard anything about mathematics in this regard.

<comment><USER19> Not sure what you are saying. Mathematicians were not even allowed to travel abroad [1] and any "concessions" were essentially as it pleased the USSR state. Only from 1990 was movement free in the true sense of the word.

[1] An example was when Margulis won the Fields medal: [https://en.wikipedia.org/wiki/Grigory_Margulis](https://en.wikipedia.org/wiki/Grigory_Margulis). There are many other examples too.

<comment><USER20> What does that have to do with sharing knowledge in the USSR and the countries in the Soviet block?

It was never in Soviet ideology to hide knowledge behind paywalls. See, for example, this [0] post about Mir publishing house and warm comments of Indians who grew up with their books. Sci-hub's ideology is just continuation of this approach.

[0] [https://news.ycombinator.com/item?id=21352277](https://news.ycombinator.com/item?id=21352277)

<comment><USER19> > Sci-hub's ideology is just continuation of this approach.

Actually, that was the point of what I was saying—the mathematicians had to be inventive and thus passed around preprints that they knew would also be read in the West.

<comment><USER20> Why did they have to be inventive? Please provide a source.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> I don't see anyone having mentioned the possibility of posting this data to Usenet at all - at minimum for archival purposes which should be good for ~8-9 years. That way at least the data isn't lost. With so many of those torrents have 0 or 1 seed, this is a serious risk I think, despite the comments elsewhere about people rotating what they seed.

I realize that doesn't solve the access problem for most people as most of the users who need this research might not know how to use usenet or even be familiar with it at all, but I think the first major concern would be to secure the entire repository on a stable network. Usenet seems like a good place for that even if it doesn't serves as a means of distribution. Encrypting the uploads would make them immune to DMCA takedowns provided that the decryption keys weren't made public and were only shared with individuals related to the maintenance of the LibGen project.

<comment><USER23> Two thoughts on that. Encoding it to a text format with CRC data for posting to usenet is highly inefficient in terms of data storage. And 33TB of stuff is not going to be retained for 8-9 years, the last I checked due to the huge volume of binaries traffic, the major commercial usenet feed providers have at most 6-9 months of retention for the major binary groups. Beyond that it becomes cost prohibitive for them in terms of disk storage requirements. This is not an issue for the majority of their customers, 6-9 months is more than long enough retention to go find a 40GB 2160p copy of some recently-released- on-bluray movie.

<comment><USER24> yEnc overhead is about 2% and there are plenty of providers with ~10 year retention.

<comment><USER23> Wow. I can't even imagine how much disk space ten years of retention of alt.binaries.* takes up. It's been literally ten years since I last did anything serious Usenet related.

<comment><USER13> Atleast in my experience, 10Y providers ask for more money and provide less high speed bandwidth (after which your up/down is usually limited to around 10 or 1mbps)

</comment></comment></comment></comment></comment><comment><USER25> To me, an aspiring scholar, LibGen is the most amazing tool ever. Things like inter-library loan and access to databases on university networks already make life so much easier to what it used to be—but nothing beats LibGen in terms of convenience. I'm in a the nowadays obscure field of patristic theology and I can't believe how much stuff I can find on LibGen, often things that even highly specialized research libraries like Harvard don't have.

The hours that LibGen saved me in gathering all the sources for my research must be in the hundreds. Thank you!

</comment><comment><USER26> There is a huge amount of duplication there (i.e. books that have many scans), I wonder if it would be better to tackle that versus doing a straight backup.

<comment><USER27> I think the duplication issue is probably overstated. I doubt tackling that would shave off more than 20% of the total backup size.

<comment><USER26> Speaking from personal experience, I usually see several results for any search. Granted, there's a big selection bias there, but 20% seems way too small.

<comment><USER28> Because you or anyone is most likely to search for relatively popular books. So those books will have a multiple copies. But for every popular book, there are many unpopular, but still useful books, that only have a single copy.

</comment></comment></comment></comment><comment><USER29> What's interesting is that 32TB is becoming more and more affordable and the research material is roughly staying about the same size.

That might change though as people start including video + data within papers and have new notebook formats that are live and contain docker containers/ipython, etc.

It's a shame we can't just mail these around.

<comment><USER30> You can buy 48TB (4x12TB) for €1000. Store some index on an SSD, and you have another full node.

<comment><USER31> If you don't care about warranty, 8 and 12TB drives routinely go for $15/TB on sale inside WD Elements.

I picked up 32TB for just under $500 with discount over the holiday that way.

<comment><USER32> Can you elaborate? What's the catch?

<comment><USER31> The only catch is that it's a minor lottery which model drive you're getting.

For instance, I got all white label WD80EMAZs (256MB cache, non-SMR, same firmware as the Reds) in this batch, so I had to insulate the 3.3V pins.

There are also true Reds, 128MB and 512MB cache drives, helium filleds, 7.2K HGSTs slowed to 5.4K, and other variants.

<comment><USER30> Or use a traditional power supply to sata cable.

</comment></comment></comment></comment></comment></comment><comment><USER33> Related: Looking at harddisk cost per terabyte, quite often extern drives are cheaper than internal ones.

For example right now in Germany I can get a WD 8TB USB 3.0 drive for 135€ but the cheapest internal 8TB drive costs 169€.

Any idea why? It's puzzling.

<comment><USER34> I noticed this yesterday while shopping for cyber Monday deals. If you want to load up a server with drives, perhaps the external drives can be removed from their cases and used internally?

<comment><USER35> Check it before you buy it. Years ago I bought a 1TB WD external drive. The usb interface was connected directly to the drive.

<comment><USER34> Thanks! I will have to read up on shucking and and /r/DataHoarder. I would think someone already has a list of which external drives can be used this way.

</comment></comment></comment></comment><comment><USER36> Let me say this: I fucking love libgen. It actually makes my life better and I'm so thankful to the people running it.

</comment><comment><USER37> Posting that here only creates problems for them. The more it's known in the west the more likely it will go down.

<comment><USER38> +1, bookwarrior has warned about this.

<comment><USER39> who is bookwarrior?

</comment></comment></comment><comment><USER40> Is there a way to just download the whole 32TB to your own machine? I see a ton of mirrors but the content seems to be highly fragmented between them

<comment><AUTHOR> There are ways to do so. The archive is made up of many, many torrents (I believe it's a monthly if not biweekly update of the database). If you have the storage/bandwidth availability for the whole 32TBs, please get in touch and I may be able to help you get the whole deal without too much hassle. Otherwise, just pick some torrents (it would be best to pick them based on torrent health, but they are so many to check manually) and try to keep seeding as much as possible.

EDIT: To find libgen's torrents health, check out this google sheet: [https://docs.google.com/spreadsheets/d/1hqT7dVe8u09eatT93V2x...](https://docs.google.com/spreadsheets/d/1hqT7dVe8u09eatT93V2xvth- fUfNDxjE9SGT-KjLCj0/edit?usp=sharing)

Thanks frgtpsswrdlame for the heads up.

<comment><USER41> I'm pretty surprised by the lack of seeders. Out of the 2438 torrents listed, a third have 0 seeders, another third have 1 seeder, and all but 5 have less than 10. Hopefully the publicity boosts those numbers.

<comment><USER42> From what I've heard a good chunk of people rotate their seeds for LibGen because their seedboxes can't handle all the connections for every torrent at once.

<comment><USER43> Is there some tool or documentation describing this practice?

<comment><USER44> I'm sure someone could get you the info to get setup as a seeder. For modern clients it's rather rather trivial to manage that many torrents. Get any decent modern CPU, 4gb+ ram, and $560 in storage and you're off.

<comment><USER42> I think the problem is that because of the size of each torrent, and there's 1000 of them, it's difficult to effectively seed all at once, so instead people would rather seed sections at once, and rotate through them.

I'm not sure how people setup the rotation though, that can't be an incredibly common feature but I could be wrong.

<comment><USER45> There are features that prioritize those with low seed/leech ratio in a sort of periodic fashion. Also it partially auto-balances because a swarm only needs a little more than unity ratio injected into it to get itself fully replicated. So each one that get's chosen because of a low seed/leech ratio will inherently drop out of that criteria as soon as the swarm is self- sufficient.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER46> Why not publish the site over IPFS, that would make P2P hosting much simpler?

<comment><AUTHOR> Currently (at least for the-eye) it's about IPFS's barrier of entry. I expect LibGen's case to be similar. Most people don't know about it, and if even those that knew about it had to learn how IPFS works etc, they would probably just try to find the book they're looking for elsewhere.

<comment><USER47> I am not fully aware how IPFS operates, but wouldn't it at least solve the back-end mirroring? Front-end servers would then "only" need to access IPFS for continuous syncing of metadata (for search) and fetching user-requested files (upon request).

</comment></comment></comment><comment><USER48> Are there any i2p torrents? I guess anonymity might be helpful if I want to mirror/seed this data...

<comment><USER49> I assume anyone could simply seed the "official" torrents via i2p? Not sure how that system actually works, it's interesting for sure but a lot less well- known than the alternatives.

</comment></comment><comment><USER50> one of the next interplanetary or Interstellar Probe should carry a copy of the sci-hub torrent in some kind of permanent storage

<comment><USER51> Do we have anything rated for a few millennia of interstellar radiation besides etched gold plates?

<comment><AUTHOR> Microsoft's project Silica [0] may hopefully provide really long term, large capacity archive grade storage on earth. I wonder what effects interstellar radiation has on them.

[0] [https://www.theverge.com/2019/11/4/20942040/microsoft- projec...](https://www.theverge.com/2019/11/4/20942040/microsoft-project- silica-glass-storage-warner-bros-features-details)

<comment><USER52> Glass is pretty inert, full stop. It would depend on the voxel size but I imagine as long as you have more than a few hundered atoms per voxel/bit you will have survivability on the order of millenia, even in high radiation environments. Someone would have to do the nuclear cross section calculations to get a real bit error rate but glass is very tough stuff.

</comment></comment></comment></comment><comment><USER53> I did not know about LibGen until this post. Too bad for me living in a cave. Anyways this is amazing project. Best luck to them and similar efforts.

</comment><comment><USER54> Imagine this:

\- A tiny well behaved client that starts with the OS.

\- It downloads rare bits of the archive at 1 kb/s obtaining 1 GB every 278 hours. It should stop around 100 MB to 5 GB.

\- It periodically announces what chunks/documents it has.

\- It seeds those chunks at 1 kb/s

\- Chunks/documents that have thousands of seeds already are not announced. Eventually those are pruned.

This escalates the situation to the point where everyone can help without it costing anything.

If someone is trying to obtain a 20 mb pfd it would take 5 and a half hours using a single 1 kb seed. With just 50 seeds it's just 8 min.

</comment><comment><USER55> I'd like to dedicate 1TB of my FreeNAS to something like this. Would be nice to run a small container with some P2P service that contained that chunk.

</comment><comment><USER56> Can't Tahoe-LAFS help with this kind of a challenge? I don't have experience with it, but it looks stable.

</comment><comment><USER29> I've thought that we could potentially build an end to end encrypted datastore within Polar and possibly add IPFS support to potentially help with this issue.

Here's a blog post about our datastores for some background.

[https://getpolarized.io/2019/03/22/portable-datastores- and-p...](https://getpolarized.io/2019/03/22/portable-datastores-and-platform- independence.html)

... essentially Polar is a PDF manager and knowledge repository for academics, scientists, intellectuals, etc.

One secondary challenge we have is allowing for sharing of research but I'd like to do it in a secure and distributed manner.

Some of our users are concerned about their eBooks being stored unencrypted and while for the majority of our users this will never be a problem I can see this being an issue in countries with political regimes that are hostile to open research.

In the US we have an issue of researchers being harassed over climate change btw. Having a way to encrypt your knowledge repository (ebooks) would help academic freedom as your employer or government couldn't force you to give them your repository.

But what if we went beyond this and provided a way to ADD documents to the repository from a site like LibGen?

Then we'd have the ability to easily, with one click, encrypt the document (end to end) and added it to our repository.

If we can add support for Polar to allow colleagues to share directly, this would be a virtual mirror of LibGen.

Alice could add books b1, b2, b3 to their repo, they could then share with Bob, only he would be able to see b1, b2, b3, then they would generate a shared symmetric key to share the books.

No 3rd party (including me) would have any knowledge what's going on.

I'm going to assume our users are not going to do anything nefarious or pirate any books. I'm also certain that they're confirming to the necessary laws ...

The challenge though is that while we'd be able to have a mirror of LibGen and more material, it would be a probabilistic mirror - I'm sure we'd have like 60% of it but the obscure material wouldn't be mirrored.

Right now our datastores support just local disk, and Firebase (which is Google Cloud basically). While we would encrypt the data end to end in Google Cloud I can totally understand why users might not like to use that platform.

One major issue is China where it's blocked.

Something like IPFS could go a long way to solving this but it's still very new and I haven't hacked on it much.

</comment><comment><USER57> I'd say IPFS, but That's a pretty big commitment from an entire community to keep alive.

</comment><comment><USER58> its best to split on small torrents on few 1-2 GB so normal users can seed

</comment><comment><USER59> If only some of the money made would reach the scientists lel. Most of em will give you their paper per mail if you aak them. The majority does not want them to sit behind paywalls...

</comment><comment><USER60> One could use FAANG data centers to host them for free, it would be really great

<comment><USER61> Look at the google books project. That got shutdown real hard due to copyright issues and litigation after they invested a ton of money in digitizing some of the most valuable library collections in the world.

<comment><USER62> It's incredibly sad: [https://www.theatlantic.com/technology/archive/2017/04/the-t...](https://www.theatlantic.com/technology/archive/2017/04/the- tragedy-of-google-books/523320/)

<comment><USER18> > Somewhere at Google there is a database containing 25 million books and > nobody is allowed to read them.

Indeed, what an intellectual tragedy..

> In August 2010, Google put out a blog post announcing that there were > 129,864,880 books in the world. The company said they were going to scan > them all.

That seems like a surprisingly "small" number.

Well, in trying to picture a physical library with 130 million books, maybe that's a realistic estimate. But compared to, say, the recently discovered data hoard of more than 2 billion online identities, it's miniscule.

SciHub and LibGen are truly the modern-day Library of Alexandria. The fact that they're being called "Pirate Bays of Science" \- and that providing free and open access to all books in the world is illegal - just goes to show that our civilization's priorities are misdirected.

<comment><USER63> Until fairly recently (historically), books were overwhelmingly scarce. A few datapoints:

\- The _total_ number of books -- not titles, but actual bound volumes -- in Europe as of 1500 CE, was about 50,000. By 1800, the total was just under one billion.

\- The library of the University of Paris circa 1000 CE comprised about 2,000 volumes. It was among the largest in Europe.

\- The Library of Constantinople in the 5th century had 120,000 volumes, the largest in Europe at the time.

\- A fair-sized city public library today has on the order of 300,000 volumes. A large university library generally a millon or so. The Harvard Library contains 20 million volumes. The University of California collection, across all ten campuses, totals more than 34 million volumes.

\- The total surviving corpus of Greek literature is a few hundred titles. I believe many of those were only preserved through Arabic scholars, some possibly in Arabic translation, not the original greek.

\- There's an online collection of cuneiform tablets. These generally correspond to a written page (or less) of text, with the largest collections numbering in the tens of thousands of items.

\- As of about 1800, the library of the British Museum (now the British Library) had 50,000 volumes. Again, among the largest of its time.

\- From roughly 1950 - 2000, roughly 300,000 _titles_ were published annually in the United States and/or English-language editions. R.R. Bowker issues ISBNs and tracks this. From ~2005 onward, "nontraditional" books (self- / vanity-published) have been about or above 1 million annually.

\- The US Library of Congress, the largest contemporary library in the world, holds 24 million books in its main collection (another 16 million in large type), and has 126 million catalogued items in total (2015).

\- At about 5 MB per book, in PDF form, total storage for the 38 million volumes of the Library of Congress would be slightly under 200 TB. At about $50/TB, that's $10,000 of raw disk storage. (Actual provisioning costs would be higher.) Costs are falling at 15%/year.

\- Total _data_ in the world comprises far more than books, and has been doubling about every 2 years. Or stated inversely: half of all the recorded information of humankind was created in the past two years.

Sources:

Some of this is off the top of my head, but partial support for the facts from:

[https://en.wikipedia.org/wiki/History_of_printing#/media/Fil...](https://en.wikipedia.org/wiki/History_of_printing#/media/File%3AEuropean_Output_of_Printed_Books_ca._1450–1800.png)

[https://en.wikipedia.org/wiki/History_of_libraries](https://en.wikipedia.org/wiki/History_of_libraries)

[http://www.bowker.com/tools-resources/Bowker- Data.html](http://www.bowker.com/tools-resources/Bowker-Data.html)

[https://www.loc.gov/item/prn-16-023/the-library-of- congress-...](https://www.loc.gov/item/prn-16-023/the-library-of-congress-by- the-numbers-in-2015/2016-02-01/)

[https://en.wikipedia.org/wiki/Harvard_Library](https://en.wikipedia.org/wiki/Harvard_Library)

[https://en.wikipedia.org/wiki/University_of_California_Libra...](https://en.wikipedia.org/wiki/University_of_California_Libraries)

[https://www.techpowerup.com/249972/ssds-are-cheaper-than- eve...](https://www.techpowerup.com/249972/ssds-are-cheaper-than-ever-hit-the- magic-10-cents-per-gigabyte-threshold)

[https://qz.com/472292/data-is-expected-to-double-every- two-y...](https://qz.com/472292/data-is-expected-to-double-every-two-years- for-the-next-decade/)

<comment><USER18> Thank you for that, very interesting and educational. I love how you led up to the punchline. It made me see that books as a technology and artifact are part of the "history of information", and how books are becoming subsumed in a shared trajectory with media/data in general.

> half of all the recorded information of humankind was created in the past > two years

That is shocking to imagine, and it's exponentially growing.

It reminds me of Vannevar Bush's "As We May Think", pointing out the emerging information overload in society. It certainly puts things in perspective, how we (humanity) have been making a conscious, collaborative effort to develop globally networked computers, one of whose important functions is to help us organize all the information, including books.

The conundrum it seems is that technology is also a massive multiplier/amplifier of the amount of data, that its capacity to help us organize would never catch up to what it's helping to produce.

> total storage for the 38 million volumes of the Library of Congress would be > slightly under 200 TB

I guess it's redundant to say, but I'm sure in the near future that would fit on a thumb drive!

<comment><USER63> Bush's essay is of course a classic. There are some precursors -- there's a BBC interview of H.G. Wells describing something similar from the 1940s.[1] E.F. Forster's _The Machine Stops_ has some similar ideas. And various encyclopaedists very much embodied similar ideals.

I've been listening to Peter Adamson's "History of Philsophy Without Any Gaps" podcast, which is _excellent_ , and spends a fair bit of time looking at the historiography of the topic -- what works were preserved, how, various interpretations, practices, preservation, and losses. Interesting to note that most of the preserved Greek and Roman works were found in _obscure_ Arabian monastaries and libraries. The mainstream collections themselves were often lost in raids, fires, or other mishaps. Which makes the LibGen situation all the more relevant and urgent.

(I'm a huge user of the site and others like it, for what it's worth.)

On the amount of total data being captured: there's a huge difference between _quantity_ and _quality_ measures of information. They're almost certainly inversely related.

Of what books were written in antiquity, up to the time of the printing press, say, odds were fairly strong that a work would be read.

At 1 million new titles being published per year, there are only 330 people in the US per book, or roughly 400 native English speakers worldwide. (With ~2 billion speakers worldwide, the total audience _might_ reach 2,000 per book). Clearly, most of what's being written will have a very small, or no, audience.

For machine-captured data, the likelihood that _any_ of it is seen directly by a human is vanishingly small. More of it will undergo some level of machine processing or interpretation, though even that only applies to a fairly small fraction of data. Insert old joke about the WORN drive: write once, read never.

As for storage costs (and/or size), at a 15% cost reduction per year, storage halves every 4.67 years (4 years and 8 months), which means that in 10 years, the $10k price tag becomes $2k, and in 20 years, it should be under $400. For the entire Library of Congress collection.

Flash drives seem to be increasing in capacity by a factor of 10 every 2.5 years. There are now 2 TB flash drives, so 200 TB might be as little as 5 years out. That ... still sounds optimistic to me.

[https://m.eet.com/media/1171702/digital_storage_in_consumer_...](https://m.eet.com/media/1171702/digital_storage_in_consumer_electronics_fig4.12.jpg)

[https://www.digitaltrends.com/computing/largest-flash- drives...](https://www.digitaltrends.com/computing/largest-flash-drives/)

The more practical problems are simply organising, cataloguing, and accessing the archives. This is an area that still needs help.

________________________________

Notes:

1\. I _think_ that's from "Science and the Citizen*, 1943, though the BBC and I have a disagreement concerning access. [https://www.bbc.co.uk/archive/hg- wells--science-and-the-citi...](https://www.bbc.co.uk/archive/hg-wells-- science-and-the-citizen/zmwcpg8)

<comment><USER18> While brushing up on the encyclopaedists, I found this little gem:

"Among some excellent men, there were some weak, average, and absolutely bad ones. From this mixture in the publication, we find the draft of a schoolboy next to a masterpiece." — Denis Diderot

Taking the quote out of context (and aside from its historical male-centered language) - it sure rings true of the current state of the web, as well as books.

About the inverse relationship of quantity vs quality, we seem to be drowning in quantity! As you've pointed out, there's great need for thoughtful organization and curation.

I like how you break down the quantifiable aspects to draw a historical trend and future projection. The rise of "data science" and "big data" in the past few decades really makes sense in this light.

I'm sure machine learning and "AI" will play an increasing role in the task of organizing and processing all this information, but at the bottom I feel that the most value probably comes from human curation.

LibGen has been an amazing resource for me as a lover of knowledge, a life- long book worm. I've got bookshelves and boxes full of physical books as well, but it's a drop in the ocean..

<comment><USER63> I love the Diderot quote. I'd also encountered earlier:

"As long as the centuries continue to unfold, the number of books will grow continually, and one can predict that a time will come when it will be almost as difficult to learn anything from books as from the direct study of the whole universe. It will be almost as convenient to search for some bit of truth concealed in nature as it will be to find it hidden away in an immense multitude of bound volumes. When that time comes, a project, until then neglected because the need for it was not felt, will have to be undertaken...."

... and on for another several paragraphs. It's an extraordinarily keen observation on the state and future of knowledge. At the always excellent History of Information website:

[http://www.historyofinformation.com/detail.php?entryid=2877](http://www.historyofinformation.com/detail.php?entryid=2877)

(Diderot is on my list of authors to explore in more depth.)

The fact that the _quality_ of any given information or exchange is often (though not always) entirely divorced from its _source_ (or author) is another interesting note. There are a few points here worth expanding on.

At least probabalistically, there are spaces (real or virtual) in which it's more likely to encounter good ideas. HN for its various failings, does well in today's Net. Google+, for all its faults, was similarly useful.

Size matters far less than selection. The tendency for centres of learning, research, and/or inquiry (and not necessarily in that order) to emerge is one that's been long observed, and their durability remarkable. The first universities (Bologna, Padua, Oxford, Paris, Cambridge, Heidelberg, and others, see: [https://en.wikipedia.org/wiki/Medieval_university](https://en.wikipedia.org/wiki/Medieval_university)) are often _still_ , 600 - 700 years later _among the best in the world._ Certainly in the US, Harvard, Yale, Princeton, M.I.T., among the earliest founded, remain the most prestigious. Though as noted in the conversation with Tyler Cowen and Patrick Collison, the list from 1920 is "completely the same, except we've added on California".

[https://conversationswithtyler.com/episodes/mark- zuckerberg-...](https://conversationswithtyler.com/episodes/mark-zuckerberg- interviews-patrick-collison-and-tyler-cowen/)

What happens as the overal quantity _and flux_ of information increases is that _more effective rejection systems are required_. That is: you've got _too much_ information flowing in, you want a way to _cheaply, with minimal effort or consequential residiual load_ , reject information that _may_ be irrelevant, with minimal bias.

There are numerous systems that have been arrived at, and many of our cognitive biases or informal tests for truth arise out of these (optimism, pessimism, availability, sunk-cost, tradition, popularity, socio-ethnic prejudice, etc.). Randomised methods are probably far fairer and less prone to category error. Michael Schulson's sortition essay in _Aeon_ remains among the best articles I've read in the past decade, if not several:

"If You Can't Choose Wisely, Choose Randomly"

[https://aeon.co/essays/if-you-can-t-choose-wisely-choose- ran...](https://aeon.co/essays/if-you-can-t-choose-wisely-choose-randomly)

Another fundamental problem is self-dealing and self-selection within institutions. Much of the failure within academia (also touched on by Cowen and Collison, who, I'll note, I don't _generally_ agree with, though they are touching on and making many points I've been pursuing for some years) comes from the fact that it's _internal_ selection of students, faculty, articles, topics, and ideologies, rather than strict tests of real-world validity, which promote these structures.

The same problems infect government and business -- it's not as if any one social domain is immune to this.

Oh, and another lecture by H.G. Wells on that topic:

"...When I go to see my government in Westminster I find presiding over it the Speaker in a wig and a costume of the time of Dean Swift, the procedure is in its essence very much the same. The Members debate bring motions and when they divide the art of counting still in governing bodies being in its infancy they crowd into lobbies and are counted just as a drover would have counted his sheep two thousand years ago...."

[https://invidio.us/watch?v=qRgP-46AC_o](https://invidio.us/watch?v=qRgP-46AC_o)

(Audio quality is exceptionally poor, 1931 recording.)

Partial transcript: [http://www.aparchive.com/metadata/INTERVIEW-WITH-H-G- WELLS-S...](http://www.aparchive.com/metadata/INTERVIEW-WITH-H-G-WELLS- SOUND/02408f42a2314a78a46c0e7b847f4107)

AI ... _may_ be useful, but seems to be result-without-explanation, a possible new form of knowledge, to go with revelation (pervasive if not particularly acurate), technical (means), and scientific (causes / structural).

Wholehearted agreement on LibGen.

Very enjoyable conversation BTW, thank you.

<comment><USER64> Nature shows us how to process information at ever increasing noise and scale - [https://www.edge.org/response-detail/10464](https://www.edge.org/response- detail/10464)

<comment><USER63> Yes and no.

Briefly: the article distinguishes "endocrinal" vs. "distributed" decisionmaking.

This applies at some levels, but not at others.

For individual humans, we don't have the option of rewiring our concsiousnesses, which are rather pathetically single-threaded, and can at best multitask poorly by task-switching, at a very great loss of task proficiency.

Even withing collective organisations (companies, governments, organisations, communities), the multiple independent actors works _where those actors ' actions are autonomous and independent of others_. Or, in the alternative, where they work _without mutual conflict_ toward a common goal.

But you get problems where either individual actors' motivations and actions are in conflict, or in which a _single_ global decision must be made (as with various global catastrophic risks), and multiple independent decisions cannot be arrived at. Even for noncritical arbitrary decisions, such as which side of the road to drive on, in which _there is no compelling argument to be made for one side or the other_ , but in which _both_ sides cannot be simultaneously selected, you need some global decisionmaking capacity.

When you reach the point of _either_ an existing decisionmaking system (as in: a single human, with the finite and largely immutable information acquisition and processing capabilities corresponding), _or_ a multi-agent system _which must reach a common decision_ , you've got the challenge of limiting data intake to that amount which allows effective function within the environment, and avoids overloading capabilities or ineffective action.

<comment><USER18> The article "Evolving the Global Brain" was thought-provoking, especially in the context of our discussion about the history of information and the exponentially increasing amount of information for humanity to gather/produce, process, curate, archive.

It's an attractive concept, that human society is structurally similar to a brain, and that an individual is a neuron. (If humanity is the brain, I suppose the rest of the Earth is the body. We're not doing too well as the self-appointed brain of the operation.)

My first reaction to the analogy of "endocrinal" (one-to-many) and "neural" (many-to-many) decision making, is that it's missing a primal psychological/biological motivation of humans to seek to dominate others of its own kind as well as all of nature. I'm not familiar enough with biology to say definitively, but I'm pretty sure the endocrinal system does not actively seek to subjugate the neural system (or vice versa) and dominate the whole body.

Social organization, it seems to me, is more a function of power, very small groups gaining advantage and dominance over vastly larger groups of people, than that of collaboration for mutual benefit. (I might be a bit too cynical of political motivations and authentic democracy these days.)

From the final paragraph:

> ..the current global brain is only tenuously linked to the organs of > international power. Political, economic and military power remains > insulated from the global brain, and powerful individuals can be expected to > cling tightly to the endocrine model of control and information exchange.

I'd disagree with this, and say that the global brain (if we mean the Internet and its empowerment of globally networked intelligence) was born from the wombs of "political, economic and military power". It never achieved escape velocity to become a truly free, autonomous and collaborative, neural model of decision making.

To backtrack a bit:

> Well-connected collective entities like Google and Wikipedia will play the > role of brainstem nuclei to which all other information nexuses must adapt.

The most powerfully well-connected collective entities are international political/financial/corporate entities, and indeed do they more or less dictate how all information nexuses (nexii?) must adapt.

One biological analogy that comes to mind, is how propaganda and "disinformation" act like neurotoxins in the social brain, introducing noise/entropy, skewing its coherence, and preventing well-informed and orchestrated cooperation.

Another is how established political powers have a well-developed "immune system", composed of mass media, legal structures, military/police force, surveillance of the public. This immune system could be seen at work, for example, at the environmental protests at the Standing Rock Indian Reservation.

The final sentence of the article:

> This formidable design task is left up to us.

By this I assume the author means, evolving the global brain. Quite a challenge! From my perspective, it's going to be a historic struggle: design or be designed.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER65> Isn't scanning a physical book and uploading a soft-copy, a landmine of hazards (both legal and moral)? Essentially you are encouraging (some) unlawful activity... I am not so sure I am onboard with this idea!

<comment><USER66> It's easy to take this stance in a rich country. But what about the people in countries where one of these books cost the equivalent of a year's wages. Not so black and white eh?

<comment><USER65> As far as I know, prices of books differ between rich and developing nations. For e.g., The C Programming Language that costs $50 in the US [1], is sold for Rs. 259 (~$4 US) in India. I believe that is the case with most "economy editions" specifically targeted at developing nations. It certainly isn't an "year's wages".

While I do understand your point, it still does not justify encouraging modern-day Robinhoods' and breaking the law.

[1] [https://www.amazon.com/Programming-Language-2nd-Brian- Kernig...](https://www.amazon.com/Programming-Language-2nd-Brian- Kernighan/dp/0131103628) [2] [https://www.amazon.in/Programming-Language- Kernighan-Dennis-...](https://www.amazon.in/Programming-Language-Kernighan- Dennis-Ritchie/dp/9332549443/)

<comment><USER67> Hi.

You're speaking to someone from the UK. When I grew up we had to walk uphill both ways, to quote the Four Yorkshiremen sketch.

I couldn't afford to buy things like books, operating systems, games, etc. We spent the money on food, rent, and on the Internet.

People like that past version of me are going to pirate regardless of what you say. Law breaking? You can't see me grinning ear to ear.

I like to think my tax bill makes up for that.</comment></comment></comment></comment>


Quibb, the Hacker News for career climbers, is raising $1M - <AUTHOR> http://venturebeat.com/2014/04/10/quibb-the-hacker-news-for-career-climbers-is-raising-1m/?source=HackerNews&referrer=Anuj+Adhiya

<comment><USER2> What makes Quibb for "career climbers?" I've seen nothing to indicate that.

<comment><USER3> All you have to do is give up permissions for your twitter account and you have a chance to be approved to find out, I guess.

</comment></comment><comment><USER4> Does it bother anyone else that one is required to use twitter to sign up?</comment>


TIOBE Index for January 2016 - <AUTHOR> http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html <comment><USER2> The Redmonk rankings are much better. As opposed to TIOBE, who look at counts of internet search results, Redmonk looks at code in Github and questions on Stack Overflow. There should be an update out in the next couple weeks; here is the latest from half a year ago : [https://redmonk.com/sogrady/2015/07/01/language- rankings-6-1...](https://redmonk.com/sogrady/2015/07/01/language- rankings-6-15/)

<comment><USER3> +1 for the redmonk.com link. This data looks more inline with what I see everyday.

</comment></comment><comment><USER4> I know it's based on some empirical data, but every time I've looked at a TIOBE index has made absolutely no sense to me.

<comment><USER5> Delphi/ObjectPascal above both Swift AND ObjC? Sure, that makes sense. If it's 1997.

<comment><USER6> Delphi is very alive (I'm moderator in a Delphi forum). Think this way: Delphi/FreePascal have almost all the market for a Pascal-alike syntax for them alone.

With the C-family you have a lot of options.

Also: Nothing beat Delphi at doing RAD apps today.

</comment></comment></comment><comment><USER7> Tiobe can't be right. It has Assembly ahead of Ruby and right behind JavaScript. What???

Redmonk looks flawed too. I would expect GitHub to create a major sampling bias in favor of smaller, throwaway projects, and in fact, a quick search(1) on GitHub appears to show 2.2 million projects smaller than a kilobyte that have never gotten a single star. Should those be counted? Do they really reflect anything meaningful about the language's _rank_?

In my opinion, the only meaningful measure of a programming language worth _quantifying_ from the net are StackOverFlow tags: they can be counted and read and provide a real reflection of the language's community without giving any special preference to language, license, or context.

Everything else is hopelessly subjective: whether or not the language maps well to a particular problem, whether it maps well to the way you like to program, and whether it fits into the ecosystem in which you plan to deploy it. You can't measure those with google searches.

(1) [https://github.com/search?utf8=%E2%9C%93&q=size%3A%3C1+stars...](https://github.com/search?utf8=%E2%9C%93&q=size%3A%3C1+stars%3A%3C1&type=Repositories&ref=searchresults)

<comment> ad-hominem Just curious, but are there proper heuristics that are easily implementable to improve the ranking a language receives from github usage?

For example, only count repositories that have more than 2 stars.

I don't like cutting repos because they are under a certain size, because even a few bytes may be genuinely useful and widely used, but a repository nobody is looking at or really using is most likely a pet project.

Maybe count only projects with > 2 stars (or forks?) OR projects with over a kilobyte of code? Perhaps such a project with no forks, no stars, but a lot of code might be a reference implementation or used in a talk or blog post or something but isn't useful other than to reference.

I agree counting all github repos is a silly way to do it.

</comment></comment><comment><USER8> Poor Objective C, nobody likes you.

[http://www.tiobe.com/index.php/content/paperinfo/tpci/Object...](http://www.tiobe.com/index.php/content/paperinfo/tpci/Objective_C.html)

</comment><comment><USER9> Notable is the search-engine-optimized rise of Groovy from #82 to #17 in a mere 12 months. The subsequent TIOBE comment "Scala might gain a permanent top 20 position soon" is probably a back-handed dig by TIOBE at the likely impermanence of Groovy's top 20 position.

The last time Groovy made the top 20, it hit #18 in Oct 2013, but 3 months later (Jan 2014), had dropped back out of the top 50 (#32 in Nov, #46 in Dec). TIOBE said the following month "The data is produced by one of the Chinese sites that we track is interpreted incorrectly by our algorithms. After we fixed this bug, Groovy lost much of its ratings." But before that fix happened, interviews with the current Apache spokesperson for Groovy (Guillaume Laforge) promoting Groovy's top 20 position were published in 5 online rags (www.infoworld.com, www.eweek.com, cacm.acm.org, jaxenter.com, and glaforge.appspot.com), and all of them quickly appeared in Google's top 30 search results for "groovy programming" and remained there for 6 to 12 months afterwards.

It's a good bet that same feedback effect will be engineered before Groovy starts losing its new top 20 ranking, perhaps before next month.

In fact, this sort of thing also happened with Groovy in December 2010. Groovy began a sudden rise from outside the top 50 when Groovy tech lead Jochen Theodorou "volunteered" his services to Tiobe in late 2010 to help them improve their algorithms. In April 2011, however, Groovy fell from #25 to #65 on Tiobe in a single month after they increased the number of search engines they monitor.

These fleeting peaks for Groovy in the TIOBE rankings (#25 in Apr 2011, #18 in Oct 2013, #17 in Jan 2016) between its usual ranking of somewhere between #51 and #100 (e.g. #82 only 12 months ago) are a bad thing for Groovy because of damage to its reputation as a solid language suitable for long-term IT solutions. Such ranking volatility gives off the stench of smoke-and-mirrors marketing intended to benefit a single stake-holder, probably the person who the groovy-lang.org DNS domain is the personal property of.

[1] [http://www.infoworld.com/t/application- development/c-pulls-a...](http://www.infoworld.com/t/application- development/c-pulls-away-java-among-top-programming-languages-230603)

</comment><comment><USER10> How in the world did Go drop out of the top 50 based on everything in the news this past year.

<comment><USER5> Look, Go might be gaining in popularity, but there's just no way it could compete with behemoths like Apex, Ladder Logic, and ABAP.

<comment><USER11> This is the epitome of sarcasm.

</comment></comment></comment><comment><USER12> I highly enjoyed the article: [http://i.imgur.com/qGEjNVL.png](http://i.imgur.com/qGEjNVL.png)

</comment><comment><USER13> It's pretty amazing that Python is basically mainstream at this point. Right?

It's also odd that they're aren't more Python jobs.

</comment><comment><USER14> I always am perplexed with how C and C++ are so high on this list. Given that some (all?) of the measure is based on search engine results. When I had some C/C++ back when I was in college, I was very underwhelmed at the number of resources available online. I had taught myself web development just by Googling everything I had an issue with. Tried the same with C/C++ and felt there was an couple orders of magnitude less available information.

<comment><USER15> C++ developer here. IMHO the easiest access to quality information are books (usually not for free) but apart from that I was always able to find everything on-line -standard draft, language changes proposals, references, SO questions, blogs, talks, OSS projects, OS/hw specifications - you name it. Do you mean anything in particular you were not able to find?

<comment><USER14> Yea, I can't remember as it's been many years but I do remember finding maybe one or two sites talking about what I needed (somewhat), and what I was working with was probably pretty entry-level stuff.

</comment></comment></comment><comment><USER16> How do you explain JavaScript going down?

Is it because of the rise of languages that compile to JavaScript?

</comment><comment><USER17> Interesting the VB.net doesn't even show up until 2010.</comment>


Google details just how good dark mode is for your battery - <AUTHOR> https://www.androidauthority.com/google-dark-mode-battery-923359/ <comment><USER3> Related and life changing, the Dark Reader extension that enables dark mode for every web site, including PDF's, with adjustable contrast. Almost every site looks better with this on. [https://darkreader.org/](https://darkreader.org/)

Make sure to check out the More tab and change the filter type on sites that it doesn't work in Dynamic mode for.

<comment><USER4> Thanks. Just tried it. Pretty awesome.

</comment></comment><comment><USER5> Funny how Google only realized this a few years after started to sell its own phones with OLED screens.

It's also funny how Google has only made Android "whiter" since then.

<comment><USER6> Phones have to be charged everyday anyway. I personally prefer prettier UI rather than better battery life.

<comment><USER7> I prefer to have a mobile device I can use for longer while, uh, mobile than a pretty UI that requires being tethered to the wall for longer.

</comment></comment></comment><comment><USER8> I like dark mode, mainly as an eye-saver when reading at night. Of course, most websites are blinding white, including this one.

However, in the daytime I would definitely prefer to use light mode, with a brighter screen. So over the course of a day I might have 80% of my usage in light mode and 20% in dark mode.

</comment><comment><USER9> I've always wanted to love dark background + white text themes; for the longest time I thought it was worth it because it was cool. About a year ago I switched back to light themes, and later found out that there's a reason why I had a more difficult time reading with dark themes: I have an astigmatism.

"People with astigmatism (approximately 50% of the population) find it harder to read white text on black than black text on white. Part of this has to do with light levels: with a bright display (white background) the iris closes a bit more, decreasing the effect of the "deformed" lens; with a dark display (black background) the iris opens to receive more light and the deformation of the lens creates a much fuzzier focus at the eye." \- Jason Harrison – Post Doctoral Fellow, Imager Lab Manager – Sensory Perception and Interaction Research Group, University of British Columbia

<comment><USER10> That's strange, I also have an astigmatism but I'm just the opposite. I can't stand light themes.

<comment><USER9> Hmm, perhaps not quite as black and white as first presented...

<comment><USER11> I also have mild astigmatism, but I find I prefer dark themes nonetheless - <USER2> think it _is_ legitimately a little harder to read, because I just tabbed over to Discord to check and realized I was using a larger font size than I'd usually use in other applications, and that the text definitely was a little harder to focus on compared to white-on-black text of the same font size.

It's just that I like having my backlight blasted in my face even less, so it adds up to a sufficiently more pleasant reading experience that I switch everything to dark theme when I can.

</comment></comment></comment></comment><comment><USER12> I really wish Android had a global dark mode theme or API where developers could tie it in. I hate having to go into individual apps to enable and only be disappointed that not all devs add this feature. It'd be great if it was just one switch!

Glad they're finally shining light on the situation ;)

</comment><comment><USER13> Those guys who were selling the fake smartphones on Shark Tank should make a 2018 luxury model of their product out of a dark, rounded slab of gorilla glass. (Which, for no reason has a "notch" decal on the top.)

</comment><comment><USER14> We've been saying this for _years_. My Windows Mobile phone ran for _days_ with it's flat black UI on an OLED screen. And Google had a great UI design that was mostly black/dark pixels in Holo, which still looks better today than Material.

In the world where Google claims energy efficiency is a key goal of theirs, their white UIs should be thrown out as even an option, as the waste of energy powering big fields of white light on billions of devices being charged in billions of homes is just wasting electricity on a massive scale.

<comment><USER15> The Material Desigb spec has a dark mode, and sites and apps implemented with material-ui can support it. Click the click bulb icon on the app bar:

[https://material-ui.com/demos/cards/](https://material-ui.com/demos/cards/)

<comment><USER14> Sure it does, but that's neither the default scheme that Google has pushed, nor one it even offers on most of it's apps. (Similarly, Holo had a light theme. The difference is that dark was the default look of it.)

And Google's apps have been going more white, instead of less white: [https://www.androidpolice.com/2018/09/25/google-testing- colo...](https://www.androidpolice.com/2018/09/25/google-testing-colorless- shapeless-play-store-redesign-large-install-button/)

</comment></comment></comment><comment><USER16> Do people often have their screen set to 100% brightness?

I never set my screen above 40% brightness on my Nexus 5X, and often turn it lower at night.

I ask since the numbers seem to indicate that dark themes have a much smaller impact at 50% brightness.

<comment><USER11> Yes, I always have my screen at 100% brightness, unless I'm in a room with the lights out or something.

...should I not be doing that?

<comment><USER16> According to the stats in this article, you save more battery just setting your brightness to 50% than using a dark theme. More than a 60% reduction in power usage for the screen.

</comment></comment></comment><comment><USER17> Anyone else noticing more apps also providing a dark mode as an option?

<comment><USER18> Doubt that has to do with preserving battery but more with being much easier on the eyes, especially at night.</comment></comment>


Groundbreaking drone coverage of tornado damage piques ire of FAA - <AUTHOR> http://www.csmonitor.com/USA/2014/0430/Groundbreaking-drone-coverage-of-tornado-damage-piques-ire-of-FAA-video <comment><USER2> That footage is excellent and definitely show how valuable drones can be during disasters.

I don't envy the FAA. They are being dragged kicking and screaming into this new age of drone flight. It is going to take a while for good policies to emerge regarding the use of drones.

I personally am conflicted. I don't especially want a drove hovering over me wherever I go. What if it crashes into me? But I also want the freedom to go buy a drone and fly it wherever I damn well please (without having to have a permit or deal with red tape).

I'm not sure what the answer is. Require insurance for drone operators? Require licenses?

<comment><USER3> I fear the stupidity and ignorance of inexperienced drone operators. Our airspace is complex with lots of rules and regulations to ensure safe separation of aircraft. Trained pilots have enough trouble obeying the rules and being safe, so I question whether someone who has no formal training is capable of doing so. What happens if an inexperienced drone operator is going down the Hudson River corridor without reading up on the radio procedures, while I'm barreling down at 140 knots? Pilots are trained to look up special airspace before doing a flight, but how would someone who just bought a drone off the shelf be familiar with these practices? There's no inherent reason that drones can't follow the same procedures and be just as safe as airplanes today, but it's dangerous to have the attitude of "buy it off the shelf and go fly it with no training."

The privacy issue doesn't concern me as much, since it's already possible to rent a helicopter and follow someone around all day. The only attribute that's changing is the cost: rather than paying $300/h for a helicopter, you can pay a couple bucks an hour for constant surveillance.

<comment><USER4> Genuine question, but does it change much compared to a wild bird ? Yhey don't respect regulations nor space rules either, and I hope it's OK. If we lump drones in the same category shouldn't it be OK as well ?

<comment><USER5> "Most accidents occur when the bird hits the windscreen or flies into the engines. These cause annual damages that have been estimated at $400 million within the United States of America alone and up to $1.2 billion to commercial aircraft worldwide."

[http://en.wikipedia.org/wiki/Bird_strike](http://en.wikipedia.org/wiki/Bird_strike)

Relatively few fatalities for humans, but very high cost. And they are not made of metal or durable plastics. They are also less likely to fall onto your head with rapidly rotating blades due to technical failure.

</comment></comment></comment></comment><comment><USER6> " _Critics say one problem with the FAA 's current policy – that amateurs can fly drones below 400 feet but commercial operators cannot – is that it doesn't seem to address any specific safety concerns._"

The critics may not realize that this restriction largely has the effect of separating drone air traffic from air traffic containing humans, which is generally limited to 500 feet above ground, modulo take, landing, and, as always, emergencies.

[http://www.flightsimaviation.com/data/FARS/part_91-119.html](http://www.flightsimaviation.com/data/FARS/part_91-119.html)

<comment><USER7> I think you've inverted the meaning of that sentence. The intent was to point out that it's odd that _amateurs_ can fly but _commercial_ operators cannot, not that the 400 foot rule does not address specific safety concerns.

I agree that drones should be regulated, having built (and crashed) a lot of them over the last four years. However, the current regulatory environment is effectively "it's OK if you're an individual and you abide by these vague guidelines for R/C planes that have been widely flaunted since their introduction 20+ years ago, but if you take a single dollar for it, we're after you."

That doesn't really make sense - drones don't get more dangerous the second someone takes money to fly them. Most commercial drone uses I've seen are for media (TV, realty/house videos, YouTube, advertisements, etc.) and tend to be conducted more cautiously and by more experienced operators than the random techies with flyaway-prone DJI gear who have popped up over the last year or so.

<comment><USER8> > That doesn't really make sense

Capitalism, man. Without a financial incentive, you dramatically reduce the use of drones. Approve commercial usage and you'll have dozens of drones flying around after every celebrity and politician everywhere they go.

Businesses will have drones hovering outside their doors all day with CCTV. Intersections and major roads will have a few drones above them monitoring traffic.

How about a reality show based on filming what goes on in backyards in suburbs all over the country. "Oooh look a pool party... lets hover around streaming this over the internet, there's sure to be some nudity or a drunk fight eventually".

<comment><USER7> I agree that without regulation, "approved" commercial drone usage could spiral out of control into the drone-dystopia you suggest.

I'm just amused by the the bizarre nature of the current regulation: amateurs regularly flaunt the supposed current guidelines (as they have been for 20+ years) and the FAA turns a blind eye until a single dollar is passed.

This kind of psuedo-regulation makes little sense and results in all kinds of hilarious (and equally illegal) workarounds. I once met a commercial drone operator who sold footage for TV by posting it anonymously on random video- upload sites, having the show producers "find" it, and then taking a kickback somewhere else for an "unrelated" job.

</comment></comment></comment></comment><comment><USER9> Last month Patrick Geraghty, an administrative law judge with the National Transportation Safety Board ruled "the FAA regulations approved for manned aircraft did not apply to unmanned aircraft any more than they applied to paper airplanes or balsa wood planes."

The FAA is appealing the decision, but as of right now it may pique their interest, but it's not illegal.

Article: [http://www.cnn.com/2014/03/07/us/drone-pilot-case- faa/](http://www.cnn.com/2014/03/07/us/drone-pilot-case-faa/) Full decision: [http://www.kramerlevin.com/files/upload/PirkerDecision.pdf](http://www.kramerlevin.com/files/upload/PirkerDecision.pdf)

</comment><comment><USER10> "Five years from now, this is going to be commonplace, that people use drones to document news stories" I realize this statement is probably obvious in the abstract, but the footage makes it viscerally obvious.

<comment><USER11> The crazy part is, this type of footage can easily be produced with less than a thousand dollars of gear, assuming we're okay with a GoPro and not a truly professional camera (and for breaking news coverage, I think GoPro quality is perfectly sufficient).

How much does a news helicopter cost? A thousand dollars an hour, plus huge initial costs?

</comment></comment><comment><USER12> In France last summer, a TV channel was allowed to take a drone footage of the city of Lourdes ( [http://en.wikipedia.org/wiki/Lourdes](http://en.wikipedia.org/wiki/Lourdes) ) just after a flood : [http://www.dailymotion.com/video/x112qvt_inondations-a- lourd...](http://www.dailymotion.com/video/x112qvt_inondations-a-lourdes-les- degats-vus-du-ciel-grace-au-drone-bfmtv-20-06_news)

<comment><USER13> And a young boy was chastised for filming his city (Nancy) from high up: [http://www.lemonde.fr/societe/article/2014/02/14/un- lyceen-p...](http://www.lemonde.fr/societe/article/2014/02/14/un-lyceen- poursuivi-pour-avoir-filme-nancy-a-l-aide-d-un-drone_4366386_3224.html)

then got a job to do precisely that from the new mayor:

[http://www.lemonde.fr/societe/article/2014/02/19/le-maire- de...](http://www.lemonde.fr/societe/article/2014/02/19/le-maire-de-nancy- propose-un-travail-au-lyceen-qui-avait-filme-la-ville-avec-un- drone_4369602_3224.html)

</comment></comment><comment><USER14> > _... a potentially vexing frontier that pits curious citizens against a > government with qualms about the spying potential of drones._

Riiiiiiight. The government exhibits such strong qualms about the spying (or other advantageous) potential of drones (or any other technology). Perhaps only when deployed and controlled by citizens instead of government. Or when lacking missile outfitting that delivers targeted destruction to state- sanctioned targets.

</comment><comment><USER15> A solution, or rather loophole the FAA might argue, would be to create a non- profit whose cost the commercial entity covers, but is free to undertake any operation they deem endearing. Police might change their mind if said non- profit proves a free and reliable asset in their own investigations.

</comment><comment><USER16> Our decision loops for things like regulation have become too slow to deal with reality. The greatest threat to the Federal Government is irrelevance.

<comment><USER17> The Federal government doesn't make rules just for the sake of having something to do. Aviation in particular is a complex and risky pursuit, and having a single set of rules has been a huge commercial and safety advantage for the US. I shudder to think what air travel would be like if there were 50+ different state regulators. Also, bear in mind that things like FAA rules are not drawn up in a vacuum, but with the participation of commercial aviators from small to large. My impression from talking to people who pilot planes is that the FAA is one of the better-functioning parts of the federal government, along with the NTSB.

<comment><USER18> You don't know the federal government very well then. It is _exactly_ why they do it. It is the sole reason why our government is deemed to have been "productive" by the number of laws it passes. Because the government is not a market governed by scarcity, make work and busy work is exactly what keeps it busy and speciously relevant and necessary.</comment></comment></comment>


This dominatrix makes men mine cryptocurrency for her – she now has over $1m - <AUTHOR> https://www.marketwatch.com/story/this-professional-dominatrix-made-men-mine-1-million-in-cryptocurrency-for-her-2017-12-20 <comment><USER2> She set herself a sweet gig of collecting mining profits instead of taking just one time payments. Probably the only way she can take a recurrent payment as a sexworker.

</comment><comment><USER3> Reminds me of the film, The Big Short, where the main characters find out from an adult entertainer that she is buying multiple houses.

The interviewer asks this woman about the future of Bitcoin when she's clearly not an expert because shes says, Bitcoin "will be too high for individuals to invest in." One could buy a fraction of a Bitcoin to "invest" as little as $5.

The question is not whether we're in a bubble, but when it will pop.

<comment><USER4> > The interviewer asks this woman about the future of Bitcoin when she's > clearly not an expert because shes says, Bitcoin "will be too high for > individuals to invest in." One could buy a fraction of a Bitcoin to "invest" > as little as $5.

I'm confused?

But 'this woman' is completely right

at ~15k$/btc only the wealthy are able to invest

Did you put quotes around "invest" because you think btc is a bad investment?

Or to protect you from people saying:" 'this man' is clearly not an expert because he thinks 5$ is an investment"

Or perhaps for some other reason?

<comment><USER3> I equate buying digital currencies hoping that they will go up in value as gambling. I do have some, so I am also gambling on it, but I would not call it an investment.

Anyone can buy any amount of BTC. One does not have to buy in whole numbers with 15k$ or more. That's like saying only rich people can buy gold. Completely false.

Essentially there are many uninformed people buying these digital currencies that know nothing about them. That is what I was pointing out.

<comment><USER4> > Anyone can buy any amount of BTC.

Actually, this is untrue.. a Satoshi is the smallest denomination of btc

> That's like saying only rich people can buy gold

It's more 'like' saying "only rich people can invest in gold"

That's why ETFs and mutual funds exist: so other people can invest in those so the wealthy people who own them can invest in gold then distribute the gains

>Essentially there are many uninformed people buying these digital currencies that know nothing about them. That is what I was pointing out.

Considering the misinformation you are spreading and your claim to be 'gambling' with 'some' 'digital currency' it sounds like you are describing yourself

In the gp you claimed "she's clearly not an expert"

Can you at least admit Theodora: someone who moves 1M+ worth of assets between multiple currencies based on first hand market research and future potential of direct and peripheral regulation; exhibits more expertise on the subject of cryptocurrencies than you?

Why did you choose this article to strawman?

</comment></comment></comment></comment><comment><USER5> Geez, that's the way you do it. Clever girl.

</comment><comment><USER4> I vouched

I feel it's unfortunate that this got killed and also that it's unclear why

This is a unique voice being heard within this community

Anyone care to discuss why they killed it?</comment>


Ask HN: How does the following apps work behind the scenes? - <AUTHOR> Ask HN: How does the following apps work behind the scenes? 1. http://heedzy.com/ 2. https://reviewcommand.co/ 3. https://feedcheck.co/<p>I know there's a Google play API which lets you get reviews for the apps you own but how then the aforementioned apps get all reviews for other apps? <comment><USER2> "We collect all reviews from various channels." I would say continuous HTML crawling of the app stores.</comment>


Mir: A lightweight JIT compiler project - <AUTHOR> https://developers.redhat.com/blog/2020/01/20/mir-a-lightweight-jit-compiler-project/ <comment><USER2> This project is trying to do too many things.

It seems like Ruby needs a profile guided optimizer, which means building an IR that is suitable for profile-guided optimization. That's way different from classic IRs like this since it means having provisions for OSR exit.

I recommend looking at these slides to learn how to do it.

[http://www.filpizlo.com/slides/pizlo-speculation-in-jsc- slid...](http://www.filpizlo.com/slides/pizlo-speculation-in-jsc-slides.pdf) [http://www.filpizlo.com/slides/pizlo-splash2018-jsc- compiler...](http://www.filpizlo.com/slides/pizlo-splash2018-jsc-compiler- slides.pdf)

<comment><USER3> I agree - I think MIR is trying to produce better machine code more quickly, but that isn't the problem Ruby faces! The problem Ruby faces is having to inline through ten levels of metaprogramming in order to get any meaningfully- sized compilation unit that you can optimise, _and only then_ it makes sense to worry about code generation.

However, the reason that this is not so simple is that Ruby is a vastly more complicated language than JavaScript (I've worked on implementing both.) Ruby has an enormous standard library and most Ruby programs are just endless calls to the that library, so your compiler must be able to understand the library semantically, either by rewriting it in Ruby (not likely at scale in MRI) or by adding tens of thousands of individually optimised intrinsics (again not likely).

Ruby is digging itself further into a local optima with these approaches optimisations, rather than looking further around for a better global optima.

<comment><USER2> Your description of optimizing Ruby sounds like it is exactly like optimizing JS. JSC's main optimizing IR (the DFG) is all about understanding the standard library semantically. We write the standard library in JS+hacks (called "builtins") or C++ (pick on a per function basis). Some of those functions have opcodes in DFG, or are built out of primitives that have their own specialized opcodes in DFG.

Maybe the reason why Ruby optimization has problems is nobody has done it the JSC way.

<comment><USER3> > Your description of optimizing Ruby sounds like it is exactly like > optimizing JS.

Yes it's the same problem and it's unique to neither Ruby nor JS - but the problem is just scale.

Ruby has a larger library, so needs more rewritten from the current C into Ruby+builtins. And then that rewrite doesn't maintain Ruby semantics (the C API currently used does not exactly match Ruby semantics) so a rewrite matching semantics is often very hard.

> Maybe the reason why Ruby optimization has problems is nobody has done it > the JSC way.

People have been trying the approach you use in Core (for over a decade, starting with Rubinius, now TruffleRuby and others), but I think (based on practical experience working on optimising both languages) that it's just a larger problem in Ruby which is why it hasn't been conquered yet.

<comment><USER2> I think that the way you're describing the solution for Ruby tells me that you don't see the problem the way that I see it. The problem isn't rewriting things in builtins. The problem is making an IR in which you can reason about the standard library at scale: reason about its speculation opportunities, all of the implied dependencies and effects, it's GC impacts, etc.

None of the IRs I've seen people try for Ruby does that. JSC's DFG IR does a lot of this. Hence I don't think folks have really tried the JSC approach for Ruby.

<comment><USER4> > The problem is making an IR in which you can reason about the standard > library at scale

How do you deal with functions written in C++ which invoke functions written in JS?

<comment><USER2> The important ones are intrinsics. The C++ function is understood by the JIT so well that it just emits the code for it itself.

Our C++->JS calling convention sucks, partly because we just avoid going down that path.

We do have a C++->JS call IC that we could use more.

<comment><USER3> > The C++ function is understood by the JIT so well that it just emits the > code for it itself.

But how would that work for Ruby, where C functions on the critical performance path are often third-party code that the compiler author has never seen before?

We'd need to let the JIT understand third-party unseen C functions. There are experiments to do that (Sulong, MIR, Rubinius sort of tried it) but I think it's more of an open problem than you're implying.

If you treat calls to unknown third-party C functions as an opaque native call then you're really going to struggle to build a meaningful compilation unit, in my experience.

<comment><USER5> It looks like the author is intending to use a C to MIR compiler so that existing CRuby code can be inlined into generated MIR code. Third party code could be compiled the same way, right?

"The blue parts show the new data-flow for MJIT. When building CRuby, we could generate MIR code for the standard Ruby methods written in C. We can load this MIR code as a MIR binary. This part could be done very quickly."

<comment><USER3> That's what MIR is doing, not what JavaScriptCore is doing.

Flip thinks this isn't needed for Ruby - '[t]his project is trying to do too many things' \- and that JavaScriptCore could do it already.

I think that MIR and TruffleRuby think they have to do something else (lifting C code into their IR) shows us that JavaScriptCore's approach isn't quite as immediately applicable as Filip thinks it is.

<comment><USER2> It's more that I think that you won't get enough semantic understanding of C extensions lifted into any IR for that to be useful.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> From GitHub [1]:

      "Plans to try MIR light-weight JIT first for CRuby or/and MRuby implementation"
      "MIR is strongly typed"

Is there an explanation of how the project bridges the gap between dynamically-typed Ruby and statically-typed MIR?

More generally, I'd love to see something like MRuby+MIR be successful. It would be great to see an alternative to the aging LuaJIT.

[1] [https://github.com/vnmakarov/mir](https://github.com/vnmakarov/mir)

<comment><USER7> Seems to me that MIR operates on a (much) lower level, basically abstract away the physical machine and its finite register set. As such, it would replace LLVM (or GCC) middle and back end. The goal is much faster compilation without sacrificing more than ~20% of performance.

Dynamic types and garbage collection would then be implemented on/for the abstract MIR machine.

</comment></comment><comment><USER8> Interesting. Seems to be the first modern non trivial compiler that avoids using SSA internally. I thought SSA had cornered the market, but perhaps now there is an opposite trend (ok one example doesn't make a trend)

<quote> No SSA (single static assignment form) for:

Faster optimizations for short optimizations pipeline and small functions (a target usage scenario) Currently SSA could be used only for two optimizations (CCP and GCSE). SSA usage would mean 4 additional passes over IR. If we implement more optimizations, SSA transition is possible when additional time for expensive in/out SSA passes will be less than additional time for non-SSA optimization implementation Simpler and more compact generator code because we can avoid to implement a lot of nontrivial code (for dominator and dominator frontier calculation, a good out of SSA code) </quote>

<comment><USER9> > Seems to be the first modern non trivial compiler that avoids using SSA > internally.

Of course it's debatable what "modern" and "non trivial" mean, and if they apply to this project. It's very small, but on the (one!) small benchmark the author cites it seems to do quite well, so it's certainly not completely naive.

For whatever it's worth, CompCert doesn't use SSA either, and that's certainly a non-trivial compiler, though arguably the non-triviality does not stem from any advanced optimizations it does.

</comment></comment><comment><AUTHOR> There are quite a few decent active Ruby Implementation going on at the moment.

CRuby with MJIT

MIR ( This )

JRuby ( Ruby on JVM )

TruffleRuby ( Ruby on Graal )

Artichoke ( Ruby on Rust )

And I remember someone mentioned making Ruby with Tracing JIT. ( Not Topaz ) Unfortunately My Google fu is not good enough I can no longer find it.

<comment><USER10> Rubinius. Here are Evan's slides from the 2009 LLVM developer conference:

[https://llvm.org/devmtg/2009-10/Phoenix_AcceleratingRuby.pdf](https://llvm.org/devmtg/2009-10/Phoenix_AcceleratingRuby.pdf)

EngineYard took Rubinius in a few directions, but I think the main lasting impact was all the RSpec work they did along the way.

<comment><AUTHOR> Edited my original post. I meant Implementation that are still actively developed. Both Rubinius and Topaz are no longer being maintained.

<comment><USER11> I believe Rubinius _is_ being maintained (again), it's just going in a direction very different from Ruby.

</comment></comment></comment></comment><comment><USER12> I would be quite interested in an IR/JIT assembler specialized for vector instructions with ARM Neon and x86-64 SSE~AVX512 output.

Ideally it handles register allocations and generated function caching as well. The current JIT assembler (ASMJIT, Xbyak) requires you to handle register allocations. LLVM is as mentioned quite a heavy dependency to have.

<comment><USER13> asmjit has a register allocator, for sure not the highest quality one, but it's there in the asmjit's Compiler infrastructure.

</comment></comment><comment><USER14> The article stresses how their JIT is much more lightweight than GCC/LLVM, which is perfectly valid, but why not compare MIR against the _other_ portable lightweight JIT engines out there? They're not the first to think of it.

The article mentions Cranelift, but that's a 'middleweight JIT' with a proper SSA IR. I was surprised to see LibJIT has more LOC than Cranelift - I thought it was lighter. (Imperfect proxy for runtime 'weight', of course.)

If you want a lightweight portable JIT engine, there's already _GNU Lightning_ [0], and the atrociously-named _Lightening_ fork [1] (used in the new JIT in the _GNU Guile_ Scheme interpreter, which turned up on the HN front page recently).

Here's a 1996 paper (preprint) on a research JIT named _VCODE_ which executed around 8 instructions to generate each instruction in its output. [2] (Sadly it was never released, as far as I can tell, and is presumably long dead.)

Anyway, with all that said, I wish this project well. No-one's managed to get good performance out of Ruby yet, so it's certainly ambitious. Google gave up on Unladen Swallow, and that was a JIT for Python, which, as I understand it, is more amenable to JIT than Ruby. Even failing that, having a quality rival to GNU Lightning would be worthwhile.

[0] [https://www.gnu.org/software/lightning/](https://www.gnu.org/software/lightning/)

[1] [https://www.wingolog.org/archives/2019/05/24/lightening- run-...](https://www.wingolog.org/archives/2019/05/24/lightening-run-time- code-generation)

[2] [http://www- leland.stanford.edu/class/cs343/resources/vcode-a...](http://www- leland.stanford.edu/class/cs343/resources/vcode-annotated.pdf)

</comment><comment><USER15> How is this meaningfully different in scope and intention vs Parrot? That project went on for a long time until every language (including its original target, Raku) decided they'd rather build their own more specialized JIT. What would prevent MIR from meeting the same fate?

<comment><USER4> MIR is much more flexible. It's like CraneLift but more basic.

Anything you write in MIR can be compiled by MIR. Parrot only JITs Parrot bytecode, so the applications are much more limited.

</comment></comment><comment><USER9> > implement the GCC C extensions necessary for the CRuby JIT implementation

That's the point where "C is nice and simple, it's easy to whip up a compiler" invariably turns into "why the #^(&)# didn't I use an existing frontend?". Real-world C code is _messy_. The entire sub-project of implementing a C compiler is a needless distraction that will turn into a huge time-suck with zero benefit to the author.

See also: "Why Says C is Simple?" [https://people.eecs.berkeley.edu/~necula/cil/cil016.html](https://people.eecs.berkeley.edu/~necula/cil/cil016.html)

</comment><comment><USER16> Good analysis on memory usage and how that negates use in mobile and IoT projects.

Ruby was my go to language for a long dry spell when I had little Lisp development jobs (except for Clojure). Ruby is a great language, as Matz says, Ruby is designed for developer happiness. I stopped using Ruby when more Common Lisp work came my way and then I used Python for five years of deep learning work. I have favorite languages but I used what customers wanted.

That said, I still keep up with Ruby news.

</comment><comment><USER17> Hey Mir display server is still alive :)

<comment> DC-3 Yet another drastic pivot!

</comment></comment><comment><USER18> Hey everyone, please stop naming things Mir for a while. Thanks!

<comment><USER19> > > [https://en.wikipedia.org/wiki/Mir_(disambiguation)#Science_a...](https://en.wikipedia.org/wiki/Mir_\(disambiguation\)#Science_and_technology)

I feel like this list is missing some more, but yeah, there's been a few 'Mir' projects.

<comment><USER20> I was thinking of this Mir:

[http://docs.mir.dlang.io/latest/index.html](http://docs.mir.dlang.io/latest/index.html)

<comment><USER19> I was thinking of Rust's MIR being missing too:

[https://blog.rust-lang.org/2016/04/19/MIR.html](https://blog.rust- lang.org/2016/04/19/MIR.html)

<comment><USER21> > That is, we are introducing a new intermediate representation (IR) of your > program that we call MIR: MIR stands for mid-level IR, because the MIR comes > between the existing HIR ("high-level IR", roughly an abstract syntax tree) > and LLVM (the "low-level" IR).

It's so terribly confusing because LLVM itself defines a .mir (Machine IR) that is a syntax between its IR and the backend (which could be thought of as mid-level IR). When I heard about Rust's MIR, I assumed it was some clever way to generate target-dependent IR.

The good news is that we now have M(L)IR: the one IR to bring them all and in the darkness bind them.

<comment><USER22> If anything I expect MLIR to make things more confusing as Rust's MIR is probably going to be around for the long haul.</comment></comment></comment></comment></comment></comment>