Open-Access JSTOR Database Accessible to the Public - <AUTHOR> http://www.universitytimes.ie/2020/03/jstor-makes-database-accessible-to-the-public/ <comment><USER2> I'm still angry over the ridiculous overcharging of Swartz and his suicide. I hope Carmen Ortiz never gets another job in law enforcement or politics again, ever, and I shall donate to any primary or general election opponent of hers if she ever runs for any office higher than dog catcher.

<comment><USER3> I was just thinking the same thing.

</comment></comment><comment><USER4> This quote from the piece, and checking at the website to confirm, says nothing about _public_ ""We are working with publishers to make more than 20,000 books available at no charge for JSTOR participating academic institutions and secondary schools that do not participate in our books program", the statement said. "The number of books available through this effort is growing daily as more publishers opt in.""

</comment><comment><USER5> Makes me think of Aaron Swartz RIP [https://en.wikipedia.org/wiki/Aaron_Swartz](https://en.wikipedia.org/wiki/Aaron_Swartz)

<comment><USER6> I thought the same, the JSTOR incident initially brought me to HN. Discussed at the time:

[https://news.ycombinator.com/item?id=5046845](https://news.ycombinator.com/item?id=5046845)

</comment></comment><comment><USER7> > Updated title: Open-Access JSTOR Database Accessible to the Public

> Correction: 19:42, March 19th, 2020

> A previous version of this article incorrectly stated that JSTOR had made > previously unavailable ebooks and journals accessible to the public for the > first time. In fact, the materials in question have been available to the > public for some time. The headline, subheading and body of this article have > been updated to reflect this information.

What's accessible here is open access in the first place. I don't see evidence of anything being opened up additionally as a response to the pandemic. Move on, nothing to see here.

</comment><comment><USER8> This isn't new according to JSTOR [https://twitter.com/JSTOR/status/1240349642191245314](https://twitter.com/JSTOR/status/1240349642191245314) Sounds like the author just misread something. (Oh there's a correction at the bottom already.)

</comment><comment><USER9> > The database is also working to expand on the amount of free content > available online to students

Don't worry JSTOR, Alexandra's got your back. [http://sci-hub.tw](http://sci- hub.tw)

Seriously though, to hell with paywalls and artificial barriers imposed on publicly funded papers by no-value-added rent-seeking publishers such as those whose material is archived by JSTOR.

</comment><comment><USER10> WOW!

</comment><comment><USER11> Nice it only took a global pandemic to accomplish the free sharing of information that Aaron Swartz was trying to accomplish:

"In 2011, Swartz was arrested by Massachusetts Institute of Technology (MIT) police on state breaking-and-entering charges, after connecting a computer to the MIT network in an unmarked and unlocked closet, and setting it to download academic journal articles systematically from JSTOR...

Swartz declined a plea bargain under which he would have served six months in federal prison. Two days after the prosecution rejected a counter-offer by Swartz, he was found dead in his Brooklyn apartment, where he had hanged himself."

[https://en.wikipedia.org/wiki/Aaron_Swartz](https://en.wikipedia.org/wiki/Aaron_Swartz)

<comment><USER12> Aaron Swartz was trying to share pre-1923 articles (this was basically the cutoff for public domain content in the U.S. at the time, ignoring more challenging issues such as whether the copyright was renewed and whether that actually means that a post-1920s item might be in the public domain). JSTOR has actually been making _that_ content broadly available for quite some time.

<comment><USER13> I think you've confused Aaron Swartz with Greg Maxwell.

[https://www.techdirt.com/articles/20110912/10132515906/jstor...](https://www.techdirt.com/articles/20110912/10132515906/jstor- freely-releases-public-domain-papers-that-greg-maxwell-already-freed.shtml)</comment></comment></comment>


Why You Should Stop Hacking Your Life and Invest in the Journey - <AUTHOR> https://medium.com/philosophy-logic/7f838b1fa228

<comment><USER2> Hacks are useful to get things done that are not important to you but need to get done. I do believe that "It's the experience you should be seeking, not the short cut. It's the voyage you should embrace, not the destination. It's the path that elevates the soul, not the destination. It's the process that ennobles, not the result." You want to feel excited for your journey, hacks be damned.</comment>


Controversial deepfake app DeepNude shuts down hours after being exposed - <AUTHOR> https://www.theverge.com/2019/6/27/18761496/deepnude-shuts-down-deepfake-nude-ai-app-women <comment><USER2> DeepNude App is offline after Developers pull plugs [https://technoidhub.com/games/deepnude-app-is-offline- after-...](https://technoidhub.com/games/deepnude-app-is-offline-after- developers-pull-plugs/16597/)</comment>


Senate Bill Would Make Credit Freezes Free - <AUTHOR> https://www.bankinfosecurity.com/senate-bill-would-make-credit-freezes-free-a-10706 <comment><USER2> I recently had to unfreeze my credit, but I was traveling so I did not have the PIN that they gave to me when I froze my credit.

Experian was fairly difficult to do, and required I prove things in a pretty non-trivial way. It involved sending a fax, which was annoying, but not exactly horrible.

Equifax, the company that got hacked, let me unfreeze it discouragingly easily. They asked me some questions on the phone that I figure were likely available in the leaked data, with only one or two questions needing a little more access.

The two questions that needed more access were frightening to me, though. They asked me to confirm an up-to-the-minute balance on either my bank accounts or the cost of a few things I purchased very recently (I forget which it was). Either way, why do they have that data? Why does the company that got hacked have that? I certainly never gave them permissions to have it, and I would like to revoke their permission to have this information.

<comment><USER3> _Equifax, the company that got hacked, let me unfreeze it discouragingly easily_

My wife and I just went through this due to a recent identity theft incident. Our experience in setting up a freeze with Equifax was the opposite of ours. My wife (and she's the financial manager of the house!) actually got refused by their online tool, and we had to call up and go through it all again with a human.

<comment><USER4> The fact that you had difficulty freezing your credit, and the person your replied to found it discouragingly easy to UN-freeze their credit, does not bode well for Equifax in either case.

</comment></comment></comment><comment><USER5> In comparison, here's what it cost me in September to place a credit freeze in a state that does not regulate its cost (Pennsylvania):

      TransUnion: $10
      Experian: $10.70
      Equifax: free (they made it free after they were hacked - until June 30th)
      Innovis: free (always free as far as I can tell)

And it worked really, really well. I had to apply for a corporate Amex at work soon afterwards. I was rejected because my credit was frozen and was asked to call. I temporarily unfroze my credit while on the phone (on Equifax, because it was free) and Amex ran my credit right then and approved me.

Having that kind of control was amazing. If I need to allow a credit check, I can do a 24 hour or 7 day unfreeze. It's such a minor inconvenience for completely blocking credit thieves.

<comment><USER6> This may sound controversial, but politicians should also limit credit reporting agencies to finite, fixed number of players. Like two or maximum three.

Sometime after Equifax hack, I froze our credit on TransUnion, Experian and Equifax. But wasn't aware that e.g. Innovis exists.

So ok, I will now go there to put a freeze in place. But now that I realized there maybe other CRAs, DDG tells me I should maybe do something about FICO[1] and PRBC[2] too.

Or of course we could do the smart thing, and tell our elected representatives to work towards abolishing/repealing whole CRA industry. I'd wager that what ever rises from those ashes, could not possibly be the horror CRA industry is today.

[1] [https://en.wikipedia.org/wiki/FICO](https://en.wikipedia.org/wiki/FICO) [2] [https://en.wikipedia.org/wiki/PRBC](https://en.wikipedia.org/wiki/PRBC)

<comment><USER5> I hadn't heard of Innovis either until I read the New York Times article encouraging everyone to freeze their credit:

 _Christina Bater, managing director at Barrett Asset Management in New York, suggests freezing your file at the little-known company Innovis, too. Hey, why not?_

[https://www.nytimes.com/2017/09/08/your-money/identity- theft...](https://www.nytimes.com/2017/09/08/your-money/identity- theft/equifaxs-instructions-are-confusing-heres-what-to-do-now.html)

<comment><USER7> Innovis Credit Freeze Online Link: [https://www.innovis.com/securityFreeze/index](https://www.innovis.com/securityFreeze/index)

</comment></comment></comment></comment><comment><USER8> I've been trying to freeze my infant son's credit and they've made it absurdly and maliciously complicated.

In addition to mailing a physical letter requesting the freeze to each organization, they require copies of his birth certificate AND social security card AND my social security card AND my drivers license AND proof of address (I.e. utility bill) AND a form from the Social Security Administration AND a court order/power of attorney or notarized statement of an authority to act on his behalf. And money of course.

That court order requirement is a doozy. I don't even know how to get one of those. So I haven't pursued it further, which I'm sure was their intent.

And of course there's the question about how wise it is to send these sloppy, negligent organizations all this sensitive info. Hopefully they fix this racket by the time he turns 18.

<comment><USER9> Why are you trying to freeze his credit? I don't think he'll even be in their systems until he gets his first financial account, presumably years later.

<comment><USER8> That's what makes his social security number an attractive target for identity theft. He won't check it for over a decade.

[https://www.npr.org/2017/10/18/556237149/to-protect- children...](https://www.npr.org/2017/10/18/556237149/to-protect-children- from-identity-theft-parents-must-be-proactive)

Quote from the article:

> A 2011 report on child identity theft from Carnegie Mellon University's > CyLab found that of the participants surveyed, the rate of identity theft > for children was 51 times higher than that of adults. Social Security > numbers are of special interest because they can be associated with any name > and birthdate, the report showed. They also offer thieves a clean financial > canvas.

</comment></comment></comment><comment><USER10> I'm a big free market guy, but the idea of a private entity profiting off of collecting my personal financial information (and charging ME to have some degree of control over it) is absolutely insane

<comment><USER6> +1. If ever there was a place for a not-for-profit federal organization, this is it. Edit: Like others have pointed out, it's just beyond believe what information these CRAs hold on us.

</comment></comment><comment><USER11> After the leak last year I've spent a few hundred bucks freezing my wife and my credit.

While the cost is annoying, the process is even more annoying.

The entire idea/industry needs to be changed. It's absurd all Americans are opted into this system.

How is credit reporting handled in other countries?

Edit: my wife's credit. ;)

<comment><USER12> Is your wife ok? ;)

But no, on a more serious note, the process should look more like:

1\. Institution asks for Credit pull

2\. Person gets alerted for approval of said request

3\. After pull, credit remains locked

<comment><USER11> Ha. I didn't realize I did that. She's cool ;)

<comment><USER13> Cool but not frozen?

</comment></comment></comment></comment><comment><USER14> Anything short of holding credit agencies, banks, and anyone who improperly leaks data or acts on false data liable for their actions is an improper solution here. Why should I have to freeze my credit, when it's the bank or credit agency's fault that someone opened a fraudulent account and it should be them and only them that pay to fix this up? This bill is a joke and only serves to reinforce the notion that it's the consumers' fault that banks act on fraudulent information.

It's not the consumer's fault that banks themselves commit fraud, so why do the consumers then have to pay for the consequences? With shit laws like this on the books, it's hard to get angry at people who rob banks: they're doing what our legislators failed to do for them. I don't even see that as immoral anymore, especially compared to current law that protects the thieves and banks and throw the consumer under the bus.

</comment><comment><USER15> While this is good, it's another instance of Congress slapping a bandaid on a gunshot wound and calling the problem solved. We wouldn't need freezes if we had an identity system which wasn't immutable (ssn). For once can we fix the root cause of a problem rather than treating symptoms.

</comment><comment><USER16> Given that it feels like everyone and their brother's camel seems to have access and input into our histories and the (just. holy shit.) levels of information these firms have access to and records for regarding just about everything we do with a dollar this is as good a start as any.

Data hoarding industries like these, those that really have too much sway over individuals' well being and access to basic services, needs to be the second most regulated thing on the planet. Or maybe third.

</comment><comment><USER17> Why not make "frozen" credit the default?

<comment><USER18> Or a better question: why are social security numbers used as a password but yet stored by every financial institution you interact with?

If only we had some way to verify an identity while keeping important information private- oh wait, we do, public-key cryptography.

<comment><USER19> They could even hash the number and it would STILL be more secure than storing the raw string in plaintext.

</comment></comment></comment><comment><USER3> I just went through this stuff after an identity theft incident. While locking my Equifax file, I learned something surprising:

 _Locking your Equifax credit report prevents access by potential creditors and lenders, but there are exceptions. ... Companies that wish to make pre- approved offers of credit or insurance to you ...

Equifax maintains consumers' credit reports and provides information to certain customers, including credit card companies and lenders, so that they may offer pre-approved offers to consumers as permitted by law. Consumers that prefer not to receive such offers should visit www.optoutprescreen.com, or call toll free at 888-5-OPT OUT (or 888-567-8688)._

So I'd really encourage everyone to do that opt out thing, too.

<comment><USER20> Read a different thread here - an Equifax "lock" isn't the same as a credit freeze.

<comment><USER21> Just to be clear, a credit freeze does not stop prescreened credit offers either [0].

[0] [https://www.consumer.ftc.gov/articles/0497-credit-freeze- faq...](https://www.consumer.ftc.gov/articles/0497-credit-freeze-faqs#offers)

</comment></comment></comment><comment><USER22> This could not come fast enough, I cannot fathom how this is not already mandatory as Credit Freeze is the best tool a consumer has to fight against identity theft. I should be able to unfreeze when I know I am going to take out a line of credit and freeze when I know I will not take out a line of credit. The fact that there are private entities out there charging me to do this is insanity.

<comment><USER20> Default to frozen by default would be the proper fix.

A credit freeze is analogous to "deleting" a hypothetical mandatory FB account (we all know they don't actually delete your data but at least you aren't publicly visible).

</comment></comment><comment><USER23> credit freeze fees and fees related to identity protection are an extortion/a shakedown plain and simple. Companies like Equifax sell your data then charge you a fee to block access to that which they sell, if you don't your opening yourself up to identity theft from behavior that they themselves have caused.

</comment><comment><USER24> I didn't see this mentioned in the article but the issue is not just the fees but the fact that a "credit freeze" only last for 90 days which means you need to reapply the freeze every 3 months. This is a game of attrition and over time consumers are less and less likely to re-up their freeze.

<comment><USER25> You might be thinking of a "fraud alert", which is different than a "credit freeze" => [https://www.consumer.ftc.gov/blog/2017/09/fraud-alert-or- cre...](https://www.consumer.ftc.gov/blog/2017/09/fraud-alert-or-credit- freeze-which-right-you)

<comment><USER24> Yes you are right. I conflated the two. That said given the the whole system is known to be compromised one should be able to put a fraud alert on their credit file for longer than 90 days. Or maybe even a permanent fraud alert.

It seems like they could be complimentary since a creditor needs to call you to get approval before extending credit in your name. With both of these even if your credit was unlocked the credit issuer would still be required to get your verbal approval.

</comment></comment></comment><comment><USER26> This should be a requirement in order to align Credit companies interests with the interests of their "users". If they get defrauded often enough that the administration costs of freezing and unfreezing become problematic, they will likely finally try to stop getting hacked.

</comment><comment><USER27> It's been free in Indiana for at least a decade. I was surprised to learn that friends in other states have to pay for what should be a right.

</comment><comment><USER28> This gives me hope but I wouldn't be surprised if they labelled it as "burdensome regulation".

<comment><USER29> Good. You know what? I want that entire industry to be hugely burdened.

</comment></comment><comment><USER30> How about also making them the _default_ state of one's credit history?

</comment><comment><USER31> Yes.</comment>


Ask HN: Where to find WebGL/Three.js open source developer? - <AUTHOR> I'm working on a hobby project and one of the next major steps is to use WebGL/Three.js or related to build a component which allows users to create maps (think water, cliffs, hills, etc.).<p>I host the project on GitHub, and as it's just a side project there is no money involved. Does anyone have tips to find such a co-developer? Have you encountered a similar issue? <comment><USER2> PlayCanvas might be an easier jumping on point for WebGL for you. See [https://github.com/playcanvas/engine](https://github.com/playcanvas/engine) and [https://playcanvas.com](https://playcanvas.com)</comment>


Microsoft Is Killing ePub Support in Edge Classic - <AUTHOR> https://www.thurrott.com/cloud/web-browsers/microsoft-edge/213350/microsoft-is-killing-epub-support-in-edge-classic <comment><USER2> Oh what? That is really bad. Epub support in edge was the one thing I used and was impressed by. The text looked great and it supported very good read-aloud that tracked the words in the text. I was impressed how well it read tricky things like numbers and sub-clauses. If you switched to one of the British accents, crank the speed to 2x and it was a great way to power through some books that might have become a slog. When you begin to tire, switch the gender of the voice and it suddenly becomes more digestible again.

I just don't know why MS spends so much time shooting itself in it's feet. So much talent and promising product gets burned for indecipherable reasons.

<comment><USER3> Killing decent products is the Google way. Now that Microsoft is using chromium maybe the Google way is seeping in.

<comment><USER4> Invasive telemetry and profiting from ads instead of selling software is definitely the "Google way" and it's been taking over MS since at least the beginnings of Windows 10 (remember the forced "free upgrade"?)

<comment><USER3> Do you support telemetry if it makes the product better, and not used for ad targeting?

<comment><USER5> Not sure why you're being downvoted. Telemtery can be a invaluable tool for a whole bunch of reasons. It should, of course, be optional.

I think only 1 type of telemetry shouldn't ask for permission, a quick, anonymous ping to say it's been installed.

<comment><USER6> _a quick, anonymous ping_

Sure, but for it to be anonymous it should not contain the sender's IP address. That probably means an UDP message with nulled source ip, which runs foul of many ISP's egress filters (and rightly so).

So, realistically speaking, there is no such thing as an anonymous ping. Or am I missing an obvious solution here?

<comment><USER7> You could have an aggregation service, run by a trusted independent non- profit. It could have a strong privacy policy and external auditing.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The biggest thing that was exciting to me was EPUB having native OS support, effectively, via Edge. You could assume an EPUB was readable in Windows without downloading something, just like you can assume for JPGs and TXT files. This feels like a huge step back for the format to lose this.

<comment><USER9> I just realized that EPUB is the CHM of the modern world.

<comment><USER10> Does CHM mean [https://en.wikipedia.org/wiki/Microsoft_Compiled_HTML_Help](https://en.wikipedia.org/wiki/Microsoft_Compiled_HTML_Help) ?

<comment><USER9> Exactly. The original html based document format.

<comment><USER11> Which was not a bad format. Even as thorough Linux user with no Windows in sight, I have often used .chm files for handy, offline documentation. PHP docs come to mind, back when PHP was something I occasionally did.

<comment><USER12> Text, markup, arborescence, embedded medias, all in a single file, supported natively by the os and comparable between apps. Before the internet everywhere there is an entire generation of windows software, especially developer tools, where chm was the norm for high quality documentation.

<comment><USER13> Yeah i really appreciated the chm format too. They killed it or is it still alive?

Mac OS guides are also a very good editorial/digital work. (but no videos?). The guides on MacOS look very simple and clean. I appreciate when the software I buy has a guide like that (pixelmator) and not just a link to a website (sketch). I know the required effort so I value them a lot.

<comment><USER12> It still works just fine but the tools haven't been updated for a while, almost no modern software supports exporting to it and it's not "cool" anymore. So now we use often inferior stuff instead, or merely get a link to some online documentation, until someone "reinvents" it five years from now probably using some kind of electron variant making it cool again.

<comment><USER14> Hm, almost every desktop program i have on Windows that bothers with help uses CHM and pretty much every documentation generator can generate CHM files, why do you think it isn't used?

<comment><USER12> Try to find node, go, vscode etc ... documentation in chm. They have third parties versions sure, but not official. And for those that still uses it, it's rarely up to date.

<comment><USER14> Of those only VSCode is a desktop application and VSCode is already a web browser. Also considering it is meant to be modular, its documentation system would also need to be modular and i'm not sure if CHM is good with that.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> > "Download an .epub app to keep reading," a notification in Edge classic > reads when you load an EPUB document. "Microsoft Edge will no longer be > supporting [sic] e-books that use the .epub file extension. Visit the > Microsoft Store to see our recommended .epub apps."

What's wrong with this? It looks grammatically correct to me…

<comment><USER16> > What's wrong with this?

This is answered in the next sentence of the article: _it's "support" not "be supporting," Microsoft_.

The is using the future continuous when the simple future is more appropriate to the message.

<comment><USER15> "Microsoft Edge will no longer be support e-books that use the .epub file extension" sounds _very_ wrong.

<comment><USER16> "support", as the author of the article suggests, should replace "be supporting", not just "supporting". The simple future is called for, not the future continuous.

</comment></comment></comment></comment><comment><USER17> This makes no sense to me. EPUB is basically HTML in a ZIP container with mandatory included files. If you already have a full-blown web browser, and a ZIP library at hand, naively rendering EPUB pages by unzipping to temp files and tweaking the URI resolver is an easy addition. If you already have a full- blown web browser that already renders EPUB pages better than a lot of EPUB- specific reader programs, removing that support is doing work to reduce functionality.

<comment><USER18> If it isn't used a lot, I'd prefer the budget to go somewhere more useful instead.

<comment><USER19> They don't have to maintain it, they just don't need to completely get rid of it. That part is baffling.

<comment><USER20> > They don't have to maintain it

Yes they do. Rarely used, unmaintained code is a great way to accrue security vulnerabilities and other bugs.

<comment><USER17> They would still have to maintain it, but they wouldn't need to add features. If they coded it well enough in the first place, that could be one person putting in 8 hours a month on the highest-priority issues in the backlog.

EPUB is a container format. It's basically an encapsulated multi-page website with some metadata. The marginal effort for a web browser is unpacking the files with a ZIP library and deciding how going from chapter001.html to chapter002.html works inside the existing UI. If you don't want to think much about the latter, render each page exactly as if it were downloaded from an online website, and use the forward and back buttons to step along the built in reading order.

I actually wonder why all web browsers don't support it natively.

</comment></comment></comment></comment></comment><comment><USER21> I'm very surprised how bad most ePub readers are. You would think that by now picking an ePub reader for the PC would be a no brainer but it seems there are very few good ones. Too bad, the ePub reader is very good. It's the only reason I use Edge.

<comment><USER22> Same with physical ebook readers, really.

<comment><USER23> How is it bad? What is the problem? The only thing that doesn't work properly is displaying PDF because many PDFs are setup bad. And sometimes - in epub - images are too small to display properly. That is something that really can be improved - image display and handling.

<comment><USER22> Not responsive enough when changing pages. Too much tech crammed in to ramp up the price of something essentially worth $15. Navigation is unintuitive, it's hard to get stuff on the device. Not all formats are supported by all readers and PDF support isn't what it should be. I can't turn pages with either my left or my right hand, no easy switching beween books and if you've got more books on there than the average person, like maybe 70, you're already screwed to find what you want to switch to reading at the moment quickly.

</comment></comment></comment></comment><comment><USER24> This was literally the only thing Edge was actually good at...

Dammit Microsoft!

<comment><USER25> It's a reasonably good PDF reader too. Because of it Windows got nearly as good as Linux on that front.

<comment><USER12> Still worth it to replace it with Sumatra right after install though in my opinion.

</comment></comment></comment><comment><USER26> They should spin off the old Edge as a reincarnation of Reader from Windows 8.

It was a great app and I hoped it would become as versatile as Preview on Mac.

It was annoying when I double clicked a EPUB (or PDF) it would open the browser and sometimes all my tabs from last session would open and in most cases just the EPUB would open and I lose the previous browsing session.

This needs to be a dedicated app separate from the browser. I hope they have something in the works.

<comment><USER27> Does windows have anything analogous to quicklook? That's one feature I think Apple did well with.. though it's obnoxious it can no longer be extended to support non-MPEG-LA video formats...

I've experimented with using mpv controlled over json IPC as a FOSS substitute to quicklook, which works well for media but not documents...

<comment><USER28> Both QuickLook and Seer do this and both have plugins making it much more akin to the Quick Look circa Leopard and Snow Leopard. QuickLook is more actively maintained but both are very good.

[1]: [https://pooi.moe/QuickLook/](https://pooi.moe/QuickLook/) [2]: [https://www.microsoft.com/en-us/p/seer- pro/9pgvfjbvbzwx?acti...](https://www.microsoft.com/en-us/p/seer- pro/9pgvfjbvbzwx?activetab=pivot:overviewtab)

<comment><USER29> Yep, I recently moved to windows 10 for my personal machine and as looking for equivalent to macOS features. QuickLook just work, and has even more options than the version from macOS

</comment></comment></comment></comment><comment><USER30> Microsoft have a weird habit of releasing niche or experimental products in a fully fledged form and then not knowing how to make them useful or popular (3D Paint, anyone?)

Did they really expect a solid ePub reader to be a significant selling point for the browser?

<comment><USER31> >Did they really expect a solid ePub reader to be a significant selling point for the browser?

Well it was the only reason I ever opened Edge, so it kind of worked?

</comment></comment><comment><USER32> I think it was a mistake integrating the browser with an Epub reader. Having said that, it was still a very nice and elegantly designed part of Edge. So I would like having it as a stand alone app, but I don't expect MS doing the right thing. They will either kill it completely or rewrite it from scratch with Electron.

<comment><USER33> Using a browser to read an Epub which is effectively just some bundled HTML seems like a very logical thing to do to me.

<comment><USER32> That is like saying a Zip file manager should also include Epub reader since Epub is basically a zip file.

Reading a book and browsing web are fundamentally different activities and need different UX and business logic. Putting book reading capabilities to a browser makes the code base bloated in my opinion.

<comment><USER34> I have to say I disagree there. "Our program for handling HTML+CSS content should handle HTML+CSS content stored in containers" is a much more reasonable idea than "Our program for opening containers should also handle whatever content is inside those containers." It's a common thing for programs to do. VLC plays videos in zip files and RetroArch plays games in zip files, for example.

And I don't think reading a book and browsing the web are fundamentally different activities. They're near-identical. The web was created as a way to access documents consisting of text, images, and links to other points within the current document or to other documents. A book is a document consisting of text, images, and links to other points within the current document (endnotes, footnotes, "see page X") or to other documents (references/citations). The UX and features you'd want for a book reader are the same you'd want for a browser's Reader Mode, as implemented by Edge, Firefox, and Safari. I don't see any difference between reading an ePub file and reading a long article on a website.

</comment></comment></comment></comment><comment><USER35> I really wished they hadn't killed Edge. And I _love_ EPUB in Edge for technical books - you can open several in different tabs and also search within them. The browser is the most natural vehicle for EPUB! Using a third party reader on Windows is going to be a pain.

<comment><USER36> It boggles the mind. Literally everything happens in the browser except reading hypertext documents.

</comment></comment><comment><USER37> Why won't they just open source the reader instead of abandoning it? I would love to have at least one 100% working epub reader on Windows...

<comment><USER38> > I would love to have at least one 100% working epub reader on Windows...

SumatraPDF is fine to read EPUBs on Windows (also supports PDF, MOBI, FB2, CHM, XPS, DjVu) - it's free, open-source software [0][1].

[0]: [https://github.com/sumatrapdfreader/sumatrapdf](https://github.com/sumatrapdfreader/sumatrapdf)

[1]: [https://www.sumatrapdfreader.org/free-pdf- reader.html](https://www.sumatrapdfreader.org/free-pdf-reader.html)

</comment></comment><comment><USER39> Bizarre decision. I've done my share of epub wrangling by hand (that is with Sigil).

Epubs are basically zipped HTML. How hard would it have been to leave basic support?

So now one has to "freeze" Edge to keep epub support.

I doubt it is as easy as moving Edge to a different folder. Probably need to mess with regedit as well.

</comment><comment><USER40> Oh, it could do that? I still use Calibre and Sigil. Then again, how can any browser not in some way support ePUB? It's just a bunch of hyperlinked XML files in a container. Rename it to a .ZIP and you're good to go.

</comment><comment><USER41> While I'm sure that there are some adequate ePub readers for Windows desktops, I find myself doubting that there are many actual good ePub readers. At the very least when I looked from a Windows phone a year or two back there was nothing very good in the Windows App Store. Maybe Microsoft's push towards the Surface family particularly the Surface Go as a tablet means that there will be something worthwhile.

At least, that's the dream.

<comment><USER41> This is quite late, but maybe useful for later thread viewers.

[https://wiki.mobileread.com/wiki/E-book_software#Windows](https://wiki.mobileread.com/wiki/E-book_software#Windows)

</comment></comment><comment><USER42> This is really unfortunate. Killing one of the best applications? I mean it's fast, reliable, simple, easy to use, integrated...

</comment><comment><USER43> "X browser is killing support for Y format"

Well, good. Now we can go back to when browsers handed files off to helper apps that supported the formats in question. I'm all for it.

<comment><USER44> You say this now but I'm, for one, very glad pdf.js exists.

</comment></comment><comment><USER45> After getting a Windows Tablet, I was looking hard for a good epub/pdf reader with annotations. To my surprise, Edge was by far the best reader out there. However, a few weeks later an update killed the annotations feature - I just couldn't save them anymore. None of the support-provided solutions has worked. I then started using Calibre to convert my epub files into azn3 and open them in the Windows Kindle app, which is also quite nice.

</comment><comment><USER46> Very annoying, but I guess makes sense as chromium doesn't support epub and they'd want to keep the different edges consistent. And the old edge will be gone soon anyways..

A natural question is: is edge still going to be the default PDF reader in Windows? And will edge's pdf viewer look just like chrome's

<comment><USER47> Probably and it does in Dev and Beta builds (which is a disappointment because "Edge Classic" PDF Viewer was better and Chrome's a distinct regression in some disappointing ways).

<comment><USER46> yeah, you'd think it wouldn't kill them to have whatever you call the opposite of continuous scrolling?

</comment></comment></comment><comment><USER48> They are starting wrapping up and deprecating Edge classic, as the Chromium one is now in beta

</comment><comment><USER49> Can someone explain why one would use ePub, vs pdf? PDF seems way more popular.

<comment><USER38> I'd say PDF is all about presentation, while ePub/Mobi is closer to real books that focus on content and legibility.

EPubs are perfect for long-lengthy documents, where you can bookmark position to given line of text (not page, so I'd say it's more precise to get back in which paragraph you've finished reading), everything is searchable/copyable/highlightable, formatted into single-column, usually justified, wrapped using hyphenation, easy to change font size, colors (i.e. day/night mode) and there isn't strict paper (canvas) size. Basically, it increases accessibility and focus.

While the PDF is just like a ZIP (I think those formats share the same popularity), that consist collection of numbered crisp, anti-aliased PNGs and SVGs, sometimes with an attached metadata. I often hear, there isn't any chance to PDF document look different on other device. However, PDF is just a subset of PostScript (hence the name) and has many versions (standardized as a public document, by ISO, by other parties or just paywalled)[1]. Fortunately, quite old PDF1.5 version has majority of features used by typical document creator, also fact that there aren't many competing implementations and they seem to be up-to-date with PDF2.0 convinced users and developers to use it. That's worth noting, PDF supports reflow [0], but it is far from perfect and I guess many bookworms will refrain from using it at all.

[0]: [https://en.wikipedia.org/wiki/History_of_the_Portable_Docume...](https://en.wikipedia.org/wiki/History_of_the_Portable_Document_Format_\(PDF\))

[1]: [https://helpx.adobe.com/acrobat/using/reading-pdfs-reflow- ac...](https://helpx.adobe.com/acrobat/using/reading-pdfs-reflow- accessibility-features.html#reflow_a_pdf)

</comment></comment><comment><USER50> Can't they develop an EPUB extension for Chrome (new Edge)?

<comment><USER47> Possibly, but they likely don't have any incentive to do that and are just going to leave it to third-party devs. Their eBooks store for mobile was the reason they added it to Edge Classic in the first place, and that store has already shut down because no one was using it.

</comment></comment><comment><USER51> I wasn't aware of _any_ browsers having epub support. Additionally, the practice of using browsers as your OS is bad. Native applications are the way to go. Pdf support should also be removed from all browsers.

<comment><USER52> > Pdf support should also be removed from all browsers.

Why? Instead of opening a PDF like any other webpage, or image, or video, etc. Which is instant and doesn't require any mental focus shifting, I should instead download the PDF, and use a PDF viewer instead?

For something so ubiquitous it seems kind of silly. Browsers are for browsing and consuming content, if a specific type of content has widespread usage, I see no reason not to include it.

<comment><USER53> > Which is instant and doesn't require any mental focus shifting

Opening URLs from the browser to native apps specialized for their format shouldn't be non-instant or require any mental focus shifting.

> Browsers are for browsing and consuming content

"Browser" is short of "Web Browser". They're meant primarily for HTML pages. General "browsing and consuming content" is too broad. You're including file managers and multimedia players in that. There's no reason to concentrate everything into a single program. Shifting between specialized applications should be seamless. If it's not, the window manager / desktop environment is lacking configuration / features.

> if a specific type of content has widespread usage, I see no reason not to > include it.

Because you end up with a bloated mess. The modern web browser is an OS inside an OS.

<comment><USER54> > "Browser" is short of "Web Browser". They're meant primarily for HTML pages.

I don't see how you get from the first of these statements to the second. Web browsers browse _the web_ —an interlinked set of hypertext documents. A "hypertext document" being anything that embeds URLs you can tell the browser to navigate to, usually by clicking. A PDF _is_ a "hypertext document."

> There's no reason to concentrate everything into a single program.

An ePub _is also_ a hypertext document, though. And one that is made out of HTML and CSS files! Would you suggest that a web browser shouldn't be able to open an MHTML archive? Because an ePub is almost exactly the same format, just with a different base CSS style + media selector. (There are some restrictions on scriptability, but from what I recall those are ePub UA restrictions, not restrictions in the standard.)

> Shifting between specialized applications should be seamless.

What if you could shift between specialized applications... inside other applications? Remember OLE? That's how web browsers have traditionally displayed most formats. The whole need for special plugins for QuickTime, Flash, ActiveX, Java, etc. was just to allow those formats to be embedded _in_ HTML pages. But if you're just opening a URL in the web browser and handing the viewport over to a COM component to render—browsers (and many other types of applications) do that just fine, without any need for "concentrating everything into a single program."</comment></comment></comment></comment>


Doctor Who 'had anti-Thatcher agenda' - <AUTHOR> http://www.telegraph.co.uk/culture/tvandradio/doctor-who/7235547/Doctor-Who-had-anti-Thatcher-agenda.html <comment><USER3> Is it a sign of a more innocent era that one could produce a fiction show without having an army of pundits decrying its "leftist media bias" - <USER2> when there was one, deliberately?

Or were they just so bad at getting their message across that "nobody really noticed or cared."

<comment><USER4> I wouldn't exactly characterize the 80s as a particularly innocent era.

Most likely, nobody cared by that point. Doctor Who was in its final stages by then.

Ever notice that in general when a fiction program decides to make a political statement, it turns to utter ham-fisted crap? I'm imagining _every show_ like that, in the kind of ultra-earnest tone of a street-corner communist, and it doesn't surprise me at all that the show was cancelled.

</comment></comment><comment><USER5> No doubt it's not the only show of the time to have an anti-thatcher bias. Many shows openly ribbed the government particularly in comedy like spitting image and Alexei Sayle. Why should anyone take notice of a show that tries to say it subtly when everyone else is saying it openly.

No wonder it got cancelled, the script-writers sound pretentious. Perhaps they thought they were Huxley, Orwell or someone.

</comment><comment><USER6> I'm shocked, _shocked_ I say! A show about an intellectual hippie who refuses to carry guns and regularly overthrows authorities is liberal?

Sorry for the snark, but I don't really see it as that surprising that Doctor Who has an explicitly liberal message when it's main character is The Doctor. He's a textbook left-wing radical - dude blew up an arms factory to grow a banana orchard, ran away from the Time Lord Presidency, was jailed (exiled to Earth as the 3rd Doctor) for violating Time Lord laws to help the less fortunate, and ultimately chose to die rather than kill (9th Doctor, "The Parting of the Ways").</comment>


Why the death of Moore's Law wouldn't be such a bad thing - <AUTHOR> http://www.techradar.com/news/computing/why-the-death-of-moore-s-law-wouldn-t-be-such-a-bad-thing-1175977 <comment><USER2> I think it would be good for it to die because it would mean the current way of making chips has to die. Maybe after that we'll be forced to think of new ways to make processors and computers. Perhaps we'll make them more like the human brain, or perhaps we'll just put a lot more funding into researching quantum computers, and so on.

That could jumpstart computers that are orders of magnitude faster than what we'll have by mid-2020's.</comment>


Launch HN: Quilt (YC W16) – A versioned data portal for S3 - <AUTHOR> We're Aneesh and Kevin of Quilt (https://open.quiltdata.com/). Quilt is a versioned data portal for S3 that makes it easier to share, discover, model, and decide based on data at scale. It consists of a Python client, web catalog, and lambda functions (all open source), plus a suite of backend containers and CloudFormation templates for businesses to run their own stacks. Public data are free. Private stacks are available for a flat monthly licensing fee.
Try searching for anything on https://open.quiltdata.com/ and let us know how search works for you. We kind of surprised ourselves with a Google-like experience that returns primary data instead of links to web pages. We've got over 1M Jupyter notebooks, 100M Amazon reviews, and many more public S3 objects on over a dozen topics indexed in ElasticSearch.
The best example, so far, of "S3 bucket as data repo" is from the Allen Institute for Cell Science https://open.quiltdata.com/b/allencell/tree/.
Kevin and I met in grad school. We started with the belief that if data could be "managed like code," data would be easier to access, more accurate, and could serve as the foundation for smarter decisions. While we loved databases and systems, we found that technical and cost barriers kept data out of the hands of people that needed it the most: NGOs, citizens, and non-technical users. That led to three distinct iterations of Quilt over as many years and has now culminated in open.quiltdata.com, where we've made a few petabytes of public data in S3 easy to search, browse, visualize, and summarize.
In earlier versions of Quilt, we focused on writing new software to version and package data. We also attempted to host private user data in our own cloud. For reasons that we would soon realize, these were mistakes:
* Few users were willing to copy data—especially sensitive and large data—into Quilt
* It was difficult to gather a critical mass of interesting and useful data that would keep users coming back
* Data are consumed in teams that include a variety of non-technical users
* Even in 2019, it's unnecessarily difficult and expensive to host and share large files. (GitHub, Dropbox, and Google Drive all have quotas, performance limitations, and none of them can serve as a distributed backend for an application.)
* It's difficult for a small team to build both "git for data" (core tech) and "Github for data" (website + network effect) at the same time
On the plus side, our users confirmed that "immutable data dependencies" (something Quilt still does) went a long way towards making analysis reproducible and trace-able.
Put all of the above together, and we had the realization that if we viewed S3 as "git for data", it would solve a lot of problems at once: S3 supports object versioning, a huge chunk of public and customer data are already there (no copying), and it keeps users in direct control of their own data. Looking forward, the S3 interface is general enough (especially with tools like min.io) to abstract away any storage layer. And we want to bring Quilt to other clouds, and even to on-prem volumes. We repurposed our "immutable dataset abstraction" (Quilt packages) and used them to solve a problem that S3 object versioning doesn't: the ability to take an immutable snapshot of an entire directory, bucket, or collection of buckets.
We believe that public data should be free and open to all—with no competing interests from advertisers—that private data should be secure, and that all data should remain under the direct control of its creators. We feel that a "federated network of S3 buckets" offers the foundations on which to achieve such a vision.
All of that said, wow do we have a long way to go. We ran into all kinds of challenges scaling and sharding ElasticSearch to accommodate the 10 billion objects on open.quiltdata.com, and we are still researching the best way to fork and merge datasets. (The Quilt package manifests are JSONL, so our leading theory is to check these into git so that diffs and merges can be accomplished over S3 key metadata, without the need to diff or even touch primary data in S3, which are too large to fit into git anyway.)
Your comments, design suggestions, and open source contributions to any of the above topics are welcomed. <comment><USER2> Congratulations to the Quilt team on the launch!

Quilt reached out to me and suggested I chime in suggesting that people interested in versioning data also check out Dolt ([https://github.com/liquidata-inc/dolt](https://github.com/liquidata- inc/dolt)) and DoltHub ([https://www.dolthub.com](https://www.dolthub.com)).

We've taken the Git and GitHub for data analogy a lot more literally than Quilt has :-) We are a SQL database with native Git semantics. Instead of versioning files like Git, we version table rows. This allows for diff and conflict detection down to the cell level. We are built on top of another open source project called Noms ([https://github.com/attic- labs/noms](https://github.com/attic-labs/noms)).

We think there is a ton of room in this space for a bunch of tools: Quilt, Noms, QRI ([https://qri.io/](https://qri.io/)), Pachyderm ([https://www.pachyderm.io/](https://www.pachyderm.io/)), and even Git. We're excited to see so many bright minds trying to solve this problem.

We're going to be populating DoltHub with a bunch of datasets we harvest from the open data community to show off the capabilities of Dolt. The coolest one so far is the Google open images dataset: [https://www.dolthub.com/repositories/Liquidata/open- images](https://www.dolthub.com/repositories/Liquidata/open-images).

<comment><USER3> Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: [https://github.com/iterative/dvc](https://github.com/iterative/dvc). Looking forward to checking out Open Images.

<comment><USER2> The coolest thing to do is diff between the V2 and V3 branch for a label_descriptions table.

[https://www.dolthub.com/repositories/Liquidata/open- images/c...](https://www.dolthub.com/repositories/Liquidata/open- images/commits/j9mmf12fuat35k8l1kf6kr969kdgd8fv)

You can start to see the power of column-wise diffs. You can start to imagine what it would be like to change this table and then merge Google's changes in V3 onto your modified copy. Very powerful. We need a query interface on top of diffs. Lots to build...

<comment><USER4> Great work on Dolt and seeing temporal data stores emerging :-)

Regarding XML and JSON SirixDB[1] already provides full blown time-travel queries using a fork of Brackit[2], that is basically XQuery to process and query both the XML as well JSON documents. That said SirixDB in principal could also store relational data or graph data. The storage engine has been built from scratch to offer the best possible versioning capabilities. However I've never implemented branching/merging as I didn't come up with good use cases. It seems it's then always more of a versioning system like Git, but more fine granular.

I always struggled to implement this as SirixDB currently only allows a single read-write transaction on a resource. Thus, if it would support branching and merging users would have to manually handling conflicts when merging (or automatically -- using a merge-strategy which is often case not good).

There's however plently of optimization potential, as SirixDB optionally stores a lot of metadata for each node (number of descendants, a rolling hash, Dewey-IDs, number of children... as well as user-defined, typed secondary index-structures). I'll have to look how to build AST rewrite rules and implement a lot of optimizations into my Brackit binding in the future, so it's just the starting point (but everything should at least work already) :-)

[1] [https://sirix.io](https://sirix.io) and [https://github.com/sirixdb/sirix](https://github.com/sirixdb/sirix)

[2] [http://wwwlgis.informatik.uni- kl.de/cms/fileadmin/publicatio...](http://wwwlgis.informatik.uni- kl.de/cms/fileadmin/publications/2013/Dissertation-Baechle.pdf)

</comment></comment></comment></comment><comment><USER5> This is great. Thank you for so openly sharing your strategic thinking and lessons learned.

I follow about 100 projects in this space "github for data" and haven't yet seen a breakout hit. Yours looks like it has potential. I like the simplicity and the "objects by file extension". Lots of these sites I think get too complex too quick.

At the UH Cancer Center we routinely deal with datasets in the TB - PB range, and that type of size definitely makes this problem qualitatively different. Your splitting of the storage (S3) from the front end is the correct technical decision, IMO.

I've worked in this space for about 10 years. My open source project is called Ohayo, and I used to try and do both front end and backend, and then similarly decided to drop the data storage backend and instead focus on my strengths, which is front end exploratory data analysis.

I think adding a "quilt" keyword to Ohayo, and access to the Quilt datasets directly in Ohayo may be mutually beneficial. Ohayo is just a single dumb web app (no online storage, no tracking, full program source code are stored in the url) and pulls in data via HTTP. Here's an example program that shows the post history from the 2 quilt founders on hackernews: [https://ohayo.computer?filename=hncomparison.flow&yi=~&xi=_&...](https://ohayo.computer?filename=hncomparison.flow&yi=~&xi=_&data=hackernews.submissions_100_akarve_kevinemoore~_tables.basic~_hidden~_filter.where_by_!%253D_~__hidden~__filter.where_type_%253D_story~___hidden~___vega.scatter~____xColumn_time~____yColumn_score~____colorColumn_by~layout_column)

We use Vega for visualization. You could imagine allowing fast simple EDA on these Quilt data sets through simple Ohayo links. Ohayo version 14 is a substantial improvement and I hope to ship next week or two, and then would love to add Quilt to the picture.

<comment><AUTHOR> Oh I would love to get the UH Cancer Center data into Quilt! Do you happen to have an S3 bucket with that data live? If the bucket is publicly permissioned it should "just work." We can talk about indexing the data for search. We are comfortable in the TB-PB range :)

I will look more closely at Ohayo.

<comment><USER6> "At the UH Cancer Center we routinely deal with datasets in the TB - PB range ..."

...

"Do you happen to have an S3 bucket with that data live?"

As someone not working in academia (or in this field at all) can you help me understand the question you have just asked ?

Specifically, wouldn't it be _tremendously profligate_ for them to have that PB range dataset living in S3 ?

Given the resources that a university has (in both Internet2 connectivity, hardware budget and (relatively) cheap manpower), why would they ever store that data outside of their own UH datacenter ?

If the answer is "offsite backup" wouldn't it be glacier or nearline or ... _anything but S3_ ?

<comment><AUTHOR> Good questions. First, services like open.quiltdata.com and Amazon's Registry of Open Data cover the S3 costs for public data. So that's one incentive. Second, the cost of cloud resources are highly competitive (if not superior) to on-premise data centers (see [https://twitter.com/mohapatrahemant/status/11024016152632238...](https://twitter.com/mohapatrahemant/status/1102401615263223809?lang=en\)--so) I don't think it's correct to think of S3 as expensive.

There are many ways to shave S3 costs (e.g. intelligent tiering, glacier), but at some point the data become so slow to access that you can't offer a pleasant user experience around browsing, searching, and feeding pipelines.

Most importantly, the "my data, my bucket" strategy gives users control over their data. A university with their own bucket has more control over their data than they do if Google, Facebook, etc. host and monetize it.

</comment></comment></comment></comment><comment><USER3> Aneesh's co-founder here. I just want to add a word of thanks to Jed Sundwall and the AWS Registry of Open Data. The support of AWS makes publishing data at this scale possible. I also want to thank Jackson Brown and everyone else who worked so hard to compile, document and annotate these large and extremely valuable datasets.

</comment><comment><USER4> So you basically store S3 Buckets in Elastic Search and you're using Git for versioning a hierarchy of buckets, right?

It's interesting that versioning now finally seems to be getting some traction in mainstream database systems (even though they are not really optimal in these systems my opinion) and for instance also in your data store. You position this as a Dropbox or Google Drive replacement, right? :-)

I'm asking all these questions, because I'm engineering a temporal, versioned Open Source storage system myself (since I studied at the University of Konstanz until 2012), possibly on a much more database oriented level -- currently for storing both XML and JSON data in a binary format.

A resource in this storage system basically stores a huge tree of database pages whereas an UberPage is the main entry point (reminiscent of ZFSs UberPage, from which SirixDB borrows some ideas and puts these to the sub-file level), consisting of various more or less hash-array based subtrees as in ZFS. Thus, levels of indirect pages are added if more data needs to be stored. I've added some optimizations from in-memory hash-array based tries.

Each revision is indexed. SirixDB stores per revision and per page deltas based on a copy-on-write log-structure.

I've thought about storing each database page fragment in a S3 storage backend as another storage option and using Apache BookKeeper directly or Apache Pulsar for distributing an in-memory intent log (it doesn't need to be persisted before committing to the data files, as the UberPage just needs to be swapped atomically for consistency).

For the interested reader:

[https://sirix.io](https://sirix.io) and [https://github.com/sirixdb/sirix](https://github.com/sirixdb/sirix)

<comment><AUTHOR> Not quite ;) S3 is the primary data and metadata store, so that the rest of the stack is a pure function of S3 data (including Elastic). We don't use git at all yet. We use S3 object versioning and then capture the version, SHA-256, etag, etc. in a JSONL-based manifest [https://open.quiltdata.com/b/quilt- example/tree/.quilt/packa...](https://open.quiltdata.com/b/quilt- example/tree/.quilt/packages/04de92de5f4c4141ee9c326a6719467293599b6478a6556e82d9f42a92440e6a). Said JSONL manifest is simply a "locked list" of all the S3 objects in that package. The same manifests can be checked into git for fork/merge of data sets, but we're still exploring the right way to do that.

I'll let Kevin answer the database fragments question.

<comment><USER7> Neat. But would this not build dependency on s3s versioning and make it hard for getting this portable across other clouds?

<comment><AUTHOR> Not quite. Abstraction layers like min.io support versioning. More importantly, Quilt manifests only require a "fully qualified physical key" that points to the data. In theory, the manifest can work with any URI: S3, local disk, etc.

</comment></comment></comment></comment><comment><USER8> While naming conflicts aren't necessarily always a problem, given that you specifically describe aspects of the problem as a "git for X", you should know that "quilt" is already the name of a popular piece of version control software.

<comment><AUTHOR> Thanks. We were careful to publish as `quilt3` on PyPI, so there are no naming conflicts with the `quilt` patch manager. We are also "Quilt Data, Inc." officially.

Our thesis is that blob storage is already "git for data" and the interesting problems, which we're working on in the open source, are to build a cross- functional data portal atop that base.

<comment><USER8> Thank you for taking that into account; sounds quite reasonable.

</comment></comment></comment><comment><USER9> Excited to see this being re-launched. "git for data" ranks pretty high on my all time list of tech I want to see succeed.

I find the business model very interesting: A kind of "middle layer" SAAS, where you provide a new front-end for an existing service. Not seen that very often. Certainly helps with the data privacy issues. Rapid on-boarding is another immediate benefit.

<comment><AUTHOR> Just determining the shape of "git for data" has been a nontrivial exercise. We found that, if you do the naive translation, you get a "one size fits none," because data and code are fundamentally different.

What would be your main use cases with said "git for data"?

<comment><USER9> One use-case is this one: [https://github.com/HeinrichHartmann/arxiv_meta](https://github.com/HeinrichHartmann/arxiv_meta)

</comment></comment></comment><comment><USER10> Is there a way to get a "dataset of datasets"? That is, all datasets you have, in downloadable tabular form with metadata for each dataset?

<comment><USER5> +1 for this.

UC Irvine has a famous repo of datasets for machine learning research but does not have a metadata dump anywhere. I had to crawl it manually and create one (1). Would be great if you offered a single URL with CSV/JSON/other dump of your available datasets.

1\. [https://ohayo.computer?filename=ucimlrDemo.flow&yi=~&xi=_&da...](https://ohayo.computer?filename=ucimlrDemo.flow&yi=~&xi=_&data=ucimlr.datasets~_handsontable.basic)

<comment><USER3> The list of datasets is pretty dynamic. Would an API call work as a URL? (assuming it could return a CSV/TSV/etc.)?

<comment><USER5> Yes! That is ideal.

<comment><USER3> We'll look into that for sure!

</comment></comment></comment></comment></comment><comment><USER11> Really excited to see this relaunched. Every DS team has issues around dataset management. We previously shared a tutorial on how to get a fully reproducible pipeline with Quilt + Comet.ml [https://blog.quiltdata.com/building-a-fully- reproducible-mac...](https://blog.quiltdata.com/building-a-fully-reproducible- machine-learning-pipeline-with-comet-ml-and-quilt-c0e682b8e25)

</comment><comment><USER12> I also really appreciated your lessons learned — pretty compelling. The showcase buckets on the site are awesome. What's the mechanism by which the public data ends up in S3, just out of curiosity?

<comment><USER3> In general, a data publisher simply creates an S3 bucket in their account and sets the access control to allow public read. In the specific case of the AWS Registry of Open Data, data providers create a clean AWS account to hold the data (the account will have only S3 and won't run any compute services). Once the dataset is accepted into RODA, AWS will cover the costs (S3 storage and egress bandwidth) for the clean account.

</comment></comment><comment><USER13> I'm very excited to see this -- data portability and management is a primary struggle we're trying to map out. Would love to see an engineering post on what you did for ElasticSearch.

<comment><AUTHOR> Can do. A lot of the magic happens in the es/indexer and search lambdas here: [https://github.com/quiltdata/quilt/tree/master/lambdas](https://github.com/quiltdata/quilt/tree/master/lambdas).

The short of what we do: we listen for bucket notifications in Lambda, open the object metadata and send it, along with a snippet of the file contents, to ElasticSearch for indexing. ElasticSearch mappings are a bit of a bear and we had to lock those down to get them to behave well.

What are the big barriers you're bumping into on the data management and portability side of things?

<comment><USER14> Seems like it'd be more elegant (and probably cost effective) if you stored the Lucene indexes inside the buckets themselves.

<comment><AUTHOR> That is an interesting idea. What kind of performance could we expect, especially in the federated case of searching multiple buckets? Elastic has sub-second latency (at the cost of running dedicated containers).

<comment><USER14> That's a bit of an open question right now, unfortunately. Using S3 to store Lucene indexes is a roll-your-own thing since last I checked, and the implementation I wrote currently deals with smaller indexes where files can be pulled to fully disk as needed. S3 does support range requests, which I'd think would mimic random access well enough.

Assuming whatever ElasticSearch implementation you're using is backed by SSDs there'd likely be more latency with S3, but I'd expect it to scale pretty well. Internally, a Lucene index is an array of immutable self-contained segment files that store all indices for particular documents. Searching in multiple indices is pretty much just searching through all their segments- which can be as parallel as you want it to be.

To be honest, I'm actually surprised the Elasticsearch company doesn't offer this as an option. Maybe because they sell hardware at markup?

</comment></comment></comment></comment></comment><comment><USER15> > Try searching for anything on > [https://open.quiltdata.com/](https://open.quiltdata.com/) and let us know > how search works for you.

I suggest adding the possibility of searching for exact matches with quotation marks, and also to ensure that it works with the quotation marks that the default keyboard on iOS has.

For example, I want to search for "Irish Setter" and only see results that include those two words next to each other like that.

<comment><AUTHOR> Here is an example of the "Irish Setter" query: [https://open.quiltdata.com/search?q=irish%20%2B%20setter](https://open.quiltdata.com/search?q=irish%20%2B%20setter)

You would type "irish + setter" in the search box.

<comment><USER15> Thank you :)

</comment></comment></comment><comment><USER16> I'm really excited to start exploring these datasets through quilt and build ML models. Thanks to all the Quilt team and everyone involved for making this possible!

</comment><comment><USER17> Congrats to Aneesh and team! We (Paperspace, YCW15) are big fans and have been following these guys for a while now!

</comment><comment><USER18> Congrats on the launch guys! Excited to read that you've already connected with Tim, more and more smart people tackling this problem is always a plus for everyone :)

</comment><comment><USER19> Does it work with digital ocean spaces?

<comment><AUTHOR> Not yet. But it's closer than one might think. Spaces has an S3-compatible API, and we have plans to use something like min.io to make Quilt work "all the blobs": GCP, Azure, Digital Ocean, etc. OSS contributions welcome :)

What would you use Quilt for in Spaces?

<comment><USER20> Is it possible to implement this using Backblaze B2 instead of S3?

<comment><AUTHOR> Contingent upon support in something like ceph or minio, yes.

</comment></comment></comment></comment><comment><USER21> Congrats. You are an awesome team.

</comment><comment><USER22> Is there any tool in this space that also handles permissions e.g. per column or table?

</comment><comment><USER23> Thanks for sharing this and driving Quilt forward @Kevin!</comment>


Bitcoin Predicted to Be the Nail in the Coffin of Climate Change - <AUTHOR> https://www.forbes.com/sites/andreamorris/2018/10/29/bitcoin-predicted-to-be-the-nail-in-the-coffin-of-climate-change/#6b0a806a745e <comment><USER2> They have the sense of the terms reversed in the title (problem is in the original, sigh). They mean the nail in the coffin of any attempts to limit climate change. Not a new thesis, but a very interesting one despite any nitpick.

</comment><comment><USER3> Maybe not correctly pricing carbon emissions to account for the externality is the nail. If we paid the true price for the energy, the renewable revolution would have been here long ago.</comment>


Deutsche Grammophon launches giant, DRM-free classical music store - <AUTHOR> http://www.boingboing.net/2007/11/30/deutsche-gramafon-la.html

<comment><USER2> That's pretty sweet. They've always released much of the best classical recording available. I'd probably buy a lot of this if not for Rhapsody type services.

Their selection is truly massive it seems. A search for Mahler alone turned up 72 products. Hell, Mahler conducted by Bernstein turns up 21. Only DG could do that.

Classical is probably the only form of music where one label could have their own site and still have awesome selection.

The site design is pretty good. I mean, it's not great, and nothing a Y C group would build, but if you told me that a major record label created a site for classical music only, I'd expect worse.

</comment><comment><USER3> Needs work. The web design is clunky and restriction to a single label means the selection is not the greatest. I'll keep them in mind and check them out again down the road, but for now I'm sticking with Amazon.

<comment><USER4> but what a label! deutsche grammophon is the finest classical music label out there

</comment></comment><comment><USER5> What I'd find really interesting is a music store that sold new classical music, by David Cope's EMI for example. Generate new scores of classical music (by an algorithm trained on Bach/Vivaldi/Purcell/&c), and synthesize them with something better than MIDI.</comment>


Fringe Benefits: Why startups mustn't appeal to the masses - <AUTHOR> http://blog.asmartbear.com/play-fringe.html

<comment><USER2> Somewhat true. Those niche people always have extreme opinions on things you wouldn't necessarily expect. Violate one of those opinions and they'll turn on you as quickly as they glommed on when you said the things they want to hear.

In short, that means that if you try to pivot, you could lose your entire 'fringe' userbase. At one shot. Hopefully you'll see it coming, but you might not.

URL sharing sites are a great example of this. Slashdot, Digg and Reddit used to all be tech sites. Slashdot is just barely hanging on, and Digg and Reddit have gone mainstream. They original userbase of techies have all gone elsewhere, except for a very few that apparently weren't quite as extremist.

Do they care? No, because they have a much larger audience now. The techies got everything off the ground, but once everything was established, the flood of non-techies was allowed and things changed.

<comment><USER3> Agreed, at scale the argument isn't as powerful. Some stay true, most don't. It's more of a technique to get started.

</comment></comment><comment><USER4> "If we understand what the extremes are, the middle will take care of itself" ~~ Dan Formosa / Smart Design. from Documentary "Objectified"

</comment><comment><USER5> Now the difficulty seems to be to explain that to investors...</comment>


Bad Memory - reasons? - <AUTHOR> Hello,
Recently (in the last 5-6 months) I noticed that my memory is getting worse and I'm not sure why.
Sometimes it happens that I forget simple daily words in a conversation. I can't remember stuff that I used to remember very good (house&telephone numbers, etc.).
For example I go to the grocery store with the idea to buy bread. When I get home, I realize that I bought 3 other things but forgot the bread. It sounds funny, but this has never happened to me before this often.
Another problem is that I'm sometimes not able to build complete sentences or put my ideas into sentences, I stutter and I need 2-3 tries before getting the sentence right. Although I think that this is due to my less reading in this language (I'm native Swiss, but read almost every day in English [hacker news, books, papers, etc.], so I'm going to up the German reading to test whether this was the problem.
My sleep habits haven't changed (6-8 hours, depending on the day) a lot. I have a stressful daily life (startup work, studying EE, girlfriend) and even there I tend to forget simple tasks and to-do's, which annoys me really bad. I exercise 3-4 times a week (weight training) and eat a balanced diet with a lot of protein (which I supplement also). I do take an omega-3 supplement and drink mostly water and green tea. I haven't used any drugs at all in my life. I was always good in memorizing and remembering faces, etc.
This problem also keeps me from focusing on important tasks. For example when I pack my bag in the morning, I go to the kitchen to get my meal ready. I then enter the kitchen see an apple and think "oh I should take an apple, too." - wash it, bag it and leave the kitchen, without my meal.
I have two possible reasons (I know they are stupid...) for it, but I can't back them scientifically:
1. too much screen time (working, watching movies, surfing)
2. the fillers in my supplements don't do me any good.
Is learning/studying somewhat connected to the short-term memory? Because when I look back, I haven't studied "real material" for a long time (after HS -> military -> EE exams in summer).
I'm not sure but I believe that only my short-term memory is affected so far. Do these brain trainer apps help?
I really appreciate any help and ideas,
Thanks. <comment><USER2> The famous Ichthyologist JLB Smith [http://cdserver2.ru.ac.za/cd/011120_1/Aqua/Ichthyology/Ichth...](http://cdserver2.ru.ac.za/cd/011120_1/Aqua/Ichthyology/Ichthyology/JLB.htm) used to say that every time he memorised the name of a new fish, he forgot the name of a student. (For his brain remembering the fish was more important that the name of a student). You are probably shifting your interests in other areas than the commonalities of every day. While you packing lunch for the office your brain is already working in the background on your work, girlfriend problems and the like.

Your screen time reason is probably right too. Our attention to topics is constantly shifting. Get off the internet for a day or two and read a book that interests you, you will be surprised at your retention.

I personally don't like the idea of any supplements or artificial food. Try and get what your body needs naturally. Coffee _can_ help.

<comment><AUTHOR> Thanks for your answer.

As I said I do have a healthy diet, but I'm supplementing because of the weight training and I don't plan to take them forever.

</comment></comment><comment><USER3> I went through a similar period too. Turned out to be burn out. It really does creep up on you. Doing a startup work and studying EE is probably too much. In my case it was a day job, moonlighting at a startup and a part-time Masters in software engineering.

Hope your girlfriend is making you have some quality down time. If she is high maintenance, then .... ;-)

My solution was to simplify my life, cut back on TV and take up meditation, ditch the Masters and the startup and build a career and a social life. Your stress levels go way down when you finally learn not sweat over all the materialistic desires. Personally I'm wary of supplements, healthy eating is a simpler and more reliable strategy.

<comment><AUTHOR> Yeah, I was thinking about a burn out, too. Unfortunately cutting one of the two isn't an option at the moment.

I guess I'll integrate some meditation techniques, too and try to cut the screen time whenever possible.</comment></comment>


Science Says Men Should Be the Ones Cleaning the House - <AUTHOR> http://www.ibtimes.com/science-says-men-should-be-ones-cleaning-house-2654468 <comment><USER2> Linkbait title. Probably men are happy with a lower standard of cleanliness, that avoids the use of harmful chemicals. The only way to know is to ensure that men and women are cleaning with the same methods, preferably make them go to the lab to clean a test room during ten tears and them measure the lung capacity again.

</comment><comment><USER3> Actually, science said: "The researchers said people should be careful choosing how they clean the surfaces in their homes."</comment>


Ask HN: NSA and their Internet connection - <AUTHOR>

While I'm infuriated by the breach of privacy, the hacker in me can't help but being intrigued. 'Collecting' and storing these quantities of data requires a massive operation, and I'm left with a lot of questions.<p>One of them is this: how do they capture all this data and transfer it to Utah? The fastest internet connection in the world (Tokyo, I believe) is 2 Gbps, which doesn't seem nearly enough for transfering the data. And that's Tokyo, not rural Utah! Do they have dedicated fiberglass cables? Originating from where? <comment><USER2> The "fastest" single pipe in the world right now is actually 100Gbps, so you're a little behind the times.

In terms of "how do they do that", its probably just store and forward pre- parsed data...</comment>


Yahoo's $8 Billion Black Hole - <AUTHOR> http://www.bloomberg.com/features/2016-yahoo/ <comment><USER2> Just give it the Digg treatment and be done with it. Too many words have been written on this subject. Yes the financials make it much more difficult but the brand is worth something to somebody. Figure that out and unwind the finances to investors in whatever the most equitable distribution is. The damage of staying as is, devalues the brand by the day. More and more money in advertising is being lost to Facebook and it's not coming back anytime soon.

</comment><comment><USER3> There was only one mention of Flickr in the article. It's really the only Yahoo! product that I'm emotionally invested in.</comment>


Apple, Google, and Facebook Are Raiding Animal Research Labs - <AUTHOR> https://www.bloomberg.com/news/features/2019-06-18/apple-google-and-facebook-are-raiding-animal-research-labs <comment><USER2> My attempt at a more accurate summary of the evidence from the article: some animal research labs focus on neuroscience, and some of those neuroscientists use machine learning. Sometimes, they get good enough at using machine learning that tech companies hire them. Some of the senior people -- say, the type that could lead a lab at a top university -- may eventually get paid over a million dollars per year.

My impression is that the article is trying to make a connection like "tech companies are trying to hack our brains...so they're getting in bidding wars over neuroscientists!" when the reality is more like "tech companies are always looking for people with machine learning expertise, and some neuroscientists fit that description, and in some cases their research is directly relevant".

</comment><comment><USER3> I think the story is exacturated. There are some people who move from computational neuroscience to machine learning research, but it's not that common.

<comment><USER4> Did you mean _exaggerated_?

<comment><USER3> Yes!

</comment></comment></comment><comment><USER5> Put another way, PhD's who know how to work hard on projects that take years to complete are leaving academia for high paying jobs at big tech companies.

</comment><comment><USER6> > To the relief of some ethicists, we're a long way from AGI, [...]

I often hear statements of this kind, but do they have basis? While there's not much to suggest that AGI will be developed soon, it doesn't seem sensible to me to say it's a long way away, as we still don't understand how difficult the problem is, nor do we have a clear path toward its development. It may be that it is achievable straightforwardly, with a breakthrough cognitive architecture.

Also, regarding ethical concerns, I don't think AGI is the problem, but rather the broader domain of super-intelligence. Plausibly, super-intelligence could result from the combination of the human mind with non-generally intelligent machinery, resulting in the same dangers.

<comment><USER7> I am convinced that we have the hardware already for AGI. Can an exaFLOPS scale Google datacenter with millions of TPUs really be less capable than the 20 watt ball of jelly we all carry around?

The barriers to AGI are in the algorithms. Nobody knows how long it will take to achieve unsupervised learning that is as general as the human brain. It might take 200 years of slow but steady progress or alternatively somebody could publish a breakthrough in a paper tomorrow and we could have parity by the end of the year.

<comment><USER8> > _Can an exaFLOPS scale Google datacenter with millions of TPUs really be > less capable than the 20 watt ball of jelly we all carry around?_

It is demonstrably less capable in some important dimensions, yes.

<comment><USER6> The brain is very complex, but it's not well understood how well that complexity contributes to its computation. Observable phenomena may contribute to the computational capacity of the brain, or may just introduce noise which impairs it.

Do you know of any tasks where the brain has demonstrated better performance than a computer, where the hardware is known to be limiting? I.e. where there can not be an unknown better program the computer could run.

<comment><USER9> Part of the complexity involves that it doesn't run purely on binary. It involves branches that have 4 to 6 nodes from a single input, with varying strength of signal (chemical) across a synapse, which could be a different chemical release. People are trying to simulate this with a simple on/off switch and things get very difficult, very quickly.

<comment><USER6> If you look at an integrated circuit like a CPU at the gate level, you will find complexity which goes beyond the logic being implemented -- the snaking physical routing of the traces; the width and layer of the traces; supply and ground lines; the orientation of the gates; the number of fins on a multi-gate finFET. Someone without a high-level understanding may wrongly assume that any of these things contribute to the logic.

Sometimes specific details are important, yet do not encode higher-level information. Supply lines must be able to carry enough current, so changing their size may break the logic. Clock distribution lines must propagate their signals at the correct rate, so changing their length may break the logic. Neither of these factors need be considered in a high-level emulation of a processor.

In the same way, complexity in the brain may be irrelevant to its function, redundant, or even problematic. As an example: the vascular anatomy of the brain is complex, but not generally considered to be instrumental to cognition; it's similar to the Vdd and ground, but shapes the neural matter around it. As we don't understand the mechanics of how intelligence arises from neurons, it would be wrong to assume all their properties are instrumental.

Given that an integrated circuit is an engineered system, and the brain is a evolved biological system, and evolution is in many ways more prone to unnecessary complexity because of local optima traps, I would speculate that much of the brain's complexity is likely not instrumental to intelligence (yet may be contributory).</comment></comment></comment></comment></comment></comment>