Introducing Google Cloud Storage Nearline - <AUTHOR> http://googlecloudplatform.blogspot.com/2015/03/introducing-Google-Cloud-Storage-Nearline-near-online-data-at-an-offline-price.html <comment><USER2> The detailed pricing is at [https://cloud.google.com/storage/pricing#nearline- pricing](https://cloud.google.com/storage/pricing#nearline-pricing)

In short, if I'm understanding correctly:

      $0.01 per GB per month storage
      $0.01 per GB retrieval
      Normal egress fees on top, so additional ~$0.10 per GB if you want to retrieve outside of Google Cloud.
      Early deletion fee. Effectively just a minimum storage charge of 1 month.

It seems this is cheaper than Glacier and quite a bit simpler. The speed restrictions are interesting though.

<comment><USER3> Unless Glacier just changed their pricing, how is it cheaper and simpler? Glacier is also $.01/GB storage, but only $.09/GB retrieval, which is cheaper than google's $.12/GB. Glacier also comes with 1st GB retrieval free/mo.

<comment><USER4> Glacier pricing is surprisingly complicated, and the actual cost can be much higher than $0.01 per GB-month if you don't read the fine print.

The biggest gotcha is that you can only access 0.17% of the data you've stored in any given day without extra charges. So if you've stored 1000 GB, you can only access 1.7 GB per day for free.

The cost for going over your daily "retrieval allowance" can be large, because cost is driven by "peak retrieval rate". They find the highest hourly retrieval rate over the entire month, and charge the whole month's retrieval at the peak rate.

This can get expensive fast. Again, with 1000 GB stored, if you retrieve 200 GB over the course of 4 hours, your hourly retrieval rate is 200 GB / 4 hours = 50 GB per hour. Your free retrieval allowance is 1.7 GB per day / 24 hours = 0.07 GB / hour. So your excess retrieval is 49.93 GB per hour.

Amazingly, that 49.93 GB per hour for 4 hours is charged for all 720 hours in the month, so that's 49.93 * 720 * 0.01 = $359.49.

That's an astonishing $1.79 per GB just to retrieve one-fifth of the data from Glacier storage.

So Glacier only makes sense for true cold storage, where you are very unlikely to touch more than 0.00007 of it in any given hour.

Google's Nearline is much simpler and faster for the same storage cost -- and far cheaper if you need to access more than a tiny fraction of the data.

Source: [http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...](http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retrieve_for_free)

<comment><USER5> Thanks for the detailed breakdown, that's really interesting! Sounds like you were bitten by this? :)

<comment><USER6> Don't know about the OP, but we were bitten by this - badly - after our logic to throttle Glacier retrieval failed.

AWS has now added the ability to set a spending limit to avoid runaway retrievals. Nonetheless, from my point of view the ultra-complicated pricing scheme is nothing short of a disaster for Glacier as a product. I think it will continue to seriously impact its uptake, and Google is smart to exploit the opportunity to offer a simple, straightforward pricing scheme.

<comment><USER7> The use case is people who have a requirement to store data and _probably_ never need to access it, either for reg requirements, or legal requirements, etc. In which case paying a couple of hundred bucks for retrieval is fine, especially if they can access at a high level of granularity

edit: Also as others have pointed out, Nearline is limited in retrieval time as well, so the cost difference isn't nearly as large.

</comment></comment></comment></comment></comment></comment><comment><USER8> Quote: "This is a Beta release of Nearline Storage. This feature is not covered by any SLA or deprecation policy and may be subject to backward- incompatible changes."

So should I believe in Google's good-will? I would be fine trying out some services, which are in Google Beta. But my valuable data? They should have a SLA right from the start to gain the user's trust.

<comment><USER9> If we place too many restrictions on how companies should offer preview releases, they'll just stop entirely. Are you suggesting that even those comfortable taking on risk to get a sneak peak should be forced to wait for general availability?

<comment><USER10> Nobody is placing restrictions, but it's understandable if people are skeptical.

<comment><USER11> Praise be the skeptical! It allows the courageous easy advantages. SLAs are almost worthless. And, really, has Google ever retired something like this (i.e., not an acquisition, not Google Reader, etc)?

<comment><USER11> ReplyTo aros: The two main criteria I would have are: 1) not an acquisition and 2) important development/developer service. I can't think of any and didn't see any after a cursory look at the list.

I think SLAs are literally worthless since I don't think they encourage even slightly more effort in minimizing whatever issues the customer is concerned about. No one wants servers to go down, no one wants to shut down a service. That a few Google bucks might be on the line would have zero impact.

</comment></comment></comment></comment></comment><comment><USER12> I added GCS nearline to my object storage comparison:

[http://gaul.org/object-store-comparison/](http://gaul.org/object-store- comparison/)

<comment><USER13> you are missing runabove edit: also missing constant.com

<comment> pi-rat Didn't know they offered object storage, thanks. $0,01/GB/month - for online content - replicated 3 times.. Wow

Any experience with them? how is their stability and speed?

<comment><USER13> No I haven't used them. But their parent-company(ohv) has the ~lowest prices you can find.

</comment></comment></comment></comment><comment><USER14> 1c/GB/mo for data stored, ~3 seconds response times (cf. tape/Glacier at multiple hours), 11 9s durability, same API as Google Cloud Storage online.

(edit: add /month)

<comment><USER15> There's no mention of durability numbers anywhere in the docs. S3 and Glacier mention 11 9s durability in their FAQ though.

<comment><USER16> Google Cloud Storage Nearline offers the same durability as the Standard- priced storage offering. See the first paragraph of [https://cloud.google.com/storage/docs/nearline- storage](https://cloud.google.com/storage/docs/nearline-storage)

</comment></comment></comment><comment><USER17> Isn't the price a little high? $10/TB/mo is the same as Dropbox, and this has a lot fewer features than Dropbox.

<comment><USER18> It's a bit apples to oranges comparison. Dropbox is a consumer product, like Google Drive. Nearline is targeted to app developers and enterprises

<comment><USER17> Exactly, so why would the lower-level service aimed at big slow transfers be more expensive than a full-featured product?

<comment><USER19> There's a few reasons:

* Dropbox doesn't guarantee 11 9s like this service does - when I'm backing critical data up, I want to make sure it's _there_. * Dropbox likely wouldn't take kindly to me storing 10PB, whereas that is what this service is designed for. * I've got SLAs and guaranteed speeds with this, Dropbox isn't designed for me to suddenly download 10PB very quickly.

<comment><USER20> Where are you seeing these durability claims? I can't find them. And what would 12 9s durability even mean? You lose one byte out of every TB?

<comment><USER19> It's 11, not 12, I misspoke.

Those claims are from glacier: [http://aws.amazon.com/glacier/faqs/](http://aws.amazon.com/glacier/faqs/) which is priced the same per GB of data.

In this case, the durability refers to the loss of data/objects stored per year - if you're sending multiple PBs of data off to Glacier, you want to be able to retrieve them many years later. Even 5 9s would mean that 1 object out of 100,000 is lost every year, which is quite poor.

</comment></comment></comment></comment></comment></comment><comment><USER21> This is a nail in the coffin of the SaaS backup industry (Backblaze, RescuePlan, etc.).

With storage becoming essentially free the last excuse for _not_ using self- hosted, secure backup tools like Arq[1] disappears.

[1] [http://www.haystacksoftware.com/arq/](http://www.haystacksoftware.com/arq/)

<comment><USER22> I'm adding support for creating nearline buckets in Arq right now.

<comment><USER23> This is exactly what I want for Linux (headless), but I haven't been able to find anything like it. The closest thing I've found is attic, which, while amazing, requires a daemon to be running on the remote end (which is what makes checking, deduplication and deltas fast).

Does anyone know an Arq alternative for linux?

<comment><USER24> duplicity ...

[https://en.wikipedia.org/wiki/Duplicity_(software)](https://en.wikipedia.org/wiki/Duplicity_\(software\))

<comment><USER23> Unfortunately duplicity is very inefficient, it needs to reupload the full set every so often, which takes up a _lot_ of bandwidth. I guess attic is my best bet in the foreseeable future.

<comment><USER25> Haven't used it, but Duplicati is another one: [http://www.duplicati.com/](http://www.duplicati.com/)

<comment><USER23> Looks very interesting, thanks! I'm investigating it now.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> So is this a blatant dig at Amazon's Glacier? "Fast performance... unlike competitors"

As they generally do tit for tat on price wars, it will be interesting to see what Amazon responds with.

<comment><USER27> > So is this a blatant dig at Amazon's Glacier?

Glacier is the other big offering from a comparable cloud vendor in this space, so, yeah. The performance claim is directed largely at Glacier, as is the consistent access one.

</comment></comment><comment><USER28> So how is this implemented? Spun-down disk drives?

<comment><USER29> My guess would be disk drives that are otherwise used for I/O intensive purposes, and where there's tons of unused/wasted space.

4MB/s is only around 4% of the bandwidth of a modern 1TB hard disk.

<comment><USER18> You can store 1 Pb and get 4000 MB/s

<comment><USER15> And at 3MB/s per TB it will always take over 3 days to retrieve all of your data. The 2-5 second retrieval latency is irrelevant when you're going to be waiting for 3 days...

<comment><USER30> FWIW it's nominally 4MB/s/TB:

> You should expect 4 MB/s of throughput per TB of data stored as Nearline > Storage.

But still, you are correct in that it will take about 1TB / 4MB/s = 2.9d to retrieve all your data.

If you need the ability to do a restore faster than this, then you need to pay more for storage, is all. For many of us, waiting 3d to recover from a catastrophic failure isn't a big deal.

</comment></comment></comment></comment></comment><comment><USER31> I own a big wholesale telco that does tons of data center business and bandwidth. Of course, margins are super thin so we need great pricing. We are no Google, but we can achieve the same pricing, including the cost of bandwidth (that is 9-9s of durability and no spin-up time, which is where the 3-seconds come from)

Maybe I should start my own service to harness all this infrastructure with something like swift!

</comment><comment><USER18> from the documentation: "You should expect 4 MB/s of throughput per TB of data stored as Nearline Storage. This throughput scales linearly with increased storage consumption."

<comment><USER15> That would mean that if you had about 1TB stored it would take more than 3 days to retrieve it (with an initial 3 second delay before it starts).

<comment><USER32> And since bandwidth scales linearly with used storage it would presumably always take more than 3 days to retrieve all of your data.

</comment></comment></comment><comment><USER18> The same egress and data transfer charges as for Google Cloud Storage Standard / DRA. Can transfer the data from Standard to Nearline bucket for free until Jun, 11 - [https://cloud.google.com/storage/pricing#network- regions](https://cloud.google.com/storage/pricing#network-regions)

<comment><USER33> That promotion is talking about transferring data between geographical regions, not storage classes.

</comment></comment><comment><USER34> slightly off topic here, but when i tried to setup an account, which should get me $300 free, it auto selects my country (Ireland) and then tells me:

Tax status Business

This service can only be used for business or commercial reasons. You are responsible for assessing and reporting VAT.

and then makes me enter a VAT number... well, I will stick with AWS and Azure, since neither of those "REQUIRE" a vat number or business status.

PS: i know a business name costs about EUR20 to setup, but the VAT requirement is the pain in the left one...

<comment><USER35> Where did you read about $300 for free?

</comment></comment><comment><USER36> Our story with Amazon Glacier (50TB):, we are a company with 50TB of data on multiple Servers and NAS, and some External drives, most of the data we needed to save/Archive for 10 years at least, it's starting to become very hard to manage locally, so we saw glacier a great solution for us! Well, not exactly

We tried to manually using it, like APIs, Scripts, etc… it's impossible, it was very had

So we tried cloudberry to help us upload, it will upload but also It won't work for huge data, with glacier you won't be able to search, list, find any file you want to download easily, also its not practical to manage all the backup and millions of files manually, also we got millions of photos we needed a way to find them easily like thumbs So Glacier had so many restrictions like 3-5 hours restore time, 5% restore quota hard to use even with utilizes like cloudberry, cant list, search, it's not useable as its! But the price was attractive

We considered Seagate Evault , but it was expensive and so many hidden fees and complicated for our case Then we tired another solution called Zoolz ( [http://www.zoolz.com](http://www.zoolz.com) ), Zoolz does not use your own AWS Account, but utilize their own account and from what I heard from them they got 5 Petabyte, so they got massive restore quota, around 5% of the 5 Petabyte, also its simple to use, they offer Zero restore cost, it's like Mozy or Crashplan but for business and on glacier , like they internally create thumbs for your photos and store it on S3, so you have instant preview, also you will be able to search and browse your data easily and instantly, they got Servers and Polices, and got a reasonable price, all we wanted , we got the 50TB for $12,000 / year, its more expensive than using Glacier by itself, using Glacier will cost around $6000, but its not practical for a company to use it as a standalone storage

The only disadvantage is that when/if we needed a file we have to wait 4 hours to get it, which is fair, it was faster than when we used to use Tapes :), awe tried to restore 1 TB with 1.2 million files, it took us around 10 hours to complete, which was okay

</comment><comment><USER37> Can you do rsync style differential backups with this?

<comment><USER14> gsutil (the Google Cloud Storage command-line tool) has an rsync option: [https://cloud.google.com/storage/docs/gsutil/commands/rsync](https://cloud.google.com/storage/docs/gsutil/commands/rsync)

</comment></comment><comment><USER38> Is azure has the product like this? Usually cloud war start from amazon, then google, and then azure is the last one

</comment><comment><USER39> I still rather use a Cloud Service provider like Zoolz as it would be more efficient to benefit from other plans. Amazing idea though!

</comment><comment><USER40> Great to finally see a cloud service competing against Glacier. Hopefully this will lead to some sort of response from AWS.

</comment><comment><USER41> I wonder what the underlying hardware is?

</comment><comment><USER42> Does anyone know of any online-storage offering that allows paying in bitcoin and allows API access? Thanks

<comment><USER32> I'm sure there are various FTP storage providers that are accepting Bitcoin.

</comment></comment><comment><USER43> "nearline, still online, but we could make a graph that is supposed to make you believe its not"

</comment><comment><USER44> The transport for this service is HTTP? I imagine most competitors use that as well, right? How does encoding factor in here? I have to transcode my data to base64 in order to put it in or take it out, I assume? I "know" I'd only get billed for the data stored as the original octet/binary encoding. But what about the egress fees? Encoded or decoded data is the input for the billing?

<comment><USER45> HTTP can transfer arbitrary binary data just fine. There is no need to encode it.

I don't know which they use but HTTP traditionally uses a `Content Length` or an empty chunk to represent the end of data.

</comment></comment><comment><USER46> Some of these Google/Amazon/Apple "Introducing Product X..." stories are starting to strike me as intellectually devoid, creeping into "free advertising" territory.

This particular story is about a Google service that is an extremely late entrant into the market (Amazon Glacier, etc) and offers almost nothing new. I'm not saying it's a bad product or not useful to folks, but it's not, by any means, groundbreaking. I'd much rather see the top spot of HN occupied by some startup's new idea or a researcher's new findings.

I also don't mean to imply that "big corporate" == bad. Certain products -- self driving cars, Space X automated landings, etc -- are absolutely worthy of our attention and discussion. I just hope that people would think twice before upvoting a story merely because it's from Google.

<comment><USER47> I suggest you read, for example, this article from TC:

[http://techcrunch.com/2015/03/11/google-launches-cloud- stora...](http://techcrunch.com/2015/03/11/google-launches-cloud-storage- nearline-a-low-cost-storage-service-for-cold-data/)

This information should challenge your argument that Nearline is not groundbreaking, at least in the "cloud archival storage" space.</comment></comment>


One Packet and it all Falls Down (Youtube) - <AUTHOR> http://www.youtube.com/watch?v=ydo945gSnms

<comment><USER2> This is a dupe of <[http://news.ycombinator.com/item?id=1038701>](http://news.ycombinator.com/item?id=1038701>).

</comment><comment><USER3> Wow, I hope Juniper can fix that soon.

<comment><USER4> apparently, they did:

 _In short, we fixed this particular problem about 350 days ago._

[http://forums.theregister.co.uk/forum/1/2010/01/07/juniper_c...](http://forums.theregister.co.uk/forum/1/2010/01/07/juniper_critical_router_bug/)

<comment><USER5> The message now being "upgrade your damn hardware('s firmware)". Security / stability upgrades are made _for a reason_. Use them, or possibly suffer the consequences.

<comment><USER6> Easier said than done. In the ideal network, sure, you'd have extra network hardware to test the update on, to make sure it didn't break your configuration. You might also have failover network equipment so that the network would stay up while you upgraded it in pieces.

Of course even with few resources, you should still strive to make updates, maybe during the middle of the night (hope your customers don't need to access your network until morning), and be prepared to roll back if things don't go according to plan.

That is, if you even knew there was an update. Unlike Windows, your router isn't going to keep popping up little bubbles to tell you to update.

So yes, it is your responsibility as an IT admin to keep the network secure, but there are still a lot of obstacles that means that overloaded admins will forget or procrastinate. I don't know much about the vuln, but it appears to affect telnet. If that was the only thing patched in that update (or I didn't care about the other features that were being patched at the same time), I would just make sure telnet was closed and leave it be.

EDIT: Ok, looked more into the vuln, apparently any open ports make it vulnerable, as it is a problem with how they handled the tcp headers. Only reasonable solution here is to patch.

<comment><USER5> "That is, if you even knew there was an update. Unlike Windows, your router isn't going to keep popping up little bubbles to tell you to update."

And if your company values its security at all, they should be certain that their IT people are checking for such things. They know the hardware, so they know the websites where exploits / updates are listed. Have a folder of bookmarks, and check it once a week, and you wouldn't be caught off-guard by (relatively) ancient exploits. The alternative is equivalent to a company using the oldest version of XP, expecting there to be no security problems because they aren't _looking_ , and don't want to risk updating to a more secure system. Bring on the viruses / script kiddies!

I'll agree that upgrades can cause troubles, sometimes extreme, but they should be _expected_ at some point in the future. Expecting otherwise is expecting your hardware / software to be eternally flawless, which is ludicrous. Even something as simple as a light switch sometimes needs to be repaired / replaced, and that's just a physical switch, almost literally as simple as it can be. If you continued using a light switch that sparked and smoked every time you switched it _for a year_ , and your house burns down, you really only have yourself to blame. Sure, replacing it means calling in an electrician, or doing it yourself, both of which bring their own dangers, but when the alternative is relatively _likely_ catastrophic failure eventually, it's probably worth it.</comment></comment></comment></comment></comment>


Straight and curly quotes: Always use curly quotes - <AUTHOR> https://typographyforlawyers.com/straight-and-curly-quotes.html <comment><USER2> This is interesting, but I don't see any explanation for why this is helpful. Obviously, the author feels that curly quotes are better typographically, but acknowledges that straight quotes are used in legal documents. They never explicitly lay out any reasoning why one should pay attention to this, but instead have more of a how-to guide. Is there any direct reason for this, aside from the author's particular sense of aesthetics?</comment>


Stirling Charger V 2.0 – Charge your phone with a candle - <AUTHOR> http://interestingengineering.com/stirling-charger-v-2-0-charge-your-phone-with-a-candle/

<comment><USER2> "This kind of engine can produce energy with a low temperature difference without emissions"

Candles emit.

</comment><comment><AUTHOR> This is a Kickstarter I'd back in a heartbeat.</comment>


Android Privacy Guide – Take Control of Your Data - <AUTHOR> https://gitlab.com/Attedz/AndroidPrivacyGuide/blob/master/Android_Privacy_Guide.md <comment><AUTHOR> F-droid,yalp store as app stores;wire, signal, conversations, qksms as messenger clients;k-9 mail, proton mail as email providers;next cloud, syncthing as cloud services;bitwarden as password store; Joplin as Evernote alternative.</comment>


56 Things from One More Thing - insights for app devs - <AUTHOR> http://www.bonobolabs.com/56-things-from-one-more-thing/

<comment><USER2> Anyone know anything about DMCA Guardian? I've never heard of them. I definitely can understand how a company like this could be useful, but I'd worry about the means they use for enforcement.</comment>


Anonymous' New Walkie Talkies Use Radio Waves to Access the Internet - <AUTHOR> http://motherboard.vice.com/read/anonymous-new-walkie-talkies-use-radio-waves-to-access-the-internet

<comment><USER2> > The hacktivist group Anonymous is working on a new communication tool to > circumvent censorship and set information free, and it's going low-tech this > time. The project is called Airchat, and it will use radio waves instead of > wifi, broadband, or phone lines to communicate data and messages between > computers. It's basically pirate radio for ones and zeros.

I wish people who write these things would reveal the history behind the methods. This technology was pioneered by ham radio operators decades ago, pre-Internet, using what's now called "packet radio". In fact, I used my own prototype packet scheme to stay in touch with home while sailing around the world -- two ham radio transceivers, separated by as much as 10,000 miles of empty water, exchanging packets.

More here: [http://arachnoid.com/sailbook](http://arachnoid.com/sailbook)

> In its current incarnation, the project uses Fldigi software to communicate > data—it's the software commonly used to broadcast amateur radio stations > from a computer.

Okay, one oblique mention of ham radio.

<comment><USER3> Indeed. Before I clicked over to the article, I had a gut feeling...

The first time I ever connected to "the Internet" was over a packet radio link from my home, in a very rural area, through a digipeater at a (relatively) nearby ham radio club and then to the nearest university. I had "borrowed" a login and was able to telnet into one of their UNIX servers and send/receive mail, use archie, gopher, etc.

Of course, I don't really miss that 1200 baud extremely high latency connection but the convers servers were really cool.</comment></comment>


Halloween party-yacht in bay area: "Largest crowdfunded party to date" - <AUTHOR> http://venturebeat.com/2012/11/07/crowdtilt-funds-pipe-dreams-into-existence/

<comment><USER2> The thing that excites me most (as a realist with a soft spot for anarchy) is the potential for crowdfunding to deliver public goods.

One example is a pitch I saw at DjangoCon, someone was running a kickstarter campaign for a detailed open-source tutorial for some software (if not a public good, open-source is definitely a commons). I just saw another on crowdtilt's [testimonials page](<https://www.crowdtilt.com/campaigns>). Apparently some enterprising Dallas residents have come up with a [direct, real-world response](<http://i.imgur.com/1LI5U.png>) to the ever-present whine, "but who will build the roads!".

</comment><comment><USER3> There are a lot of industries where crowdfunding makes the most sense as a payment mechanism. There are always ads in college dorms for package deals on spring break trips to Florida or ski weekends - they will be so much easier to make happen with crowdfunding.

As the activation energy for getting folks together to do fun stuff gets lowered, I'm excited to see what event planning companies start coming up with. Surely there are new party/travel/event offerings that now make business sense thanks to crowdfunding.

<comment><USER4> and it's natural behavior too. doing stuff as as group requires making sure everybody is financially committed. I think that's the whole value of not collecting the payment until everybody paid/committed.

That's where technology trumps real life. if you collect cash from people, you have to return it.

</comment></comment><comment><USER5> For those of you that missed it, I took some video on my iPhone: <http://youtu.be/1Rl1TJG17Wk?t=35s>

Way to go Crowdtilt! Looking forward to the next one!

</comment><comment><USER6> way to go James and co!!!! (plus damn great party)

</comment><comment><USER4> really shows what happens when you reduce the friction of collecting money from a lot of people. can't wait to see more of this.

</comment><comment><USER7> Way to go guys! Keep killing it :D

</comment><comment><USER8> so pissed I missed it</comment>


Ask HN: What are some good educational resources for machine learning? - <AUTHOR> So for some background, I'm a junior software engineer for a big, old household-name company.  I have a recent Bachelor's in Computer Engineering (basically electrical engineering/computer science hybrid, focusing on computer hardware and OS/embedded level software development), and most of my current work is classic c++/java development with some python, perl and SQL here and there.
Recently my small corner of the company has taken an interest in machine learning and other modern data analytics techniques, and is planning some informal programs to develop/brainstorm ideas for potential applications of said techniques to our line of work.  Given that I've always been interested in such things in the abstract, I'd like to jump aboard.  However, aside from a passing familiarity with some of the high-level concepts (aka reading Wikipedia when I'm bored) I have zero direct experience with machine learning and data analytics in general.  So I'm looking to learn as quickly as practical in my spare time.
I started to search through past HN comment threads for resources, then figured it might be quicker if I just asked. :)
Given the circumstances, if possible I'd like resources that involve applications.  Theory is fun, and over the medium-term I will enthusiastically dive deep, but my immediate goal is to have the capacity to deliver something practical; that's dictated by the company's timeline.
Thanks for your time! <comment><USER2> Andrew Ng's machine learning course on coursera is great for beginners. It'll show you some theory backed by intuition, and if you do the assignments you'll implement ML algorithms yourself (in Matlab).

After that, I recommend Stanford's CS231n. It's focused on computer vision tasks, but you can learn a lot about neural nets in general from it. The videos were recorded around a year ago I think, so it's up to date with the latest deep learning techniques and libraries. For some reason the videos were taken down from the CS231n site but you can watch them here: [https://www.youtube.com/watch?v=g-PvXUjD6qg&list=PLlJy- eBtNF...](https://www.youtube.com/watch?v=g-PvXUjD6qg&list=PLlJy- eBtNFt6EuMxFYRiNRS07MCWN5UIA)</comment>


Show HN: Free genetic reports from your genome - <AUTHOR> https://meports.com <comment><USER2> Uh...I'm definitely not sharing my genome data with some guy named Ethan at unnamed company.

[https://meports.com/privacy](https://meports.com/privacy)

<comment><AUTHOR> Valuable comment. How would we convince you that the genome file is not stored beyond the report generation process?

<comment><USER3> Honestly, short of me building and running the code on my personal machine, I don't think there's any way I'd trust you on your word.

</comment></comment></comment><comment><USER4> I don't understand why you have your privacy policy as it is right now, at all.

 _Privacy is important to us. We want to do it right. (...) At no time is your DNA data shared - or sold - to any external party, period. (...) If a genome is uploaded, but the user does not continue and generate a report, the uploaded genome is automatically deleted immediately. As soon as a report has been generated (no more than 10 minutes) the uploaded genome is deleted._

Wow! A company that isn't out to steal your data! Great!

 _When using Meports you are uploading your genome to our central website for analysis. (...) By uploading your genome you grant us a temporary, limited, revocable, royalty-free, world-wide license to process and use your genome for the purpose of providing you with the service._

 _" Vision: We believe in using data and software in order to maximize everybody's quality of life. "_

(Actually - you even have a different company "vision" on gene.meports.com. Which one is it?)

So what does this mean then? Are you using the word "analysis" to trick people into thinking you don't store data derived from the genome on your servers, but you're not storing the literal file someone is uploading? While, at the same time, reassuring clients you'll never store or sell their data?

<comment><USER5> Not sure I understand your concern: It's standard boilerplate for the case where private data is required to run an analysis, but is not otherwise used.

Not saying that it should necessarily be trusted but the wording isn't problematic. What _is_ problematic is the complete lack of legal security.

<comment><USER2> Definitely not boilerplate. The privacy policy is quite light, doesn't define terms like "external parties", doesn't even have the company's name or origin.

<comment><USER5> I was talking specifically in the context of OP's comment. I otherwise agree with you. But the wording that OP seems to complain about simply means that, in order to perform the requested analysis, the company needs to temporarily store your data.

<comment><AUTHOR> I don't see any other way to do it haha

</comment></comment></comment></comment></comment><comment><USER6> [https://gene.meports.com/report/2347380d-ac31-4f4a-b973-6e18...](https://gene.meports.com/report/2347380d-ac31-4f4a-b973-6e18ff4c05a3)

sample report lists GBA under Gene column. I click link, get sent to wikipedia entry for Game Boy Advanced...

<comment><AUTHOR> Hahahaha, looks like the disambiguated Wiki page is more relevant:

[https://en.wikipedia.org/wiki/GBA_(disambiguation)](https://en.wikipedia.org/wiki/GBA_\(disambiguation\))

and the gene:

[https://en.wikipedia.org/wiki/Glucocerebrosidase](https://en.wikipedia.org/wiki/Glucocerebrosidase)

</comment></comment><comment><USER7> I don't even see a name of the creator on this site. Personally, I won't ever submit personal genome data to an anonymous site like this, and no one else should either. Aren't there also legal implications to hosting personal health data online like this? I understand they claim to delete the data after the processing, but can we really trust an opaque process like this?

<comment><AUTHOR> My name's Ethan, you can email me at email@example.com and my personal website is ethansteininger.com. All open !

What aspects of transparency would you be looking to learn more about in order to grant more trust?

<comment><USER6> I think the biggest concern that people might see is how are you making money. With promethease, we give them a small amount of money. People are going to assume that the only way to generate income on this project is the selling of user data or keeping it to build a database to use for analysis and then selling that data.

This site could use a bit more information about how the analysis is being done and what benefits it offers over similar sites.

<comment><AUTHOR> Great feedback, I appreciate it. Our office's page is how we make money. Offices embed the button on their webpage and receive your report upon which they're able to prescribe certain preventative plans.

</comment></comment></comment></comment><comment><USER8> Looking at the OP history, they only ever posted content related to meports.com.

Judging by the titles of the submissions and on content archived by the wayback machine, meports was originally intended to be a "life dashboard" (mood, workouts, etc..) and pivoted approximately a year ago towards its current form, a meta analysis of genome.

<comment><AUTHOR> Yep, it was a pivot. Is there something wrong with that?

<comment><USER9> No judging here. I'm curious why you pivoted away from health tracking?

<comment><AUTHOR> I wanted to do something that had larger humanity potential. IMO human genetics is a top 3.

</comment></comment></comment></comment><comment><USER10> No indication of how this site makes money, nor who they are. I would be _very_ suspicious.

<comment><AUTHOR> Fair enough, the offices page (custom report generation for offices) is our revenue model: [https://meports.com/office](https://meports.com/office)

<comment><USER11> Does this mean, you don't store or distribute my DNA, but the risks you calculated from it?

<comment><AUTHOR> The risks are stored, but deleted after 48 hours.

</comment></comment></comment></comment><comment><USER12> Who's running this? What country are they based in? Why should I trust meports.com not to misuse my genome?

<comment><AUTHOR> I'm running this - based in US. That's a great question and I'm open to ideas. How can I prove that the genome file isn't stored beyond report generation?

<comment><USER12> Hi, thanks for replying! I think the only way would be to make this a client- side application somehow. This is extremely sensitive data, so the only way I can trust that you'll never misuse it is if I never give it to you in the first place.

</comment></comment></comment><comment><USER13> This looks like it might need FDA approval.

<comment><AUTHOR> How is it any different than Promethease from an FDA perspective?

<comment><USER14> Tangentially, what does this offer that Promethease doesn't? I ask because I've run genomic data through Promethease and I'm wondering if running through your service would be redundant.

<comment><AUTHOR> Yeah it would be redundant. the call to action once you generate a report is to send it to your provider.

</comment></comment></comment></comment><comment><USER11> Is this safe?

<comment><AUTHOR> Define safe haha. The genome doesn't stay on our servers for more than 5 mins.

</comment></comment><comment><USER15> can't get it to use any data sets that i have in the normal .vcf formats, etc

<comment><AUTHOR> What testing providers share your raw genome in an .vcf format?

<comment><USER16> Nobody shares your genome.

Promethease though allows for gvcfs and other formats so that you can upload more comprehensive sequencing data with full coverage.

<comment><AUTHOR> Maybe I should've rephrased... my question is which genetic testing providers give you a raw genome file in .vcf format? If none then I think we're talking about two different things.

[https://customercare.23andme.com/hc/en- us/articles/212196868...](https://customercare.23andme.com/hc/en- us/articles/212196868-Accessing-and-Downloading-Your-Raw-Data)

<comment><USER16> 23andme is casual genetics, and misses a lot of the variants that are important clinically. These ancestry sites are the entry point, but not the definitive answer.

SNP chips are on the way out and whole genome/whole exome are becoming much cheaper. For many people (myself included) who are trying to diagnose a disease or understand our personal genetics at a base-pair resolution then VCF is the standard format.

Ancestry and 23andme use proprietary formats but all independent labs/clinics/commercial operations use VCF and it is the format that patients will have. If a patient is dropping a grand or three to get their personal genome, they are going to be receiving the raw data in VCF or BAM.

As a side note, your site asks if people "had their genome sequenced" but 23andMe results aren't a genome. It is a peek at a fraction of the genome. Your site doesn't accept whole genome data.

Also, if by "offices" you mean doctors offices or any kind of medical professional, then you are putting them at risk. Variant calls delivered to patients from a medical professional need oversight. Calls need to be made in a CLIA laboratory and pass through a genetic counselor. It is fine to skirt regulations if an individual is analyzing their own genome, but once you insert a third party, then you are bring in legal issues, reporting issues, and HIPAA issues.

<comment><USER17> Your comment was very insightful. I've considered 23andme and similar services before, and it is helpful to understand that a fractional amount of one's genome is all that is provided/analyzed. I'm excited to see that whole sequencing has broken the $1000 barrier some time ago, and was available recently for $200 as a promotion! ([https://www.wired.com/story/whole-genome- sequencing-cost-200...](https://www.wired.com/story/whole-genome-sequencing- cost-200-dollars/))</comment></comment></comment></comment></comment></comment>


Lastpass Is Replacing the Native Mac App with a Web App - <AUTHOR> https://www.theverge.com/2020/1/29/21113505/lastpass-native-mac-app-store-replacement-web-safari-extension-update <comment><USER2> Title is misleading, it's still a desktop app but will probably use electron (so "technically" a web app, I suppose) but title implies a fundamental change in functionality/usability on Mac which is not happening.

<comment><USER3> According to the article the loss of system functions like hot keys. Electron apps don't feel like "real" apps on the Mac (I most frequently use the Slack desktop app and continually stumble on its un-macness).

</comment></comment><comment><USER4> just switch to bitwarden.

<comment><USER5> I recently did and it's the best thing ever. LastPass for some reason slows down my Chrome to a crawl.

<comment><USER6> Same here, it's nice having the luxury of being able to view it's source code. LastPass is clunky and always threw errors for me. Hell, even deleting my account threw an error, but it still deleted my account, but I didn't get any "Your account has been deleted dialog" or get logged out like you might expect, I was stuck in a phantom session where I was still able to add entries for some reason. That'd not how you get me to trust your service.</comment></comment></comment>


Peter Thiel's CS183: Startup - Class 19 Notes Essay—Stagnation or Singularity? - <AUTHOR> http://blakemasters.tumblr.com/post/25149261055/peter-thiels-cs183-startup-class-19-notes-essay

<comment><USER2> (1) From a dialectic point of view, I think Kurwzeil makes an impressively compelling argument that the rate of progress has been exponential, _historically_ (e.g. using others' milestones).

So it's striking that he doesn't argue for an underlying mechanism, nor for whether it can continue - or even mention it explicitly.

The mechanism seems to be similar to _standing on the shoulders of giants_ \- once an improved technology is developed, it can be used for improved search for other technologies. The search can then be faster, more efficient, can take place in new domains, with greater accuracy - whatever is the the nature of the improvement.

But there's another assumptions: that there _will_ be more to discover, and with a constant density. But why should the frequency of potential discoveries be constant, such that if you seek faster, you'll find faster? To be clear, it _does_ seem to be that way... I'm just wondering if there's an argument as to _why_ it's that way, and why it will continue to be that way as we keep searching. There's the assumption of mediocrity - that we are not at a privileged center of the universe - but is there a better argument? For example, why shouldn't it be that discoveries become exponentially rarer? So that we have to keep searching faster and faster just to maintain a linear rate of progress.

EDIT: e.g. consider primes, infinite but become less frequent as you go.

(2) I also like Hofstadner's argument that it's not necessarily true that an intelligence is sufficient to understand itself e.g. a giraffe can't understand itself. Of course, we can divide-and-conquer, and create hierarchical understandings, such that we can understand one level in itself, by assuming the concepts below, and ignoring concepts above. But not all things can be neatly composed into modular hierarchies, such that each level is easily understood - though we are biased towards seeing those that can, because that is all that we can see. In other words, perhaps we can one day duplicate a human mind... yet not understand it.

(3) There's a fascinating thought in these class notes, that people assume stagnation, and don't like to generalize beyond extrapolating single variables. This is very pragmatic, because it's almost impossible to predict with any fidelity. But it results in a very interesting effect: people are very confident as they stride the same well-trod paths, only ever taking one step away from it, so only finding big wins transitively (by hill climbing). This means that just two or three steps off the beaten track can be miraculous improvements... all you have to do is find it, though that might take an enormous number of attempts.

EDIT (4) Re: exponential progress in software (cf. Moore's Law for hardware), there are dimensions of progress that seem to be exponential: the release rate of new software; the productivity due to using others modules (SOTSOG, esp open source); "software is eating the world" as more problems are solved with software; software is being used on more devices, in more places (eg mobile devices). One could argue this is cherry picking, and none of these are equivalent to Moore's Law - but exponential improvement only requires some kind of improvement, that can itself be built upon. NOTE: I don't have figures, so I don't know for sure whether the above are actually exponential, though "eating the world" seems to be. Also found this: [http://multiverseaccordingtoben.blogspot.com.au/2011/06/is-s...](http://multiverseaccordingtoben.blogspot.com.au/2011/06/is- software-improving-exponentially.html)

<comment><USER3> 6ren: Perhaps the reason that discoveries don't seem to become exponentially rarer is that, in an infinite universe all possible probabilistic outcomes must occur. Thus, if there is a non-zero, finite chance of something being discovered it will and can be discovered in an infinite universe.

Each new discovery in such a universe will produce non-zero speculative probabilities of something new to discover -- resulting in an infinite stream of problems and solutions that present new problems.

It doesn't matter if the probabilities are speculative or eventually discovered to be real: when speculations are "proven" false, the proof of falsehood is itself a new discovery that also provokes further speculation.

I guess this would mean that technological progress is inevitable given an infinite universe where some technological progress has already occurred and where speculation or "imagination" or unguided thought exist.

</comment></comment><comment><USER4> Hi - this is omri (founder of genomecompiler) - our kickstarter project is still awaiting amazon payments authorization and kickstarter authorization for the project. See <http://www.youtube.com/watch?v=BLhU1RGTHN4> and <http://www.youtube.com/watch?v=F8qcDQaY8Mw> for details

</comment><comment><USER4> Cool, they talked about our genomecompiler (<http://genomecompiler.com>) and the kickstarter project for glowing plants we're in the process of posting.

firstname.lastname@example.com

</comment><comment><USER5> I know that de Grey has a rough outline to reach his "Methuselahrity", but does anyone know of a reference to other milestones needed to reach the singularity's state of radical abundance.

The notes indicate that they discussed the idea of milestones to the singularity, but I was wondering if anyone has sat down and really thought it through on how to get there.

Thanks!

</comment><comment><USER6> Does anyone know the link to the Kickstarter project "that involves taking an oak tree and splicing firefly genes into it"?

</comment><comment><USER7> Can anyone link me to the kickstarter project they mentioned?</comment>


BoringTun, a Userspace WireGuard Implementation in Rust - <AUTHOR> https://blog.cloudflare.com/boringtun-userspace-wireguard-rust/ <comment><USER2> Jason also sent an email to the WireGuard mailing list addressing this. [1] In the email, he stated that Cloudflare repeated refused to work directly with him and his team as part of the upstream project.

[1] [https://lists.zx2c4.com/pipermail/wireguard/2019-March/00404...](https://lists.zx2c4.com/pipermail/wireguard/2019-March/004048.html)

<comment><USER3> One of CloudFlare's goals is to have a BSD-Licensed result. Seems reasonable to maintain isolation from the core WireGuard team that is publishing under GPL to support that goal, just to ensure they don't accidentally end up with GPL code in their codebase.

<comment><USER4> With the exception of the kernel-specific code, we're actually pushing more and more of our stuff as MIT/Apache2 licensed. If there's a project we haven't done that to yet that's not permissively licensed, just email team at wireguard dot com or ask on the mailing list. wireguard-go, for example, is MIT.

<comment><USER5> It seems that the official rust implementation wireguard-rs[1] is currently GPL licensed.

[1]: [https://github.com/WireGuard/wireguard- rs](https://github.com/WireGuard/wireguard-rs)

</comment></comment></comment></comment><comment><USER6> Nice to see more userspace implementations of wireguard.

I've been testing out wireguard's linux kernel module, but experienced too much jitter due to wireguard not respecting isolcpus.

Probably possible to make the kernel module behave, but working with userspace is so much easier. Wireguard-go seems to behave (the drop in performance is not necessarily a showstopper), although the description says that it shouldn't be used on Linux :-)

Would be nice if a high performance rust implementation can become a real alternative. A userspace implementation shouldn't need to be much slower than a kernel module if network card with kernel bypass is available, if you really need that extra performance.

</comment><comment><USER7> Unfortunate that Cloudflare didn't just contribute time to the first party wireguard-rs client. But it's their wish to do what they want with their engineers.

Nonetheless, it's great that Wireguard have a path to a solid user space implementation that others could build from.

Jason Donenfeld's post about this surprised me when he said the go implementation is flaky. Is the health of the non-kernel parts of wireguard not great? The macOS clients work really well for me.

<comment><USER4> The Go implementation is fine. I don't think you have much to worry about.

But we're pretty frequently running into limitations and fighting with the runtime; we've submitted quite a few random patches to it. I think Rust is probably a better-suited language for a high performance and relatively secure WireGuard implementation. Hence, we'd really like to have a strong first-party Rust implementation for the project.

<comment><USER8> Haven't looked at the other implementations yet but with wireguard-go I've been struggling to run it as non root on OpenBSD.

I've created a _wireguard user+group, setup a /dev/tun0 interface to have the _wireguard group with read/write however from the little tracing i've done it looks like I still need root when it tries to do the unix sys call to set the MTU.

</comment></comment></comment><comment><USER9> OpenVPN is a piece of shit performance wise, the best userspace VPN tool I tested was curvetun. It was moving encrypted gigabit speeds with very low CPU usage on MIPS openwrt routers.

[http://netsniff-ng.org/](http://netsniff-ng.org/)

</comment><comment><USER10> Looks like pretty good code, but interesting that they only have a few dependencies and decided to implement a lot of stuff, like kernel polling, themselves rather than using the mio crate. I wonder why.

They also implemented the noise handshake in a non-generic way tightly coupled to wireguard protocol, again written themselves instead of using an external dependency, which I think is also the approach other wireguard implementations have taken.. which is fine, but harder to independently verify. There's a lot of code there.

<comment><USER11> I have been getting the mio question a lot.

Mio, in theory, is not very far off of what I ended up implementing, but it is missing a few features that I really needed, and building those features on top of Mio would be harder than just using epoll/kqueue directly. Also I don't understand the fear people have of using epoll/kqueue. Those are very basic features that every engineer has to know how to use.

So what are the features:

No locks - Mio polls under lock, but epoll/kqueue are multithread safe by nature as long as you use the correct filters. Mio has considerable overhead over pure epoll/kqueue.

No need for tokens, instead I store closure pointers. Real zero overhead. Of course I could cast pointers to usize and use as Mio tokens, but I would still have to build the ownership model on top, which is most of the work anyway.

Timers in Mio run in a thread??? I use platform idiomatic conventions instead, which with kqueue require direct access to kqueue.

Signal handlers, can't make idiomatic one with Mio (or maybe I missed how).

Same with notifications. Could use something awkward like a channel, but prefer idiomatic approach.

</comment></comment><comment><USER12> Nice, Rust is a great fit for this kind of project.

</comment><comment><USER13> What does the context switching overhead look like in this implementation, vs kernel space? Are there any throughput graphs comparing the two?

<comment><USER11> It is still quite expensive, the kernel module is still faster. Performance depends a lot on your kernel version and network card. We measured anywhere from 20% to 40% advantage to kernel module. But we are working on additional performance features, such as using send/recvmmsg and raw sockets.

<comment><USER13> Thankyou, that is even better than what I would expect. I have been trying to tune around similar issues in tinc-vpn which also runs in user space and depends on newer kernels and code changes to work around.

</comment></comment></comment><comment><USER14> Blog post mentions windows support but there is no real code support for it. Will it eventually be ported?

<comment><USER11> Currently the library part supports Windows, and it can be used from C# or C++, if you want to use something like [https://docs.microsoft.com/en- us/uwp/api/Windows.Networking....](https://docs.microsoft.com/en- us/uwp/api/Windows.Networking.Vpn)

However there is no native support for tunnel interfaces in Windows, and we are still thinking about the best way to add full support.

The WireGuard project for example developed a whole new driver just for that. Don't know if we are going to follow that route.

<comment><USER14> They did and the code is surprisingly short at only ~1300 lines of C. Why not just use their driver?

</comment></comment></comment><comment><USER15> I've been interested in using WireGuard for a while, though I don't really have a use-case for it currently.

In the interest of honest understanding, what would be reasons _not_ to use WireGuard over existing VPN software (excluding the obvious legacy interop) ?

<comment><USER7> I am a huge fan of Wireguard and use on all the major production servers I currently manage.

But user management is exactly as sophisticated as SSH key management. That is to say, easy for 2-3 people, a pain otherwise.

There are attempts at creating UI based user managers for wireguard, and I hope those go somewhere.

I bet the major wireguard-supporting VPN providers have solved this problem for themselves though.

Update: Just wanted to add that I don't think that user management (the work of actually provisioning and managing users) should fall under the umbrella of Wireguard. It's perfect the way it is. But I suspect there will be an ecosystem of tools that come up around it.

<comment><USER15> Thanks, this is exactly the kind of useful information I was looking for.

(this wouldn't be an issue for my personal use, but it's useful to be aware of it)

<comment><USER16> It's not an issue for like 95% of the users, which is why Wireguard went that way. For the people who really need something better they are left to figure it out for themselves. Normally this is OK, but sometimes it leads to situations like PGP Email where key distribution is never really solved and the whole thing kind of lumbers around in a half dead state forever.

</comment></comment></comment></comment><comment><USER17> is really wireguard that better than other VPNs? I always wondered if there is need for a "better" vpn.

I mean a more reliable VPN is probably needed, i.e. something that isn't detectable. However I guess wireguard has all the problems than most VPNs once it's known how it works it's detectable with a DPI. What I probably want to see is a VPN over h2c or over quick, its probably impossible to detect it, it looks like regular web traffic.

<comment><USER18> Wireguard focuses on being a better vpn where better is vaguely defined as using modern crypto in a way that is hard to get wrong. As a result, it is a lot simpler with far fewer bells and whistles. This means it tends to be orders of magnitude faster than an equivalent usage of say openvpn (I measured 15mbps vs 52mbps awhile back with openvpn and wireguard respectively).

What you want is a slightly different albeit very important problem to solve reliably.

</comment></comment><comment><USER9> Any Dockerfile?

<comment><USER19> Open Issue [https://github.com/cloudflare/boringtun/issues/38](https://github.com/cloudflare/boringtun/issues/38)

</comment></comment><comment><USER20> Is the name resemblance with GloryTun (another VPN) just pure coincidence?

<comment><USER19> According to the blog post:

"The name might sound a bit boring but there's a reason for it: BoringTun creates a tunnel by 'boring' it. And it's a nod to Google's BoringSSL which strips the complexity out of OpenSSL. We think WireGuard has the opportunity to do the same for legacy VPN protocols like OpenVPN. And we hope BoringTun can be a valuable tool as part of that ecosystem."

so coincidental.</comment></comment>


Vaccines Could Be Emailed and Printed at Home - <AUTHOR> http://mashable.com/2012/10/19/vaccine-email-print-at-home/

<comment><USER2> This is an amazing idea, but the economics really don't check out as stated. Home printers caught on because for a brief flash of history, people needed to bridge the gap between digital and paper based systems often enough to warrant owning a device for that specific purpose. People buy cars because they need a flexible transportation option that they can use every single day. Even though these investments are expensive, the benefits outweigh(ed) the costs. Nowadays, it's not uncommon for your situation to be that printing things is a rare need, and on those rare occasions you can just go to a place like FedEx. As usage becomes more occasional, the need for decentralization drops.

And that is the case with a vaccine printer. This is something you would use maybe once or twice a year, and then it would just sit there. For a piece of equipment that will be expensive initially and then probably require regular maintenance, that simply doesn't make sense.

But distribution and availability of vaccines _are_ problems, and this technology could be a solution to them. You wouldn't have the printer in your home, though. That's crazy talk. Instead, your doctor or clinic would have the printer. Instead of paying based on the scarcity of the vaccine and the difficulties of shipping them around the country, you'd be paying for the licensing of the vaccine and the maintenance of the printer (which makes way more sense if spread across hundreds of patients). Assuming old business models don't die too hard, this would mean higher availability and lower costs.

<comment><USER3> This is much bigger than another dust-collecting machine in everyones home. The "vaccine printer for everyone" is just what you write on the grant application form to get the simpletons at the government excited.

<comment><USER4> >>the simpletons at the government

Don't you think this was a little unnecessary?

</comment></comment></comment><comment><USER5> The availability of an affordable vaccine biofactory implies general manufacture of complex molecules.

It's science fiction on the order of fusion power plants or hard AI: there's nothing that we know will prevent it, but our technology level is just not there yet, nor can we make good estimates of when it will be.

I will be happy when it shows up, but I wouldn't expect it anytime soon.

<comment><USER3> I think when it does show up, no one will be happy. People are already running around like chicken with its head cut off because someone managed to 3d-print the lower receiver of an AR-15, fearing regulatory pressure. There are hour- long talks on the topic of computational freedom.

Now imagine a world where you can print molecules. It's either one where theres no freedom left or complete and total freedom.

<comment><USER2> >I think when it does show up, no one will be happy

Lawyers will.

</comment></comment></comment><comment><USER6> If you think RIAA/MPAA is evil, wait until you try cutting out big pharma.

<comment><USER7> Especially considering that they already have a massive government enforcement apparatus defending their turf (the FDA and friends).

<comment><USER8> Big pharma would be extremely happy if the FDA ceased to exist. It would make them more cost-competitive with homeopaths and supplement makers, because they would also be able to stop all of their testing and quality control.

</comment></comment></comment><comment><USER9> What happens when the vaccine printer gets a virus? ;)

<comment><USER10> Not quite what you were thinking, but I think viruses might be a piece of the puzzle:

print DNA -> PCR -> insert into vector of choice (i.e. some virus) -> transform organism of choice (i.e. some _E. coli_ strain) -> purify drug

(this is going to be one complicated printer with lots of expensive inks)

<comment><USER11> nope - because once you have the (open-sourced) recipes it will be QUITE cheap. Do you really think your Phusion master mix is expensive because it has a lot of material in it? You do realize you can purify your own polymerase for thousandths if not millionths of the cost of which you buy it from Invitrogen.

</comment></comment></comment><comment><USER12> Rewording of [http://www.theatlantic.com/sponsored/bank-of- america/archive...](http://www.theatlantic.com/sponsored/bank-of- america/archive/2012/10/synthesizing-life-means-faster-cures-biologist- says/263360)

</comment><comment><USER13> and the only thing I imagine is hackers making people inject themselves bad things instead of vaccines.

<comment><USER14> I'm wondering how long it'll take for someone to print acid on it.</comment></comment>


Apple Selling a Mac Monitor Stand for $999 - <AUTHOR> https://www.bloomberg.com/news/articles/2019-06-03/apple-just-announced-it-s-selling-a-mac-monitor-stand-for-999 <comment><USER2> There once was a time when a whole display was $999. How the times have changed...

</comment><comment><USER3> Chinese knock offs for 100 usd should be available soon. Of course if you pay so much for the screen you probably don't mind the cost of the stand.

</comment><comment><USER4> Ok this is silly but 24K gold is about 40$ per gram. Does anyone know the weight of this stand? Could I make one out of solid gold for that price :)?

Probably not with 24K gold, assuming the stand weighs more than 250g but 6K gold is about 10$/g so it's still semi-possible :D

<comment><USER3> Reddit style comment but I admit I laughed

</comment></comment><comment><USER5> Not to mention the PC at $6K with 256GB SSD. Still consider the most innovative company. Those guys one day will sell smart coffee cups for 500 dollars.

</comment><comment><USER6> This is a wonderfully unsubstantive article.

</comment><comment><USER7> Not saying I think the pricing is "right" but separate for $999 is actually great for people who would've previously paid $999 for the whole display & thrown that part away in favor of a VESA mount.

Look at the bright side: you save $800 if you buy the VESA adapter instead … BYOA VESA mount, though; because the $199 adapter is just a dongle.</comment>


Show HN: Vimsical.com – Interactive Screencasting Tool - <AUTHOR> http://vimsical.com/new <comment><USER2> I'd love to see this with a toggle for an "Ignore Mistakes" feature to exclude any text that was typed and eventually erased. Cool project so far</comment>


Learn C Programming and the Rest Will Come - <AUTHOR> https://www.zeroequalsfalse.press/posts/learn-c-programming#.XG6MqIfCUvY.hackernews <comment><USER2> I started to learn programming (and C) by doing Harvard's CS50 course. It was only after couple of years had passed that a lot of the things I learned then - allocating memory, pointers, writing functions that in other languages would be abstracted away into a library - have begun to really seem of value to me.

I wonder whether it is better to start with Python or JavaScript so you feel productive and can see more immediate results from your programming, or to start with C so that very little of programming is "magic" to you. I still haven't arrived at an answer, but I agree that some exposure to writing C is something that I would highly recommend.

</comment><comment><USER3> I see this sentiment a lot around here and while I agree that C should be _one_ of the first languages people learn, I don't think that it should be the first.

I started with Pascal in my first year CS course and then moved to C. Pascal with its clear structure (begin, end) but similar process to C was a good lead in, and I doubt going straight to C would have been anywhere near as helpful when learning programming from a base level of nothing.</comment>


Estonia – the Skype effect - <AUTHOR> http://www.bbc.com/news/technology-36286157 <comment><USER2> I can completely agree with this article. I've worked at Skype, and now I see many of my ex-colleagues having large impact at other startups and tech companies - some Estonian, some international. Transferwise. Twillio. Pipedrive. Fleep. Just a few ones where ex Estonian Skypers and their network have a huge impact. Twillio and Transferwise are two good international examples who did this and have had excellent results - the Tallin offices producing some of the best engineering outputs globally.

Almost every talented Skype engineer I know of has joined either a new startup, or a promising tech company with European offices. And many of them have not moved from Tallin - having these companies open offices here, and hire more locals, who the seniors then mentor and train.

It's incredible what effect a single company - Skype - has had to the tech community for all of Estonia.

<comment><USER3> I am curious is the lingua franca English in the office generally? I find it amusing that Estonia's weather was mentioned here, is the winter in Tallin worse than Stockholm or Berlin? Also poor connections? Is that train only? Certainly Ryan Air flies there no?

Are salaries on par with the rest of Europe?

<comment><USER4> Yes, in tech it's pretty safe to take English as the lingua franca here.

Connections - we do indeed have Ryanair and easyJet but not that many destinations. Tallinn Airport has a good list[1] of what's available directly.

Salaries - we at Teleport (with deep Skype roots as well) just launched a salary comparison tool[2] this week, hope it helps to get an overview of salaries but also costs.

[1] [http://www.tallinn-airport.ee/en/flight- info/destinations/](http://www.tallinn-airport.ee/en/flight- info/destinations/) [2] [https://teleport.org/salaries/](https://teleport.org/salaries/)

<comment><USER3> Thanks for the links, what's the Visa policy?

<comment><USER5> There is website called Work in Estonia[1],

I guess there is more specific information for different backgrounds.

[1] [http://www.workinestonia.com/coming-to- estonia/](http://www.workinestonia.com/coming-to-estonia/)

<comment><USER3> Thanks

</comment></comment></comment></comment></comment></comment><comment><USER6> > Estonia has quite a few disadvantages - a tiny home market, a winter that > seems to go on forever (although the weather was lovely for our visit), and > poor connections with the rest of Europe.

The weather? Sweden and Finland both have longer winters and it has not stopped them. Poor connections? Estonians are everywhere (London, NYC, SF) and their tech companies are all "global first", with investors from around the world and targeting the global market directly.

<comment><USER7> Yeah, I think the weather trope is getting old. Winters are fairly mild in the Baltic region and most people would prefer below freezing over rainy and windy anyway. It's not even the winter as such that gets to you, but the darkness.

Estonia isn't really comparable to Sweden and Finland though, mostly because of history and size. I want Estonia to become a tech hub, but realistically it's not currently that big of a player.

<comment><USER8> Perhaps it's a trope because the computer business basically originated in California, which is famed for its good weather.

</comment></comment></comment><comment><USER9> You know, a winter that lasts forever doesn't sound such a bad thing after all in terms of productivity. I live in Greece and every year when the summer approaches the last thing I want is staying inside and code.

<comment><USER10> Maybe, but you'd have to reconcile that with California's good productivity and good weather.

<comment><USER11> SF specifically is notorious for its mild climate, no? Average high in July (the hottest month) is 23 degrees. Compare that with e.g. 31 for DC or 34 for Athens.

<comment><USER12> "The worst winter I ever spent was a summer in San Francisco" \-- Mark Twain

<comment><USER13> That was the summer following a volcanic eruption, indeed, it snowed during that summer!

[https://en.wikipedia.org/wiki/Year_Without_a_Summer](https://en.wikipedia.org/wiki/Year_Without_a_Summer)

also: While usually attributed to him, the actual source of these words is unknown and there is no evidence that Mark Twain ever said this.

</comment></comment></comment></comment></comment><comment><USER14> Estonia totally puzzles me. The first time I ever met anyone from there was when I attended a Linux conference in Singapore in 1999. What puzzled me was that there was representatives of many countries businesses of course but the density of Estonian company was totally mind blowing. A country with just about 1.2 mio people and here they are with quite a lot of companies representing their passion for Open Source. Also 4/5 from them were selling infosec products like VPN solutions and firewalls etc. But it shows that there was already something fundamentally right with how open they are in their mind towards adopting technology (e-citizenship etc, ... and the ease it is to open and run a business there)

There is no doubt that Skype had a massive impact as a role model to form a strong local entrepreneurial identity and pride. But what I want to say is that Skype was already in the right place to start with.

<comment><USER15> i'll tell you why: they've been one of the first countries in the eu to push digital infrastructure and ftth since around 2000. you'd be astounded what big pile of clusterfuck germany's infrastructure is, by comparison.

ftth is far away because deutsche telekom is eager to grab subsidies, squeeze every bit out of copper and customers, and not get out of their bloody way.

</comment></comment><comment><USER16> Have worked with a few Estonians, and Seasonal Affective Disorder appears to be pretty prominent. One of the developers stopped working all of a sudden, and had to relocate to Thailand to get more sunlight.

<comment><USER3> I think theres another name for the affliction characterized by a sudden impulse to relocate to a beach in southern Thailand : )

</comment></comment><comment><USER17> Interesting, I think it's a good thing to see people at least trying to avoid the clichés of starting up in silicon valley or NYC. Even the US for that matter, it's only gonna help the world economy if people say "hey, I can make the best of things right here" instead of thinking they NEED to be a US based startup company

<comment><USER18> Having been in a "ho-hum that's nice" environment and a "let's murder the competition" environment, I would say it goes beyond THINKING you NEED to be a US based startup company.

The fact of the matter is that your environment defines who you are, as a person. Why would that not matter for your company?

Can the environment be replicated somewhere else? Sure, with the right amount of luck. But people who think environment doesn't matter are fooling themselves.

</comment></comment><comment><USER19> > Estonia has quite a few disadvantages - [...] a winter that seems to go on > forever

Awesome. In the Netherlands it's currently 23 degrees and it's already too warm. I'm dreading the approach of July.

<comment><USER5> Temperature is not the problem.

Lack of sunlight from October till February is depressing.

<comment><USER20> Agreed. I've never been depressed in my life, then I moved to Finland!

Seriously though, my first winter was rough, and despite how obvious it was, despite how much I'd heard about it, it took me a long time to recognize the symptoms of not wanting to do anything, and not being productive.

I'm sure now I know what to look for things will be better in the future..

</comment></comment></comment><comment><USER21> Yeah, I'm with the article's author; those delivery bots won't last 5 minutes in most American or European cities once people understand what they are.

<comment><USER5> Do you know that there are things full of cash called ATMs?

Yeah, some robots will be stolen in the beginning, but they are filled with all kind of sensors & cameras.

You'll be caught pretty quickly.

<comment><USER22> They're secured to buildings, pretty much immobile, usually have CCTV covering them (in the UK at least) ... but they still get stolen occasionally, eg ripped out the wall using heavy plant (diggers and such).

</comment></comment></comment><comment><USER23> And now they're just spyware for Microsoft.

</comment><comment><USER23> And now it's just more spyware for Microsoft.</comment>


Browser Wars: Is Internet Explorer 10 a Relevant Browser? - <AUTHOR> http://www.7tutorials.com/browser-wars-internet-explorer-10-relevant-browser

<comment><USER2> Browser Wars: Does it matter any more? To my pleasure, I can say that it mostly doesn't - modern pages will work in any modern browser. Hip hip hooray!

<comment><AUTHOR> That's because everybody agreed to support open standards but, if a major player stops doing so, then we are back to the dark ages. :)

<comment><USER2> Alas, then it has started again. But it says "Upgrade to Chrome" instead of "...to IE 4"; worst of all, the Chrome-pushing sites work equally well in recent Opera, or Firefox, or Safari (yes, I know that uses the same core), once you get past the browser-sniffer - but noooo, damn any capability checking, there's gotta be "CHROME" in the User-Agent string.</comment></comment></comment>


Show HN: La Vue - a cinematic iOS app with realtime police radio & ambient music - <AUTHOR> http://bortomheten.com/lavue/

<comment><USER2> This is pretty trippy. Nice work! I really like the UI as well. Where are you getting the police feeds from?

<comment><AUTHOR> Thanks! Check out the credits ( <http://bortomheten.com/lavue/legallicensecredits/> ) for a list of all sources that the app relies on, including the police scanners.</comment></comment>


How I improved our Jade compilation time by 99.7% - <AUTHOR> http://paulyoung.tv/jade-inheritance

<comment><USER2> FYI - I am getting the following warning on FireFox 22.0 for Ubuntu 12.04:

              Reported Attack Page!
              This web page at paulyoung.tv has been reported as an attack page and has been blocked based on your security preferences.

<comment><AUTHOR> Oh wow. Did not know that.

Apparently Firefox uses data from Google SafeBrowsing, which doesn't report anything out of the ordinary.

[http://www.google.com/safebrowsing/diagnostic?site=paulyoung...](http://www.google.com/safebrowsing/diagnostic?site=paulyoung.tv)

<comment><AUTHOR> Source: [https://support.mozilla.org/en- US/questions/956343](https://support.mozilla.org/en-US/questions/956343)

</comment></comment></comment><comment><USER3> > Currently, jade-inheritance depends on its own version of Jade to parse > templates. Since jade-inheritance does not offer compilation, this must be > done using other tools which may use a different version of Jade.

Why not use the jade that's already installed by the parent app? Due to the way dependencies work in npm, by just removing the jade dependency from jade- inheritance it should just traverse up the directory structure until it finds a jade package. Just instruct users to install jade alongside with jade- inheritance, and instead of using node_modules/jade- inheritance/node_modules/jade, it should use node_modules/jade.

<comment><AUTHOR> Good question.

I had tried this approach and had issues in getting it to work. Particularly, how to specify that jade was still a dependency in package.json. Both regular dependency and peer dependency didn't seem to fit the bill.

A simple proof-of-concept would be to take the built-in binary and have that be it's own package which depends on jade and jade-inheritance.

If you can offer some advice on how to go about this, could you create an issue? [https://github.com/paulyoung/jade- inheritance/issues](https://github.com/paulyoung/jade-inheritance/issues)

Thanks!

</comment></comment><comment><AUTHOR> My site is having some issues so I've added this blog post at [http://paulyoung.github.io](http://paulyoung.github.io).

</comment><comment><USER4> What's with the formatting on android - the right hand margin is off the screen and I cannot pinch to zoom out?

<comment><AUTHOR> I was using this theme: [https://github.com/soulmedia-me/wordpress- svbtle](https://github.com/soulmedia-me/wordpress-svbtle)

Is this any better? [http://paulyoung.github.io](http://paulyoung.github.io)

</comment></comment><comment><USER5> Same message on Chromium.

Probably you shall request site re-check by GGL.</comment>


How to manage images in a fashion website - <AUTHOR> http://cloudinary.com/blog/how_to_manage_images_in_a_fashion_website

<comment><USER2> It really annoys me (much more than it should) when software claims to do "management" of something and then offers absolutely no management features at all. This thing is doing delivery and manipulation of images, which is very useful, but actually _managing_ what's there is left up to the user. Argh.

See also, every "content management system" ever.

<comment><USER3> Hey onion2k, thanks for checking us out :) Have you registered to Cloudinary yet? (free registration available btw). Make sure you login to our dashboard for nice insights about your image usage. Click the media library console to interactively manage your websites images, transformations, etc. If that doesn't cut it, we have a RESTful admin API that let you do all of your image housekeeping programmatically. Hope this helps!

</comment></comment><comment><AUTHOR> This blog post describes the challenges that modern fashion sites and mobile apps face while managing their photos. The post shows how Cloudinary can be used by fashion sites for managing their images in the cloud: easy to use & powerful uploading, manipulation and fast CDN delivery to their users.</comment>


Genetic and Early-Life Environmental Influence on Dental Caries Risk: Twin Study - <AUTHOR> https://pediatrics.aappublications.org/content/early/2019/04/24/peds.2018-3499?sso=1&sso_redirect_count=1&nfstatus=401&nftoken=00000000-0000-0000-0000-000000000000&nfstatusdescription=ERROR%3a+No+local+token <comment><USER2> A failure to reject the null is not evidence for the null. Most of the (6yo or younger) children in this study didn't even have any cavities!</comment>


Launch HN: Syndetic (YC W20) – Software for explaining datasets - <AUTHOR> Hi HN,
We're Allison and Steve of Syndetic (https://www.getsyndetic.com). Syndetic is a web app that data providers use to explain their datasets to their customers. Think ReadMe but for datasets instead of APIs.
Every exchange of data ultimately comes down to a person at one company explaining their data to a person at another. Data buyers need to understand what's in the dataset (what are the fields and what do they mean) as well as how valuable it can be to them (how complete is it? how relevant?). Data providers solve this problem today with a "data dictionary" which is a meta spreadsheet explaining a dataset. This gets shared alongside some sample data over email. These artifacts are constantly getting stale as the underlying data changes.
Syndetic replaces this with software connected directly to the data that's being exchanged. We scan the data and automatically summarize it through statistics (e.g., cardinality), coverage rates, frequency counts, and sample sets. We do this continuously to monitor data quality over time. If a field gets removed from the file or goes from 1% null to 20% null we automatically alert the provider so they can take a look. For an example of what we produce but on an open dataset check out the results of the NYC 2015 Tree census at https://www.getsyndetic.com/publish/datasets/f1691c5d-56a9-4....
We met at SevenFifty, a tech startup connecting the three tiers of the beverage alcohol trade in the United States. SevenFifty integrates with the backend systems of 1,000+ beverage wholesalers to produce a complete dataset of what a restaurant can buy wholesale, at what price, in any zipcode in America. While the core business is a marketplace between buyers and sellers of alcohol, we built a side product providing data feeds back to beverage wholesalers about their own data. Syndetic grew out of the problems we experienced doing that. Allison kept a spreadsheet in dropbox of our data schema, which was very difficult to maintain, especially across a distributed team of data engineers and account managers. We pulled sample sets ad hoc, and ran stats over the samples to make sure the quality was good. We spent hours on the phone with our customers putting it all together to convey the meaning and the value of our data. We wondered why there was no software out there specifically built for data-as-a-service.
We also have backgrounds in quantitative finance (D. E. Shaw, Tower Research, BlackRock), large purchasers of external data, where we've seen the other side of this problem. Data purchasers spend a lot of time up-front evaluating the quality of a dataset, but they often don't monitor how the quality changes over time. They also have a hard time assessing the intersection of external datasets with data they already have. We're focusing on data providers first but expect to expand to purchasers down the road.
Our tech stack is one monolithic repo split into the frontend web app and backend data scanning. The frontend is a rails app and the data scanning is written in rust (we forked the amazing library xsv). One quirk is that we want to run the scanning in the same region as our customers' data to keep bandwidth costs and transfer time down, so we're actually running across both GCP and AWS.
If you're interested in this field you might enjoy reading the paper "Datasheets for datasets" (https://arxiv.org/pdf/1803.09010.pdf) which proposes a standardized method for documenting datasets modeled after the spec sheets that come with electronics. The authors propose that "for dataset creators, the primary objective is to encourage careful reflection on the process of creating, distributing, and maintaining a dataset, including any underlying assumptions, potential risks or harms, and implications of use." We agree with them that as more and more data is sold, the chance of misunderstanding what's in the data increases. We think we can help here by building qualitative questions into Syndetic alongside automation.
We have lots of ideas of where we could go with this, like fancier type detection (e.g. is this a phone number), validations, visualizations, anomaly detection, stability scores, configurable sampling, and benchmarking. We'd love feedback and to hear about your challenges working with datasets! <comment><USER2> Neat! Congrats on the launch - the demo is very helpful to understand the product. Having consumed long, painful PDF data dictionaries in the past, this is a big breath of fresh air. Excited to see where Syndetic goes!

For me, the most painful part of working with 3rd party data was actually figuring out the "match rate" to internal data. For example, you might be a consumer-facing company who hopes to add more context to your internal data by pulling in 3rd party information for existing clients. To match your internal data to a 3rd party dataset, you usually match on some hashed email (or similar identifier) to see what percentage of your consumer records will be available in the 3rd party dataset. Have you thought about something like that with your tool? Maybe you can upload a sample of hashed emails and see how different match rates pan out.

<comment><AUTHOR> Yes! This has come up across multiple industries and is probably the feature on our roadmap I'm most excited about. The implementation is tricky but customers definitely care about the intersection of a provider's data with their own. Some more sophisticated providers have internal tools for generating things like sample sets customized to a prospect.

We're going to be adding a feature where we can flag fields as identifying keys and index them. We'll start with a simple intersection count ("upload 100 stock tickers, see how many records match"). Then we'll add an interactive feature to let a prospective customer generate all of the stats in the dictionary scoped down to the subset of data they care about. It's important to be able to answer questions like "for the 100 tickers I care about, how many NULLs are there for this other column?".

Maybe someday we'll even get into the more general record linkage problem when there's no reliable matching key.

<comment><USER3> That sounds very useful.

I am also super impressed that you managed to present your product without mentioning "big data" or "machine learning" or AI - given that anyone that does anything these days crams those big words in.

Thats is good, good luck.

<comment><AUTHOR> Thanks, I'm with you on the big data buzzwords and trying to avoid overengineering things (one of my favorite HN posts ever is [https://news.ycombinator.com/item?id=8908462](https://news.ycombinator.com/item?id=8908462)).

Right now the data scanning is just a fork of [https://github.com/BurntSushi/xsv/](https://github.com/BurntSushi/xsv/) running in a container with plenty of ram, and we've handled files in the ~20GB range with no problem. I think we could actually scale up to ~100GB files with xsv, which seems to cover 99%+ of data providers we're running in to. Providers might be processing massive amounts of data but the eventual deliverable they share with their customers is rarely too big for one machine.

That said, we will probably move away from our super simple stack towards running a Spark cluster in the medium term. Not for "big data" (actually I expect Spark to have higher latency and possibly to be slower for a moderate sized dataset than the rust solution) but because we want to be able to run multiple parallel scans over the datasets for upcoming future features. Some of that will involve a DAG of dependencies (e.g., do type detection first to figure out fields that are categorical, then generate visualizations where the plots are grouped by whatever the values are of the categorical field). There are also a bunch of nice libraries in the spark world for more comprehensive stats.

</comment></comment></comment></comment><comment><USER4> This (or something like it) makes a lot of sense to me. I've been at multiple organizations where there have been efforts to create these "data dictionaries" explaining the meaning of the data, especially when the schemas or APIs are not well designed.

But then manually writing documentation is obviously tedious and can typically only be written by the data team that knows the underlying data well, which is not always the best use of their time.

I'll definitely be following Syndetic and hope they can help crack this problem.

<comment><USER5> (This is Allison) - thank you! It's interesting for us to see how the problem is handled at different types of organizations, because as you point out the data team knows the underlying data best but is not often customer-facing.

</comment></comment><comment><USER6> Great concept, although worried about uploading datasets with sensitive data to your cloud.

I usually use this for Pandas Profiling to accomplish EDA tasks [https://github.com/pandas-profiling/pandas- profiling](https://github.com/pandas-profiling/pandas-profiling)

</comment><comment><USER7> Super great idea. I've been talking to data scientist/eng in B2B SaaS space on how we should bring best practices like that to the Sales/Marketing/Business ops world too.

What would you say are the differences between syndetic and qri.io (not affiliated in anyway)

<comment><AUTHOR> Thanks for the kind words! I hadn't seen qri.io before, but from a read of their website I think it's broadly similar to Dolt ([https://www.liquidata.co/](https://www.liquidata.co/)) which is git for datasets. Kaggle has a similar data hub at [https://www.kaggle.com/datasets](https://www.kaggle.com/datasets) that's not open source but is in the same space.

Our approach is to scan the datasets wherever they currently live in production rather than being a new way to store the data. The industry seems to have settled on FTP and S3 for now, and we think it's important that we connect to the same exact thing a customer would access. That lets us keep the dictionary up to date automatically without the data providers needing to change their storage infrastructure.

</comment></comment><comment><USER8> I wonder if you could combine your service with freely available datasets like Google Dataset Search [1] to demo what a large amount of various datasets would look like under your service.

[1]: [https://datasetsearch.research.google.com/search?query=puppi...](https://datasetsearch.research.google.com/search?query=puppies&docid=OhxQw53Vy%2FmwMc68AAAAAA%3D%3D&filters=WyJbXCJ1cGRhdGVkX2RhdGVcIixbXCIzeVwiXV0iXQ%3D%3D&property=dXBkYXRlZF9kYXRl)

<comment><USER5> We would love to do that at some point. There is tons of open data out there, but not a lot of it has useful descriptions at the field level (only the dataset level), so it would take some time to put a collection together that is robust. Also the demo on our splash page is a demo of the artifact we create (i.e. the published dictionary) only. The other side of the web app is a management layer to bundle datasets into collections, annotate fields, configure sample sets, and share the artifacts. We'll work on fleshing out our demo to show what the system looks like when there are hundreds or thousands of datasets.

</comment></comment><comment><USER9> I have been doing this for my company, when there is a sector you have data on - there is also external factors like how much of the market does your db cover for prices etc. and NLP on different items and metrics for different types of data in your offerings. What do you do with this when the actual field names are very general (i.e. item metric region unit)

<comment><USER5> We actually built in the concept of "display names" for exactly this purpose. You want to keep the actual field name in the schema so that ingestion works properly, but you also want to describe the field as helpfully as possible. In another comment, Steve mentioned that we are trying to tackle the first problem you mention (how much overlap is there between my dataset and other external factors) with the concept of creating dataset intersection or relevance scores.

</comment></comment><comment><USER10> Any plans on a hosted version? Either regular on prem or as something that can be privately hosted on e.g. AWS like DataBricks or Snowflake?

I love the idea but we could never expose most of our data to a public SaaS. There are all kinds of restrictions we have on things like data privacy and data needing to stay in specific regions.

<comment><AUTHOR> Yeah, on-premise or at least private cloud has come up a few times. Beyond the data privacy and licensing requirements it'd also just be plain faster in some cases. We haven't offered it yet just because we're a small company and are rapidly adding features. Our backend is mostly running in k8s so I don't think it'll be a _huge_ technical rewrite to get it running in private cloud. Frankly I just don't have experience supporting software running outside my control and want to make sure we take the time to do it correctly.

<comment><USER10> Cool. We're an enterprise B2B SaaS company that does a ton of data interchange with our customers. Each project burns a bunch of engineering hours and calendar time because customers send us data that doesn't match the spec or we just don't have anyone outside the engineering org who can confirm that the data looks how it's supposed to.

Something which simplified the process of analyzing sample data and provided a view of it to a non-technical user would be very valuable. But as I mentioned in my previous comment we could never expose any of our data outside of our private infrastructure, so we can't use this until there are other options for hosting.

</comment></comment></comment><comment><USER11> This looks really cool! I've worked with large data sets before and one of the most annoying things was when they were split up into multiple files. Do you currently support statistics across multiple datasets?

Also, how did you come up with the pricing? 500 and call us seems like a lot per month

<comment><AUTHOR> Thanks! We can definitely combine multiple files into one dataset so long as they share the same fields. We've got one customer that keeps their data in an S3 bucket as one JSON file per record, so for them we're scanning ~480K files to construct the stats. If you've got multiple different datasets we've got a concept of "collections" for organization.

We came up with the pricing strategy based on conversations with early customers. We want to be able to say yes to integrations with whatever system they're using to store their data so we need flexibility at the early stage.

</comment></comment><comment><USER12> This reminds me of readme generator for Frictionless 'Data Package' [https://frictionlessdata.io/](https://frictionlessdata.io/)

<comment><USER5> Interesting, I just came across Frictionless recently as part of the FISD group that's trying to implement a set of standards across the financial industry for documenting datasets.

</comment></comment><comment><USER13> What data types do you plan to support? Have you considered supporting datasets with images+labels used in computer vision? How would you like to handle them?

Are you going to support data labeling tasks?

<comment><AUTHOR> We've talked with some image data providers who are creating datasets for use in machine learning. We're not running any of our own models on image data right now, so I think the place we can be most useful is in summarizing metadata about the images in cases where the dataset isn't just an image file. For example, if the dataset is images of intersections plus bounding box coordinates of street signs we can tell a prospective consumer what % of images have a street sign in them. If you have a little more metadata (e.g., what time of day the photo is) the stats get much more useful out of the box.

I don't understand the data labeling question. How would you imagine us getting involved there?

</comment></comment><comment><USER14> Makes a ton of sense to me, I really wish the datasets that I work with had this or some kind of equivalent.

Would be really great if you could generate a fake dataset of similar size for testing purposes. It would take some thinking but it would be really useful for building a consumer before getting the full dataset.

<comment><USER5> I missed this comment earlier - we can pilot with sample/fake data to give the user a sense of how the system works and what workflows within it work best for their company. The versioning/diffs wouldn't make much sense, but that's probably ok.

</comment></comment><comment><USER15> very cool product! I've worked on much smaller datasets with pandas and their inbuilt profiling report method can slow things down to a crawl!. Hoping to see more from you guys :)

<comment><AUTHOR> A very reductionist version of our company that Allison hates when I use is "csvstat but on the internet" :-). I think the problem of auto-summarizing datasets has hit kind of a local maximum in what pandas dataframe summaries (csvstat is a similar python tool) can do on one machine. We will be able to add much fancier things like sophisticated type classification (e.g., is this field a stock ticker) without burning your CPU.

<comment><USER15> hah! but this is a very interesting area. You're right on the auto-summarizing issue becoming a problem these days with the usage of larger datasets. Data versioning also is starting to become a larger problem and I saw that you guys already have addressed it in your enterprise product. Hoping to see some sort of API-like version for comparison of data troves from different timelines in the future.

</comment></comment></comment><comment><USER16> What industries do you see this being most useful to?

<comment><USER5> We think data-as-a-service is a new and growing category, and this includes startups we would consider "pure" data companies (e.g. a company that sells data on airfares across the web) and companies that have been around for decades selling csvs that they deliver over FTP (e.g. a giant company like ADP that sells payroll processing data). Based on our backgrounds we have a fair amount of experience in the alternative data space, which is basically any data that might have some signal to a hedge fund that isn't market data. I'm finding that the providers in that space are interested in expanding their customer base to corporates (e.g. Walmart, McDonalds) whereas the providers currently selling to corporates are interested in expanding into alternative data.

</comment></comment><comment><USER17> Awesome! Congrats on the launch! Excited to check it out.

</comment><comment><USER18> Hey! I'm Greg of ReadMe... think Syndetic but for APIs rather than datasets!

Congrats on the launch :) I'll find you at Alumni Demo Day, or feel free to reach out if I can help with anything! And welcome to the war on PDFs :)

<comment><USER5> This made our day : ) Thank you!

</comment></comment><comment><USER19> Does this support more complex data structures - for example parquet files?

<comment><AUTHOR> The manual data upload is restricted to well-formed CSV files w/ headers on the first row right now. For the "contact us" higher tier we'll handle any file format that we can extract columns from, so parquet would be fine.</comment></comment>