Evio – Fast event-loop networking for Go - <AUTHOR> https://github.com/tidwall/evio <comment><USER2> This project is not intended to be a general purpose replacement for the standard Go net package or goroutines. It's for building specialized services such as key value stores, L7 proxies, static websites, etc.

You would not want to use this framework if you need to handle long-running requests (milliseconds or more). For example, a web api that needs to connect to a mongo database, authenticate, and respond; just use the Go net/http package instead.

There are many popular event loop based applications in the wild such as Nginx, Haproxy, Redis, and Memcached. All of these are single-threaded and very fast and written in C.

The reason I wrote this framework is so I can build certain network services that perform like the C apps above, but I also want to continue to work in Go.

<comment><AUTHOR> > It's for building specialized services such as key value stores, L7 proxies, > static websites, etc.

First of all, thank you for publishing this project. It's very interesting in my opinion since I never thought about the benefits of an event loop. Would you mind explaining briefly why an event loop is a better suit for these applications? Is it due to performance and efficiency?

<comment><USER3> I'd suggest that's not the right way to look at it. To a first approximation, "everything" is using an event loop nowadays, in that everything is using the same fundamental primitives to handle and dispatch events. In particular, this includes the Go runtime; run "strace" on a Go network program and you'll see these same calls pop up in the strace.

What this does instead is give a Go program _direct_ access to the event loop. The benefit is that it bypasses all of the stuff that Go wraps around the internal event loop call that allows it to implement the way it offers a thread-like interface for you, and integrates with the channel and concurrency primitives, and maintains your position in the call stack between events, etc. The penalty is... the exact same thing, that you lose all the nice stuff that the Go runtime offers to you to implement the thread-like interface, etc., and are back to a lower-level interface that offers less services.

The performance of the Go runtime is "pretty good", especially by scripting language standards, but if you have sufficiently high performance requirements, you will not want to pay the overhead. The pathological case for all of these nice high-level abstractions is a server that handles a ton of network traffic of some sort and needs to do a little something to every request, maybe just a couple dozen _cycle 's_ worth of something, at which point paying what could be a few hundred cycles for all this runtime nice stuff that you're not using becomes a significant performance drain. Most people are not doing things where they can service a network request in a few dozen cycles, and the longer it takes to service a single request the more sense it makes to have a nice runtime layer providing you useful services, as it drops in the percentage of CPU time consumed by your program. For the most part, if you are so much as hitting a database over a network connection, even a local one, in your request, you've already greatly exceeded the amount of time you're paying to the runtime, for instance.

It does seem to me that a lot of people are a bit bedazzled by the top-level stuff that various languages offer, and forget that under the hood, everyone's using the event-based interfaces. What differs between Node and Twisted and all of the dozens or hundreds of other viable wrappers over these calls is the services automatically provided, not whether or not they are "event loops". Go is an event loop at the kernel level. Node is an event loop at the kernel level. Erlang is an event loop at the kernel level. They aren't all the same, but "event-based" vs. "not event-based" is not the distinction; it's a question of what they lay on top of the underlying event loop, not whether they use it. Even pure OS threads are, ultimately, event loops under the hood, just in the kernel rather than the user space.

<comment><USER4> > It does seem to me that a lot of people are a bit bedazzled by the top-level > stuff that various languages offer, and forget that under the hood, > everyone's using the event-based interfaces.

Yup. It's all very similar under the hood.

The most important difference between I/O models is whether the paradigm involves _explicit_ vs. _implicit_ management of the event loop. Callback models like Node, async/await style models like those of C#, and low-level primitives like IOCP, epoll, and kqueue fall into the former category. Go/Erlang, plain old threads, and even Unix processes fall into the latter category. There are advantages and disadvantages of each model.

Within each of these broad categories, the distinctions are, IMHO, much less interesting, and they're often made out to be more significant than they actually are. In particular, the distinction between runtimes like Go and regular OS pthreads is often made out to be more important than it really is, when the difference ultimately boils down to the CPU privilege level that thread management runs at.

<comment><USER5> Patrick, on the 2.6+ Linux kernels, is there a significant difference between threads and processes? It seems like both threads and processes are created via clone and the only difference is memory access?

I often hear "context switching between threads is cheaper" but pthreads still have their own PID and everything, so is this really the case?

Is there really much advantage to pthreads over the way PostgreSQL does things with efficient CoW sharing between processes for the binary?

<comment><USER6> The significance of the distinction depends entirely on the use case.

Yes, they're both created with clone, but with different levels of sharing. A pthread will share the virtual address space of its parent, which makes shared memory simple to implement; use the same pointer and you're done. CoW is not "sharing" really, because you can't communicate over it, it just saves some creation overhead.

With CoW, technically nothing gets copied initially, but as soon as the new process starts executing, it's going to start copying the stack frame and any other regions it's using. With a pthread you can be certain it will just copy the stack.

Context switches are usually cheaper when you don't need to throw out the old virtual address space (and invalidate the Translation Lookaside Buffer). Pthreads share virtual address space, so there is no need to flush the TLB.

In a use case like Postgres, you don't necessarily need to optimise for context switches. If you have a lot of concurrent connections, each of which has one process, then you'll only hit limits with context switching overhead if very few of those connections are fighting over any locks or spending much time in IO at all. This is atypical, so usually those other factors hit you first.

<comment><USER7> > The significance of the distinction depends entirely on the use case.

Indeed.

> Context switches are usually cheaper when you don't need to throw out the > old virtual address space (and invalidate the Translation Lookaside Buffer). > Pthreads share virtual address space, so there is no need to flush the TLB.

I believe the cost of that has been reduced somewhat due to tagged TLBs on modern hardware.

> In a use case like Postgres, you don't necessarily need to optimise for > context switches. If you have a lot of concurrent connections, each of which > has one process, then you'll only hit limits with context switching overhead > if very few of those connections are fighting over any locks or spending > much time in IO at all. This is atypical, so usually those other factors hit > you first.

Yea. There's a number of limitations in postgres due to the process model, but they're imo not TLB / context switch related. The biggest issue is that dynamically sharing memory between processes is harder, because there's no guarantee that it's possible for all post-fork memory allocations can portably be put at the same virtual addresses. Which then makes it more complicated to have shared datastructures, because you need to use relative pointers and such. That's not a problem for the main buffer pool etc, which is allocated when postgres is started, but it is problematic e.g. for memory shared between multiple processes working on the same query (say the memory for a shared hashtable in a hashjoin).

<comment><USER6> > you need to use relative pointers and such

I don't think this qualifies as a performance overhead, though, beyond the odd isub.

<comment><USER7> > > you need to use relative pointers and such

> I don't think this qualifies as a performance overhead, though, beyond the > odd isub.

It ends up as one. The reason is less the additional instruction(s), but that you actually need to ferry arround additional data. In common scenarios you'll end up with a number of mappings shared between processes, so you can't just assume a single base address per-process. Instead you've to associate the specific mapping with relative pointers, and that does add to overhead. Both programming wise and runtime efficiency wise.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> One of my favorite things about Go is that it cuts through the "threads vs. events" debate by offering thread-style programming with event-style scaling using what you might call green threads (compiler assisted cooperative multitasking that has the programming semantics of preemptive multitasking).

That is, I can write simple blocking code, and my server still scales.

Using event loop programming in Go would take away one of my favorite things about the language, so I won't be using this. However I do appreciate the work, as it makes an excellent bug report against the Go runtime. It gives us a standard to hold the standard library's net package to.

<comment><USER4> It doesn't really "cut through" the debate any more than any other implementation of threads does. The only difference between Go and plain old one-thread-per-connection is that regular threads run in the kernel, while Go threads run in userspace. That's not a _semantic_ difference, only an implementation detail (a large detail, to be clear, but still an implementation detail).

There were historical implementations of pthreads, such as NGPT, that used precisely the same model as Go, and they were abandoned because the advantages over 1:1 were not sufficient to justify the complexity.

<comment><USER9> What you call a "Go thread" has a precise name (goroutine) and running in userspace is hardly the only difference between a goroutine and a kernel thread.

Creating and destroying kernel threads is significantly more expensive.

A kernel thread has a fixed stack and if you go beyond, you crash. Which means that you have to create kernel threads with worst-case-scenario stack sizes (and pray that you got it right).

Goroutine has an expandable stack and starts with very small stack (which is partly why it's faster; setting up kernel page mappings to create a contiguous space for a large stack is not free).

Finally, goroutine scheduling is different than kernel thread scheduling: a blocked goroutine consumes no CPU cycles.

In a 4 core CPU there is no point in running more than 4 busy kernel threads but kernel scheduler has to give each thread a chance to run. The more threads you have, the more time kernel spends and pointless work of ping-ponging between threads. That hurts throughput, especially when we're talking about high-load servers (serving thousands or even millions of concurrent connections).

Go runtime only creates as many threads as CPUs and avoids this waste.

That's why high-perf servers (like nginx) don't just use kernel thread per connection and go through considerable complexity of writing event driven code.

Go gives you straightforward programming model of thread-per-connection with scalability and performance much closer to event-driven model.

You work on Rust and are well informed about this topic so I'm sure you know all of that.

Which is why it amazes me the lengths to which you go to denigrate Go in that respect and minimize what is a great and unique programming model among mainstream languages.

<comment><USER4> > What you call a "Go thread" has a precise name (goroutine)

I call goroutines threads because they are user-level threads.

As an analogy, NVIDIA calls local threadgroups "warps", but that doesn't make them not local threadgroups.

> Creating and destroying kernel threads is significantly more expensive.

Because kernel threads usually have larger stacks. But they don't always have large stacks: that is configurable. Other than the stack size, the primary difference is simply that kernel threads are created in kernel space and user threads are created in userspace.

> A kernel thread has a fixed stack and if you go beyond, you crash. Which > means that you have to create kernel threads with worst-case-scenario stack > sizes (and pray that you got it right).

You can do stack switching in 1:1 too. After all, if you couldn't, then Go couldn't do stack switching at all, since goroutines are built on top of kernel threads.

Go's small stacks are really a property of the moving GC, not a property of the threading model.

> In a 4 core CPU there is no point in running more than 4 busy kernel threads > but kernel scheduler has to give each thread a chance to run.

> Go runtime only creates as many threads as CPUs and avoids this waste.

Not if they're blocked doing I/O!

If they're not blocked doing I/O, then Go tries to do preemption just as the kernel does. (I say "tries to" because Go currently cannot preempt outside function boundaries; this is a significant downside of M:N threading compared to 1:1 kernel threading.)

> That's why high-perf servers (like nginx) don't just use kernel thread per > connection and go through considerable complexity of writing event driven > code.

High-performance servers like nginx use an event loop because it's the only way to get the absolute fastest performance, with no overhead of stacks at all. The fact that the project described in the article gets better performance than Go's threads is proof of that fact, in fact.

It would be possible, and interesting, to do Go-like 1:1 threading with small stacks.

> Go gives you straightforward programming model of thread-per-connection with > scalability and performance much closer to event-driven model.

Sure. But that's mostly because of the GC, not because of the M:N threading model.

> Which is why it amazes me the lengths to which you go to denigrate Go in > that respect and minimize what is a great and unique programming model among > mainstream languages.

It's not unique. As I said, NGPT used to do M:N for pthreads. Solaris used to do M:N for pthreads. The JVM used to do M:N.

<comment><USER10> Nope, the JVM used to do M:1, it's very different from M:N.

</comment></comment></comment></comment></comment><comment><USER11> The Go network stack already makes use of epoll and kqueue: [https://golang.org/src/runtime/netpoll_epoll.go](https://golang.org/src/runtime/netpoll_epoll.go) So I'm not quiet sure why this would be faster since almost all I/O in Go is event driven, including the networking stack.

<comment><USER12> The benchmarks at the bottom of the readme show quite an improvement (with a single thread it seems).

I would speculate the performance win is because there is no stack switching and less channels.

I've done lots of event loops in the past (eg hellepoll in c++) and think that the cost of that is on the programmer - keeping track of things, callbacks, state machines and things and avoiding using the stack for state etc is all hard work and easy to mess up.

I am reminded of this post I saw on HN a while ago [https://www.mappingthejourney.com/single- post/2017/08/31/epi...](https://www.mappingthejourney.com/single- post/2017/08/31/episode-8-interview-with-ryan-dahl-creator-of-nodejs/) Ryan Dahl, creator of node.js, would just use Go today ;)

<comment><USER13> > I've done lots of event loops in the past (eg hellepoll in c++) and think > that the cost of that is on the programmer - keeping track of things, > callbacks, state machines and things and avoiding using the stack for state > etc is all hard work and easy to mess up.

This is improving, even in C++. This is what the core loop of a line-based echo server could look like in C++17 (and something very similar compiles today on my machine)

        void echo_loop (tcp::socket socket) {
            io::streambuf buffer;
            std::string line;
            std::error_code ec;
            do {
                ec = co_await async_read_line (socket, buffer, line);
                if (ec)
                    break;
                ec = co_await async_write (socket, line);
            } while (!ec);
        }

<comment><USER2> Wow. That looks really simple.

<comment><USER13> Unfortunately it's just exposition, but here[0] is a version that works with Clang 5 + Boost

Echo specific code starts on line 167. Everything above will hopefully be provided by the standard library once both the Networking TS and Coroutine TS merge in to C++20.

One nice thing about lines 1 - 165 though, is that it demonstrates how easy it is to extend the native coroutine capabilities in C++ to support arbitrary async libraries, even if the author of those libraries didn't know anything about coroutines. All this happens without breaking the ability to call these coroutines from C. You can even use async C libraries that only provide a void* argument to your callback.

[0] [https://gist.github.com/anonymous/d9a258136431a352516122d1c9...](https://gist.github.com/anonymous/d9a258136431a352516122d1c9c2ca58)

</comment></comment></comment></comment></comment><comment><USER14> I love Go because I never had to write asynchronous, callback driven programs in this language. I hope it won't become the norm in Go, too.

<comment><USER2> It won't become the norm. I promise you, cross my heart and hope to die, stick a needle in my eye.

</comment></comment><comment><USER15> I'd be interested in the level seven reverse proxy application. As well as unix domain socket message queues. There are probably many other places in the networking pipeline evio could provide a boost.

It's a testament to what is possible through the "syscall" and "golang/x/sys" facilities. As well as your confidence in playing with Linux internals ;)

<comment><USER2> Thanks! The L7 proxy should be pretty sweet. :)

</comment></comment><comment><USER16> Not sure i understand what the use case is. As soon as you start doing something on the event loop , you need some kind of way to perform the operation in another "thread" ( or goroutine or whatever). And then you start to need some kind of concurrency mechanism, and pay the price.

Stripping those mechanism to pretend the event handling is faster only works if you never intend to have some real computation performed. That's never true in practice... Or am i missing something ?

<comment><USER17> Not the OP but typically you resort to these tactics when you want to shave the last ms of the server's response time, and/or get that last 1000 requests/s/core performance. You have a "fast path" that is simple and event driven and hand off operation processing to regular threads for the (less frequent) more complex operations.

So you're not missing anything.

<comment><USER18> Aren't you losing a lot of the benefit by using a GC'd language in the first place?

</comment></comment></comment><comment> brian-armstrong At this point, why not just use C++? I feel like people are trying to stretch Go way past what it's good for. It's not going to replace C++ where C++ is effective, and it shouldn't :)

<comment><USER4> Because memory safety is much easier to achieve in Go than in C++, even "modern C++".

</comment></comment><comment><USER19> Worst of both worlds, only faster?

</comment><comment><USER20> This is single-threaded? What are you going to do with the other 31 or 63 cores?

The single-threaded nature of applications liked Redis an Haproxy is a singificant impediment to their vertical scalability. CPUs aren't getting faster, we're just going to get more cores, so anything that assumes there's only a single core seems like a dead end.

Haproxy literally just added multithreading support in 1.8.

<comment><USER21> The CPU is rarely the bottleneck and for both Redis/HAProxy the vertical scalability solution has been to launch multiple processes or forks with different core affinities. There are downsides of course (no IPC) but I still argue that CPU is not the bottleneck for 99% of usage scenarios.

HAProxy added threading support in 1.8 as you pointed out and Redis has started the same (for a certain subset of processing) in 4.0 as well. They're getting there but concurrency is tough.

To suggest that his product is a "dead end" due to not supporting threading seems a bit premature, as Redis and HAProxy are extremely well-regarded in their niche and they made it there without threading, and we've been at maximal clock speed for nearly a decade.

<comment><USER20> > There are downsides of course (no IPC) but I still argue that CPU is not the > bottleneck for 99% of usage scenarios.

I suppose my experience might be unusual, but I frequently log in to c3.8xlarge redis machines that have a single core pegged at 100% and the rest doing nothing. Yes multiple processes help, but that requires updating clients and makes it harder to share memory.

> To suggest that his product is a "dead end" due to not supporting threading > seems a bit premature, as Redis and HAProxy are extremely well-regarded in > their niche and they made it there without threading.

Well yeah, CPUs hitting their GHZ limit and the dramatic increase in the number of cores per machine is a relatively recent phenomena.

I just think its weird to start a brand new project making those same assumptions, especially when the underlying programming language was explicitly designed with concurrency in mind.

It'd be like building a new networking library in Rust which ditches memory safety.</comment></comment></comment>


Shopify CTO: our platform is now handling Black Friday level traffic every day - <AUTHOR> https://twitter.com/jmwind/status/1250816681024331777 <comment><USER2> I hold a pessimistic viewpoint because it's a lot more insightful to look at losses than it is gains.

With so much brick-and-mortar retail shut-down, retail has to happen somewhere, so it's going to happen online. Increases in product sales online could result in net declines for the product group when you factor in the loss of B&M sales.

I think the logical result here is pretty well-represented in how some public companies in this space's stocks prices are playing out. Distribution companies are likely to prosper because their competition has decreased substantially:

* Amazon is at an all-time high * Overstock has recovered their dip * Wayfair is recovering their dip * Chewy is at an all-time high

None of this means the manufacturers who these distributors are selling are doing well, though.

On the flip side, the product categories that are down substantially online are likely facing devastating loses. They already aren't selling in retail so if ecomm is it and they're down there, it's _really_ bad.

Outside of distributors, I think the D2C ecomm space is a huge mixed bag but likely net negative for a few reasons:

1\. A lot of DTC companies play in the product categories that are seeing lower online sales

2\. Most DTC companies are positioned as being accessible, but still premium. There are cheaper alternatives people may be more inclined to go with, given the economic uncertainty.

Given that, layoffs at Away, Everlane, Third Love, Stitch Fix, Casper, etc. make sense, even if online sales as a whole are up. Some of them may even need to start paying attention to unit economics now! I'd guess a lot are scaling back advertising and trying to focus on lower-cost acquisition channels to keep a tighter grip on cash flow.

<comment><USER3> Amazon not only recovered their dip, but renewed their 52 week max several times already. And they don't even make much from online retail. To make matters more interesting, AWS seems to be _much_ busier than usual, and spot capacity is all but gone, at least at the price we were paying before. I'm not even sure why that is.

<comment><USER4> Amazon makes almost 50% of their annual revenue from online retail. Many billions of dollars. It's by far their biggest product category.

Yes, through accounting magic, they're making the majority of their profits via AWS, but that's part of Bezos's long-term strategy and could be changed on a dime (at the potential expense of the long-term growth he's targeting).

Top line, most of their business is from online retail, not AWS.

(edited because my original was a bit of a flame, and didn't clarify profit vs revenue distinction)

<comment><USER5> Amazon makes substantially more from AWS than its retail segment. Retail revenue is higher but revenue does not equal profit. For instance last quarter AWS revenue was 13% compares to retail but it contributed 52% of operating income.

<comment><USER6> Those are accounting tricks

<comment><USER7> Citation needed. It seems pretty common-sense that retail would be lower- margin than digital services.

</comment></comment></comment></comment></comment></comment><comment><USER8> I am presently scaling a client on Shopify who started at a 30k a month ad budget. Today we scaled past $8,000 per day (So a little under 250k a month). I fully expect and we'll hit $500,000 a month in ad spend for them monthly probably in the next 2 weeks. The combination of increased screen time and major companies pulling ad dollars has cratered media prices, and in the process, created a once in a decade opportunity for DTC brands that have cash on hand and are ready to scale up production. This announcement from Shopify themselves surprises me not at all.

<comment><USER9> I am not being judgemental, rather trying to understand how you get fish of that size with your far-from-well-designed website (from your profile), and far-from-rich marketing contents. Is that luck? Or, your lack of time to work on your own presentation?

Again, I am seriously not being judgmental, I am curious.

<comment><USER8> You aren't wrong it's been on my list forever to work on my site. FWIW some of the best media buyers I know though literally don't have websites.

I've been doing Facebook ads for 9 years. My team and I are a white label agency so we do the media buying for creative shops, designers, content marketers, etc. So I didn't land any client. I partnered with the folks that did. I got the partnership by putting out a ton of content on Youtube (Over 20 hours worth), through doing a 30 episode deep podcast on running an advertising agency, and being super helpful in a number of Facebook groups, Linkedin groups, etc. for years.

Tl;DR Content Marketing + Networking

<comment><USER10> It looks good to me.

Putting out relevant content and networking is where it's at, too. Even if your website actually was bad, you'd probably be fine.

I used to get a lot of work for web design without a website. I had a resume with good references and urls leading to good work samples. That's the stuff that counts. There was no shortage of jokes about not having a website though, and everyone thought they were the first person to tell me 'the cobbler's children go barefoot', haha.

</comment></comment></comment></comment><comment><USER11> Same here. We have provide a warehouse management system specifically for e-commerce warehouses, as SaaS. We are (one of) the biggest player in the Netherlands, and since "social distancing" started here 4 weeks ago we see Black Friday 2019 levels of orders every day, for 4 weeks straight now.

We have some customers that saw there orders come to a stop, even a few bankruptcies already. But almost all see a 30-40% bigger order flow. Some customers that sell DIY hair products see a doubling or tripling in orders per day and revenue.

When I talk to the biggest e-commerce players in the Netherlands and the top 3 parcel shippers, they all see (near) black friday levels of orders/shipments.

<comment><USER12> Same in France, most makeup brands are x2-x3, fashion is x1.5 to x6

In France there is no increase of unemployment, instead they are at home getting paid, I guess they have a lot of money to spend as they can't party or go the restaurant

It is actually quite visible here : [https://contentsquare.com/covid-19-ecommerce-impact-data- hub...](https://contentsquare.com/covid-19-ecommerce-impact-data-hub/)

<comment><USER13> Only 9 million people in furlough (with gov benefits); so a third of the working population. The economic shock is going to be brutal.

<comment><USER14> Don't think so for France. The state gives too good benefits and it already announced it will be maintaining paychecks for all unemployed.

However that's billions over billions of new debt taken, and it's gonna be brutal to pay back in the next decade.

</comment></comment></comment></comment><comment><USER15> I've been talking to as many companies as I can during all of this. The surge is not evenly distributed but a lot of companies are seeing surprising upticks in online traffic and sales.

The big question though is how long will this last? I think the general population has not yet appreciated the possible severity of the ensuing economic downturn.

A lot of trade publications are already starting to sound the alarms on how ugly this could get. A return to 2008 era aggressive discounting is likely going to happen. Retail partners may be crippled by the quarantine. A lot of brands are going to be assailed from many directions simultaneously, it's going to hurt.

<comment><USER16> > I think the general population has not yet appreciated the possible severity > of the ensuing economic downturn.

My fear as well. The swiftness and ferocity of this downturn is like nothing ever, AFAIK. 22 million out of work in 4 weeks. Many are unlikely to get their jobs back. So while certain companies are seeing upticks due to logistics of the lockdown, I don't think it's safe to assume it would continue long after SIP ends.

<comment><USER17> > Many are unlikely to get their jobs back.

Their original job? Sounds plausible. Any job? Surely that would be a political choice for their respective governments? Unless there's a reason we can't repeat the famous infrastructure jobs programs from a century ago - I'm no economist, so I genuinely don't know, but Hoover Dam Mark 2 sounds good to me.

<comment><USER18> > Hoover Dam Mark 2

Is this like the Hoover dam but with 100x longer environmental reviews and 10x more expensive due to special interest grifts that then cripple the investment before it fails 10yrs later? And the only people who made money were a bunch of private consultants hired by the government to do the reviews and executive staff the pseudo-market company behind it?

I haven't heard of a single major infrastructure project that hasn't followed that trajectory. Reading about Obama's two solar projects are a good starting point. But nothing beats Keystone XL.

It's no wonder it hasn't happened and it's not for lack of trying or tax dollars being spent. Even here in Canada it's the same story every time and we have a far more receptive political base for spending the tax money. So I'll never be convinced that's the real hurdle.

<comment><USER17> If the point is to be a jobs program, wasting money and massive overspend are a feature.

<comment><USER18> That was my point. It won't end up in workers hands unless they actually start work on the project. Otherwise it will go to a small group of consultants, gov connected executives, and most importantly endless amounts of lawyers and activism groups.

</comment></comment></comment></comment></comment></comment><comment><USER19> Shopify is majority GCP, minority AWS for its cloud infrastructure [0]. Curious how they are splitting workloads between these two clouds during this extended usage spike.

[0][https://www.digitalcommerce360.com/2018/03/29/shopify- picks-...](https://www.digitalcommerce360.com/2018/03/29/shopify-picks-google- for-e-commerce-site-hosting/)

<comment><USER20> Everything is on GCP

</comment></comment><comment><USER21> Just want to add a datapoint: I run a skincare brand (www.mendskin.co) for men. No discernible difference in traffic, but a significant drop in conversions. I've stopped all Ads because the click through rate actually increased, but the conversions plummeted.

I've been hearing over the past 2 weeks that online traffic and revenue has been soaring for others, but I'm skeptical. I think it's just traditional brick & mortar shops that never had the time nor the impetus to set up an online presence now doing so. Just my 2c.

</comment><comment><USER22> Most of the items I have ordered from non-Aamazon small businesses during COVID-19 crisis is backed by Shopify platform. Their checkout form has a very unique layout so it is easy to tell without looking at the page source.

<comment><USER23> Is Shopify's form the one that has awesome address autofill?

I love that because my street name is like 40 characters long, it doesn't even fit in the legacy IT systems used by utilities.

<comment><USER24> Yes, they do have autofill.

</comment></comment></comment><comment><USER25> There's also the ripple effect of Shopify's data.

We run an omni-channel API connector service for our wholesale distributors to connect Shopify/marketplaces to our own back-end on-premise ERP, and the distributors that have utilized this service are seeing record breaking numbers that are in-line with what Shopify is saying.

The downside? We launched this service less than a year ago and could've showcased some staggering numbers if we launched earlier. We're trying to show the limited data we have to our customers to convince them to pivot to e-commerce while their wholesale channels are suffering.

</comment><comment><USER26> Their stock is booming as well.

This is a fundamental repricing, going forwards ecommerce will become the norm as the slow adopters move onto it quicker. SHOP could be worth way more under a post pandemic economy. This isnt the same economy as three months ago, certain companies are worth more and certain ones are worth less. The money to be made in options is in understanding these repricings and exploiting them, a stock at ATH right now is a stock that may actually grow more in the future, its post pandemic outlook is only in the process of being priced in......

</comment><comment><USER27> question as I've been looking at Shopify direct ship: Is it viable anymore? I mean sourcing from hundreds of suppliers listed on their Oberlo service all the while competing with thousands of others who are trying to sell the exact same products as well as the suppliers themselves selling it at much lower price (same price as their wholesale price) on Alibaba, etc.

Shopify material makes it sound like you can easily sell it for 4x the cost if you spend enough on marketing, etc. Has anyone here have experience setting up dropshipping on shopify recently and making a decent profit?

</comment><comment><USER28> It'll be interesting to see if this results in much extra revenue for Shopify. They have extended their free trial from 14 to 90 days, and if a lot of the traffic is via new merchants, I assume some percentage of them may close the store once their brick and mortar opens back up. (That may be sooner than a full resolution of the pandemic - Texas for example announced retailers can all open back up in a week for pick-up orders)

</comment><comment><USER29> In the past 3 weeks, we have helped over 150 brick and mortar retail stores (our customer base) open Shopify stores. They are not only opening stores, but doing meaningful revenue.

Shopify has not be helpful as an organization, and their processes and red tape have been a challenge. But, it's apparently working for them.

</comment><comment><USER30> We run an e-commerce platform and have seen our traffic/signup numbers jump from initial launch fairly dramatically in recent months, which we can only attribute to the current climate at the moment as we're not yet doing a hard marketing push.

<comment><USER31> which e-commerce platform ?

<comment><USER30> Link is in my profile.

</comment></comment></comment><comment><USER32> I work around stuff related to the online shopping supply chain. We are absolutely seeing black Friday numbers. Really wondering if this forces people out of old habits and into new ones. Retail might never be the same.

</comment><comment><USER33> Might be a good time to revisit the old rumour that Rails doesn't scale.

<comment><USER34> I dont think anyone said that actually meant it doesn't scale. What they meant was it doesn't scale cheaply.

Rails is still great ( or possibly best ) for SaaS or Paid per usage model. Not so good for ad based or traffic driven per revenue model.

</comment></comment><comment><USER35> I'm quite surprised at the lack of comments about the fact that a little over a week ago, Shopify withdrew guidance and now an insider made comments during their quiet period before earnings that materially affected stock price. Elon Musk has been called out by SEC a few times for this type of activity

</comment><comment><USER36> traffic != sales

The whole industry is experiencing a surge in online-window-shopping. Amazon is probably seeing a few blackfridays increase.

<comment><USER37> I work for an ecommerce business and can attest that orders are up in accordance with the headline.

<comment><USER38> Same here. We're seeing very out-of-season numbers. In fact we saw another massive uptick this week that was suspiciously timed with the stimulus direct deposits going out (Tuesday/Wed morn).

</comment></comment></comment><comment><USER39> Really? What are people buying?

<comment><USER40> Good story here

[https://www.visualcapitalist.com/shoppers-buying-online- ecom...](https://www.visualcapitalist.com/shoppers-buying-online-ecommerce- covid-19/)

Food, workout items, entertainment. A lot of categories are through the roof!

<comment><USER41> > workout items

I wonder if gyms will ever come back from this. Gold's announced yesterday they are permanently closing 30+ locations. Even when people go back out, gyms were never the cleanest places to start with. Some places will require masks for the near future, so how does that work at a gym? Now that I'm been forced to finish my home gym, I'll likely cancel my membership once things open again.

<comment><USER42> I'm worried about the YMCA. They do an admirable job of serving the community and poorer populations, but they apparently do so (at least here) by discounting on income.

If they lose any significant portion of the people who could afford to buy home equipment, can they still serve those who can't?

<comment><USER43> I still keep my YMCA membership at the moment. YMCA and Boys and Girls Club had a huge impact on my childhood.

<comment><USER42> Yeah, I'm fine with my small contribution to the cause; it's been very important to me over the last few years, so I'm not in any hurry to terminate my membership.

Whether I'll use it again once it's open, though, is a much trickier question.

</comment></comment></comment></comment></comment></comment><comment><USER44> Strange. I'd expect less traffic from novelty type e-com sites ...

<comment><USER45> I expected the same. Colleague of mine runs a local SaaS to give restaurants an online presence, including online ordering. This week orders went up 60%...because people got their $1200.

Americans don't know how to budget and will buy stuff just because they feel like it.

Edit: I did buy a pricey swing set for my 2 toddlers. But I wanted to buy one last year was denied. This lockdown convinced my wife otherwise.

<comment><USER44> People have it really easy there. I'm from India and the needy are getting $10 per month.

I think its $70 per month accounting for PPP, but even then the difference is massive

<comment><USER46> Citation please. I haven't heared about it.

<comment><USER44> Sorry not $10, more like $7

Source: [https://www.bloombergquint.com/coronavirus-outbreak/not- rs-5...](https://www.bloombergquint.com/coronavirus-outbreak/not- rs-500-indias-poor-need-at-least-rs-3000-a-month-george-mason-university- researchers)

>India announced a Rs 1.7 lakh crore economic relief package >ensuring cash transfers of Rs 500 a month, and food >security. However, the package is less than one percent of >the country's GDP.

</comment></comment></comment></comment></comment><comment><USER47> I think it would be really interesting to collect some data on web traffic before, during, and after this pandemic.</comment>


Air India Sets Record with All-Female Crew World Flight - <AUTHOR> http://fortune.com/2017/03/06/air-india-record-setting-female-crew/?xid=soc_socialflow_facebook_FORTUNE <comment><USER2> Is this really true? Not the fact that this happened but the fact that this is a first! Seriously?

If it is, way to go Air India! That's classy and brilliant!

Edit: The linked CNN article has more details and although the 'record' seems gimmicky, that's still something to be celebrated. I'm still surprised that this is something that needs celebration, rather than just being something commonplace.</comment>


Ask HN: Resources for software project management practices? - <AUTHOR> What are some suggestions for good books or links people have found helpful, for software management practices and insights? Either long-term, or short-term projects.<p>There is a wealth of information out there, no doubt HN can increase the signal/noise ratio a little for valuable resources. <comment><USER2> [http://shop.oreilly.com/product/9780596517717.do](http://shop.oreilly.com/product/9780596517717.do) is a good project management book.

and this is a more advanced one with some good ideas:

[http://en.wikipedia.org/wiki/Critical_Chain_(novel)](http://en.wikipedia.org/wiki/Critical_Chain_\(novel\))

</comment><comment><USER3> I am not the best source, but into mind jumps..

organizing your team: trello, openproject.org, workflowy as a basic knowledge base for recerche

if you need time tracking: clockinit

bugtracker/ticket system: rt from best practical

git: bitbucket

all are free to degree</comment>


Google Fortunetelling – Predict your future - <AUTHOR> http://betagoogle.com/ <comment><USER2> This was the response I got after testing it..[http://betagoogle.com/refugees.html](http://betagoogle.com/refugees.html)

</comment><comment><USER3> It doesn't say on the site, but it's not affiliated with Google - WHOIS info:

        Registrant Organization: BrainMedia</comment>


Preview of the new Erlang site - <AUTHOR> http://demo.erlang.org/

<comment><USER2> I don't mean to sound snarky, but I hope they didn't pay their designer too much. What's with that odd drop-shadow effect to the left and right of the header image? Is it to denote that the header image is bulgy? And what's up with the rest of the drop-shadow effects? What angle is the light coming from?

And why use Google Sitesearch? Surely a custom-written search interface would be a great way to show off the power of Erlang?

<comment><USER3> I wish this were two comments so I could upvote the first paragraph and downvote the second.

</comment></comment><comment><USER4> There are free templates that look better than this.

</comment><comment><USER5> Quick thoughts: Logo in the top header seems really small and cramped in that space. Colors are too dark and monochromatic. Mix in lighter shades of the blues and reds in the middle box.

<comment><USER6> With all respect, the designer tried to deliver an awesome 2.0 web feeling, however they fell short.

I agree, the logo needs some spacing, the "e" especially needs some spacing on the left side. I think a better logo is needed period.

For started the typography is not all that appealing. The default font verdana looks like blah, I think arial looks nicer, at least to me.

Another thing that bugs me is the content area. The contents are not flowing right; it needs to be re-arranged or something. The the gray area with black text on top is not doing it for me, maybe a lighter gray to increase contrast.

Anyway, it's a good start, but a lot of work needs to be done. Good luck.

</comment></comment><comment><USER7> This demo looks really great, is the software driving the site written in erlang?

<comment><USER2> At the bottom of the page it says "Powered by Erlang Web". Erlang Web is a script written in procedural PHP to simplify website maintenance.

...

<comment><USER8> Huh?

Erlang Web is pure Erlang, it doesn't have a single line of PHP. [<http://www.erlang-web.org/>]

<comment><USER2> It was a joke :(

<comment><USER9> I thought it was funny.

</comment></comment></comment></comment></comment><comment><USER10> Hmm, contrary to other comments, when first looking at site my thoughts were "Wow, looks cool. Looks like something modern and active rather than archaic."

The first thing I want, "about/wtf is erlang", is right there on the home page without even having to go to some about page. [Too many modern projects have no intro/assume everyone knows about it]

Two second things I want the most are big and prominent, Documentation and Download

The third thing I want, News, has it's own section.

Everything I want, on the homepage. No clicks, no menus. I think that rocks.

<comment><USER11> Agreed and let's not forget it's a massive improvement compared to the old site.

</comment></comment><comment><USER12> Does not look good. Bad logo.</comment>


I Dreamed of a Perfect Database - <AUTHOR> https://newrepublic.com/article/124425/dreamed-perfect-database <comment><USER2> Some perspective on this subject:

\- A graph database is essentially a relational database plus join recursion. Many modern relational databases support join recursion; support for graph models is ubiquitous, they just don't call themselves "graph databases". There are other reasons graph-like computation is relatively rare.

\- There are some important types of relationships that are effectively not representable in graph data models. Relationships that have a topological nature, such as negative constraints, spatiotemporal, etc are notoriously problematic.

\- General graph-like operations have terrible scalability and performance characteristics as commonly implemented. Consequently, people almost never organize their data models this way unless it is absolutely unavoidable. Even Facebook materializes common traversals as non-graphs so that they do not have to execute them dynamically. The heavy reliance on secondary indexing for performance in most graph databases ensures they will be marginal for large- scale data models.

\- Semantic Web fails at scale because no two people map reality to definitions in the same way. This is particularly obvious at global scales because of cultural influences on how we interpret the world. If you've ever done global data model integrations, and I have, you quickly realize that the only values that are approximately consistent globally are physic-based measurements e.g. "650nm wavelength" (instead of "red"). The nodes have no common definition in practice, just commonly overlapping parts of countless subjective interpretations of the definition, which leads to pollution of the data model in real systems.

<comment><USER3> Hi Andrew - Have you explored holographic algorithms/encodings that use shape to represent elements such that you can succinctly and uniquely define an object/relation in terms of the composition of its constituent parts in a way that enables fast decomposition back into its constituent parts?

<comment><USER4> > Have you explored holographic algorithms/encodings[...]

As far as the wording in mainstream science is, holographic algorithms are something entirely different, see [https://en.wikipedia.org/wiki/Holographic_algorithm](https://en.wikipedia.org/wiki/Holographic_algorithm).

<comment><USER3> Vladimir Kornyak touches on some of these ideas in these papers:

1\. On Compatibility of Discrete Relations (2005) [http://arxiv.org/pdf/math- ph/0504048.pdf](http://arxiv.org/pdf/math-ph/0504048.pdf)

2\. Structural and Symmetry Analysis of Discrete Dynamical Systems (2010) [http://arxiv.org/pdf/1006.1754.pdf](http://arxiv.org/pdf/1006.1754.pdf)

3\. Discrete Dynamical Models: Combinatorics, Statistics and Continuum Approximations (2015) [http://mmg.tversu.ru/images/publications/2015-vol3-n1/Kornya...](http://mmg.tversu.ru/images/publications/2015-vol3-n1/Kornyak-2015-01-05.pdf)

On a somewhat related note -- applying quantum principles to graph database processing -- see this new paper by Marko Rodriguez (the creator of the Gremlin graph programming language):

4\. Quantum Walks with Gremlin (2015) [http://arxiv.org/pdf/1511.06278v1.pdf](http://arxiv.org/pdf/1511.06278v1.pdf)

</comment></comment></comment></comment><comment><USER5> Graph databases are exciting, but I'm far more interested in the potential of append-only stores. Rather than recording data at all, you store events (item added, item deleted, etc.).

This allows auditability and for you to look back at the state of the data at any time, but the largest benefit, in my opinion, is that it decouples data from its data structure. This allows you to treat data structures like "caches" that are efficiently structured for how they will be used. If you want, you don't have to choose between relational databases or graph databases or anything else: you can play the same set of events into different structures and query the appropriately structured database for the kind of query you're doing. It also allows you to implement security at the data storage level in a very simple and granular way: you can reject events based on predicates which update as themselves as they receive modifications to the permissions, and distribute filtered streams of data to users based on what events they are allowed to see. Overall, the power of this method is very large.

<comment><USER6> Datomic much? :)

<comment><USER7> For folks who are interested, some other examples are the Kafka/Samza ecosystem and blockchains like Ethereum.

</comment></comment></comment><comment><USER8>
      All of these tables relate to one another

      Where all this gets particularly interesting is wherever,
      by traversing the relations between table...

      These are the merits of the "relational" database.

That's not what "relational" in "relational databases" means.

He even links to Codd's paper which defines a "relation" explicitly:

      The term relation is used here in its accepted mathematical sense.
      Given sets X1 , S, , . . . , S, (not necessarily distinct), R is
      a relation on these n sets if it is a set of n-tuples each of which
      has its first element from S1, its second element from Sz, and so on.
      ...

</comment><comment><USER9> Sorry, this is a terrible article. It reads more as a stream of consciousness about his own mistakes and sad regrets, and does not add any new or interesting insight into anything regarding databases.

There are plenty of new cool things we can do now - and the graph database we're building ( [http://github.com/amark/gun](http://github.com/amark/gun) ) is doing several of those things. Like realtime by default, like what Firebase and RethinkDB are starting to push. Fully decentralized and fault tolerant, like what Riak and Cassandra tried to do. Graph (and not just triples) so you can have _relational documents_ as ArangoDB and Neo4j allow. And absolutely totally offline-first, like what Pouch/CouchDB wanted to pioneer.

The truth is the world of databases are only getting better and more exciting - in large part due to new algorithms like CRDTs that push what we can do. GUN is one player in that along with others. But this article? Just depressing and overlooking all the new opportunities.

<comment><USER10> I often see you plug your project, Gun, whenever someone posts about distributed databases. You always write about how great it is. But every time I look at your repo, it's still the same few hundred lines of, I'm sorry, quite shoddily written JavaScript, with huge chunks clearly being temporary placeholders or even commented out. You even have scratch test data in there, intermingled with your main code. Your web site also make some grandiose claims about how it solves certain synchronization problems, but I can't find any technical explanation of exactly how it is supposed to work.

The lack of code or documentation means it's very hard to take your comments seriously. But please correct me if I'm mistaken.

<comment><USER11> I attended @marknadal's talk at Distributed Matters Berlin ([https://2015.distributed- matters.org/ber/speakers/#284925727...](https://2015.distributed- matters.org/ber/speakers/#2849257271562)). It promised a lot.

After the talk, I was left with many unclear messages, lots of promises, few detailed explanations, IMVHO many flawed statements, and only one clear outcome: gun is a "script" (I wouldn't call it database) that offers "conflict resolution" based on data's lexicographical order (!!!).

I just probably understood everything wrong, or I'm simply inexperienced enough to appreciate gun. My apologies in advance.

<comment><USER9> I'm sad to hear I did a poor job explaining the concepts, thank you for this honest review. Next time around, I'll see if I can be more clear.

One minor detail - while lexical sort is used in the conflict resolution algorithm it is only part of the equation (see [https://github.com/amark/gun/wiki/Conflict-Resolution- with-G...](https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns) ). As I mentioned in my talk, lexical ordering is intentionally naive because I rely upon deterministic behavior ( _not_ PAXOS, RAFT, consensus, or gossip protocols) which guarantees convergence.

<comment><USER11> Maybe it was just me. However, if my feedback would help, I'd say that there was a lack of technical detail of the stuff mentioned. It was only covered from a superficial point of view, and many different complex issues where mentioned somehow together and lightly.

But the way, why do you easily discard consensus protocols and rather rely on conflict resolution?

<comment><USER9> What is the best medium for presenting more technical details? In my talk I attempted to use stories (which is a very laymen approach) to explain algorithms - this may have been too high level. So instead, what is easiest for you to digest and verify? Actual code and working samples/demos that prove the behavior? Mathematics? Case studies by large customers?

\</comment></comment></comment></comment></comment></comment><comment>-

I dislike consensus protocols because they are difficult to scale. Deterministic algorithms however are not. Why? Consensus requires communication, and communication takes time. As you add more peers, you then have to do more communication in order to maintain consensus. But as more communication occurs, things bottleneck and get even slower. However _some problems require consensus_ (like finances, traditionally) and therefore GUN would not be a good choice.

Deterministic algorithms only need the same inputs and they then spit out the same output as any other machine (running that algorithm) anywhere in the universe. This is why "immutable data structures" are all the rage lately. This is incredibly scalable because as long as the inputs are received (which might have been sent slowly over the network) then all the databases will converge to the same result (for the same inputs) regardless of whatever current state those databases might be in. And because a database maintains state, this type of guarantee is really important for databases.

WARNING NOTE: Missing input causes the databases to be temporarily out of sync because their inputs are different, however when the inputs are finally received (via retries or what not) the databases will sync up _regardless_ of the ordering. Making sure the inputs can be sent in any order or retried is the idea of idempotency and is explored more with CRDTs and commutativity. This means GUN is "Eventually Consistent" and not Strongly Consistent, so don't use GUN in those scenarios.

</comment><comment><USER12> "without the motive power of capital, things move slowly"

 _slowly_?! We've amassed huge collections of detailed knowledge in mere decades, and it's all searchable and discoverable.

"not one big pool of knowledge"

The real fantasy is centralization and taxonomies of everything. Distributed isn't a lazy thing we've settled on; it's the best approach. Each island grows about as far as it makes sense, and connects to other islands where it makes sense.

Sure, it may be imperfect and some datasets are hidden in corporations, but open data is certainly a healthy and growing ecosystem in no danger of extinction.

<comment><USER13> > The real fantasy is centralization and taxonomies of everything

Precisely. Currently there are plenty of graph-based databases to choose from - GraphDB, ArangoDB, Neo4j, Allegro, etc etc. They are all pretty good, and some also support rule-based inferencing, i.e. you can put a rule in the database like "if X is a person and belongs to team Y and the manager for team Y is Z, then Z is X's manager".

Systems like that have a few disadvantages:

1\. They absolutely implicitly trust all data you put in them. If in the above example you add Z as a member to team Y, then it will infer that Z is Z's manager.

2\. You cannot assert negatives, nor can you search based on negative predicates.

The first is the absolute killer for such technologies if you try to deploy them internet-wide and gather "data" from any random source publishing RDF documents.

<comment><USER14> ad 1: why not attach a credibility score to relations (probably aggregated from multiple sources) and query against those too?

<comment><USER4> What inference rules for unreliable data do you suggest?

<comment><USER15> is there something wrong with inductive logic? [https://en.wikipedia.org/wiki/Inductive_reasoning](https://en.wikipedia.org/wiki/Inductive_reasoning)

<comment><USER16> Inductive logic needs an implementation. These are things like Bayes Nets, Inference using Gibbs sampling, PSL, etc.

You can sorta, mostly get them to work if you have a clean graph, and are prepared to spend a long time debugging.

The problem is that no one has managed to get them to work well enough to be useful on dirty, web-scale knowledge graphs.

</comment></comment></comment></comment></comment></comment><comment><USER17> > There is as yet no absolute challenger to the relational model. When people > think database, they still think SQL. But if there is a true challenger, it > is in the graph model. Because graph data structures power social networks, > and social networks are the dominant technological organism of the era.

You can use a fact database model (like Datomic's) and then you don't have to choose between row, column, document and graph based databases, those are projections on top of some index.

If entities are universally unique and attributes namespaced you can get a "database of everything". Or you need at least a mapping between entities from different vendors. That's the hard part :)

</comment><comment><USER18> I'm old enough to remember being excited about FOAF too.

The idea came out of the world of blogs; if you have a blog, so the thinking went, you basically have an online identity endpoint. So if there was a simple way to describe relationships between those endpoints, you could write software to follow those relationships, and then... _voila!_ A completely open social network.

The dream collided unpleasantly with reality in two places. First was the problem that running a blog implies a commitment to continual writing that most people are never going to make, so when your first step is "set up a blog somewhere" people think you're asking them to make a huge commitment and run away. And second, FOAF was built on RDF; RDF makes my brain hurt just thinking about it, and I'm a professional nerd! (The refrain was always "it doesn't matter how complicated the format is, users will only interact with it via tools that make it simple." But every time I hear this about a format it reminds me of all the other formats I've heard that about in the past, all of which are deader than doornails now.)

</comment><comment><USER19> Perhaps I'm stereotyping, but whenever I read a love letter to graph databases, it usually begins with taking MySQL as the limits of the relational world.

</comment><comment><USER20> The real power of relational databases is:

* it's fantastic for many kinds of queries you didn't have in mind when you created the data model

* it's good enough for most use cases

* it's been around long enough that safety and correctness features like transactions, foreign keys, UNIQUE and NOT NOLL constraints etc. are well implemented and understood.

Any DB model that wants to challenge the ubiquity of RDBMS must address these points, and offer something on top. So far I haven't seen anything that would come close. Sure, there are lots of specialized use cases and specialized tools that are great in some areas (like geospatial data, graphs, time series, ...), but nothing that I'd use as a default choice for a project with vague requirements.

</comment><comment><USER21> I'm in love with Paul Ford's writing about technology. His What Is Code? cover story for Business Week was amazing: [http://www.bloomberg.com/graphics/2015-paul-ford-what-is- cod...](http://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/) [https://news.ycombinator.com/item?id=9698870](https://news.ycombinator.com/item?id=9698870)

This is just a short piece, but I think it is so interesting to try to introduce non-technical readers to some of the knotty problems that everyone on HN takes for granted. And, the fact that the world's largest graph database (Facebook's) is built on top of a SQL database really is ironic, and it's great that he succeeds at explaining all the concepts well enough to get that across.

For dinosaurs like myself who still use an RSS reader (I'm a big fan of Inoreader), here is a full-text feed of his New Republic writing, made with Feedity and Five Filters: [http://ftr.fivefilters.org/makefulltextfeed.php?url=https%3A...](http://ftr.fivefilters.org/makefulltextfeed.php?url=https%3A%2F%2Ffeedity.com%2Fnewrepublic- com%2FVFRWVFJa.rss&max=3)

</comment><comment><USER22> I think the future "may" be the JCR Standard (Java Content Repository), or at least something build 'on top of' JCR. The only real contender IMO is Apache Jackrabbit Oak. In general what this article describes is a Content Repository, but a real world-wide semantic web has not yet been built with it. My little project meta64.com is a mobile front end for some of this kind of data processing/storage.

</comment><comment><USER23> "There is as yet no absolute challenger to the relational model. When people think database, they still think SQL. But if there is a true challenger, it is in the graph model."

This article is quite biased towards graph databases with regards to the SQL versus NoSQL tension. This video presents a much more balanced view of SQL versus NoSQL, in my opinion. [https://www.youtube.com/watch?v=qI_g07C_Q5I](https://www.youtube.com/watch?v=qI_g07C_Q5I)

</comment><comment><USER24> This article doesnt address what is useful as we migrate into the cloud. First, the data may be larger than any disk image in the cloud. The data could be distributed across hundreds of disks in disparate geographical locations. Plus there may be hundreds of process or services reading and modifying the data. Certain NonSQL architectures seem to work better then.

</comment><comment><USER25> A guy writing about "graph databases" without once mentioning IMS, hmm [http://www-01.ibm.com/software/data/ims/](http://www-01.ibm.com/software/data/ims/)

</comment><comment><USER26> I woke up and find riak, hope they make search on by default but one could argue that perfection is not a thing.

</comment><comment><USER27> Check out Brodlist [http://brodlist.com](http://brodlist.com)</comment>


Show HN: Bt – BitTorrent library in Java 8 - <AUTHOR> https://github.com/atomashpolskiy/bt/blob/master/README.md#----bt <comment><AUTHOR> There are many great BitTorrent clients out there, yet I've made my own and willing to share and attract collaborators!

<comment><USER2> Just curious, how many hours do you think you've spent on this project?

<comment><AUTHOR> I'd say something in the range of 500-1000 hours.

<comment><USER3> That's less than I was expecting. Did you study any existing library to get up to speed?

<comment><AUTHOR> No, after I had studied BEP-3, wiki and and a few papers (most importantly "BitTorrent economics"), I had a very clear understanding of what needs to be done.

</comment></comment></comment></comment></comment><comment><USER4> Thank you, your work is terrific! Good docs, good code, and I especially like the ability to create custom code to write to alternative storage.

<comment><AUTHOR> Thanks!! Yeah, I was pursuing such design that would allow to replace/customize most of the parts, including the internals. As a bonus, it makes it easier to continue working on the library as the codebase grows (now nearing 25K lines of code)

</comment></comment><comment><USER5> Man this looks slick, going to give it a shot. Love the ability to be able to download a magnet link from the CLI and not worry about what some fancy GUI is doing in the background cuz I am paranoid.

<comment><AUTHOR> Thank you! Make sure to install [http://www.oracle.com/technetwork/java/javase/downloads/jce8...](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html) to allow 160-bit cryptography and use -e flag to be completely safe!

I also have a simple shell script locally, that wraps the java command, so I have to just type `btmagnet AF0D9AA01A9AE123A73802CFA58CCAF355EB19F8` to download to a pre-determined location (hardcoded in the script). I wonder if I should share it on the CLI README page

<comment><USER6> > Make sure to install > [http://www.oracle.com/technetwork/java/javase/downloads/jce8...](http://www.oracle.com/technetwork/java/javase/downloads/jce8..). > to allow 160-bit cryptography

Or don't use the Oracle proprietary distribution but the OpenJDK of your operating system which normally doesn't exclude proper crypto algorithms.

(be aware that the JDKs downgrade silently if certain libraries are missing in your host)

<comment><AUTHOR> That's some really useful information, thanks! Need to add this to the readme

</comment></comment></comment></comment><comment><USER7> It looks great but the top GIF makes it seem like you are offering a command line BT client. If I were you I would have it slowly scroll through the code of the client in the GIF or put a few frames of code of the client and or some compiling and then show the current top GIF.

<comment><AUTHOR> Oh, I didn't expect this :) Sounds good, but I'm not sure I have patience do this right, it's such a pain to record gifs - I always misspell or misclick something, and by take #47 go completely nuts

<comment><USER8> Try using asciinema. It records your terminal session and works so well. Of course typos and such are still an issue...

</comment></comment></comment><comment><USER2> Nice work. I'm wondering, how would you test/benchmark such an application. Besides manual testing on a variety of torrents, how would you do regression test there? same question for benchmarking. How to compare this work with another client?

<comment><AUTHOR> I have several hundred unit tests and also a bunch of integration tests. While UTs usually test API of individual classes, each of the ITs creates a swarm of local peers and launches a torrenting session with certain conditions: seeders to leechers ratio, downloading from .torrent file vs using a magnet link, PEX enabled/disabled, encrypted vs raw message streams, etc.

Benchmarks (like in libtorrent) is something I'm looking forward to, but the project is still in the early stage -- there's a lot to be done with regards to optimization, i.e. switching to using NIO selectors, adding a caching layer, etc. That's a lot of work, and that's why I'm looking for collaborators, before investing more time into performance-tuning.

<comment><USER9> I would advise against implementing a custom cache for piece data. Caching file data in-app is counterproductive because it wastes memory duplicating data which the OS already holds in its page cache. Instead memory mapped I/O should be preferred.

<comment><AUTHOR> It makes sense, what would you advise in case of big files that don't fit into memory (or when it's undesirable to use big amounts of RAM, and speed can be sacrificed)? Maybe caching should be disabled by default but with the option to turn it on if the user wants it?

<comment><USER10> you can memory-map files in chunks and keep references to the mapped buffers or evict them once a certain limit is reached.

note that mapped buffers are not necessarily backed by RAM, they act more like swap space, except it goes straight to the original file instead of a swap partition. the downside is that anything reading from the buffer can unpredictably incur IO overhead if there's memory pressure and the OS decides to not back them with memory. java doesn't have an api to check residency of buffers. a possible workaround is to first queue them up on a separate thread and forcing them in before using. if they're already there it should have low latency. if they aren't then only the victim thread eats the IO penalty.

<comment><AUTHOR> So, something like LRU cache of MappedByteBuffers, one buffer per chunk (i.e. "piece")? I wonder what queueing strategy would look like, just enqueue a couple of subsequent pieces when one is requested and loaded?

<comment><USER10> Start simple, then heap on optimizations if there are latency or throughput issues.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> Really like the approach around using Guice as a central point for hooking your own code into the system.

Will look forward to reading through the code later!

<comment><AUTHOR> Thanks! God bless IoC, it saves my ass each time I make some silly design choice

</comment></comment><comment><USER12> I'm curious, it looks like the screenshot shows the author is using a Mac. But Mac OS hasn't shipped with Java in several years. So why Java? Is it simply familiarity, or are there other reasons?

<comment><USER13> Most OS's don't ship with a C compiler, go, python 3, rust, arc, etc, what does that have to do with anything?

<comment><USER14> You're conflating developer tools with a language runtime. Every popular OS ships with everything necessary to run a C, Go, or Rust binary. Python 3 and Java both require a separate runtime to be installed, but I will say that every Java CLI I've used has had terrible startup performance.

<comment><AUTHOR> I'm using Lanterna, a Java library for creating text-based terminal GUIs ([https://github.com/mabe02/lanterna](https://github.com/mabe02/lanterna)), and the startup is 1-2 seconds. Can't recommend it enough

<comment><USER14> Good to know, but 1-2 seconds is still very slow compared to a native app. I have an 8MB Go CLI application that can startup and print its help in .05 seconds, and even that is probably slow compared to a C or Rust equivalent.

<comment><USER15> Any savy Java developer knows how to AOT compile to native code, if that really matters.

There are plenty of JDKs that support it, and even Oracle is finally adding support for it with Java 9, initially only for Linux x64.

By Java 10 timeframe no one that only knows about OpenJDK and not the several other JDKs can state that Java starts slow vs C, because both will be AOT compiled to native code, loading native .so files.

<comment><USER16> From what I've seen Java/JVM AOT won't produce binary executables but shared libraries that will be loaded on startup by the JVM. When I first read about Java AOT-compilation I expected it to produce real executables that's why I think this is important to mention. But please correct me if I am wrong. So if this is true even with AOT compilation you still need the JVM as dependency (although I think you can get a minimal JVM in under 10MB). It also seems that the AOT compiled code needs to be recompiled on Java updates. Although I quite like Java and the JVM, I still wouldn't choose Java for writing simple command line tools.

Nevertheless it is quite useful for improving startup for tools like Gradle or server applications. But sure, I also don't expect Java startup performance to be a problem in the mid- to long-run.

<comment><USER15> Correct, but that is no different than having to ship glibc.so or MSVCRT.dll alongside your application, specially if you want to be certain which version gets used.

This is only relevant to Oracle's implementation though.

Other JDKs do support static linking, as long as you restrain yourself from features that could require dynamic compilation.

There was a talk at JVM Languages Summit about the current state of AOT compilation regarding Java 10.

[https://www.youtube.com/watch?v=n5DCg6M2MDM&index=11&list=PL...](https://www.youtube.com/watch?v=n5DCg6M2MDM&index=11&list=PLX8CzqL3ArzXJ2EGftrmz4SzS6NRr6p2n)

<comment><USER14> I would rather ship glibc (8MB) or musl (.5MB) than the JVM (100s of MBs?), but it's not necessary given that nearly every target already has a libc. Also, Go programs don't depend on libc, and C and Rust can probably statically link against some libc in a pinch.

<comment><USER17> Rust can use musl, yes.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> Amazing work, how long did it take to build? Where you working on it full-time or part-time?

<comment><AUTHOR> Thank you! Calendar year, in the evenings and on weekends, with sprints of several weeks and a month or two of rest/AFK after each sprint to keep sanity (I have a full-time job, a wife and a 3 year old kid). Happy that I had the persistence to keep working on the project

<comment><USER18> Very impressive! Thanks for sharing, it's encouraging to me and I hope to others to know that with proper time management we can put out amazing work.

</comment></comment></comment><comment><USER19> I remember using Java based client Azureus and the memory usage was huge. Overnight it would leak to about 1Gb. Later uTorrent appeared and it used 2mb... Forever

I guess nothing to do with this but was triggered by two combined words: Torrent and Java

</comment><comment><USER20> It's great to see projects like this on HN. Congrats man, well done!

</comment><comment><USER21> Hello, thank you for writing the library.

On my context it is useful for downloading large files without depending on a single point of failure or fixed location.

Have a question in regards to the example syntax:

client.startAsync(state -> { if (state.getPiecesRemaining() == 0) { client.stop(); } }, 1000).join();

This is a feature from newer Java syntax that I'm not yet familiar enough with lambdas to understand in full what is happening.

Perhaps I can ask for an example that replaces the "->" into the older Java syntax? I know this would likely come at the cost of a few extra lines, it would certainly ease a lot for many developers looking at the example. Many thanks.

<comment><USER22> In the older syntax, something like this:

      client.startAsync(new Consumer<TorrentSessionState> {
    	void accept(TorrentSessionState state) {
    		if (state.getPiecesRemaining() == 0) {
    			client.stop();
    		}
    	}
      }, 1000).join();

<comment><USER21> Thank you. Really useful.

</comment></comment></comment><comment><USER23> Wow thanks man! This can be useful in a variety of circumstances!

</comment><comment><USER24> Not very familiar with the Java world yet. I've heard Android now supports at least some features from Java 8, but would this library work in an Android app?

<comment><AUTHOR> I guess not. There is a ticket for backporting this project to Java 7 or RxJava ([https://github.com/atomashpolskiy/bt/issues/7](https://github.com/atomashpolskiy/bt/issues/7)), but I personally think it's not worth it.

</comment></comment><comment><USER25> It's amazing how this made it to the front page of HN and I had no idea until now. I actually discovered this by googling a couple hours ago :)

</comment><comment><USER26> Wonderful Lib!

</comment><comment><USER27> It's possible in Azureus, but it might be a lot of effort to turn the functionality into a library.</comment>


Blue Apron falls 9% on fourth day as a public company - <AUTHOR> https://techcrunch.com/2017/07/05/blue-apron-falls-9-on-fourth-day-as-a-public-company <comment><USER2> These companies are going to get absolutely destroyed when the big supermarkets come on tap.

Several are working on identical services where you pick up, curbside, the ingredients and instructions in a bag, and pay significantly less for doing so.

Why have your ingredients mailed to you when a supermarket on your way home will offer the same thing for 50% cheaper?

<comment><USER3> While I don't disagree, people said the same thing about Amazon.

The supermarkets have changed very little about the way they've done business in the last half-century, and as such I have a difficult time imagining them doing a good job executing on something like this.

I've also tested the majority of the meal and food delivery services out there, and my experience meshes well with the above: the local supermarket chain in some ways provides a better offering than anyone (as they should–you're absolutely right about that), but for a few years now they've persisted in making sure the actual execution is the worst I've experienced of any service. I can't see that changing until they completely overhaul their hiring practices and put sufficient processes in place to ensure good execution.

(For the record, Blue Apron strikes me as a pretty mediocre product at a pretty uninteresting price, executed in a pretty bad manner.)

<comment><USER4> >> While I don't disagree, people said the same thing about Amazon.

just to play devil's avocado, Amazon's advantage was not having the overhead of stores, now they just purchased whole foods. Seems as if they are moving towards being a supermarket

<comment><USER3> Sorry, I was referring to how it was obvious to so many that Barnes & Noble and Borders would crush this fledgling website founded by the kook who set out in a van to compete with them.

<comment><USER5> There was some of that... generally, skepticism that internet startups would become large enough to challenge businesses like Walmart, ever. [https://www.forbes.com/sites/ericjackson/2015/09/24/revisiti...](https://www.forbes.com/sites/ericjackson/2015/09/24/revisiting- amazon-dot-bomb-16-years-later/#6b40b816b08f)

And also people saying things like tiny businesses of people with bookstores in their spare time will make Amazon obsolete... [https://mobile.nytimes.com/1999/02/26/opinion/foreign- affair...](https://mobile.nytimes.com/1999/02/26/opinion/foreign-affairs- amazonyou.html)

Ah, Thomas Friedman. Thanks for the advice to not buy amazon stock in 1999.

<comment><USER6> Out of curiosity, I went to have a look at AMZN's stock price at the date the article was published, to see how the stock price during the heady heights of the dotcom bubble compared to today.

Interestingly, if you view the entire stock history price chart, the dotcom bubble is barely a blip.

<comment><USER7> IMO, the right way to look at long-term performance of an investment is on a semi-log chart. (Day or week charts, I prefer linear; anything 3 month or longer, I prefer semi-log.)

Semi-log (where constant percentage changes are constant-sized), it's more than a blip. Linear (where constant dollar changes are constant-sized) charting hides past market moves if the overall bias has been up and exaggerates past market moves if the overall bias has been down. This can lead investors to improperly discount the pain/gain cycle that happened over that time period.

Linear charting: [http://bit.ly/2tk5Ulj](http://bit.ly/2tk5Ulj)

Semi log: [http://bit.ly/2sJs7ec](http://bit.ly/2sJs7ec)

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The difference between companies like Blue Apron and Amazon is that one is created with a "I think I want to do a startup, any startup that's in vogue nowadays. I know people who can take me to IPO" mindset and the other "there is a clear business opportunity in bringing efficiencies in the retail books business. The time is right since all the underlying infrastructure (PCs, wide internet access, Credit cards, USPS/UPS, etc.) needed for such an opportunity to succeed are present; all it needs is a strong execution path and leadership to have a go at it" mindset.

<comment><USER9> I call this "playing startup."

Edit: possibly not my own thought, for whatever that means. I just don't want to accidentally suggest to others that I steal thoughts. See child comments.

<comment><USER10> oh really? you sure it's you and not Paul Graham that calls it that? like don't you realize you're literally on a site started by the guy that coined that phrase: everyone here is already familiar with it and the correct attribution.

<comment><USER9> I'm not here to convince you of anything, but I will say with all genuine honesty I don't know who Paul Graham is. I'm here because I got tired of reddit and my friends said there's more tech news here.

I also think that "playing startup" is so obvious that many people could have come up with it. That being said, it's also quite possible that I saw it somewhere and absorbed it without fully remembering the source and now have a false memory of coming up with it myself.

I will add though that it's pretty fascinating as someone who is so disinterested in (and at times has a disdain for) startup people and startup culture, to be accused along the lines of, "how dare you feign not knowing who Startup Person X is!"

<comment><USER11> Surfing Hacker News without knowing who Paul Graham is, is like shopping at Amazon without knowing who Jeff Bezos is. There's absolutely nothing wrong with it but it seems a bit odd to some of us. That doesn't make it okay to be rude to you.

<comment><USER12> Would be willing to bet you less than 1% of Amazon shoppers worldwide know who Jeff Bezos is...

</comment></comment></comment></comment></comment></comment><comment><USER13> Does anyone actually use Blue Apron? They seem to do promotions all the time and yet, I don't know anyone who actually uses their service.

<comment><USER14> In the last big Blue Apron thread I participated in, I found it a bit suspicious how many people were just rabid about the company, the product, and the service and how it was "simply gourmet and such better food than [insert fast casual chain]" and "how dare you compare the quality!", etc.

I'm not so paranoid to think that the whole crowd were interested/biased parties, but I can't help wonder if they've convinced themselves that the absurd cost is actually worth it. I've tried it and I don't find the quality to even as good than that of say, Modern Market or some other slightly-upscale eatery.

<comment><USER15> Blue Apron seems to inspire surprisingly passionate discussions here. Detractors tend to be very negative and even a bit hostile, and I suspect that maybe puts fans of the service on the defensive or makes them feel like they have to praise it more than it really deserves. So far this thread is very civil though.

I've tried it and enjoyed it. I found it to be a high-quality service. Negative points were (1) trash; (2) I didn't like most of the vegetarian meals, and I'm trending towards a more vegetarian diet; (3) I prefer to cook simple meals at home that generate leftovers (so that I don't have to cook every night), but BA meals tend to be complex, messy to prepare, and do not create any leftovers. These three points were easily sufficient to stop me from using it long-term.

(1) is a topic of a lot of argument. Some insist BA creates mass amounts of waste compared to normal grocery shopping. Others contend the difference is much smaller or even nonexistent when looking at "average" grocery shoppers. I'm in the middle. I think the waste is exaggerated, but still greater than what the average shopper would produce.

<comment><USER7> We used it for about 6 months (or maybe a little more) a few years ago. I got it for my wife who enjoys cooking and enjoyed the "semi-forced" variety. Several of the meals (or close variants) are now in our regular rotation at home.

I'd say we were glad on balance that we tried it.

Cons:

Incredibly expensive for what you get. (Obviously; they're paying a large amount for shipping and a large amount for marketing.)

Ingredients were generally acceptable quality, the chicken was quite good but the beef was pretty uniformly terrible. (I often thought back to Rodney Dangerfield's quote in Caddyshack: "this steak still has marks from where the jockey was hitting it!")

After a few months, the variety/novelty factor wears off and then it just becomes an incredibly expensive way to get a small amount of groceries delivered.

I don't want to seem like I'm grinding an axe, but it also annoyed me greatly that there's no web UX accessible way to cancel your account online. (You have to call and talk to a human.) Solution: google for the hidden cancellation page or just remember every few weeks to google "blue apron" or "plated" and click on a paid ad to bring you to the site and cancel the deliveries for the next six weeks. :) That makes their churn numbers look lower, but their SEM expenses higher...

<comment><USER16> Weird, I canceled Blue Apron twice from their website. Maybe they removed that because it was too easy to cancel.

</comment></comment></comment></comment></comment><comment><USER17> I find it interesting that none of the discussion here has been around CSAs. Are these less of a thing, even in other cities? In the D.C. area there are _tons_ of them. They provide really great variety and I suspect many also include recipie ideas with each box (the one my wife and I used did). Most will deliver to nearby farmers markets and in our case they had enough customers in our apartment building to deliver there directly. We learned so much about different types and combinations of ingredients in that time. It was a lot of fun and made for some delicious meals.

<comment><USER18> It could be that there's no discussion of CSAs because nobody here knows what they are. I certainly don't. Can you explain? Searching, I get Client-Server Architecture, Community Supported Agriculture, Compliance Safety & Accountability, and more. I assume it's the second one? What does it involve?

<comment><USER17> Adding to undersuit's response, here's a wapo article about the CSA scene in the D.C. area (in 2015): [http://wapo.st/1AlQgSF](http://wapo.st/1AlQgSF)

Basically you pay a set amount directly to the farm ahead of the harvest season and get weekly boxes of a variety of different veggies (and some bigger ones will also do meats and cheeses). Typical they cost around $30-40/week in this area, but can also be considerably more than that. Usually pick up at a local farmers' market is necessary. We lucked out with delivery to our door so some added value in that case for people in larger cities.

</comment></comment></comment><comment><USER19> I find the whole concept a bit sad in a way. Kind of like Russian mail order brides or something.

</comment><comment><USER20> RCSS does home deliveries from the app now

</comment><comment><USER21> We use one of their (numerous) competitors.</comment>


Pittsburgh cancer center warns of cell phone risks - <AUTHOR> http://hosted.ap.org/dynamic/stories/C/CELL_PHONE_WARNING?SITE=WIRE&SECTION=HOME&TEMPLATE=DEFAULT

<comment><USER2> Linkbait: It's not just for kids! Sometimes, respected scientists get into the act.

I'm impressed, though -- this is a much better article than the title would suggest. It does a good job of running down the evidence that this guy is being hysterical for no good reason. All the old data -- and there's quite a lot of it -- is inconclusive, and there's no new data:

 _Herberman is basing his alarm on early unpublished data. He says it takes too long to get answers from science and he believes people should take action now - especially when it comes to children._

"Early, unpublished data." Perhaps we should also design some cold-fusion power packs for our children.

Interesting how it's only the answers that _agree with this guy's gut feeling_ that take such a long time to be found by science.</comment>


Russian Hackers Leak U.S. Files from Doping Agency - ezequiel-garzon http://www.nytimes.com/2016/09/14/sports/simone-biles-serena-venus-williams-russian-hackers-doping.html <comment> ezequiel-garzon I have a hard time believing this wasn't sponsored by the Russian government in retaliation for the ban on Russian athletes for the Rio Olympics. The digital battlefield could get nasty.</comment>


RUB researchers compile Milky Way photo with 46 billion pixels - <AUTHOR> http://rubin.rub.de/en/largest-astronomical-image-all-time <comment><USER2> Is there a downlink link anywhere?</comment>


Ask HN: Best country (apart from USA) to hack in the summer? - <AUTHOR>

<comment><USER2> If you like Japan, Japan is great. If you like Sweden, Sweden is great. If you like Peru, Peru is great.

What exactly are you asking?

North Korea may be the only country bad for a hack-cation. No Internet there.</comment>


Your bank account: The next thing to go obsolete - <AUTHOR> http://money.cnn.com/2015/06/08/technology/abra-bank/index.html This article appeared on the CNN homepage. Abra relies on the Bitcoin Blockchain. What I found particularly interesting is neither the article nor the company's site, http://goabra.com, make any mention of Blockchain or Bitcoin.<p>This must be intentional, which I find very interesting. <comment><AUTHOR> The article itself was shallow. I found it interesting that the featured startup, Abra, is a company that relies on Bitcoin Blockchain technology yet the article makes no mention of Bitcoin or the Blockchain -- and neither does their site.</comment>