BSD socket API revamp - <AUTHOR> https://raw.githubusercontent.com/sustrik/dsock/master/rfc/sock-api-revamp-01.txt <comment><USER2> Am I missing something? It seems to propose to ditch all non-blocking APIs under the assumption that "runtimes" will provide lightweight concurrency. But AFAIK most (all?) runtimes that provide lightweight concurrency make extensive use of the non-blocking network APIs to work - including Golang, that is even cited as an example:

       During the decades since BSD sockets were first introduced the way
       they are used have changed significantly.  While in the beginning the
       user was supposed to fork a new process for each connection and do
       all the work using simple blocking calls nowadays they are expected
       to keep a pool of connections, check them via functions like poll()
       or kqueue() and dispatch any work to be done to one of the worker
       threads in a thread pool.  In other words, user is supposed to do
       both network and CPU scheduling.
       [...]
       To address this problem, this memo assumes that there already exists
       an efficient concurrency implementation where forking a new
       lightweight process takes at most hundreds of nanoseconds and context
       switch takes tens of nanoseconds.  Note that there are already such
       concurrency systems deployed in the wild.  One well-known example are
       Golang's goroutines but there are others available as well.

<comment><USER3> I'm not sure we have a sufficiently efficient concurrency mechanism that doesn't simply give the user control over scheduling decisions.

Depending on how many tens we're talking about, those tens of nanoseconds for a context switch might themselves blow your time budget even on ordinary hardware. 10 gigabits per second means one byte is about 800ps long, so a 64-byte ethernet frame (plus the gunk before and after) is about 7 tens of nanoseconds. To keep up with this traffic pattern, we need to be able to process packets at least this quickly.

A goroutine context switch plus one unbuffered channel read and one unbuffered channel write on my machine seems to take about 18 tens of nanoseconds on my machine when Go's runtime is made to use the same thread for everything. (I'm not sure how to better isolate context switch overhead in Go, but a context switch, channel read, and channel write per packet does seem to be the anticipated use case.)

<comment><AUTHOR> The question to ask is: Can you do better?

Say you use a state machine instead of using coroutines. Will one state transition of the machine be faster than one context switch?

180ns can definitely be improved on (with the C implementation in the project it's more likely to be 60ns or so) there's no fundamental reason why one approach should be faster than the other, given that both are doing basically the same thing, albeit on the different layers of the stack.

<comment><USER3> I'm not clear on why coroutines or an explicit state machine are necessary to implement network protocols. We have been doing it for decades by writing straightforward imperative C code and for years by writing C++ template classes that are templated on the downstream type (CRTP). Both of these approaches result in near-zero overhead (computers are good at fetching and decoding consecutive instructions) or even negative apparent overhead (if the compiler sees caller and callee, inlining and dead code elimination can potentially eliminate lower-level protocol handling irrelevant to the higher- level protocol).

I would guess that transforming coroutine-based code into state machines hampers performance over the long term by scaring the programmer away from wanting to touch it. However, I would also guess that the direct cost of a coroutine context switch (save and restore all callee-save registers and stack pointer then indirect-jump somewhere unpredictable) outweighs the direct cost of a state machine transition (update some variable).

I do not want to pass judgment on the approach beyond saying that its attendant overhead is too high for some workloads.

<comment><AUTHOR> Ok, but doesn't "straight imperative code" rely on blocking API underneath it? If it does not, the implementation has to store the current state in case it cannot proceed immediately, then restore it later -- which is a context switch.

As for state machine vs. coroutine performance, agreed. But the cost is somewhere around 10-15ns per context switch. Which may be worth paying to get a maintainable code.

<comment><USER3> You can use the usual BSD socket API (in which case the context switches to/from the kernel for calling read() puts the contemplated workload out of reach) or any number of vendor-specific userland network stacks. There is no need to context switch. (Function calls and function returns are not context switches.)

</comment></comment></comment></comment></comment></comment><comment><USER4> Didn't notice this on my first read, but this is by Martin Sustrik. Sustrik is a former ZeroMQ contributor and, more recently, author of the abandoned zmq alternative nanomsg [1], as well as the libmill [2] and libdill [3] lightweight concurrency libraries.

      [1] http://nanomsg.org/
      [2] http://libmill.org/
      [3] http://libdill.org/

<comment><USER5> Is nanomsg truly abandoned? I was hoping it was simply converging toward stability.

<comment><USER4> Hmm true, my language was a bit ambiguous. I meant it was abandoned by Sustrik [1], even if some of the community are doing the best they can manage to keep it alive.

      [1] http://www.freelists.org/post/nanomsg/New-fork-was-Re-Moving-forward

<comment><USER5> That seems a bit outdated; looks like main nanomsg repo has regular commits from gdamore now[1], so maybe the fork was resolved?

1\. [https://github.com/nanomsg/nanomsg/commits/master](https://github.com/nanomsg/nanomsg/commits/master)

<comment><USER4> Yep it was eventually resolved midway through 2016, D'Amore stepped up, then stepped down, then stepped up again. The link I cited is not a commentary on the current status of nanomsg, it is more to illustrate Sustrik's role in the situation. If, ultimately, this discussion is about Sustrik's IETF proposal, it is important to consider his relationship with previous similar (zmq/nanomsg) or related (libmill/libdill) projects which have borne his name.

</comment></comment></comment></comment></comment><comment><USER6> Why is this being proposed to the IETF? It's not an issue visible at the wire protocol level. It doesn't affect interoperability. Sockets belong to the POSIX spec, and to some extent to language specs.

<comment><USER7> There are already other RFCs regarding the socket API, such as RFC3493 (IPv6 socket API).

<comment><USER8> There even are RFCs (RFC 2783 comes to mind) that specify OS-level APIs that are only tangentialy related to networking.

</comment></comment></comment><comment><USER9> Very interesting idea overall. I've long thought the current segregation of network protocols between userland and the kernel is rather arbitrary and inflexible - why should reliability (TCP) be in the kernel but security (TLS) be in userland?

I don't think the idea of requiring Go-style lightweight threading is viable. Lightweight threading has many nice properties, but also inherently requires more expensive operations to deal with split stacks, and tends to use more memory than a manual approach. It's also rather poorly supported in general by existing languages and OSes (other than Go), while any new OS-level socket API should be as universally accessible as the existing one. In particular, _many_ scripting-ish languages either have no concept of threading at all or only support isolated threads whose communication is limited to relatively slow message passing. In theory the language implementations could write a C shim to expose a non-blocking interface around blocking underlying operations - this is what libuv does today for file I/O, for instance - but the result would be a lot of unnecessary overhead.

<comment><USER10> How would a design with TCP done entirely in userland work? A given socket address might in one instant belong to uid 10, and in the next, after uid=10's connection closes, to uid 20.

(I'm not challenging you, just asking what you think the semantics should look like).

Worth adding for everyone else, though I'm sure you know: high speed network implementations already pull a lot of this stuff into userland.

<comment><USER9> There are a few different ways to answer that question.

First, I believe that for most applications, it's not that TCP should be in userland; rather, TLS should be in the kernel, or otherwise system managed. See the wall of text I just replied to someone else with.

Second, if you _do_ want to customize the transport layer, which is of course fairly common today - multiplayer games often go for some custom "reliable UDP" protocol that drops ordering guarantees to improve latency, and then there's Chromium with QUIC, and uTorrent µTP, and WebRTC... well, the reality of networking today is that that has to be done on top of UDP rather than IP directly. I doubt there's much point writing a new API for TCP itself that exposes all the bells and whistles, because the design of TCP is dated. Rather, there should be more standardized APIs for applications to more easily use replacements for TCP that run over UDP. Ideally I should be able to switch from TCP to QUIC just by changing a few lines of code.

(Edit: By the way, if we could redo networking protocols from scratch, I think ports ought to exist at the IP layer rather than being replicated in TCP and UDP, and then UDP could be abolished entirely. But we can't.)

Actually, that ties into high performance networking too. As you know, those alternate stacks require userland to talk directly or near-directly to hardware, meaning you just have to give up on different ports belonging to different applications. But beyond that, one thing holding those stacks back is that they require a lot of custom code and don't work with existing applications. With a standardized networking API that supported userspace plugins, it could be possible to add "DPDK+rumptcpip" as an option next to "kernel TCP" and "QUIC", and configure any random process to use it.

<comment><USER11> > I doubt there's much point writing a new API for TCP itself that exposes all > the bells and whistles, because the design of TCP is dated. Rather, there > should be more standardized APIs for applications to more easily use > replacements for TCP that run over UDP.

The advantage TCP has over UDP is that middleboxes know what TCP FIN is and so are willing to use much longer timeouts for TCP sessions. For example the default Linux connection tracking timeout for established TCP connections is five days but for UDP streams it's three minutes.

So if you need a long-lived session to receive event-based messages your choices are to use UDP with a mapped port using NAT-PMP or PCP (ideal but not always available), use UDP with frequent keepalives (expensive), or use TCP.

Being able to do TCP in userspace would be very useful for any VPN-like thing because you could get the long timeouts but still deliver packets immediately even if an earlier one was lost, and avoid the TCP-over-TCP performance degradation by deferring congestion control to the inner packets.

</comment></comment></comment></comment><comment><USER12> OT: How is text files like this (or like any RFC) written? Is there any standard format and is it done in a specific way?

EDIT: Found it! There was actually a RFC for that. [https://tools.ietf.org/html/rfc7322](https://tools.ietf.org/html/rfc7322)

<comment><USER13> So RFCs are self-hosting? ;)

</comment></comment><comment><USER14> I'd like to see some comparison with the STREAMS layered/stackable API from SVR4 which let a developer do just the sort of layering this API proposes. What makes it better than STREAMS, does it share the same pitfalls, etc.?

<comment><USER15> Also Plan 9.

</comment></comment><comment><USER16> There doesn't seem to be much of anything interesting here. I do think there's a lot to consider in a new system design, but the fact (brought up at the start) that you can't send any new protocol over the internet, and can't send anything but HTTP through half of it, really limits you.

If I were designing an API, the RX side would be asynchronous and batch multiple connections, and the TX side would let you assemble your own TCP packets. This rules out ideas like sendfile() which IMO became nonstarters when we moved to HTTPS.

And obviously the accidental complexity like fcntl vs. setsockopt, bind/connect, shutdown/close, multithreading+signals, have to go somehow. It doesn't really matter though. I can't imagine any major world crises are being caused by BSD sockets.

</comment><comment><USER17> A very long time ago and coming from VMS a big reason preferred BSD over Sys V was non blocking sockets. Now that was over 20 years ago but now going away?

</comment><comment><USER7> Something this API gets right is having a unified interface for both IPv4 and IPv6. With the sockets API, you have to decide for one of them. Changing isn't easy as the constants and structures are all named differently.

While it's possible to use IPv6 sockets for IPv4 connections, this doesn't cover all use-cases. For example, you can't do IPv4 broadcast with an IPv6 socket. Additionally, as most examples are written for the classic IPv4 API, that's what everyone uses per default. Later on, when people complain about missing IPv6 support, they are turned down because it's a ton of work to change.

<comment><USER8> For majority of applications, supporting IPv6 boils down to using getaddrinfo()/getnameinfo() instead of gethostbyname()/gethostbyaddr(), which results in code that supports both IPv4 and IPv6 and is simpler than the IPv4-only original.

<comment><USER18> The catch is you also have to be prepared to listen on multiple sockets. There are lots of servers out there which get this wrong, including qemu-nbd.

For reference here's my implementation of this which (I think) gets it right: [https://github.com/libguestfs/nbdkit/blob/master/src/sockets...](https://github.com/libguestfs/nbdkit/blob/master/src/sockets.c#L105)

</comment></comment></comment><comment><USER19> The problem with proposals like this is that they paper over the fundamental problem:

Time is an input variable and really should be part of the API.

The problem is that sometimes you want the end user to have control over time and sometimes you might not want the end user to have control over time.

If I'm on Linux, I certainly should not be able to manipulate the "time" of the communication stack in order to send faster/more aggressively than I should be allowed.

If I'm on embedded, I _really_ want to be able to manipulate the "time" of the communication stack depending upon what I am doing.

<comment><USER20> Could you clarify? I don't get where you are heading with that.

At least the concept of a "stream" isn't really linked with time. It's just a sequence of bytes.

<comment><USER19> The issue is that abstractions "leak".

Let's take your "stream", for instance.

It's a sequence of bytes.

But is it a "maximum bandwidth" stream of bytes (gigantic file transfer) or is it a "minimum latency" (audio packet) stream of bytes?

If I know or can control what the "time" the stack is working toward, the stack doesn't have to know the difference between those two.

In addition, when on embedded, I quite often want a stack API which is "foo_init(), foo_deinit(), foo_queue_send(), foo_queue_action(), etc." which all update internal state

And then "foo_make_incremental_progress(foo_state, foo_now, ...) where _ONLY_ that incremental progress function _actually_ carries out actions like reading hardware, writing hardware, timing out, etc. Now, I can use whatever concurrency construct I like without the stack getting in the way.

Now, that isn't necessarily the fastest performance as the stack needs to be structured such that it only carries out one "action" at a time on each incremental call. However, it's very flexible. It also has the wonderful property of being _repeatable_ and _testable_. Something which TCP stacks are notoriously resistant to.

</comment></comment></comment><comment><USER21> > To make the API less tedious to use, short protocol name, e.g. "ws", SHOULD > be preferred to the long name, e.g. "websockets".

Well that's a classic dumb mistake.

Programming isn't about typing.

Why not save everyone the effort of having to learn pointless things or look things up? There is no ambiguity with "websockets" where as "ws" describes nothing without context.

<comment><AUTHOR> Another example: use "tcp_connect" instead of "transmission_control_protocol_connect".

<comment><USER21> i'd make an exception for that, but it has nothing to do with the length of the function name.

TCP is much more well known as an abbreviation than long hand (most people who use it do not actually know what it stands for)

communication is about getting the intent across and not obscuring it after all...

<comment><AUTHOR> Yet one example: "dccp_send" or "datagram_congestion_control_protocol_send" ?

</comment></comment></comment></comment><comment><USER22> Specifying a deadline timestamp instead of a countdown could run into problems if a given machine has an improperly configured clock.

I think the risk of any machine having an incorrect timestamp is greater than a library implementer not recalculating the countdown correctly. Whenever possible, I prefer to stay away from timestamps. They're a fickle thing.

<comment><USER23> Why would it matter? The deadline would be specified in terms of a monotonic clock that's unaffected by wall time adjustments. So even if some crude NTP implementation steps your clock by seconds (or a leap second is inserted), it shouldn't matter for the deadline calculation.

<comment><USER24> I think most of my timeout/deadlines relative to the monotonic time. Hence specifying the absolute monotonic time would just involve yet another syscall (to query the time). Is there a really good argument to specify a absolute monotonic time?

</comment></comment></comment><comment><USER25> To clarify, none of the BSD projects have anything to do with this "revamp" not-a-real-RFC.

<comment><AUTHOR> "BSD sockets" is the name of the existing socket API.

</comment></comment><comment> Ono-Sendai Sounds good. Please add a flushHint() call as well: [http://www.forwardscattering.org/post/3](http://www.forwardscattering.org/post/3)

<comment><AUTHOR> The interesting question here is: Once we have flush() function does that means that protocols are allowed to delay data indefinitely if flush() is not called?

<comment> Ono-Sendai Yeah that is an interesting question. I think if the provider of the socket interface could be sure that the client code using the interface would know about flush(), then it wouldn't need to set timers or anything like that to flush after X ms etc. However there wouldn't be any point, I think, to buffering up more than e.g. a MTU worth of bytes. So it wouldn't be delaying indefinitely.

</comment></comment></comment><comment><USER26> There's a typo in section 3.4 paragraph 4 s/bystream/bytestream--unless it's an attempt to be clever.

<comment><AUTHOR> Fixed.</comment></comment>


Leaving Spellcheck Enabled Is a Privacy Risk (2016) - <AUTHOR> https://github.com/signalapp/Signal-Desktop/issues/824 <comment><USER2> The conspiracy theorist in me thinks that Google deliberately makes Chrome's offline spell check be as poor as possible, to give people incentive to give all text to them under the guise of spellchecking. It seems like every day or so, there will be some common word that is marked as misspelled, usually a plural or possessive form of a word that is in their spellcheck dictionary.

The developer in me thinks that this is because nobody at Google cares about the experience if the user chooses incrementally slightly more privacy, and so it simply gets no attention.

</comment><comment><USER3> Title of submission should be: 'Leaving spellcheck Enabled On Chrome Is a Privacy Risk'.

In the spirit of trying to do constructive discussion, the original post references the long dead Signal Chrome app. The new desktop garbage they push is a Node.js app called Signal Desktop. They use native spellchecking provided by Node and Hunspell (even on Windows, which could use native windows API available for 8 and newer and which probably is privacy violating)

To the best of my knowledge, Hunspell is privacy conscious software.

EDIT: Additional context, [https://github.com/electron-userland/electron- spellchecker](https://github.com/electron-userland/electron-spellchecker) also confirms it only uses Hunspell on windows.

<comment><USER4> Thank you this will help anyone with paranoia of being watched.

</comment></comment><comment><USER5> What about things like spotlight? If I work in healthcare and I search for "Jane Doe medical records.txt" does part or all of that search end up hitting a webservice somewhere?

<comment><USER6> You can use an SSL proxy/snooper to find out. I use Charles to see what is actually happening at that level, you would be surprised (or not) at how much stuff is sent to third parties.

[https://www.charlesproxy.com/](https://www.charlesproxy.com/)

<comment><USER7> That used to be straight forward, but now most of the apps come with certificate pinning and for seeing through network request, toying with tools Frida are now needed.

<comment><USER8> Can certificate pinning be defeated?

<comment><USER9> It's not something I'd expect your average user to be able to do, but it is possible and is regularly done for various reverse engineering purposes. The approach I've seen involve changing what certificate is pinned to one that is locally controlled.

That being said I don't think it's nearly as common for certificate pinning to be present as people think- in fact it was deprecated in 2017 because it caused so many problems. Instead things like HSTS and transparency logs are used to prevent damage from malicious issuance of certificates, and organizations can typically override things with their own CA.

If your software is pinned to a specific certificate (even if that certificate is a root CA) your software won't work in many corporate environments. One of the exact reasons mentioned- medical data- is a big driver in this. My company has APIs used by medical companies, and they have to whitelist our service in their firewalls because many of them MITM all traffic to ensure that PHI isn't sent out over an unapproved service by mistake.

<comment><USER10> > in fact it was deprecated in 2017 because it caused so many problems. > Instead things like HSTS and transparency logs are used to prevent damage > from malicious issuance of certificates, and organizations can typically > override things with their own CA.

Still kinda salty that pinning was deprecated from HTTPS. It wasn't perfect (accidental/malicious pinning was far too easy seeing how it was merely controlled by an HTTP header), but the current alternatives (Certificate Transparency, CAA DNS records) aren't an adequate replacement. Sure Certificate Transparency helps to detect a misused certificate, but it doesn't actively prevent it from being used, and CAA requires that the CA isn't "lying" about the header through a "bug" or otherwise.

</comment></comment></comment></comment></comment></comment><comment><USER11> I don't believe Signal Desktop runs as a Chrome app anymore, can anyone confirm?

<comment><USER12> That's correct.

<comment><USER13> Just an FYI, you can run it in Linux on Chromebook:

[https://support.google.com/chrome/a/answer/9025903?hl=en](https://support.google.com/chrome/a/answer/9025903?hl=en)

</comment></comment></comment><comment><USER14> The title ought to include "in Signal", or more generally, "in apps that use online dictionaries". I always disable that. And it's easy to check; just test with no uplink.

Also, misspelling is a profiling risk.

</comment><comment><USER15> AFAIK leaving the text translation feature enabled (even if you disable for a particular language) is even more of a risk.

<comment><USER16> Text translation in Chrome doesn't send anything to Google servers unless you actually translate a page.

It detects what language the page is written in with an entirely client side model (which is why it's accuracy is poor).

</comment></comment><comment><USER17> Is this a chrome/electron only issue, or is it OS level as well? What about Firefox?

<comment><USER18> Chrome issue, you can disable it in the settings page.

</comment></comment><comment><USER19> (2016)

<comment><USER20> Have any of the relevant issues changed since 2016?

<comment><USER19> For one, Chrome Apps for non-ChromeOS aren't a thing anymore.

Also, it's customary to add that notice on HN regardless of how timeless the content is.

<comment><USER20> Yes, of course the year is always there if it's not this year.

What I mean is to that this same insecurity principle could apply to other situations.</comment></comment></comment></comment>


Did Scientists Just Solve the Bee Collapse Mystery? - <AUTHOR> http://www.motherjones.com/tom-philpott/2014/05/smoking-gun-bee-collapse

<comment><USER2> Previously discussed at [https://news.ycombinator.com/item?id=7723456](https://news.ycombinator.com/item?id=7723456), but this article does add some context.

</comment><comment><USER3> As far as I can see the only reason colony collapse is seen as a mystery is because companies selling pesticides issue press releases saying so that get spit out verbatim by journalists.

</comment><comment><USER4> > a grain of table salt weighs 64,800 micrograms

Um... citation needed. Setting aside the issue of measuring weight in units of mass, where are these people getting their mega-salt?

(ETA: Source appears to be Yahoo! Answers, as far as I can tell. Clearly reputable.)

<comment><USER5> [http://en.wikipedia.org/wiki/Grain_(unit)](http://en.wikipedia.org/wiki/Grain_\(unit\))

So the interesting question is whether the confusion was introduced by deception or not.

</comment></comment><comment><USER6> We're solving the bee collapse mystery on average about 2.31265 times a day now. I'm starting to fear that we're solving it so well, we're gonna produce the opposite effect - bee explosion.

And now just think about this for a second - a bee explosion. Is this _really_ a better problem to have, seriously?

My scientific opinion is that some problems are just not meant to be solved. Instead, we should look into quadcopter beebots.

</comment><comment><USER7> [http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines](http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines)</comment>


Show HN: Leash.co – Easily transfer URLs between devices - <AUTHOR> http://leash.co

<comment><USER2> Nice thinking, love the zero login aspect like pastebin, the keyword is an interesting idea.

I have an idea on how to support reserved keywords

When you submit a url to a keyword you can optionally include a 4 digit pin

so

[http://leash.co/4567@neilellis/google.com](http://leash.co/4567@neilellis/google.com)

When you read them you just use

[http://leash.co/neilellis](http://leash.co/neilellis)

You list all 'reserved' usages first and then unreserved after (say if a user clicks a button or something to unhide them)

The first person who submits a pin for a unreserved keyword gets to reserve the keyword for a month.

Anyway, just thoughts. Like the idea, keep em coming!

<comment><AUTHOR> I like this idea!

</comment></comment><comment><USER3> Neat.

I remember, in a time far from Twitter and URL shorteners popularization, joining two computers in a chat room (any random one over the web) to exchange URLS. These two computers in the very same lab room.

I'll already memorized (leash.co!). Definitely going to use (and retire shortners and pastebins just for single URLs as I use often).</comment>


Yahoo Nears Investment in Snapchat - <AUTHOR> http://online.wsj.com/articles/yahoo-nears-investment-in-snapchat-1412361684

<comment><USER2> You know why Alibaba is profitable? Because they don't make stupid investments like this one.

First Tumblr and now this. Yahoo should stop playing 'save a VC'.

<comment><USER3> Alibaba is a pretty bad example to use, because Yahoo made $9.4 billion because they DID invest in Alibaba early on.

<comment><USER2> Actually it's a perfect example.

Jerry Yang made the investment in Alibaba circa 2005. If you told him Yahoo! will be investing in Snapchat he would probably laugh at you.

Now in 2014, Yahoo! is buying a blogging platform and investing in picture messaging with a timer.

The driving force overseas is money, money, money. Not impressions or # of useless users.

Comparing Alibaba investment to Snapchat is simply absurd.

</comment></comment></comment><comment><USER4> I'm going to have to eat my words at this rate: [https://news.ycombinator.com/item?id=6824666](https://news.ycombinator.com/item?id=6824666) ...

I really don't understand this. What is so special about snapchat? What future does this thing have? Can Evan Spiegel see the future?

<comment><USER5> For more perspective on this question, I'd recommend this recent video (~6 min) by Casey Neistat - [https://www.youtube.com/watch?v=kKSr6h5-fCU](https://www.youtube.com/watch?v=kKSr6h5-fCU)

<comment><USER6> As someone of the "Facebook generation", I have to say my mind was blown by this video. Thanks for sharing.

I feel there is clearly a trend line one could draw here in the evolution of social networking and social media. Since the initial era of social networking, friction was reduced (e.g. Instagram / Twitter) and now it's moving to be both frictionless and ephemeral (Snapchat).

It's interesting because the trend almost leads you something like life broadcasting (e.g. Justin.tv) - but we've already been there. I wonder what the next phase will be.

</comment></comment></comment><comment><AUTHOR> For non-subscribers: [https://www.google.com/search?q=Yahoo+Nears+Investment+in+Sn...](https://www.google.com/search?q=Yahoo+Nears+Investment+in+Snapchat)

</comment><comment><USER7> Man, its worth $10bn but doesn't make any money, that's pretty amazing. Another solid investment by Yahoo.

</comment><comment><USER8> "Separately, this week Yahoo continued its acquisition spree of small startups with the purchase of another mobile messaging company, MessageMe, according to two people familiar with the deal."

This is why activist hedgefunds are lining up to destroy Yahoo and make a buck. The board clearly complained about Marissa bailing out failed silicon valley startups for millions. Messageme was an unoriginal product created by people with a favorable track record but the actual product had nothing to offer users except growth hacking by using the facebook friends graph api. Spending millions on this aquihire makes no sense.

What Yahoo needs is a core product that is a big win, Marissa said Tumblr would bring this, but instead we have seen competition on mobile increase and time across social media and blogging apps fragment. We have seen them lose their weather app monopoly on ios and yahoo mail on mobile is not gaining traction. Across the hot core areas being, search, social, cloud services, document services, smart hardware and artificial/assisted intelligence, yahoo is nowhere to be seen. Yahoo needs to seize the initiative and focus on a core competency and win big.

Snapchat at a $10 billion valuation has too much risk for someone that needs a big win. It really seems like snapchat investors are running some kind of pump and dump scheme the way they pushing this valuation. As things stand Snapchat has to go public to make good on a $10b valuation.

</comment><comment><USER9> Ah yes, Yahoo investing in another company with no hope of actualizing it's valuation by actual revenue (as opposed to investors bloating it until an IPO).

I'm sure it'll look great next Tumblr...

<comment><USER10> "Revenue" isn't the only thing that companies compete for.

Many of these online services companies (Google, Facebook, Yahoo) compete for user timeshare - e.g. you're awake 16 hours a day, how many of those hours are spent under company X's umbrella?

Tumblr had doubled their number of blogs and grown their userbase by 1/3 since being acquired. The jury is still out on whether it was a good or bad investment.

<comment><USER11> I read Tumblr was a few weeks away from running out of cash at the time Yahoo bought them. While I agree the jury is still out on the Tumblr/Yahoo acquisition, I do think they could have bought a nearly insolvent company a bit cheaper than they did.

That being said, maybe they did get it on the cheap if WhatsApp went for $19B.

<comment><USER12> Exactly. How do you know the price they acquired Tumblr for wasn't its "insolvent" price? There was probably competition for the deal - passing in the hope of getting a lower price is dangerous if you know other players might snap up Tumblr.

</comment></comment></comment></comment><comment><USER13> People seem to not take into consideration the concept of staying in the game. Facebook bought WhatsApp for the users, not necessarily to become Facebook users but to prevent them from becoming users of a competitor. Yahoo needs to make acquisitions so they can stay relevant and connected, otherwise they may not have any chance of succeeding in a much bigger way with all of their offerings.

<comment><USER2> > concept of staying in the game

That's actually the opposite of that concept. WhatsApp actually solved a real problem and aligned closely with Facebook's overall strategy (communication).

<comment><USER14> WhatsApp definitely solved a problem when they launched and they did an incredible job, but I'm not sure what I'm missing, what problem were they solving when Facebook bought them That Facebook couldn't? I'm not saying it was a bad investment - I think it was smart, but to me it just seemed like Facebook Messenger and WhatsApp were competitors by the time Facebook bought them.

</comment></comment></comment><comment><USER15> This would be an amusing way to funnel lots of money directly into Google's pockets, I guess.

</comment><comment><USER16> I get the feeling they'll do better on this than they did on Ali Baba

<comment><USER17> Weren't they recently valued around $10b[1]? They'd need to grow a _lot_ to make Alibaba-level money on Snapchat. And Snapchat would have to really find a way to monetize eyeballs without losing them.

[1] [http://money.cnn.com/2014/08/27/technology/innovationnation/...](http://money.cnn.com/2014/08/27/technology/innovationnation/snapchat- valuation/)

Edit: Apparently the round hasn't closed yet, so they would be a part of the $10b round. I think my point still stands!

</comment></comment><comment><USER18> how does yahoo actually generate revenue now? the only people I know of that mainly use Yahoo mail are in countries like India where ads are worth next to nothing, and nobody uses Yahoo search. Is it just from ads on their properties?

<comment><USER19> These days it seems to be from investing. Alibaba, now apparently SnapChat.

<comment><USER20> That does not constitute revenue under accounting rules.

</comment></comment></comment><comment><USER21> God please Melissa do not buy anything!

</comment><comment><USER18> Yahoo right now(and when they bought Tumblr)

[http://a.pomf.se/gsxzwx.jpg](http://a.pomf.se/gsxzwx.jpg)</comment>


In the US, wells being drilled ever deeper as groundwater vanishes - <AUTHOR> https://arstechnica.com/science/2019/07/wells-are-getting-deeper-as-groundwater-gets-depleted/ <comment><USER2> A local groundwater management agency in Ventura County, CA recently started a pilot program that established a groundwater market among farmers in the county.

Following the early 2010s drought in California, laws were enacted to cap and eventually reduce the amount of groundwater that can be extracted. Recent advancements in water usage tracking infrastructure, combined with the law has led to a natural environment to create a cap and trade system.

Interestingly, farmers view water usage as a trade secret, and so the pilot marketplace is using block-chain to provide an anonymous marketplace. Groundwater management in California (and I assume across the nation) is regulated by a patchwork of different local government agencies and laws, but there is an awesome opportunity to provide a unified software based marketplace to establish a groundwater commodity market.

A market also facilities conservation as farmers now have incentive to lower their water usage and sell excess water on the market. A big challenge however is transportation of water across vast distances.

<comment><USER3> > Interestingly, farmers view water usage as a trade secret

I wonder why. Do slight changes in irrigation patterns have a significant effect on crops?

<comment><USER4> Definitely! If someone is able to figure out how to use less water to grow the same amount of crops they could totally outcompete and undercut other producers.

There's an interesting Planet Money episode about it: [https://www.npr.org/2019/06/12/732147345/the-water- marketpla...](https://www.npr.org/2019/06/12/732147345/the-water-marketplace)

<comment><USER5> >If someone is able to figure out how to use less water to grow the same amount of crops

We've already had that for years. Farmers don't want to make the structural investments required to get there. Water for farming needs to become prohibitively expensive for them to even consider moving away from their wasteful large fields. Tax the hell out of the land, too.

What takes an acre of land and 100,000 gallons of water (a crop of wheat grass) can be done in a building 1/8 of an acre, 90% lower water requirements if not more, and produce 6x more than what the open plot of land does, on a yearly basis, on average.

I did and proved that almost a decade ago. Looks like nobody still wants to listen.

</comment></comment></comment></comment><comment><USER6> When some aquifers are drained, they take many thousands of years to replenish, a heavy rain season doesn't really help with these depleted aquifers.

Here's a great video on Cody'sLab that explains how confined aquifers work and why they take so long to replenish: [https://www.youtube.com/watch?v=tF4iCgxsZX4](https://www.youtube.com/watch?v=tF4iCgxsZX4)

<comment><USER7> It even shows why some aquifers may never replenish. The layer of the aquifer is not tightly compressed and can hold water. When the water is released and the layer starts to compress from the weight from above, its ability to hold water is permanently diminished.

<comment><USER8> Happens to Teheran and many Indian cities. Viewable via satellite.

<comment><USER9> Las Vegas has a lot of subsidence due to ground water use.

<comment><USER10> In the Santa Clara valley. Used to be the water table was so high in a lot of places you could drill artesian wells. When orchard farms got going they pulled so much water that by the 19920's some places sunk 12 feet. The fix was irrigation systems and replenishment.

I read my great grandfathers diaries and water was a big deal in the 1880's. One spring after dry winter he had to slaughter all of his calves and the older dairy cows. Later that year his older brother shot himself in despair. Next year his sisters and younger brother went up to Oregon to live with relatives and never returned.

Water has been a big deal my whole life in California. That's unremarkable. What's new is it's becoming a big deal in a lot of places.

</comment></comment></comment></comment></comment><comment><USER11> The same phenomenon is happening is happening in China & India and probably many other parts of he world as well. As we become more a more affluent society, our on water consumption increases. We need to raise awareness at a global level about preserving and saving water.

[https://www.theatlantic.com/china/archive/2013/04/28-000-riv...](https://www.theatlantic.com/china/archive/2013/04/28-000-rivers- disappeared-in-china-what-happened/275365/)

[https://www.nationalgeographic.com/environment/photos/rivers...](https://www.nationalgeographic.com/environment/photos/rivers- run-dry/)

<comment><USER12> Why not just price water to reflect supply realities?

Why would anyone care about saving water when it's so dirt cheap? "Awareness" seems like a waste of time in comparison.

<comment><USER13> Does market economics work if there's no way to prevent the tragedy of the commons?

<comment><USER14> That's a confusing question. The tragedy of the commons happens specifically when a resource is under- or un-priced.

<comment><USER15> If a free market is left alone to set prices, do any resources end up under- priced?

<comment><USER14> No- but that presupposes the existence of a well functioning free market, which everyone agrees we do not have with water.

</comment></comment></comment></comment></comment></comment><comment><USER16> Lets stop wastefully watering lawns as a starting point. Some places are probably already doing that, but the use of native landscaping needs to spread further.

<comment><USER17> The largest water user, at least in California, is agriculture. Fortunately they're now planting fewer acres of low-value crops such as cotton. But other crops use an insane amount:

       1 walnut     4.9  gallons
       1 almond     1.1  gallons
       1 pistachio  0.75 gallons

And it looks like Palm Springs is a big water user. Must be all those golf courses?

[https://www.motherjones.com/environment/2014/02/wheres- calif...](https://www.motherjones.com/environment/2014/02/wheres-californias- water-going/)

<comment><USER13> I heard meat/milk production uses a significantly larger amount of water (in addition to the crops eaten by the animals).

<comment><USER18> It's the waste from cattle that's a bigger problem. And areas with little rainfall seem to be particularly bad at handling all the excess nitrogen.

</comment></comment></comment></comment><comment><USER19> For those interested in exploring the political and social consequences of this in a near-term (albeit fictional) world, I highly recommend reading "The Water Knife" by Paolo Bacigalupi. From the Goodreads description:

> In the American Southwest, Nevada, Arizona, and California skirmish for > dwindling shares of the Colorado River. Into the fray steps Angel Velasquez, > leg-breaker, assassin, and spy. A Las Vegas water knife, Angel "cuts" water > for his boss, Catherine Case, ensuring that her luxurious developments can > bloom in the desert, so the rich can stay wet while the poor get dust.

<comment> et-al And if anyone prefers non-fiction, _Cadillac Desert_ by Marc Reisner describes the (completely fucked) water supply situation out in California.

<comment><USER20> One of my all-time favorite non-fiction books, perhaps even more topical today than it was when it came out in 1986. It's a shame that the author died in 2000 at the young age of 51; the book deserves a sequel or two.

</comment></comment></comment><comment><USER21> When I was a kid, we owned a house out in a small farming community in western Washington State. One of the things we had to do is drill the well a solid 100 feet below the water table, because the upper 60 feet had measurable levels of fecal coliform bacteria. This was fairly rural. Our nearest neighbor was a quarter mile up the road, and we were surrounded by livestock pastures with cattle herds.

I would imagine the current owners of the property have had to re-drill the well at least once as population density in the area increased.

</comment><comment><USER22> Unpopular (or at least unconventional) opinion: depleted aquifers are a good thing, because it's turns one of the externalities of outrageously exploited agriculture into a true cost. More expensive food means less land use and less energy spent irrigating what we do grow.

(I felt the same way about "peak oil" 20 years ago, but alas fracking messed that guess up.)

<comment><USER23> That's the issue, I've found, with hoping that calamity will make the problems visible enough to shock us into making systemic changes. Humans are very good at inventing clever short-term "solutions" that ameliorate the symptoms but wind up making the long-term problem worse. I have no doubt that if/when global civilization fails due to global warming, we'll wind up burning everything in sight to power our air conditioners for a few minutes of respite.

<comment><USER24> GMO is supposed to save us this time, if we have to pay for it some other way is still being discussed.

</comment></comment></comment><comment><USER25> Have any more improvements been made in the process of removing salt from ocean water in large volume? Is anyone actively still working on this?

<comment><USER26> Did a quick google search and found that Israel launched a large desalination plant back in 2015 (uses reverse osmosis). Here's the source: [https://www.technologyreview.com/s/534996/megascale- desalina...](https://www.technologyreview.com/s/534996/megascale- desalination/)

<comment><USER25> Thankyou. That would suggest it is feasible on a large scale. Hopefully more countries will start to invest in desalination plants. With ocean levels rising, now should be a good time to optimize that process even further.

<comment><USER27> Desal works fine, but uses a lot of energy. With ever cheaper solar panels, there will soon be a point where it makes sense to power a desal plant on just solar power.

But there is still the small matter of the salt that is left over. There is so much salt on this planet that no one wants it. You could dump it back in the ocean, but wherever you do so becomes too salty, which would be really bad if you dump it back right where you're doing your desalinization.

Which means we either need to ship it to the middle of the ocean (which maybe you can do with autonomous solar powered boats) or you need to figure out some other use for all that salt.

<comment><USER25> Perhaps that could augment or replace our existing salt mining.

</comment></comment></comment></comment></comment><comment><USER28> Its just amazing to look at population growth in the desert states, Nevada, Arizona, Utah should be barren wasteland, not with rapidly growing cities of millions. Running out of water seems inevitable.

</comment><comment><USER29> This is poor reporting and possibly poor science. It's not enough to say that it will vanish. We need to know on what timescale it will vanish. I suspect that it will last at least several centuries at current extraction rates. Several hundred years from now, the climate may be completely different, energy might extremely cheap, new continental water distribution might be constructed, etc.

<comment><USER30> Aquifers are a local resource, you can think of them as somewhat like lakes that rest in depressions in geological strata rather than in depressions in the ground. There is not one large aquifer which spans the US. As a result, the details of the situation depend greatly on the area. In the Southwest, where aquifer depletion has been a major concern, there are concerns that some aquifers may deteriorate in quality due to overextraction to the point that expensive water processing is necessary over a scale of one to two decades (e.g. along the Mexico border). It's also somewhat difficult to give a year range because there is generally not actually a cut-off point where you completely "run out" \- instead, the cost of extracting and processing the water increases as overextraction continues, due to a lower water table and, worse, increased salinity. Alamogordo, NM has already initiated the construction of a desalination plant because of this problem - a very expensive proposition, and one the public rarely thinks of inland.

Concern about aquifer depletion is increasing across the nation, not just in the desert southwest, but the severity of the problem varies. What is reasonably consistent is that the situation is getting worse in most places.

The USGS has general information here: [https://pubs.usgs.gov/fs/fs-103-03/](https://pubs.usgs.gov/fs/fs-103-03/)

</comment></comment><comment><USER31> It doesn't have to be this way.

Off the top of my head:

Fresno County, California is a quiet and overlooked pioneer in water rights and water development. They are a modern "Hanging Gardens of Babylon" in that about 150 years ago or so, it was essentially desert with greenery growing only near the rivers. That changed drastically when they built canals.

After building canals, they found that some low-lying areas turned into ponds. It was a side effect of the canals raising the groundwater level.

From there, they actively created groundwater recharge systems. In the last 100 years, while most aquifers in the US only went down, groundwater levels in Fresno actually improved at times, helping to protect the area's water supply.

I lived in Fresno for over two years while homeless. I considered staying in part because I expect it to be a quiet haven of secure water supplies in coming decades as our water supply issues spiral into ever deepening crises globally. I bet some systems could borrow best practices from them and at least slow the problem.

 _Water for a thirsty land: the Consolidated Irrigation District and its canal development history_ is a history of water development in Fresno. It's a fascinating read. I highly recommend it.

[https://www.worldcat.org/title/water-for-a-thirsty-land- the-...](https://www.worldcat.org/title/water-for-a-thirsty-land-the- consolidated-irrigation-district-and-its-canal-development- history/oclc/36954078&referer=brief_results)

Earthships can provide adequate residential water supplies via rain catchment in areas with as little as 8" of annual rainfall.

One of the arguments for eating vegetarian is that beef uses a lot of water to produce. You don't have to go full on vegan to reduce the amount of water burden your diet represents. Just cutting back on your consumption of beef can have an impact.

As a child, I grew up with a set of encyclopedias at home. One article that really stuck in my mind was about successfully growing trees in the desert for agricultural purposes by shaping the ground to be a local catchment system to extend the usefulness of natural rainfall.

You make a square or rectangular catchment area with one corner being the low point. It's not very different from designing a shower with a low point for the drain.

You plant your tree in the low point corner so it is downhill from the rest of the catchment area. When it rains, the catchment fills up and the tree has its own personal water supply for at least a few days.

The interesting and technical part is that you need to develop standards for how large of a catchment to build based on plant species and rainfall levels. Too large of a catchment can cause root rot and kill the tree. Too small and it won't get enough water to thrive.

There are many ways we can work on maintaining a high quality of life while reducing our use of water. We don't have to throw our hands in the air and accept that it is inevitable that we will just hurtle towards some doomsday scenario until we get some mass die-off event for humans, in essence.

</comment><comment><USER32> It hasn't "vanished", it is just hiding in the ocean and the atmosphere. Same as the (vanished) water at the icecaps.

As the Earth heats, the water evaporates. When that happens it adds to heat retention. Expect the water to make a bold reappearance as we experience tropical storms.

<comment><USER27> The _ground_ water has vanished and become _sky_ water, which is much less useful to a farmer.

</comment></comment><comment><USER33> On the flipside the great lakes are nearly at an all time high. Maybe the growth in those places requiring deeper wells have exceeded what they can support?

<comment><USER34> The Great Lakes sources their water from glaciers melting. Too little and too much of a thing it's still worrying.

<comment><USER35> Do you have a source for this? I thought they were sourced was almost all rainfall both directly into the lake and from the streams and rivers that surround them

<comment><USER36> Glaciers formed them, but probably have little to do with sustaining them [https://en.wikipedia.org/wiki/Great_Lakes](https://en.wikipedia.org/wiki/Great_Lakes)

</comment></comment></comment></comment><comment><USER37> If this were actually a serious issue, we would be discussing NAWPA again for real.

[https://en.wikipedia.org/wiki/North_American_Water_and_Power...](https://en.wikipedia.org/wiki/North_American_Water_and_Power_Alliance)

The original blueprint used nukes to dig the trenches, but conventional explosives work just fine.

<comment><USER38> Actual water resources researcher focusing on climate change here. My job is focused on what can we do to continue to deliver water to society under future scenarios (and what steps we can take to get there) from policy to infrastructure investments. The article doesn't present anything new for us but the problem is that the general public isn't informed.

While we're the oldest engineering discipline, remember that even until around the 80s people didn't think or know about groundwater contamination. They've always thought that sticking contaminated water back into the ground would filter it naturally like magic. In the late 80s more and more people started to understand that's not how it works. Even then, these policies and ideas are still in the process of being accepted in developing countries almost 40 years later and even then "what else can we do? My son needs water now and he could get hit by a car tomorrow so this is a problem I'll deal with later."

The problem (as the article states) is that the issue is so distributed. We're talking about traditionally individualized solutions for a problem that needs a more coordinated solution. However, noone is willing to pay for that or considers other investments as "more critical" than major water projects (also many environmental groups aren't as risk averse as water utilities). I mean the biggest cost in water isn't the actual product but the transportation costs. We have water, we just don't have enough funds to deliver them everywhere in a sustainable manner at a price that people are willing to pay.

NAWPA idea was conceived even before we understood our environment. It's the wrong solution. I mean even 10 years ago we had a landmark article that changed the paradigm of how we decide policy and build infrastructure[0]. I'd never take NAWPA for anything now than the "grandiose-ness of the 50s", especially since that was during a "water resources renaissance" in the United States where even crazy ideas were taken seriously.

[0] [https://www.law.berkeley.edu/files/CLEE/Milly_2008_Science_S...](https://www.law.berkeley.edu/files/CLEE/Milly_2008_Science_StationarityIsDead.pdf)

<comment><USER39> Why are so many water researchers opposed to letting the price of water vary and drive market-based conservation?

<comment><USER38> Great question! There's really no simple answer to this unfortunately also I don't think I'm the best person to answer this question, but I'll try my best!

Water is a basic human right. In most countries, water is a public good that the government subsidizes and, to some extent, the market is adjusted to allow vulnerable households to afford the resource as well. If we have a 100% free market-based solution then in many locations the poor and underprivileged have a considerably less access to this vital resource (some won't get any). An example of this can be seen in India. Many districts there suffer from intermittent water supply (where the local piped water comes at irregular intervals instead of continuously). These people might not be able to shower regularly and they aren't given a schedule of when the water will come, so they can't schedule their day or week. Many of these people might even have to leave work for a day (losing that income and economic productivity) and jump out to fill up their water tanks in their homes. As they don't know when the water comes (no regular scheduling), this uncertainty can be nerve racking and (even if you're trying to stretch it out to last) may not be enough to last your family until the next "supply". In these cases, you call in a water tanker to refill your tank but even then you don't know if it'll come.

On the tanker truck company's side, they don't want irregular domestic clients, they want more regular commercial clients (because the total customer value is much higher, you get more money from those clients than a household that needs a one-offer). So those companies aren't really incentivized to service residential clients. So if another call comes in, those commercial clients would take higher priority than those vulnerable households.

Those solutions don't really help this problem, and this problem is just going to grow for the foreseeable future. Not exactly what I'm talking about, but IWA recently published an article about how many cities are shifting from 24/7 to intermittent water supply solutions [0], and if this becomes irregular and the system starts to fail, then this will just keep that problem growing. One of the projects I'm involved in works on trying to solve this exact problem and we recently got some support from the US National Science Foundation as well as the Gates Foundation. However, there's still a long way to go. (Sidebar: Anyone want to give us more funding or help? Send me a message!)

[0]: [https://iwa-network.org/running-out-of-water-cities- shifting...](https://iwa-network.org/running-out-of-water-cities-shifting- from-24x7-to-intermittent-water-supply/)

<comment><USER40> Surely water security is also a basic human right, and that is directly impacted by destructive consumption subsidies.

> If we have a 100% free market-based solution then in many locations the poor > and underprivileged have a considerably less access to this vital resource > (some won't get any).

If we set water prices to something sane, how much cash do we have to give the poor to leave them in the same financial position?

Surely it can't be much: domestic water usage is but a fraction of agriculture and industry, and the number of poor which couldn't afford water is but a fraction thereof.</comment></comment></comment></comment></comment>


HubSpot Gets $12 Million To Drive Traffic to Your Site - <AUTHOR> http://www.techcrunch.com/2008/05/15/hubspot-gets-12-million-to-drive-traffic-to-your-site/ <comment><USER2> Congrats Dharmesh!

<comment><USER3> Thanks a bunch. Glad the news is finally out so I can talk about it (and blog about it).</comment></comment>