Client side caching in Redis 6 - <AUTHOR> http://antirez.com/news/130 <comment><USER2> This might be a bit of a strange question, but it's interesting to think about and I know antirez will probably read it here, so I want to bring it up:

Could there be some potential privacy or data-leakage concerns with this, since the server is now storing and acting on info about what data has been accessed, by which clients, and even when (to a limited degree)?

For example, say that some kind of sensitive data is stored in redis (e.g. user private messages). By watching for relevant invalidation messages (or if there's any way for clients to read the data in the Invalidation Table directly), it could be possible to learn some info about how that data was accessed.

Without knowing more about the implementation, I don't know if this is even a concern at all, and I haven't come up with a specific method of abusing it even if it is. I think it's probably something to keep in mind though.

<comment><USER3> Everything could be a risk, but in the specific case of a software device holding data, I believe that the main problem is the exposure of the data itself, so in the specific case of Redis the fact that the database stores informations about which key was retrieved by which client should not be an issue AFAIK. A few random arguments:

1\. The database anyway sees such connections and there is a continue stream of activity about clients, so there is also an instantaneous state that discloses who does what, at any time.

2\. Companies trace everything in order to have data to do debugging when there are problems, so this information is stored at a much better level of details in other places as well.

3\. There is no obvious way this could be exploited.

4\. In general it is supposed that the clients can trust the server.

5\. If you already violated Redis in order to read such info (not accessible via APIs), you can collect a lot more.

So based on initial thoughts I don't think this is going to be ever a security problem.

<comment><USER4> Also, the client needs to opt-in.

</comment></comment></comment><comment><USER5> Redis continue to wonder me, it's yet another feature I thought to implement, already implemented by redis itself. And I had same reasoning about hash slots.

<comment> core-questions I always appreciate antirez' very candid writing style. Comes across as such an earnest and reasonable person. Very low ego, very excited about the technology, but with an eye for how people will be able to adopt it. Wish we saw more tech leaders like that.

</comment></comment><comment><USER6> Interesting. I built something somewhat similar about 2 years ago (not big on blogging so it's not published - probably end up hurting my career :( )

The gist is using pubsub to send invalidation messages consisting of item ID and time stamp, but also store the latest invalidation message into a set that is polled on every longer time period (10 seconds in my case). Listening to pubsub and polling are done per server that hosts the in memory cache for all items in the cache.

One interesting complication was how to make sure I don't end up with the wrong (stale) cached item due to race condition between the read and the message passing. Imagine 2 thread, thread A reading and populating the cache, thread B listening to invalidation events. If thread A get held up (context switching, slow network, etc) it can actually populate into the cache after thread B has received and processed the invalidation message. To avoid that I ended up holding the invalidation messages and their receive time in the receiving server for a short duration (20 seconds - twice poll time). I required and in memory cache population to tell me the time from before the read command was sent to the cache. At the time of cache population, if I saw in had any invalidation message marked after the time the read command started I discarded that item instead of caching it. In highly concurrent, highly distributed application this turned to be a must-have.

Also, mainly to placate my fears, I had "panic mode". If I didn't see any message (I sent periodic maintenance messages) or if I couldn't execute multiple polling attempts I dumped the whole cache and wouldn't let any new item be ca he'd until communication was restored.

<comment><USER6> Forgot to mention - writes also invalidated the cache locally so maintain "read your own writes" for multi step algorithms. That was crucial for us and I suspect for a lot of other use cases.

</comment></comment><comment><USER7> Redis is already super useful, but this would be an absolutely killer feature. Server assisted client side invalidation! Something people have always wanted but has never been built.

Implement it quick so I can use it on AWS in a few years!

<comment><USER8> Any reason you can't achieve this today with DynamoDB+DAX?

<comment><USER9> Isn't DAX only a network-based in-memory cache, and not a in-process in-memory cache? Or is it both?

It's definitely at least network-based, b/c you have to spin up instances of it in your VPC.

However, they also require custom clients, which is a little odd afaiu (b/c in theory DAX is API-compatible with DynamoDB, so why would regular dynamodb sdk clients not work?) so maybe the custom clients also add some in-process-based caching?

</comment></comment></comment><comment><USER10> I read 'time', I think 'jepsen'... as the saying goes, there are only two hard problems in software engineering: naming things and cache invalidation.

Disclaimer: happy user of a redis cluster :)

<comment><USER3> If you re-read you'll find that I suggest an incremental epoch to avoid any issue :-D however starting with the idea of time to later hint about a better solution is a lot more understandable IMHO.

</comment></comment><comment><USER11> It reminds me of OVSDB + IDL (C client): [https://tools.ietf.org/html/rfc7047](https://tools.ietf.org/html/rfc7047) . The C client keeps a partial replica of the database, and the database notifies all the clients with the changes.

It makes super fast to work with data (it is already in process memory), and the centralized server handles consistency and transaction serialization.

</comment><comment><USER12> You can see the diagram I did of Ben Malec's multi-level cache system here:

[https://images.app.goo.gl/RgQcYt3MTU3AA3CF9](https://images.app.goo.gl/RgQcYt3MTU3AA3CF9)

</comment><comment><USER13> I feel like this would be neatly implemented as a Redis Proxy, keeping the same semantics but removing that concern for simpler clients.

<comment><USER2> The simple clients just wouldn't enable client tracking, it's off by default.

It also wouldn't work very well as a proxy because then if any clients don't go through the proxy but change some of the data, there's no way for all the proxy-using clients to know.

<comment><USER14> You can make all the changes propagate to all the relevant servers and clients.

</comment></comment></comment><comment><USER15> What's the effect of this system on consistency guarantees? Seems like clients could end up in a state where they serve stale data due to (undetectably) not receiving the push updates.

It also looks like client / protocol implementations are about to get a lot more complex, even if the feature is being used by the majority of users?

<comment><USER3> This is discussed in the blog post itself. If you want to ask a more specific question I'll be happy to reply to my best knowledge.

<comment><USER15> My bad for not being clear enough. I was thinking of, for example, a networking issue where both server and client are still "connected" but delivery never happens. Can be solved with tuning timeouts / client-side TTL as mentioned elsewhere, but I was wondering specifically if this would change Redis' known guarantees for data consistency, what happens during partition etc. More of a theoretical question.

On the second point, the post talks about having the feature disabled by default; I was curious regarding the protocol itself - it is currently quite simple, and by adding extra cache-control metadata it sounds like correct client implementation might become a lot more complex, but it's hard to tell without having seen what the new protocol looks like.

<comment><USER3> Yep your concern about consistency is definitely correct, but the good thing is that the invalidation connection can be pinged continuously by the client, as often the client wants (for instance 10 times every second). As soon as the connection is timing out, the client may decide to no longer serve the cached data. Of course every time the invalidation connection is closed at all, the client should flush the cached data completely or apply a very short TTL to the set of cached data.

About your second point: today I implemented even the feature (in the unstable branch) to support RESP2, that is, the current Redis protocol. But even using RESP3 the complexity is very low. On the data connection side you have just to use the CLIENT command to activate it. In the data connection, what you could receive, if you associate a callback in your client, is push data that notifies you the invalidation connection is no longer active. Push data is normally discarded by the client implementation of RESP3 if no handler is associated. On the invalidation connection, in RESP3 mode, you get again push data with a callback, or if you are in RESP2 mode, you just subscribe to the special channel __redis__:invalidate, and you get the messages about the caching slots that need to be invalidated.

</comment></comment></comment></comment><comment><USER16> > But later also client ID 444 will ask about keys in the slot 2, so the table > will be like:

I think you mean slot 5 in this sentence.

<comment><USER3> Thanks, fixing

</comment></comment><comment><USER17> How will client libs that do connection pooling handle this? If a client opens a pool of 50 connections, will they be identified as 50 different clients, and receive 50 different pushes for invalidation? I guess the client libs will have to build a layer to balance caching on top of the pool.

<comment><USER18> Just curious, why would you pool redis connections?

<comment><USER17> High throughput. Pooling is the default behaviour for many Redis clients too, isn't it? eg: [https://github.com/gomodule/redigo](https://github.com/gomodule/redigo)

</comment></comment></comment><comment><USER19> Is this similar to Hazelcast's "near cache"?

</comment><comment><USER20> Shameless plug for in-language storage. I've found that using a fast language/framework with a local cache gives you most of the advantages of a shared cache like Redis without all the complexity. This feature feels like a tacit admission of that.

In "fast" languages (C# Java Go Rust Cxx) allocation tends to be fairly efficient so you're not wasting tons of space keeping objects in RAM.

These fast languages can max out the network interface most of the time using the right frameworks. In them, memory storage is also over 100x faster than remote like Redis.

Now if you've got something in PHP or Rails that can only manage 100 requests/second remote caches make tons of sense. The network overhead is nothing compared to how slow the language is. This advantage erodes when you're closer to native speed and local memory access is 1000 times faster than retrieving cached data over network.

TLDR: Redis and alike make sense when your language or framework is slow and the bottleneck is CPU use. They don't give you much when networking and memory bandwidth are your bottleneck because the language runs near native speed.

<comment><USER21> Yeah but this doesn't work with something like microservices, or even Kubernetes based deployments, right?

It might take you 3, 40, or a 100 tries for the same request till you hit a node that has your data cached. It could work with extremely high req/seconds, though in an unpredictable way.

<comment><USER22> To be fair, what does it take you to write a simple caching microservice based on protobuf or similar?

<comment><USER21> A lot more knowledge in the relevant space that I have. I could spend a week or two learning necessary information, analysing our use cases and then implementing it finally (which I would never get the time to, from a business perspective).

Or I could use an off the shelf product that is probably better than whatever I could come up with, given, even, unrealistic amounts of time to perfect it. I could come up with something more suited to our use case only, but the moment the use case changes, I'd wonder why I didn't just go with Redis.

I haven't actually looked into a DIY method though. I just know I want something like Redis for our user table, because it's hit a lot, and the traffic tends to be repetitive.

<comment><USER22> >> A lot more knowledge in the relevant space that I have.

Relevant to what? You've already got a microservice mesh, which means you've also got a microservice architecture that's supposed to handle all of the RPCs.

Adding a layer of Redis on top is probably going to introduce more entities into the system and make it less homogenous.

</comment></comment></comment></comment></comment><comment><USER23> Whether or not you like this feature or think it's a good idea, the brunt of this will be born by library writers. Yeah you can use a dedicated thread to listen to these push messages, but really this is tacitly blessing the asyncio's and whatever else.

As you can probably tell, I'm not at all a fan of the way Redis is moving. It hit a sweet spot as a "better memcached" but apparently aspires to be a real database, so now this nice simple tool is all crazy. The new protocol just kinda sucks for synchronous use.

I'd go with a hand-written in-memory caching scheme before I ever considered a Redis sandwich with this feature. Seems like a complexity nightmare and very difficult to reason about. Yeah maybe I'm being crusty, but I'm just thinking about maintaining and debugging something like this and having cold sweats. The linked redisconf talk clearly bears this out. Sounds like they had a hell of a time. But then that also provides some measure of job security?

<comment><USER24> Your hand tuned client cache won't have active invalidation like this new proposal does, so you are stuck waiting for ttl to expire and ttl windows where data is inconsistent between nodes.</comment></comment>


How Apple Could Kill Google's Holiday Buzz - <AUTHOR> http://brooksreview.net/2010/08/kill-the-buzz/ <comment><USER2> I wonder, why do people write such useless stuff?</comment>


The N.S.A.'s Spying on Muslim-Americans - <AUTHOR> http://newyorker.com/online/blogs/closeread/2014/07/the-nsas-spying-on-muslim-americans.html

<comment><USER2> If we make it about names then we have Barack "Hussain" Obama who shares a middle name with Islam's prophet Mohamed's prominent nephew, son of Ali, who was the original holy imam of the Shia sect and to whom most Hezbollah members have sworn allegiance. So I don't think it's a case of racism at the highest levels. I think there are some idiots at any government agency who are mentally backward and terribly lacking in moral aptitude.</comment>


Are Women Colder Than Men? The Science Behind Office Temperature Wars - <AUTHOR> https://www.wsj.com/articles/are-women-colder-than-men-the-science-behind-office-temperature-wars-11594373401 <comment><USER2> Non-paywall link: [http://archive.is/XboPO](http://archive.is/XboPO)</comment>


On Putting Things Off - <AUTHOR> http://www.lrb.co.uk/v37/n17/robert-hanks/on-putting-things-off <comment><USER3> I'm very passionate about the problem of procrastination, having suffered terribly from it all my life. For some people it's a relatively minor thing, but for others it's debilitating– ruining educations, careers, relationships.

What has been blowing my mind lately– and I'm always doing all the reading and experimentation I can about this– is how much of this is PHYSICAL. Hanks talks about headaches and stomach disorders– for me, it's always been anxiety and loss of appetite. Which leads to low/volatile blood sugar, which leads to mental fogginess, inaction, and worsens the cycle of procrastination.

If I could go back in time to my teenage days and change one thing, it wouldn't be scheduling, prioritization, visualization, environment-management, monotasking, and breaking down tasks into little chunks. (Though all of those things are very helpful.)

It would be to eat a hearty breakfast every morning. I'm thoroughly convinced on hindsight that it makes the biggest difference. You need a clear mind to do all of the hard, messy work of dealing with reality, and to have a clear mind you need to eat, hydrate, sleep and exercise.

<comment><USER4> If I could tell my freshman year self one thing it would be that the folks who say "sleep is for the weak" are full of shit.

<comment><USER5> I attribute my success as a _CS_ undergrad to this, and to not having Internet connection at home.

A fresh mind and not having the whole world a click away makes wonders. My home kind of promoted long sleeping hours, no late computing and always refused to give me access to Internet.

I studied with books and did programming assignments offline. Virtually everyone else was online already at the time (mid 00s). Even my 80 year old neighbour.

I missed one discrete math deadline that got changed because I did not check my email. The prof. couldn't believe my excuse, no Internet at home.

I ranked top 1 of my country. Now I'm happily surfing the web, and being a lot less productive.

</comment></comment></comment><comment><USER6> "It seems to me that Hughes wanted to be a writer more than he wanted to write; the difference isn't always obvious, even to the person doing the wanting, and talent, which you feel ought to be a clue, may be a red herring."

Wow, that sounds like me and research!

"Some see procrastination as a rational preference: the procrastinator has chosen immediate over deferred gratification, pleasure over work. But generally the failure to work goes along with a failure of appetites: a lot of the time I'm chained to my desk as a ghost is chained to the spot they haunt. It doesn't even have the glamour of writer's block."

So much this—instead of working hard during the day then doing something fun in the evening, one procrastinates all day doing half-fun things (like reading low-brow internet news sources, _ahem_ ), then spends the evening wallowing in self-flagellating guilt, and angst, since the deadline slowly comes ever closer! How to break the cycle? The next time i write a paper, my time management will be better, i promise! The next review will be in on time, really! Who is being deluded? The habit must change. That sounds like the self-flagellation phase, cue the coda!

</comment><comment><USER7> "Reading as a way of putting off thinking; thinking as a way of putting off feeling."

Poetic ending.

<comment><USER8> Perhaps... but it's also psychological science. People routinely play out that exact pattern.

</comment></comment><comment><USER9> The article is written with great humility and self-effacing humour and is a confession of Robert's battle with a condition we all suffer from: a universally human problem of procrastination. That alone makes this writing magnanimous.

There is so much humanity and compassion in Robert Hanks's appreciation of this aspect of his life - and ours. We all wish we could say that's cure enough though! Self-forgiveness & confessions aren't going to fix the problem of chronic procrastination ... but what they do is make us _feel_ better. And that's always a start. Feeling better is Stage 1 of fixing any of our problems. Starting to fix the real thing, like making lists and ticking things off and doing Coveyian shit is always going to be Stage 2.

</comment><comment><USER10> I put off lots too, but I think I am more like Bartleby. There is just an endless array of stupid, boring shit that I do not feel like doing. And it doesn't matter how hard I try to screw up the gumption to power through tedious shit - I really would prefer not to. I suppose on some level I feel that it is wasted time either way, so, might as well enjoy wasting it in some modicum of comfort.

<comment><USER11> Just don't complete non-central tasks until someone shouts at you. Then you do it. Can cut out a fair chunk of the 'boring shit' that most large organisations ask for. Over time - like a bit of machine learning - <USER2> develop criteria that enable you to identify the 'boring shit' that will lead to shouting, then you do that first.</comment></comment>


Saudi Arabia arrests economist after he criticises Crown Prince's plans - <AUTHOR> https://www.independent.co.uk/news/world/middle-east/saudi-arabia-economist-arrest-mohammed-bin-salman-terror-charges-mbs-name-of-economist-edited-a8564991.html <comment><USER2> The Crown Prince is incredibly influential in the tech sector as I believe he controls the purse strings that fund Softbank's +100B Vision fund.

</comment><comment><USER3> the absence of HN comments is deafening

<comment><USER4> You would be wrong to think HN is against establishment.

<comment><USER5> What an amazing misunderstanding of what today's establishment amounts to.</comment></comment></comment>


Chinese law will force game makers to reveal loot box drop rates - <AUTHOR> http://www.pcgamer.com/chinese-law-will-force-game-makers-to-reveal-loot-box-drop-rates/ <comment><USER3> I'm surprisingly okay with this, although I feel it should really only apply when the "box" contains any rewards that can be traded/resold.

It's not perfect, but it's an 80/20 thing that would give a clear exemption to a lot of innocent loot scenarios while still covering most of the problematic/scummy ones.

</comment><comment><USER4> Diablo 2 would have been no fun if it were like this. One of my favorite things about playing was discovering newer and more powerful items. This basically forces every publisher to publish spoilers. It's not the end of the world but it kinda sucks.

<comment> norea-armozel I think for China it's a bit more complicated since you didn't pay Blizzard for keys to open lockboxes which decides what drops. In Diablo 2 you farmed like everyone and that pursuit was fun in itself. Whereas these companies and their games are basically digital slot machines where you farm for the lockboxes and then spend extra to just open them in the hopes you get any return. I remember reading about one woman in China who literally spent thousands of US equivalent dollars on this. It's really sad and it should be treated like gambling where you are told your odds.

<comment><USER4> I played Diablo 2 in China.

</comment></comment></comment><comment><USER5> Can anyone explain what problem is being solved by this law?

<comment><USER6> Without odds, players can't make informed decisions about their spending. They have no idea whether a rare drop is 10%, 5% or 0.002% of the time. So this regulation will help them calculate the probable cost of acquiring the item.

Example: If it costs $0.99 to open a box, and the desired item drops from 4% of boxes, then it will cost you $9.90 for a 33% chance of getting it. But if the drop rate is 0.5%, $9.90 only gives you a 5% chance of getting the item. At some point you might just decide to buy the item on the secondary market (and indeed, these odds will influence secondary market prices).

(I personally think that these loot crates are effectively gambling. Even if the developers claim the items have no monetary value - because they have to, otherwise they're clearly offering you odds of a return on investment - <USER2> same developers often operate secondary markets which assume the items have a value, e.g. Valve and its Steam trading service.)

<comment><USER5> In games where there's no way to 'export' real cash (at least not without breaking the TOS), does it matter? I can see the usefulness of the rule in that case (although I still don't think it should be a law), but here it's the difference between making one set of pixels appear on your screen vs another. Could an unbalanced/unfair/opaque game be less fun? Yes, for some players. But I have difficulty grasping why even an authoritarian regime feels it needs to regulate _how fun a game is to play_.</comment></comment></comment>


How do we block domains once apps incorporate their own DNS over HTTPS? - <AUTHOR> Original DNS was easy to see and manipulate, and that was a problem once it left the private network - it enabled ISP's and other close-hop providers to catch a glimpse of what is being viewed online.
Inside the private network it was great because it could be detected and manipulated as necessary - redirecting rogue DNS queries to the desired DNS servers if specific software and apps were attempting to use their own hard-coded DNS servers (to bypass ad-blocking etc.), blocking unwanted domains and sites, etc.
Using Pi-Hole my current DNS request block percentage is 45.8%. I specifically redirect DNS requests through it using firewall rules. If I remove or disable the DNS redirection firewall rule this percentage falls to around 25 - 30%, meaning that there are a number of internal devices and software that are using specific, custom DNS servers rather than what my DHCP server hands out.  I don't like that, but vanilla DNS lets me close that potential security hole.
What if these devices and software start using internal DNS over HTTPS (or any of the other forms of non-inspectable or non-specific-DNS-port DNS requests) settings that are non-configurable?
How do we even detect that it's occurring, never mind being able to manipulate it to comply with the 'policies' of our private networks? <comment><USER2> If this becomes prevalent, there will be considerable outrage. So people will be discovering which devices and apps use tunneled DNS. Restricting the universe to open-source stuff, that shouldn't be that hard, just from the code.

Also, although blocking domains will be harder, you can always block IP addresses and ranges. DNS requests may be tunneled, but apps are still just connecting to IP addresses.

So you can compare IP addresses resolved by trusted DNS servers with those that an app is requesting. And block whatever you don't want. It could become tedious, but one could build something like Pi-Hole that was easily configurable on the fly.

</comment><comment><USER3> Right now, it's quite easily to turn off Firefox's use of DNS-over-HTTPS, using about:config (or, like me, blocking the "canary domain" via my internal DNS servers).

On a larger scale, however, this won't work. We can't expect every application to honor such "workarounds" \-- they will, of course, do what's best for them.

That's my biggest problem with this "movement" \-- removing the choice from the end user (at home) or those managing the network (at work).

Yet again, I think it mostly comes down to "we know what's best for you".

</comment><comment><USER4> Chromecasts already do this with Google's DNS. If you try to reroute DNS traffic to the servers of your choice, or block Google's DNS servers, the Chromecast will refuse to function.

</comment><comment><USER2> Just don't use those devices or apps.

Or perhaps there's some way to MitM them?

<comment><AUTHOR> It won't necessarily be obvious which devices or apps are doing it.

One of the reasons DNS over HTTPS has gained popularity is because it's not possible to MitM it (easily, at least). That's kinda the point of it's existence.

<comment><USER2> If devices do it, that will undoubtedly come out.

For particular apps, you can check against wget, w3m, etc. Or just dig. Using whatever DNS server you specify.</comment></comment></comment>


D3 v4.0.0 released - <AUTHOR> https://github.com/d3/d3/releases/tag/v4.0.0 <comment><USER2> D3 has the reputation of being super-complicated because of all the libraries that are based on it, "simplifying" it so that everyone can use it. In the past year I wanted to create pretty unique type of data visualisation, so I dived into D3 and discovered it a makes a lot more sense than I though. Of course, if you only want a regular bar chart, you'll do better with things like C3, nvd3 etc'. But if you want anything a bit special, D3 itself is very powerful and the documentation in pretty good - there's no reason to avoid using it directly.

Definitely looking forward to try the new release.

<comment><USER3> To add to that, if you are a complete newbie to any data visualization, do not _start_ with d3. If you want to make pretty charts programatically, using R/ggplot2 or Python/Seaborn is good enough. Even Excel is fine if you tweak the defaults.

D3 is good if your visualization _benefits_ from interactivity, either with dynamic data adjustment or rich tooltips. But static visualizations are important too. (I recently restructured my workflow so I can output static images _and_ interactive charts with the same code, which makes it the best of both worlds.)

<comment><USER4> Any recommendations for doing realtime charts that update a few times per second?

<comment><USER5> I have used Epoch[0] and Rickshaw[1] before for similar purposes, and have heard that Plotly works well too (especially if the process that is generating the data is Python-based). For simple line charts, maybe Smoothie[2]?

Currently, I am trying to make a dashboard for visualizing many different data sources with WebSockets and Chartjs but it's still not quite ideal-- there's a bit of lag, and sending too much data via JSON really slows things down. If anyone has solved that this problem I would appreciate it if they told me how they did it.

0\. [http://epochjs.github.io/epoch/](http://epochjs.github.io/epoch/)

1\. [http://code.shutterstock.com/rickshaw/](http://code.shutterstock.com/rickshaw/)

2\. [http://smoothiecharts.org/](http://smoothiecharts.org/)

<comment><USER6> for a fair bit of interactive data updating every second over websockets, ive had some success with DCjs.

On ios, the performance wasnt the best though

it was replaced with straight d3 i think, or maybe something rendering on canvas

</comment></comment></comment></comment></comment><comment><USER7> Fantastic work! D3 has a large API surface area, due to the functionality it covers. So even a relatively conservative major upgrade is a TON of work, and some of it might not have been pure fun only.

1) What's your opinion on the apparent case that the bulk of changes come from you, as society promotes teamwork so much? Is it a case of 'small team efficency'? (Btw. I know of others' contributions too, and reliance on Rollup, ColorBrewer, matplotlib/Viridis etc.)

2) What kept you going? It's probably at least a person-year of work just on your side that's not paid for by companies, not to mention some kind of startup bet. Have you alternatively toyed with the idea of starting a value- added layer, since so many for-profit organizations benefit from your work?

3) While it changes the API, and most everything is rewritten, it's IMO a conservative upgrade in that it doesn't steer away from familiar concepts and structure. It still does DOM binding, transitions, layout etc. fairly similarly. Have you considered much more disruptive departures? As an example, React's DOM diffing is more radically different from D3 selections than the change from D3 3.* to 4.* or the Grammar of Graphics as an API concept is more radically different from D3 than the 3.* -> 4.0 API changes. Or something like moving away from the 'functional objects' concept in favor of curried, fixed- argument pure functions, perhaps prefaced with the familiar chaining API would have been a large change (these are mostly examples rather than wishlist). What were YOUR ideas, had you wanted more disruptive changes, or 'D4'?

<comment><USER8> IIRC a key point of d3 was to not be declarative, like the gog. That was more an influence in protoviz, its precursor.

You might be interested in Vega and Vega lite if you want to go declarative.

<comment><USER7> I'm contributing to plotly.js right now, which is also declarative, and I see the pros and cons of levels of denotative semantics (mostly depends on goals and library user requirements) but it was just an example for what I mean by more radical changes. Good you mentioned it anyway, protovis -> D3 also represented a rather disruptive change. But perhaps I was late with these questions for this specific AMA.

</comment></comment></comment><comment><USER9> Changelog (weekend reading): [https://github.com/d3/d3/blob/master/CHANGES.md](https://github.com/d3/d3/blob/master/CHANGES.md)

Really excited about this release. It seems like nothing has been left untouched with major changes to everything from selections to layout generations and the modularisation of the entire lib (think lodash).

There are major namespace changes with basically everything being flattened ( d3.geom.voronoi -> d3.voronoi) and many of the typical patterns will have to be relearned (major changes to selections), so I don't see upgrading any existing projects and I may have to hold off on using 4.0 in new projects until I actually understand what has happened.

Favourite thing so far has to be the changes to selections, which I think will make things a lot clearer to newer users: [https://github.com/d3/d3/blob/master/CHANGES.md#selections-d...](https://github.com/d3/d3/blob/master/CHANGES.md#selections-d3-selection)

<comment><USER10> Aye, a lot has changed! I know change in itself is a pain; despite the quantity of changes I did try to avoid unnecessary change, and batch the changes into a rare major release. I believe such change is worthwhile, and that it's better in the long run to keep improving the API than to be tied down to past mistakes.

The renames (like d3.voronoi instead of d3.geom.voronoi) are superficial changes, so although it might seem disruptive, it should be pretty easy to update your code and habits. At least, I haven't found it difficult… The more design-y changes, like the new d3.stack and d3.treemap, require thought to migrate existing code. But the new designs are much better, so I hope you find the effort worthwhile.

<comment><USER11> I know Perl6 and Python3 have shown the dangers of changing things in disruptive ways. But I still appreciate your willingness to change things carefully to make them better. The length of time your software will be used in the future almost certainly outweighs the length of time it was used in the past :-)

<comment><USER10> I thought about the Python 2/3 debacle a lot this past year, and yes, I hope this isn't a repeat. The Python migration was especially hard because of dependencies between libraries (you could only migrate your library or application to Python 3 after your dependencies migrated), and because Python is used for many large, long-term projects. Certainly D3, too, has its own plugin ecosystem and long-term projects, but it's also commonly used as a standalone library in short-term work, such as in news graphics—where migration should be much easier. So hopefully that pain is lessened.

</comment></comment></comment></comment><comment><USER12> Just want to say how much I appreciate the refactoring into smaller libraries. d3 has enough traction that it's not something they had to do, but it's the right thing to do, and probably took a good deal of effort and thinking.

<comment><USER10> Thanks. It's definitely more fun to work on the pieces of D3 individually, rather than feeling the weight of a massive monolith. Hopefully it will encourage others to develop more reusable libraries that depend on (the relevant parts of) D3, too.

<comment><USER13> Just wanted to say that your work is very much appreciated. I've been using D3 since it was Protovis and I love it. Excited to dig into the changes and try it out.

</comment></comment></comment><comment><USER14> What's the best way to integrate D3 and react? Or there's a better approach to charts/visualization in react than using D3?

<comment><USER15> I've recently been experimenting with pure SVG and React, which seems like a really good fit. Just draw SVG in your render function as naively as possible, and let React figure out the DOM transitions. Super easy to debug and reason about.

Not being too familiar with D3, what would I get by adopting it? I would hope for higher level primitives, but the API seems imperative and stateful, which doesn't seem like a good fit with React. Would I end up duplicating and syncing my existing app model state into D3 state?

<comment><USER16> We built a workflow design ui with svg under react and for the most part it is an absolute pleasure. Also pretty neat that you can treat the svg as an image/thumbnail. If you need the svg to be interactive, overlaying html works well too.

Example at [http://flowgear.me/s/1Ae7Pso](http://flowgear.me/s/1Ae7Pso) if you're interested.

One thing to watch out for is that there are some svg object attributes (and some objects) that aren't declared in react.

<comment><USER7> You can generally use them with 'is', for example: <marker id={'bar'} is markerHeight={viewModel.height} refX={viewModel.xCenter} markerWidth={viewModel.width}>

</comment></comment></comment></comment><comment><USER17> Does anyone know if there is a good way to do server-side rendering (svg files) of D3 charts while being able to reuse the same code in a browser?

We do a lot of charts at work to visualize electricity consumption and are currently thinking about how to rewrite the js we've accumulated over the years. I've found some attempts at doing server-side rendering but nothing really compelling for now...

D3 is a well thought out library to do interactive visualizations in the browser and having an easy way to do svg exports (for things like automated emails) would be a big plus. Doing screenshots through selenium ain't always so pleasant ;)

<comment><USER18> Not sure if it fits your use case, but; for one project I needed to have charts that the user could modify in their browser, and once they were happy they could hit 'download PDF' to get a PDF version.

Achieved this by (admittedly slightly clunky way) of sucking the SVG out of the chart and sending it to the server via ajax. Server intercepts SVG and passes it to inkscape which does pretty good command line svg -> pdf... then, sends back completed PDF.

One gotcha is that you won't get any CSS this way, so all styling has to be done via SVG props.

<comment><USER19> That's exactly what we are doing but running the SVG code through Apache Batik Rasterizer to turn it into png. I like this approach because the user can customise the chart in the front end and then the same resulting svg code is rendered by the backend. The only issue is styling as you mentioned.

</comment></comment></comment><comment><USER20> 1\. Are there any plans to have a TypeScript version of the lib?

2\. Would it be possible to add jsdoc comments for autocompletion?

3\. Are the polyfills for Map and Set still necessary now that we have that in es2015?

4\. Why is the lib so heavy on argument overloads?

5\. Are there any code style restrictions? I often saw loops and conditions without curly brackets and alike.

6\. Are there any fields you'd need support in for maintaining or reworking the library?

I really like D3.js although I haven't found an application for me yet. I find it hard to get/update the date from a none standard REST endpoint and display them, especially if they are kind of multichanneled, like working times of multiple users unordered.

</comment><comment><USER21> Looks like quite a lot to take in, where should we start? - are there updated tutorials, upgrade guide etc?

<comment><USER10> The place to start is the release notes: [https://github.com/d3/d3/releases/tag/v4.0.0](https://github.com/d3/d3/releases/tag/v4.0.0)

And the detailed change list: [https://github.com/d3/d3/blob/master/CHANGES.md](https://github.com/d3/d3/blob/master/CHANGES.md)

I will update my tutorials and examples (and probably write new tutorials) in the near future, and I expect others will, too. But as I now have thousands of examples, I didn't want to hold the release on updating all of them. My newer examples use 4.0 already, and I'll be going through the others in the coming weeks.

<comment><USER22> Does this mean you intend to update your existing examples in place? It might be nice if you forked them with v4 instead to create a defacto upgrade guide.

I know many tutorials reference your examples and it might create more confusion if they are updated in place.

I suppose it depends on how much you want to keep supporting v3, but I'd argue its worth keeping the old knowledge base intact for a while.

<comment><USER10> Yep, I'm going to update my examples in-place. Consider it my heavy-handed encouragement to upgrade to 4.0. (For one thing, there are lot of 3.x bugs fixed in 4.0, and I have limited resources to provide free support to old versions.) My examples are backed by git repos, so you'll still be able to find the 3.x versions if you need them.

<comment><USER22> fair enough!

</comment></comment></comment></comment></comment><comment><USER23> Is it possible to have V3 and V4 coexisting in a project? Like, adding some of the awesome v4 new things to an already working v3 project without having to rework all the v3 code to v4 compliance?

I've tried doing this with v4 beta, with strange behaviours :)

<comment><USER10> Hmm. It might be possible, though it doesn't sound like a great idea to have both D3 3.x and the D3 4.0 default bundle loaded simultaneously, since there will be a lot of overlap. A better strategy might be to load just the new modules you want to use. Like if you just want the new force layout, then load d3-force: [https://github.com/d3/d3-force](https://github.com/d3/d3-force)

But it might be best to just migrate to 4.0. I know the changelist is long, but it might not be as hard as you think to migrate. And you can always send your questions to the d3-js Google group or d3.js on Stack Overflow and I'll try to help.

<comment><USER24> On a project I work on* we've found that this technique has worked well so far. We're slowly moving from v3 to v4 by splitting out groups of components into their own packages and upgrading them as we go. We've been heavily inspired by d3's approach e.g. with the use of rollup for bundling.

We're probably at about 60% of the library upgraded/split-out but we've yet to tackle the more tricky changes like d3-selection.

* [https://github.com/d3fc](https://github.com/d3fc)

</comment></comment></comment><comment><USER25> Exciting release! I played with the force layout on canvas to graph reddit conversations [http://ramiro.org/tool/graphit/](http://ramiro.org/tool/graphit/)

It's harder to get interaction done with canvas compared to SVG, but for large graphs the performance gains are huge. Curious to dive into the other new features.

Thanks for this incredible piece of software mbostock!

</comment><comment><USER26> For those interested making d3.js reusable for data analysis, check out [https://vida.io](https://vida.io). We create templates out of d3.js visualizations with dynamic properties. You can create dashboards from custom d3.js components. For more examples, see [https://vida.io/explore](https://vida.io/explore).

</comment><comment><USER27> Modularization aside (and it's terrific to have in the library), what additional functionality are you most excited about with this release?

<comment><USER10> Let's see, in no particular order…

* New pretty continuous color schemes (viridis, rainbow, ColorBrewer)

* New axes that are easier to use.

* Vastly improved circle-packing layout.

* More stable, extensible force layout.

* Cleaner implementation of immutable selections & transitions.

And a ton of other stuff. Taken overall, it feels like a big step forward in robustness and cleanliness. And I'm hopeful that it'll be easier for folks to develop more D3 modules.

</comment></comment><comment><USER28> For simpler need such as rendering a SVG map from Topojson or GeoJSON data, is there an alternative than D3 that can utilize React.js' rendering of SVG? With no animation, no charts, just map with zoom in and out and drag around, is there a benefit of using D3?

</comment><comment><USER29> Did they remove the emphasis n the .enter() pattern yet? This is the weirdest thing compared to how just about every other library works, and it can totally be avoided by just using the data() stuff directly.

</comment><comment><USER30> Any 4.0 tutorials available yet to dive in and try it out for a first-timer?

<comment><USER25> Many of mbostock's recent "blocks" use D3 v4 [http://bl.ocks.org/mbostock](http://bl.ocks.org/mbostock)

These are not tutorials, but working examples with some introductory texts. They are very helpful for diving into D3.

</comment></comment><comment><USER31> Why didn't they name it D4

<comment><USER32> Because D3 is the actual name of the library. The full name is "Data-Driven Documents v4.0.0". Shorten that and you get "D3 v4.0.0".

<comment><USER6> and there's a derivatived library called D4 as well

<comment><USER33> Derivatived Data Driven Documents?

</comment></comment></comment></comment><comment><USER34> dsadsds

</comment><comment><USER35> More comments here [https://news.ycombinator.com/item?id=11994410](https://news.ycombinator.com/item?id=11994410) including from @mbostock. Not sure why it didn't stay on FP.

<comment><USER36> Thanks, we've merged that thread with this one.</comment></comment>


Time Machine: H.L. Mencken's 1925 Review of 'The Great Gatsby' - <AUTHOR> http://www.chicagotribune.com/lifestyles/books/ct-prj-great-gatsby-f-scott-fitzgerald-hl-mencken-20141010-story.html#page=1 <comment><USER2> A fine piece of writing: "The thing that chiefly interests the basic Fitzgerald is still the florid show of modern American life — and especially the devil's dance that goes on at the top. He is unconcerned about the sweatings and sufferings of the nether herd; what engrosses him is the high carnival of those who have too much money to spend, and too much time for the spending of it. Their idiotic pursuit of sensation, their almost incredible stupidity and triviality, their glittering swinishness — these are the things that go into his notebook."

</comment><comment><USER3> Also see Mark Twain's _" Fenimore Cooper's Literary Offences"_[1], reviewing some of the works of James Fenimore Cooper (of _" The Last of the Mohicans"_ fame).

A sample:

      In his little box of stage properties he kept six or eight cunning
      devices, tricks, artifices for his savages and woodsmen to deceive
      and circumvent each other with, and he was never so happy as when he
      was working these innocent things and seeing them go. A favorite one
      was to make a moccasined person tread in the tracks of the
      moccasined enemy, and thus hide his own trail. Cooper wore out
      barrels and barrels of moccasins in working that trick. Another
      stage-property that he pulled out of his box pretty frequently was
      his broken twig. He prized his broken twig above all the rest of his
      effects, and worked it the hardest. It is a restful chapter in any
      book of his when somebody doesn't step on a dry twig and alarm all
      the reds and whites for two hundred yards around. Every time a
      Cooper person is in peril, and absolute silence is worth four
      dollars a minute, he is sure to step on a dry twig. There may be a
      hundred handier things to step on, but that wouldn't satisfy
      Cooper. Cooper requires him to turn out and find a dry twig; and if
      he can't do it, go and borrow one. In fact, the Leather Stocking
      Series ought to have been called the Broken Twig Series.

[1] - [http://www.gutenberg.org/files/3172/3172-h/3172-h.htm](http://www.gutenberg.org/files/3172/3172-h/3172-h.htm)

</comment><comment><USER4> Mencken describes Fitzgerald's increasing skill as a writer as a rare thing.

I would argue that among really good writers, it's not rare at all. The usual course is for an author to keep learning for years, figure out what works well for them, and perhaps lose a little ground towards the end of their lives due to failing health or lack of care.

Similarly, good technologists build a workshop full of techniques that experience has given them, do their highest quality work in middle to late middle age, and then retreat from active building to teach, critique or advise.

<comment><USER5> >Mencken describes Fitzgerald's increasing skill as a writer as a rare thing.

It's not rare among really good writers, Mencken is saying a really good writer is what is rare.

<comment><USER6> With "a really good writer" being defined _not_ as one who produces one good book.

</comment></comment></comment><comment><USER7> "The trouble with H.L. Mencken is that he believes he is H.L. Mencken. There is no cure for a delusion of that magnitude."

... is a quote I distinctly remember reading many years ago, and which I cannot find a single reference to anywhere on the internet. Guess I'm just crazy?

<comment><USER8> If that's the extent of your craziness, you're doing pretty well:

[http://www.newstatesman.com/science- tech/internet/2016/12/mo...](http://www.newstatesman.com/science- tech/internet/2016/12/movie-doesn-t-exist-and-redditors-who-think-it-does)

</comment></comment><comment><USER9> I'd be curious to know how this was transcribed: it has "Totentons" evidently for "Totentanz". I seem to reread the book every _n_ years, and often as not change my mind from the previous reading.

<comment><USER10> > it has "Totentons" evidently for "Totentanz".

Same goes for "fidelis ad urnum", which should read "urnam".

<comment><USER11> And probably gaudy in "gandy villas and bawdy house parties".

<comment><USER12> I imagine if not originally dictated, his draft was hand written and submitted directly for typesetting.

While possibly Mencken's errors, a linotype operator botching a slug or an editor misreading a partially open 'a' wouldn't be the severe error it is today in the age of autocorrect and instantaneous type production.

When you post a comment, there's just a screen between you and all of the internet and its information. Mencken might have had weeks or months and a dozen people's hands between his review and its printing. Not to mention reprintings, revisions, and the eventual OCR close to a century later.

</comment></comment></comment></comment><comment><USER13> I don't think I'd describe Gatsby as a 'glorified anecdote'. It's fairly tightly plotted, as much as any hollywood blockbuster. But the writing does far outshine the story. It's absolutely beautiful, if perhaps a bit mawkish.

</comment><comment><USER14> For those with adblockers: [https://archive.fo/eazBq](https://archive.fo/eazBq)

</comment><comment><USER15> It's surprising how much Gatsby has in common with Dandy characters like Dorian Gray. Never noticed until Mencken's review.

<comment><USER16> Two books I loved when I read them in my early to mid twenties. I fell in love with their main characters and the decadence.

</comment></comment><comment><USER17> If you've not read his piece on Chiropractic, it is worth it. [https://www.chirobase.org/12Hx/mencken.html](https://www.chirobase.org/12Hx/mencken.html)

<comment><USER18> Most of that piece is contemptuous rhetoric, like "the simple sentimentality of a somewhat sclerotic fat woman" from the OP but without the more interesting analysis. It takes the worth of doctors of his time for granted, mentioning tonsillectomy and adenoidectomy: at least the former was much overprescribed then, I don't know about the latter. A chiropractor won't offer to remove any organs you're not sure of the use of. (I've never seen one myself.)

I do like older manipulative writing for being more obvious to us, because we're not quite the target anymore. A kind of vaccine.

</comment></comment><comment><USER19> The recent Family Guy spoof nailed its weaknesses pretty well, I thought.

</comment><comment><USER20> ++spoiler alert++

</comment><comment><USER21> > To find a parallel for the grossness and debauchery that now reign in New > York one must go back to the Constantinople of Basil I.

H.L., honey, you have _no idea_...

<comment><USER11> Between Mencken's day and today, there was a book titled Hollywood Babylon, with the expectable content.

We're probably into solid Paleolithic territory by now ...

</comment></comment><comment><USER22> Regardless of the subject matter and opinions expressed, I think any reviewer who decides to give away all of the plot details within the first few sentences of a review should not be given much (if any) merit.

<comment><USER23> That's an interesting perspective. I guess there's a philosophical question: what is the point of reviews? If it is to help determine whether to read or see something, then I guess I agree with you that this sort of thing largely defeats the purpose. But if it is instead to have a sort of "conversation" with the reviewer about their thoughts on something you've already read or seen, then this sort of recap is less problematic.

I think I use reviews for both purposes. Using movies as an example, I'll check rottentomatoes before-the-fact to see whether there is at least a vague consensus that the movie I'm about to pay to see isn't going to be a total waste of my time. But then after I've seen it, I'll sometimes also go read what a smaller set of reviewers who I enjoy had to say about it, and compare and contrast their thoughts with my own. Both things have value to me, but in very different ways.

<comment><USER24> It's unfortunate that there aren't separate widely-used terms for recommendation style reviews and critical breakdown style reviews, because it makes it hard to find the latter when I want it. For an example, Jamahl Epsicokhan has written reviews of all the episodes of Star Trek on his site Jammer's Reviews. They break down both the plot and give criticism of the entire work, including parts that would be spoilers. Sometimes you can find these kind of articles under the term "rewatch" though.

</comment></comment></comment><comment><USER25> The last three paragraphs are interesting if you think of Trump as a descendant of Gatsby's NY.</comment>


Tool's survey - <AUTHOR> https://www.surveymonkey.com/r/JJ2B5L7 <comment><AUTHOR> My name is Shogoufah, I am a master student in the Faculty of Computer Science at the Free University of Bolzano in Italy. Currently I am working on my final project in which we are evaluating online tools to support startups, so that we can better suggest them to startups that are looking for supporting tools. I would like to kindly ask you to give us 10-15 minutes of your time to fill in our survey on Show hacker news. Thank in advance

</comment><comment><AUTHOR> Reminder: I need your collaboration to regarding the survey, please give us 10-15 minutes to fill in our survey. Thanks</comment>


I will be visiting Tokyo and Japan in January. Non obvious recommendations? - <AUTHOR>

<comment><USER2> Robot Restaurant - [http://www.shinjuku- robot.com/pc/?lng=en](http://www.shinjuku-robot.com/pc/?lng=en)

I haven't been yet but friends have said it's one of the most wild and crazy shows you'll ever see.

</comment><comment><USER3> Not sure what is non-obvious to you or what kind of recommendations you're looking for...

Anyway, there's a semi-regular HN meetup going on in Tokyo:

[http://hntokyo.doorkeeper.jp/](http://hntokyo.doorkeeper.jp/)

There's also Mobile Monday and Startup Weekend.

For general touristy info, check /r/JapanTravel on Reddit.

<comment><AUTHOR> Thanks a lot! By non-obvious, I meant things that I can't easily find on the internet by myself.

</comment></comment><comment><USER4> Bitcoin Meetup. Every Thursday, usually in Roppongi.

[http://www.meetup.com/Tokyo-Bitcoin-Meetup- Group/](http://www.meetup.com/Tokyo-Bitcoin-Meetup-Group/)</comment>


Convore (YC W11) Introduces iPhone App - <AUTHOR> http://blog.convore.com/post/3748606742/convore-iphone-app

<comment><USER2> At the very top in the header of your blog you have an about button, but I didn't click it, I clicked the close tab because you didn't say what in the world Convore does. That is fine if you come from some other page or hit the blog every day, but when coming from a news site you get deep linked to some random blog entry it isn't clear.

<comment><USER3> Or, more succinctly: The page should open with the about info pre-opened.

I had no idea what convore is, clicked on "Convore", and promptly ended up at the exact same page (just with the full timeline visible), then finally ended up clicking the tiny "About" link.

</comment></comment><comment><USER4> I really like Convore, so far. Hopefully it does't turn into another troll land.

<comment><USER5> unfortunately, that means it's probably too unsuccessful to make any money.

</comment></comment><comment><USER6> Interesting timing -- it wasn't very long ago that some iOS developers were organizing together to build an iPhone app. Not sure if this is a product of their work or if it was developed independently of those efforts.

<https://convore.com/iphone-development-sdk/convore-for-ios/>

<comment><USER7> We built this in-house, but we're excited to see that other people are building out apps too. (I'm an Android user myself, and use the third-party Convorsation app <https://convore.com/convorsation/> )</comment></comment>


Lyft adds a new app for drivers - <AUTHOR> http://www.businessinsider.com/lyft-adds-scheduled-rides-new-app-for-drivers-2017-5 <comment><USER2> I think this is a very good idea! I know many people around where I live who use local Uber drivers whom they trust to schedule early-morning rides to the airport and things like that. The rider will have the driver come to his/her house at the specified time, and then the driver and the rider will both turn on Uber and the rider's request will be filled by that driver because nobody else is nearby that early. This would would make it much easier to do something like that for both the rider and the driver, and I see this as an important use case because it's really the only remaining one which many people would prefer to use a car service or taxi service than Uber/Lyft. It's also much more profitable for the driver so drivers will like this a lot. Great move on Lyft's part, maybe they could consider allowing riders to directly send requests to certain drivers. The point here being that riders using Lyft like a car service have drivers they trust to come on time, not cancel, etc. which are the only potential problems I see here.

<comment><USER3> If they're going to the trouble of pre-arranging the ride, why don't they just use cash/Venmo/Square, and keep Uber's cut of the fare?

<comment><USER4> Assuming you mean, other than the ethical concerns?

A) Who's insuring the ride? If it's not insured and the driver is at fault in a collision, there could be no one to pay for injuries to you -- Lyft/Uber certainly wouldn't owe anything for stuff booked outside their platform.

Usually when they drive for a TNC, the TNC provides the insurance for the driver in the course of using the app. It's really unlikely for the driver to have a policy that covers them for non-TNC freelance rides -- especially since the big insurers haven't caught up to the realities of this use case and seriously overprice them.

(AIUI, they just have policies that assume you drive full time commercially, nothing that works on a per mile basis -- and even if they do have such policies, that would defeat most of the purpose of cutting out middlemen, as there would now be a written record of the transaction that can't be hidden on taxes.)

B) In order to get enough information to re-book out-of-band you'd have to accept it and then cancel. Do that a lot and it will be grounds for being kicked off the platform.

(The same applies to the people in the anecdotes in the sibling comments.)

<comment><USER5> Out of curiosity, what are the ethical concerns? The arrangement is not being made by the middleman company. I don't see how it is unethical to not pay them for something they did not do.

Point (B) doesn't make sense to me. Why do you have to accept and then cancel? You just get the driver's phone number and talk to them directly.

In fact, if this is _not_ common already, I think it undermines the contention of these companies that they are merely a service being provided to independent contractors. If drivers are independent contractors, _of course_ they should be trying to build their own brand and dedicated clientele.

Point (A) is a good one, though. Sounds like there should be independent insurance providers that these independent contractors can use.

If you're instead saying that drivers are _employees_ of these companies, then maybe you're right about all the rest of it. But you can't have it both ways!

<comment><USER4> >Out of curiosity, what are the ethical concerns? The arrangement is not being made by the middleman company. I don't see how it is unethical to not pay them for something they did not do.

Their service was connecting drivers and riders -- allowing them to find each other for the transaction. They connected you. That was their service. (They provide other related intermediary services, but the "finding" function is the big one.)

Same issue as when a recruiter finds leads and the employer makes an offer to the employee directly and cuts out the recruiter, which is why they set up a bunch of contractual barriers to stop it.

EDIT: Sorry, I misunderstood the use case -- that wouldn't apply to future bookings _after_ the first one's been completed and they've exchanged information. But the "moat" there is the difficulty of iterating through your network of past uber drivers for each future ride.

>Point (B) doesn't make sense to me. Why do you have to accept and then cancel? You just get the driver's phone number and talk to them directly.

AFAIK, they don't give you any way to contacting each other until the ride is accepted. Am I mistaken here?

>If you're instead saying that drivers are employees of these companies, then maybe you're right about all the rest of it. But you can't have it both ways!

No. You can have a legitimate claim to compensation for connecting two people without the seller thereby being your employee -- see the recruiting example (though it's a bit confusing in that there is another employment relationship in the picture but I'm sure you see the mapping -- a recruiter can legitimately claim that they're owed a cut while not saying that their the employer of the, er, hiring manage or the candidate).

EDIT: As above, that wouldn't apply to future rides or job offers after that contract.

<comment><USER6> There is no moral or ethical obligation to compensate Lyft or Uber for future rides not on their platform.

Also, it seems anyone could replicate this feature by using Stripe Connect. It's cool, but expect rapid commoditization of preplanned ride coordination.

Edit: comment recinded due to OPs edit.

<comment><USER4> >There is no moral or ethical obligation to compensate Lyft or Uber for future rides not on their platform.

Sorry, you're right, I misunderstood -- see the edit.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> So this is of course anecdotal but the only time I book taxis is when it's very important I get somewhere in time, like catching a plane early in the morning. Given the nature of Uber/Lyft it would take a lot before I would trust them with this.

<comment><USER8> Are taxis more reliable than lift/uber where you live? My experience in Australia is the opposite.

I never had a uber (no lift here) not showing up , while booking a taxi is almost like a lottery. Few years back our office manager booked 4 cabs for 6pm of the following day because we had a work function to go to. The first cab showed up at 6.50 after countless calls to the taxi company, the other never did.

We ended up calling 3 uber and we were on our way within 10 minutes.

That was the day I swore to never get on a taxi again, unless I really really had to.

<comment><USER3> I've had bad experiences like this with both taxis and Ubers. With Uber, I've had dozens of experiences where either (1) There were no drivers available in my area, (2) The drivers repeatedly canceled on me, or (3) The driver didn't cancel, but also never came to pick me up, and instead just drove in the opposite direction for 10-15 minutes.

I now don't trust taxis, Uber, or Lyft to get me to the airport very early in the morning. I live in DC, not exactly a small town.

<comment><USER9> This is why I still use an old-school limo company ("limo" as in fancy black car, not a real limousine) for airport pickups/dropoffs in my home city. Years ago I found one that's extremely reliable and trustworthy. Nearly five years later I haven't had a single issue.

Uber in general is much better than taxis, but I still wouldn't really trust it for time-critical early morning pickups.

</comment></comment></comment></comment><comment><USER10> I don't think scheduling a ride a few days in advance actually helps the drivers. One of the best things about Uber or Lyft for the drivers is that they don't have to think about scheduling, they're told exactly where to drive.

If they have a scheduled ride and they forget about it, or they're stuck waiting around to make that ride and lose business, that doesn't seem like it will be popular.

One big thing that Uber needs to fix is the incentive scheme. One of the Uber drivers I talked to said she hated how they were, in my words, gamified into earning their incentive bonuses. And the incentive bonuses change every time, so it makes it hard for them to keep track. That's something Uber needs to change in order to increase driver satisfaction, making it clearer and more consistent on how to earn the incentives which is where the drivers make more. I don't see how Lyft's "Power Zones" are any different from Uber showing where surge is occurring, but if it works, then great.

<comment><USER11> This could be something for Lyft drivers people who don't drive so regularly. You could for example spot gigs which you can handle while riding to work. If you know day or two beforehand, you can then adjust your schedule accordingly.

</comment></comment><comment><USER12> This could be a really good thing if you're an hour away from the airport where you have a morning flight, but a really bad thing if the driver decides that morning to not take you. :)

<comment><USER13> Presumably another driver would pick it up - it wouldn't just get dropped.

<comment> Paul-ish That would require there to be another driver, which in a less populous area at 4AM might not be the case. It seems like there would need to be steep penalties on both parties if they cancel near the arranged time.

Eg: Driver cancels 2 hours before pickup. Lyft penalizes driver x% of total fee, then uses that to "bid up" the price on that pickup to entice another driver to fulfill the request at the last minute. This has the potential to help attract drivers to go out of their way, which could help in less urban areas.

<comment><USER14> I assume/hope the driver app has reminders and confirmations before the ride.

</comment></comment></comment></comment><comment><USER15> With the caveat that I don't work for Lyft DevOps or anything (I only used to be a driver way back when)... If you need to make a critical update to the driver side your code is solidly firewalled from having to pass integration tests with the passenger side, and there's a possibility that some problems bleed through between the two different parts of the app. In the early days, there were a few updates where such things leaked through (if I remember correctly, there was one version update where the passengers accidentally could see the primetime map).

Quite frankly, I'm suprised they hadn't done it sooner.

</comment><comment><USER16> Grrrr... BI no longer works when you have an ad blocker installed.

<comment><USER17> It works with uBlock Origin with "Anti-Adblock Killer | Reek" checked under "3rd-party filters"

<comment><USER16> Suite! Thanks

</comment></comment></comment><comment><USER18> As a rider, I haven't noticed the Lyft app to be slow or especially bloated, so it will be interesting to see if it is significantly faster once the update pushes through with the driver portion removed.

<comment><USER19> I doubt it. If you aren't a driver I think it would be senseless to be running driver code at the same time as rider code. Maybe a little less disk (flash?), but I'm not certain. I'm neither a mobile dev or a Lyft employee.

</comment></comment><comment><USER20> Interesting, there's obviously a demand for this feature but I wonder if it will change the way the end user books rides substantially or if its just a nice to have.

</comment><comment><USER21> Chinese Didi had it long ago...

</comment><comment><USER22> Anyone got a non-BI link?

<comment><USER23> Here are the direct Lyft pages on the two new driver features mentioned in the article: Scheduled Rides [0] and Power Zones [1].

[0]: [https://blog.lyft.com/posts/introducing-scheduled- rides](https://blog.lyft.com/posts/introducing-scheduled-rides)

[1]: [https://thehub.lyft.com/power-zones/](https://thehub.lyft.com/power- zones/)

</comment></comment><comment><USER24> Everyone loves the underdog

</comment><comment><USER25> Why is this front page worthy?

Uber has had these features for a while now. As someone else mentioned, Didi has had these features too. 71 upvotes so far. This just further reinforces my view that there is an active voting ring on HN manipulating Uber and Lyft stories in Lyft's favor, which is ironic since many of the comments on such stories talk about Uber being shady.

Throwaway because I fully expect the voting ring people to downvote this.

Downvoters and those who upvoted this story, please leave a comment explaining why this is front page worthy.

<comment><USER26> > _Downvoters and those who upvoted this story, please leave a comment > explaining why this is front page worthy._

We don't owe you an explanation, but I'll offer one anyway: I upvoted because it's a story about a popular tech company, and the way they're trying to improve their offering for their semi-employees. Why wouldn't I upvote this?

</comment></comment><comment> e-sushi Let's see how that goes...</comment>


Microsoft: No browserless Windows 7 after all | Beyond Binary - CNET News - <AUTHOR> http://news.cnet.com/8301-13860_3-10301299-56.html <comment><USER2> Why force people to make a choice? It's in the best interest of the computer user to have a browser installed by default. Since Microsoft makes a browser, it makes sense to use that one with their operating system. End of story.

Yeah, there's history behind this, but really, it was TWELVE YEARS AGO. It's time to move on.

An operating system should come with some basic tools like a command shell, file manager, web browser, text editor, calculator, media player etc. As a user, if I want to upgrade any one of those things, I can do so. But please don't leave them off the system just because there are 3rd party alternatives. And please don't show me a screen full of options for every one of them.

This was a silly issue to squabble over back in the 90's, and it's only gotten sillier since.

<comment><USER3> Back then people didn't take issue with IE shipping _with_ windows, but _as_ Windows. MS tried its typical bullying tactics to squeeze out competitors. IE became so tightly integrated into Windows that it a) couldn't be removed without wrecking the OS, and b) enjoyed low-level (performance enhancing) OS access that was refused to competing browser/app makers.

So it was not silly to "squabble" over it. MS wanted to dominate the web as a proprietary platform and extension of Windows. This was a real threat back then.

You could be using "IE 5.8 service pack 14" right now, writing jscript (not javascript), MSHTML, and active-x plugins, enjoying horrific security issues exacerbated by a browser monoculture, and being treated as a second class citizen if you use any other browser than said IE 5.8, since you'd have to reverse engineer all the interfaces and get sued in the process to make active-x etc. work in competing browsers.

<comment><USER4> Your first paragraph made sense. I think you lost it in the IE5.8 world. Which is why you may have been down-voted.

I think the point being that removing/unbundling the browser may not be as important as making sure that users aren't only locked in to using your browser.

</comment></comment></comment><comment><USER5> This is a big win, IMO. As much as I despise Microsoft's monopoly over the desktop OS market (and to a lesser extent, the browser market), the decision to ship Win7 without a browser was douchebaggery on a whole new level. A browser is now an integral part of every OS, just like a file manager or an image viewer. Removing it does more harm than good, and only serves to confuse the average user.

</comment><comment><USER4> I was always suspicious of this 'no browser' thing. How did they expect people to select and then download a browser without a browser already installed?

Sure you could have installed files which connect to ftp or something like that, but is it really worth making new users go through that? I think the solution they are going with is great, as it lets average users know other browsers exist. But i suspect most will still stick with IE.

I am caught wondering if the placement of Google, and the big google logo might be a boon to Chrome, as lots of people I'm sure will start and think 'oh, i want to go to google', clicking the link and thinking that is what it does.

<comment><AUTHOR> r.e. the Google logo. That was my initial thought too - a great place to have your brand.</comment></comment>


Emacs now has multi-tty in CVS HEAD - <AUTHOR> http://blog.mwolson.org/tech/emacs_now_has_multi-tty_in_cvs_head.html

<comment><USER2> I'm really looking forward to emacs 23. Between this and freetype support it's going to be a big improvement.</comment>


Apache Killer Is The Biggest Little Internet Threat - <AUTHOR> http://www.fastcompany.com/1776321/the-biggest-little-threat-to-kill-the-internet-you-didnt-know-about

<comment><USER2> Yeah (gulp) I've checked this out and it's pretty serious. KingCope is a white hat security professional and luckily he's drawing more attention to the problem. If anyone hears more about this, post information here on HN.</comment>


Ask HN: Does the website design matter? - <AUTHOR>

One of my friends said that he does not care what a site looks like as long as it has what he needs. I have been designing and redesigning mine to make it look well good and I am just wondering whether I should bother at all. <comment><USER2> Others have mentioned that there are different definitions of what "design" means, but I will assume you're talking primarily about visual/graphic design more so than the other ways the word is used.

As designers/developers or creative people in general, it's natural to think of our creations as art, they are.

But when it comes to art that has a concrete function, like websites, I find it helps me to remember that the average user will not be coming to a website I build for the sole purpose of marvelling at that lovely gradient that I made or that Ajax UI element that took me hours to put together. They're coming to the site to use a service or obtain information. That's all they're really interested in.

I'm not trying to say that the prettiness of the site doesn't matter, it does (for some sites more than others). But it's secondary to making sure the site is usable/accessible, making sure the visual design doesn't get in the way of what your users are trying to do. I'd also include in this making sure the design conveys the image you're tying to get across. That's part of the design not getting in your users way.

Users by and large will not squint at your visual design like you do and critique it in anywhere near as much detail as you might.

</comment><comment><USER3> I think the term 'design' can be misleading as it has a few connotations. If you think craigslist or drudge report, you can say design doesn't matter because, they are ugly sites. But there are lots of sights that have nice colors and images but aren't usable.

I couldn't find a link to your site, so I can't figure out where you fit in.

For design, go usable first, then dress it up. like building a house, foundation, make sure the door is in the right place, you've got a nice driveway, etc. Get a site people can look at. Then add the layers of 'design'.

Take a look at apple.com (the mac page is probably better than the home page for a comparison <http://apple.com/mac>). As usual via their minimalism, there is a ton of design, but you almost want to click on stuff. Compare it to linux.com, less enticing, then try microsoft.com.

I personally fall victim to overloading people with content, and not giving enough direction. Less is often more.

</comment><comment><USER4> It matters a lot to me. I often hit the back button within 3 seconds of the page loading. The information needs to be cleanly presented and the site needs to look credible.

</comment><comment><USER5> If design is tantamount to usability, it is the second most important aspect of your site (behind functionality, in my opinion). If design = drop shadows and rounded corners, I would put it on the bottom of my priority list.

</comment><comment><USER6> If you look sketchy, why should I trust you? Your friend can use Lynx if he doesn't care about Web design.

<comment><USER7> huh. see, I am always more suspicious of people who put more effort into appearance. I mean, if you show up to a programming interview in a suit, my first impression is that you are trying to distract me from things that matter with your appearance.

But yeah, that's taste, and varies based on who your customers are. Functionality matters to everyone.

</comment></comment><comment><USER8> I matters much more than you think.

<comment><AUTHOR> but, surely if the site has what the person wants, what the site looks like doesn't matter?

<comment><USER8> Well, yes it does.

\- Assuming it's the first time someone visits the site, he has to know that what he wants is to be found there. That requires design.

\- Someone new to your site will give it roughly 1 second before he decides whether it is any good or not. I'm not making this up, it's been proved in many studies. Taht's design.

\- If you're selling something, physical bits or thoughts, you need to convey trust in what you're selling. That's design.

Not that when I talk about design here I'm not so much talking about nice colors and fancy gif's but more about things like overall style, usability, etc.

<comment><USER9> To add to what max is saying, I've read that we think of sites as a kind of book that the user is going to take their time reading. So we fill them up with directories, gizmos, lots of information. We assume an information-dense page is worth more than a sparse page.

In actuality, web sites are much more like billboards that people pass by on the highway. If you get 1 or 2 seconds of quality interaction time, you're ahead of the game.

I've heard of designers literally squinting their eyes and walking by a monitor quickly to test what kind of impression the site has on the new viewer. For most sites, it's not so good.

</comment></comment></comment></comment><comment><USER7> look at myspace. I mean, design matters in the sense that you need to be able to do what your users want to do, but no, rounded edges don't matter.</comment>


New way to test self-driving cars could cut 99.9 percent of validation costs - <AUTHOR> https://phys.org/news/2017-05-self-driving-cars-percent-validation.html <comment><USER2> > _the new accelerated evaluation process breaks down difficult real-world > driving situations into components that can be tested or simulated > repeatedly, exposing automated vehicles to a condensed set of the most > challenging driving situations_

Chess engine programmers do something like this. Rather than playing thousands of full games to determine your engines strength, you can get a pretty good snapshot by running custom test-suites that contain a curated selection of known challenging positions [1]. One well known downside to this is that if you optimize your engine for test-suites, it invariably performs worse in overall general gameplay. Maybe we'll end up in a situation where cars learn to drive in an extremely over-cautious and risk-averse manner in order to pass all of the dangerous test scenarios.

[1] [https://chessprogramming.wikispaces.com/Test- Positions#Test](https://chessprogramming.wikispaces.com/Test-Positions#Test) Suites

<comment><USER3> Perhaps I'm pulling teeth here, but at 70+ MPH, I don't think there's any such thing as overly-cautious until it backfires and becomes less safe in the process.

<comment><USER4> I think the "overly cautious" bit would be never thinking it's safe to come up to 70MPH in the first place.

<comment><USER5> Indeed, one thing I look forward to is SDCs returning residential streets to their original primary users: humans.

<comment><USER6> Self driving cars will actually take us away from a world where all car trips are humans accomplishing their goals, to a world where a decent proportion of car trips are machines serving their own purposes.

As an aside: please do lean further into this "people whose transportation needs are best served by cars aren't humans" rhetoric. It's going to be a lot easier to destroy at the polls.

I say this having just sold my car to move to a tiny studio in an urban high rise near work, because I recognize that my top-10% salary and lack of children are the only reason I can do that. I'm excited by the possibility of self-driving cars to bring the benefits of the automobile to more people, and most excited about those it can liberate from slow, crowded buses.

<comment><USER5> As derefr said I meant use of the roads for walking, playing, sitting, etc like they were before the "motor car" became common. The SDCs will have to be more careful.

Roads are now intended for cars, and new infractions (like jaywalking) were invented to give automotive use primacy over others. Cars should merely be one use of the roadways.

Not sure what you mean by "It's going to be a lot easier to destroy at the polls."

<comment><USER6> "Cars should merely be one use of the roadways" is equivalent to "people who can't afford decent urban homes should live in shitty urban homes." The primacy of the car is what lets us currently live in decent suburban homes.

Urban and suburban residents have competing interests here: urban residents (for whom cars are a nuisance) would like to safely frolic in the streets. Suburban residents (whose lives depend on fast travel by car) would like to continue affording decent homes.

This competition will appear in city-level politics in various ways. The anti- car side is handing the pro-car side some ammunition when talking about them as subhuman (i.e. framing drivers and people and mutually exclusive categories).

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Hi admin I examined your post about Facecord / Bushcord measuring Firewood. It was quite an impressive blog and it carries so much important and useful information, and I also liked the way you presented the blog i was amazing. Thanks for sharing! [http://www.juegosdefriv.live/](http://www.juegosdefriv.live/)

</comment><comment><USER8> Lets pass on the issue of memorizing the training data. The article also says the test is designed

       "to consider human drivers the major threat"

I'm thinking that degenerate driverless-algorithm behaviors will be even more problematic. Imaging an imminent head-on collision between two driverless cars. They could play 'dance with me', each dodging to the right or left only to see the other car do the same. In a few milliseconds they could lose the opportunity to avoid the accident.

With a human driver and their limited response time, the other car is actually _more predictable_ and interactions easier to manage.

<comment><USER9> You can make a rule, just like ships already are doing:

> When two power vessels are approaching head on,both vessels should alter > course to starboard to pass port-side to port-side.

> When two power-driven vessels are in crossing situation on a collision > course, give way to the vessel to starboard (right).The give way vessel must > take early and obvious action to avoid a collision by either stopping or > altering course to starboard.

[https://maritime.college/Boating-Rules.php](https://maritime.college/Boating- Rules.php)

\---

Plus, driverless cars can potentially use more efficient short-range radio and figure out themselves.

</comment></comment><comment><USER10> _it would take nearly a decade of round-the-clock testing to reach just 2 million miles in typical urban conditions._

...if you only use one car.

<comment><USER11> As stated in the article, they need to reach 11 billion miles to get the confidence level they are looking for. So extrapolating this, it would take 5500 cars a decade of round the clock testing to actually meet the guarantees they need (90% safer than human driven at a 80% confidence level). You still could do it in parallel; it'd just require a lot of test cars.

One part that is unclear is that the 11 billion mile requirement doesn't specify urban conditions and the 2 million mile per decade estimate does. So there might be some errors there.

</comment></comment><comment><USER12> And then, car makers will try to detect these tests and make their cars more cautios if in such a situation. Its probably going to be great for training but testing should be run in the real world, could finally also reintroduce realistic benchmarks for mpg and other running costs.

</comment><comment><USER13> There will always be unanticipated circumstances that will foil the sensing systems. My go-to example is this situation where road re-paving artifacts and shadows combine to create a false lane marker. This vehicle's video-based system determined that the bright area between the shadow and lane pavement was the left lane marker and sounded a lane departure alert. Had the autonomous lane following feature been activate, it would have reacted and attempted to center the car on the false boundary:

[http://imgur.com/a/KnsXU](http://imgur.com/a/KnsXU)

</comment><comment><USER14> Hmm.. Randomly generated real-world traffic based from traffic sensors and cameras. The key here is, if they use the same data and feed every car the same, then it can be easily gamed with overfit techniques.

The better question is, what happens if you give the AI car a no-win situation? How does it handle it? The Kobayashi Maru can be looked at as a way to detect outliers and failure modes... even if the game is rigged.

</comment><comment><USER15> Unfortunately, the linked white paper is _very_ thin on methods and analysis. It, itself, is not peer-reviewed, and it points to only one peer-reviewed publication to justify the approach.

</comment><comment><USER16> seems like a good dataset for training a deep learning system.

<comment><USER17> This needs to be an out-of-sample test.</comment></comment>


Oculus Rift review - <AUTHOR> http://www.theverge.com/2016/3/28/11284590/oculus-rift-vr-review <comment><USER2> What is this shit? I'm only seeing an empty website, even after accepting all the third-party JSs (except for Google/Twitter/Facebook).</comment>