Underreported Yakuza murders in Japan, FBI liver-transplant-for-info swap [2009] - <AUTHOR> http://www.npr.org/templates/story/story.php?storyId=120237244

<comment><AUTHOR> You can hear NPR audio from the link. Transcript is at [http://www.npr.org/templates/transcript/transcript.php?story...](http://www.npr.org/templates/transcript/transcript.php?storyId=120237244).

This was an interesting story I heard a few months back, detailing how:

* Yakuza/organized crime figures carry business cards and operate in the open

* Companies and private citizens contract the Yakuza to settle disputes

* Most construction companies have some Yakuza tie-in

* Most yakuza murder victims do end up as cliches ... in building foundations ... and aren't reported as murder victims

* a Yakuza boss promised to inform on cronies for the FBI in return for entry/exit to/from US for liver transplants at UCLA (UCLA got a nice sum for expedited liver matching and transplant services too) ... and he brought along three friends for identical procedures.

* The Yakuza boss did not keep his end of the deal. (I guess the FBI just trusted he'd do his part.)

* The American crime reporter/author giving the interview blew the story, disgraced the Yakuza figure, and suffered some consequences.

Fascinating story overall. The Japanese official homicide rate doesn't reflect the truth.</comment>


Anyone interested in giving each other facebook app usability tips as the facebook apps are developed? - <AUTHOR>

<comment><AUTHOR> If anyone is interested, let me know so I can add you as a developer for my app (so you can try it out) and you can do the same for me. Moreover, we should probably establish a friendship (at least temporarily) to test the friend features.

<comment><USER2> me too!

firstname.lastname@example.com

<comment><AUTHOR> I've sent you a friend request.

</comment></comment></comment><comment><USER3> Definitely. I love Slide's My Questions app but its usability is horrid,,, Max... are you around? Love your app but I got a few ideas to make it a helluva lot more usable. Start by returning me to the question I've just answered instead of my own question page. (I'm familiar enough with my question, I don't need to return to it.) etc etc (And Slide's Facebook apps are hardly the worst offenders here. ALL of them could use a little bit of improvement IMHO.)

Web 3.0 = where _everyone_ contributes to the development of web apps, not _just_ engineers.</comment>


Ask HN: European Hosting Providers and CDN? - <AUTHOR>

Hi Guys,
I've got fairly substantial project going live shortly, and it's localized for the european market. I'd like to arrange some hosting for it, and a CDN.
All of my experience is with North American providers, so I'm hoping that someone can suggest a good provider for physical or virtual (cloud) hosting aiming at the big three (UK, France, Germany).
Thanks very much <comment><USER2> Amazon EC2/S3 has servers in europe (Ireland). Though it's a litte bit more expensive.

Also I've met the guys from Flexiant.com/Flexiscale (<http://www.flexiant.com/products/flexiscale/>) at FOWA:London years ago. Not sure how they deliver.

</comment><comment><USER3> Very satisfied leaseweb customer here. Leaseweb.com is their website.

Large volume about E1450 / gigabit including server, OVH is cheaper but has a bit of a bad rep.

</comment><comment><USER4> cdnXite - <http://www.cdnxitehosting.com> \- offer VPS Hosting with datacentre in UK/USA/SWEDEN. Just add Microsoft free CDN - <http://www.microsoft.com/windowsazure/windowsazure/>

</comment><comment><USER5> Linode has a datacentre in London nowadays.</comment>


Solicitor General Advises Supreme Court to Not Grant Cert in Google vs. Oracle - <AUTHOR> https://drive.google.com/file/d/0B941PmRjYRpnZ2lqd09vbDlZWXM/view

<comment><USER2> Essentially, the US Solicitor General says that the lower court ruling in favor of Oracle should stand. And Google should pursue a "fair use" defense of their adoption of the Java standard library APIs.

Hypothetical worse case for Google and Android: the US Supreme Court refuses to hear the case, and then Google subsequently also loses a "fair use" defense. Oracle then is awarded big damages and the Android ecosystem is hurt bad.

Oracle wants smartphone vendors to use its version of Java.

<comment><USER3> The worst case isn't just for Google and Android, it would mean that API's are no longer in fair use. No more IBM PC compatible BIOS or any such industry advances.

<comment><USER4> I think that interoperability might better stand if much more of the APIs were translated verbatim, and restricted in runtime on the devices... Though that would be only a technical work around.

That said, I think the argument is bullshit, as given conventions used in the language (and in similar language) most of the organizational structure for given APIs and libraries are what makes the most sense, and there are very much in fact a fairly limited number of means of expression that can be made in order to establish a platform with a familiar language/expression. If you compare say, the .Net environment, you will see it is very similar, with very similar structure... For a language whose implementation is very unique from Java, the structure, and call patterns for common methods are in fact similar.

<comment><USER3> Exactly right about .Net, just having Capitalized method names doesn't make it vastly different. There's an issue with the even _Supreme_ court making decisions regarding technology. We may be in need of a more specialized ruling body that as an appreciation of the implications a decision.</comment></comment></comment></comment>


Show HN: Tired of Bad News? Meet SentiNews (Android) - <AUTHOR> http://www.sentinews.ml/ <comment><USER2> Interesting article on why negative news can have positive effects. [https://www.gsb.stanford.edu/insights/positive-effect- negati...](https://www.gsb.stanford.edu/insights/positive-effect-negative- information)

Still love the idea of this kind of product, will read!

</comment><comment><USER3> I really appreciate this idea, but is there a reason is has to be an app? I check all news through the browser (phone and desktop), and I'm leery of installing an app to read news. I think such a concept would make a great website though.

</comment><comment><USER4> Not sure that there is much of a market for good news If there were then surely the existing news media would make more effort to disseminate it.

And it's not a particularly stunning implementation either. There is no help, no documentation at all.

Next to the thumbs up and down buttons is an icon for something else but it is not described anywhere. What is it supposed to do?

As usual when used in landscape mode it wastes a lot of screen area, looks like it was only ever tested on a mobile phone.

Interesting idea but I would want to have some influence on the choice of story, perhaps by voting a story up or down, before I would make much use of it.

</comment><comment><USER5> Will be very surprised if this makes any money, let me know.</comment>


Germany's Defense Against the N.S.A.: Typewriters - <AUTHOR> http://takingnote.blogs.nytimes.com/2014/07/15/germanys-defense-against-the-n-s-a-typewriters/

<comment><USER2> Sometimes the lo-tech approach works the best.

Shades of the 'Windtalkers', who used Native American language during WW2 as a form of unbreakable code.</comment>


Taking Kindle To The Pool - <AUTHOR> http://www.npr.org/templates/story/story.php?storyId=92079896 <comment><USER2> This article is essentially contentless:

>As for the pool, Pegoraro says that while a few drops won't hurt the device, you probably won't want to bring Kindle on the raft with you.

>"This is not water resistant, so far as I know. I haven't put it to the test yet," he says.

<comment><USER3> Yeah. NPR? More like PR.

<comment><USER4> But is PR NPR complete?</comment></comment></comment>


Using Python to Get Free Chicken Sandwiches - <AUTHOR> <comment><USER2> It's mildly interesting that ethics are flexible depending on the target, but from a technical standpoint this project is not interesting. The code just builds a proxy list, generates a random number, opens a page, and posts the payload. I work with selenium and this isn't solving any interesting problems whatsoever.

Posting it is unethical, and it's disappointing to find on the front page of HN.

<comment><USER3> 30 years ago, on IRC, doing the same would have been considered a fun hack, and a tongue-in-cheek motivational project for beginners.

Different generation I suppose.

<comment><USER4> Some people will try this exploit to get free stuff. in the 90s it would have been a few dozen at most, now it would be thousands. It ceases to be tongue- in-cheek when it becomes onerous.

<comment><USER3> My guess is more than the anomally will be detected and the hack will be blocked in 2 days.

Compared to the free phone hack with the cereal whistle that everybody though was so cool...

</comment></comment></comment></comment><comment><USER5> This is hacking and is unethical at best and at worst, extremely illegal. Congratulations, you found out a way to systematically steal other people's reward points and are helping others do the same. Should this really be on HN?

<comment><USER6> they're not advocating that you do it (and I think stealing from Chick-fil-A is probably ethical to a lot of people)

edit: for people not aware, the last part was in reference to pride month and Chick-fil-A's "relationship" with the LGBT community. I'm not advocating one way or another, just replying to the above comment.

<comment><USER7> I'd like to hear an explanation as to why stealing from Chik-Fil-A is a moral act. The only one I can think of is that mass-produced chickens suffer horribly, and so anything that hastens the demise of the company is a good thing. But that makes a number of flimsy assumptions.

<comment><USER8> They donated to anti LGBTQ groups for years. Money given to Chik-Fil-A ended up supporting policies of hatred and bigotry.

<comment><USER9> Then don't patronize their business? Why do private opinions give you the right to steal public earnings?

</comment></comment></comment></comment></comment><comment><USER10> What happened to responsible disclosure? I mean, fair enough demonstration of how to do something like this, but at least give the company a chance to fix it before publishing to the world.

</comment><comment><USER11> This repo violates GitHub TOS, specifically it "contains or installs any active malware or exploits"

[https://github.com/contact/report- abuse?report=baileywj+%28u...](https://github.com/contact/report- abuse?report=baileywj+%28user%29)

</comment><comment><USER12> I've never seen a receipt survey in person so I did a google search. Apparently they used to have authentication built into the redemption process by requiring the original receipt [0]. However, it appears only the generated QR code is required for the new process [1].

[0]: [https://www.yelp.com/biz_photos/LI3vSjDZGxCTKs785eTkeQ?selec...](https://www.yelp.com/biz_photos/LI3vSjDZGxCTKs785eTkeQ?select=JnBJ8PFZl6EVrD9fPhJMuA&utm_source=ishare&utm_content=photo)

[1]:[https://i.redd.it/fzxy7rff0g211.jpg](https://i.redd.it/fzxy7rff0g211.jpg)

</comment><comment><USER13> On the one hand - awesome, free chicken sandwiches!

On the other, well, every little thing has to be thought through and secured these days doesn't it? Can't have a nice little rewards program without some asshole taking advantage of any perceived flaw in your armour...

<comment><USER14> Some people see it as a form of challenge. Sure, it might be fun to figure it out on your own, but sharing it is a whole different question [0][1].

The challenge, especially for places that aren't really tech-first types, kinda becomes stale (for me) since they inevitably have a flaw somewhere. It's really just me pitting my free time against their threat model [2].

And yeah, ultimately, this is one reason why we can't have nice things.

[0]: I'm not saying publicly releasing flaws is always a bad thing. There's a subtle etiquette (an art-form, even) to using publicity to pressure change.

[1]: There are different levels of publicizing. e.g. sharing (complete) code, social media post about said code, etc.

[2]: This is also not meant as an insult to the OP.

</comment></comment><comment><USER15> Could the last number be:

What you ordered?

Number of items you ordered?

</comment><comment> 1-6 Would Jesus want you to do this?</comment>


Introducing a simpler, faster GitHub for Mac - <AUTHOR> https://github.com/blog/1862-introducing-a-simpler-faster-github-for-mac <comment> squeaky-clean I really wish GitHub would open source these clients. If not for community contributors to poke at (it's kind of funny that a client based around open source code collaboration isn't open source), I'd at least like to see the code behind the UI. Github for Windows 2.0 is absolutely beautiful, and Github for Mac looks to be just as good.

I've read somewhere that Github for Windows uses WPF for the UI. There are no good looking open-source WPF apps that I'm aware of, so I'm very curious as to how it was made.

<comment><USER2> You could try loading the assemblies up in dotPeek and poking around. You miss out on the XAML (though you'll see the code the XAML was transformed into), the compiler generated stuff for await is a little ugly, and obviously you wouldn't be able to re-use any of the code you see, but if you're just curious...

</comment></comment><comment><USER3> Who uses this? Is it an effective part of your daily workflow?

<comment><USER4> (Biased, I work at GitHub and helped design the original GHfM)

I use it every day, almost all of the time. I don't use a lot of it, but what I do use is tremendously useful. My workflow is usually:

1\. Navigate to the folder in terminal (old habit, or starting servers) and type in `github .` which launches the client for this repo.

2\. Sync to pull in new changes (cmd + s)

3\. Create a new branch (cmd + b)

4\. Commit as I code (visual diff helps me do light code review on the spot). This is definitely the core usage for me — seeing what uncommitted changes I have, selecting out partial commits, drafting up good commit messages, amending bad commits, etc.

5\. When I'm ready to publish my branch, sync again (cmd + s)

The key for me really comes down to some really simple stuff: a visual editor for creating commits, and quick keyboard commands to common actions (branching, switching branches, push/pull). It's possible to do fast in terminal, but muscle memory serves me personally much better with real keyboard commands.

<comment><USER5> I don't get it. How in the world is that any easier than:

      cd /your/src/dir
      git pull

      <edit stuff>

      git diff

      <look at stuff; possibly edit more stuff>

      git commit -m "edited stuff"

      <oops...forgot something; edit another file>

      git commit -a --amend
      git push origin

Which, of course, has the added advantage that you're _using git_ , instead of using a GUI obfuscation layer on top of git, and therefore learning your tools.

I mean...I sort of get why people do git integration in editors (even thought it tends to lead to ignorance of git), but opening up another, non-console, non-editing app, just for git?

<comment><USER6> for one thing, you are forgetting that people are human. this is what that workflow would more realisticly look like:

        cd /yor/src/dir
        bash: cd: /yor/src/dir: No such file or directory

_examines closely to look for typo_ cd /your/src/dir

        <edit stuff>

        git diff

        git comit -m "edited stuff"
        Did you mean this? git: 'comit' is not a git command. See 'git --help'.

Did you mean this? commit git commit -m "edited stuff"

I think this is enough to get the picture.

<comment><USER7> > cd /yor/src/dir > bash: cd: /yor/src/dir: No such file or directory <tab> ?

And on anecdotal evidence, I've observed that people who use git intensively everyday often have aliases to these commands that have a low probability of typos.

eg gcom => git commit

</comment></comment></comment></comment></comment><comment><USER8> Does someone involved in the project know what is the status of desktop notifications? Last time I checked, it had been removed from the app, which was a deal breaker for me.

<comment><USER9> This is the only reason I have installed the app. Patiently waiting for a good desktop notifications system. I'm on far too many projects at any time, some high volume open source, and I really would like notifications built in. Currently giving The Hub App a spin.

[http://thehubapp.com](http://thehubapp.com)

</comment></comment><comment><USER10> Probably a dumb question but can I use this for repositories hosted anywhere or does this only work with Github repositories?

I currently use SourceTree but only for solving conflicts and manually adding things to staging in increments. I would like to give this software a try.

<comment><USER11> I just tried it out and found that it can work with repositories that are hosted elsewhere, but I haven't yet found a way to clone a new local repo from a non-github remote repo.

<comment><USER10> Yeah that's more than good enough. I only clone in terminal anyways. I'll give this a shot for a couple of days. Thank you!

</comment></comment></comment><comment><USER12> I was using it only to see a nice diff with colours instantly. It bit me once, though. I couldn't figure out why my new folder 'icon' was being ignored. I checked the gitignore, update etc. The problem was the app added global gitignore config for mac icons.

</comment><comment><USER13> Nicely done! If someone from the GitHub team is reading this, is there any plan in the future to integrate "issues" into the desktop client? P.S. like the new style logo, seems Yosemite ready :)

</comment><comment><USER14> I installed this today and now it crashes on opening

</comment><comment><USER15> Why use it and not now Atlassian's SourceTree?</comment>


A new game that your character is a monster - <AUTHOR> http://gamekatana.com/you-are-a-monster-in-this-game-darkborn/ <comment><USER2> I don't know anything about the game, but the premise reminds me of Creature Of Havoc - an 80s role playing game from the Fighting Fantasy series, where you play a monster. It was such a cool idea! ([https://en.wikipedia.org/wiki/Creature_of_Havoc](https://en.wikipedia.org/wiki/Creature_of_Havoc))</comment>


Tree traversal without recursion: the tree as a state machine (2007) - <AUTHOR> http://plasmasturm.org/log/453/ <comment><USER2> I've always found these kinds of tricks fascinating. There's all sorts of crazy things you can do to save some bytes here and there. One of my personal favorites is the XOR linked list[0]. It's a doubly linked list that uses as much memory as a singly linked list. It works by XORing the previous pointer with the next pointer. As you're doing a traversal you can compute the next pointer by making use of _next = prev XOR (prev XOR next)_ giving you the address of the next node in the list.

Anyways, I did some digging into TAOCP. In section 2.3.1 Knuth mentions threaded binary trees[1]. They are trees that you can traverse without a stack _and do not use more memory than a regular binary tree_. This is compared to the method used in the post which requires you to store the parent pointers. The trick is that in a binary tree, lots of pointers are going to be null. Instead of storing null there, you can store a special value that provides information on how to traverse the tree. Indicating the difference between a regular pointer and one of these special values requires only a single bit. You can often store that bit within the pointer itself, requiring no additional space over a regular binary tree.

For a threaded binary tree, if the left child is null, you instead store the predecessor node. If the right child is null, you store the successor node. Finding the successor now becomes:

        1) Check if the right child is null. If so the right child pointer points to the successor. We are done.
        2) Follow the right child pointer. Then keep following the left child pointer until the left child is null. This gives us the smallest value greater than the current value.

One small advantage of threaded binary trees is you traverse the tree slightly less. If the right child is null, instead of traversing back up the tree as in the post, you can jump straight to the successor node.

[0] [https://en.wikipedia.org/wiki/XOR_linked_list](https://en.wikipedia.org/wiki/XOR_linked_list)

[1] [https://en.wikipedia.org/wiki/Threaded_binary_tree](https://en.wikipedia.org/wiki/Threaded_binary_tree)

<comment><USER3> >You can often store that bit within the pointer itself

Alternatively you can follow what java did with compressed ops and use 4bytes per pointer on 64bit archs (instead of 8) and shift the pointers prior to dereference - effectively enabling 32GB heaps with 4bytes pointers.

<comment><USER4> It's a good idea IMO, but the two aren't alternatives, i.e. not mutually exclusive — you can do both. See Knuth's "A Flame About 64-bit Pointers" from 2008 ([https://cs.stanford.edu/~knuth/news08.html](https://cs.stanford.edu/~knuth/news08.html), starting with "It is absolutely idiotic to have 64-bit pointers when I compile a program that uses less than 4 gigabytes of RAM"). Work on making this possible in Linux started as the x32 ABI in 2011 (see Wikipedia [https://en.wikipedia.org/w/index.php?title=X32_ABI&oldid=887...](https://en.wikipedia.org/w/index.php?title=X32_ABI&oldid=887611857) or this LWN article: [https://lwn.net/Articles/456731/);](https://lwn.net/Articles/456731/\);) unfortunately it looks like there's discussion about removing it (Dec 2018 thread starting here [https://lkml.org/lkml/fancy/2018/12/10/1145](https://lkml.org/lkml/fancy/2018/12/10/1145) though apparently I can't figure out how to navigate the LKML tree using constant space).

<comment><USER5> I've found myself using 32-bit "array indexes" to half my "pointer-sizes" for some code I'm writing (for fun). When doing linked data-structures (linked lists, trees, etc. etc.), a huge amount of the data ends up being a pointer.

Consider a typical binary-tree with a 64-bit integer as its value. You use 24-bytes per node (value, left child, right child) if you use 64-bit pointers, but only use 16-bytes per node if you use 32-bit pointers (or array indexes).

Now, a 32-bit pointer can have at most 4billion nodes across the whole tree. But 4billion is more than usable for many programs.

EDIT: Consider that 4billion nodes of size 16 bytes (where each node is the value + left child + right child struct discussed earlier) will take up 64GB of RAM.

EDIT2: And half the time, my brain gets short-circuited and I end up recreating some terrible form of segmented memory before having to slap myself for thinking up such a horrible idea. In any case, a surprising amount of memory is used up as pointers in almost all the code I write. If you care about fitting as much data into L1 cache (64kB on modern machines), you will absolutely want to minimize your data usage.

</comment></comment></comment></comment><comment><USER6> If you require the highest performance and your trees are write once, read often (and immutable), it's worth looking into if you should just sort the nodes in traversal order in one contiguous block of memory. Storing the edges as indices into the block is handy in this case.

The article used parent pointers in node to avoid using a stack (native or custom made). One downside of a parent pointer is that your identical subtrees can no longer share structure.

<comment><USER7> Could you expand more on what you mean by "Storing the edges as indices into the block is handy in this case."?

This is super relevant to me right now, thanks!

<comment><USER8> You add integer slots to each node and store the index of the child nodes in them. So you have, for a binary tree,

      [actualTreeNodeData1,child1Idx,child2Idx][actualTreeNodeData2,child1Idx,child2Idx][...]

instead of just

      [actualTreeNodeData1][actualTreeNodeData2][...]

Vary this to include whatever you actually need, could also be links to parents instead of or in addition to links to child nodes.

<comment><USER9> That feels like it'll start hurting when you don't know the (maximum) degree of your nodes. As you won't have fixed-size object anymore.

<comment><USER10> Hurting because of memory alignment or something else?

Given we have indices along the serialisation for child nodes I would have thought not having nodes start at the consistent intervals doesn't hurt us for traversal at least.

I'd quite like to use this idea for a file backed immutable tree. The array of children would have variable size with bit field to index then nth child in the sparse array.

<comment><USER9> Constant time array indexing depends on a fixed array member size. So, the actual memory layout of storing the nodes with non-fixed size could not use an array of nodes.

I suppose you could (should) write your own allocator to ensure nodes are allocated in a contiguous piece of memory, and have a separate array of indexes / pointers that tells you where the n-th node starts. That is another array that'll have to be kept in cache though.

</comment></comment></comment></comment></comment></comment><comment><USER11> > You can get rid of any stacks whatsoever by keeping a parent pointer in the > tree node data structure.

You can actually in-order traverse a binary tree without any memory overhead at all (no parent pointer nor an implicit or explicit stack) using Joe Morris' traversal.

I found that this can be done to be surprising but effectively it uses the NULL slots as temporary storage.

Here's one explanation: [https://yuyuan.org/MorrisAlgorithm/](https://yuyuan.org/MorrisAlgorithm/) \- although there seem to be a few youtube videos on it also.

</comment><comment><USER12> Reminds me of Sean Parent's recounting[1] of how his son's beads and string toy could represent a tree data structure with stateless traversal and some other interesting properties, like:

> ...to erase elements from some point in my model to some other point in my > model, what I do is grab those two locations on the string, and I pick the > whole thing up, and the beads that fall off are the ones that get erased ... > (48:41)

1: [https://youtu.be/sWgDk-o-6ZE?t=2721](https://youtu.be/sWgDk-o-6ZE?t=2721)

</comment><comment><USER13> It seems like by adding parent pointer one is shifting complexity from the traversal into the data structure, in order to avoid a stack. I know there are places where avoiding an explicit stack can be nice (perhaps a BVH tree in a shader or other places where constant space is required?) but are there more good reasons for it? A DAG (such as a singly linked tree) has so many benefits for memory safety, GC performance, ability to extend it from a binary tree to N children etc., that it seems like an odd tradeoff in most scenarios.

<comment><USER14> Obviously, adding read-only data in the tree and to save mutable data in the traversal state can be amortized if we have many ongoing traversals rather than one. For example, in a massively parallel GPU computation running an arbitrary number of tree searches simultaneously in a fixed memory amount.

</comment></comment><comment><USER15> Learn some Prolog, one of the best langauge for studying your algorithms and data structures. [https://rosettacode.org/wiki/Tree_traversal#Prolog](https://rosettacode.org/wiki/Tree_traversal#Prolog)

preorder(nil). preorder([Node, FG, FD]) :- format('~w ', [Node]), preorder(FG), preorder(FD).

inorder(nil). inorder([Node, FG, FD]) :-inorder(FG), format('~w ', [Node]), inorder(FD).

postorder(nil). postorder([Node, FG, FD]) :-postorder(FG), postorder(FD),format('~w ', [Node]).

</comment><comment><USER16> > _You can get rid of any stacks whatsoever by keeping a parent pointer in the > tree node data structure. Effectively, this turns the tree into a (sort of) > state machine._

How can it be a state machine if you aren't mutating it?

In fact, you can traverse a tree without recursion (or any equivalent stack- like structure) _and_ without parent pointers. And then you _actually_ use the tree as a state machine: you stash the reverse path pointer in the tree itself by overwriting the downward links temporarily and then putting them back.

This is useful for garbage collection; you can have a garbage collector that is guaranteed never to blow the stack in the marking phase.

<comment><USER17> > How can it be a state machine if you aren't mutating it?

You are confusing a state machine (which is an abstract model of computation) with the behavior of a program implementing it. The set of possible states and the conditions of transition from one state to the next never change.

These conditions can depend on mutable state, as is the case described by the author (in this case, mutable state is kept in the current and previous node pointers).

</comment></comment><comment><USER5> > You can get rid of any stacks whatsoever by keeping a parent pointer in the > tree node data structure. Effectively, this turns the tree into a (sort of) > state machine. While traversing, you need no memory other than the current > and previous node/state. The traversal algorithm is very simple:

While this sounds innocent, truly consider the ramifications of this decision.

Your node has now grown to be +8 bytes (the size of a 64-bit pointer). If you have 1-million nodes in your binary tree, you are using +8MB of memory.

\</comment><comment>-

Lets consider the alternative: lets say you have 1-million nodes, and you have a red-black tree (imperfect balance: instead of a perfectly balanced 20-depth tree you have some branches which are 40-depth).

Traversing to 40-depth requires 40x pointers on the stack, or 320 bytes.

\</comment><comment>----

In effect, you're spending 8-bytes __per node __(which could be millions, or even billions of nodes), to save log2(depth) space off of your stack frame.

While the state-machine is very "clean", it seems like a very bad tradeoff from an algorithmic / memory space point of view. I'd rather spend log2(depth) space on the stack, rather than O(+8 bytes * number nodes) in the heap.

I think there's something to be said about the clarity and cleanliness of the state-machine design. Its quite possible that some algorithms are easier to write with this "parent pointer in node" methodology. However, anyone seeking the highest performance per unit-memory will have to see that the +8 bytes per node is a terrible, terrible tradeoff.

\</comment><comment></comment><comment>-

> Update: Todd Lehman pointed out that given node-level locks, this algorithm > allows concurrent traversal and update of the tree. Any atomic operation > other than detaching a non-leaf node is safe.

Hmm... the parallel-angle I haven't thought about. But it does seem to be a potential building block as maybe even a lock-free data-structure.

Or just use locks, since locks are easier to think about.

Insertion and deletion into the tree seems possible, but "rebalance" is very difficult for me to think about (since it possibly requires modifying many, many nodes). Locking all nodes involved would be a heavy approach and probably work.

Red-black colors are kept track of for self-balancing purposes. Maybe more colors are needed to make the methodology "clean" for multi-threaded / lock free purposes.

</comment><comment><USER18> A complete aside: the site design on linked page is beautiful. Very minimalist and totally functional.

<comment><USER19> I don't know, maybe I get a different color scheme than everybody else? It's light gray on white for me and the text is almost invisible.

[https://i.imgur.com/YM3dIcG.png](https://i.imgur.com/YM3dIcG.png)

Surely that's not considered a great design?

EDIT: ok, was fault of a chrome extension "NightModePro" :)

</comment></comment><comment><USER20> Note that if your language has non-precise garbage collection, having parent pointers guarantees that any spurious pointer into any node on the tree will keep the entire tree alive.

</comment><comment><USER21> Excellent!

Generally, trees are poor for performance as you zig-zag around memory and stall on cache misses. But in situations where you have to do tree traversal and you have to be as fast as possible, trading the memory for the parent pointer (which probably fits within the same cache line as the rest of the node anyhow, so in practice basically free) vs recursion or an explicit stack which stress memory in the tight loop, it is probably a winner! Definitely worth profiling in those situations.

<comment><USER22> Man, hold your horses.

Assume you have 64GB of memory devoted to binary tree (of just integers), assuming the tree is balanced.

Your node is two pointers + integer (24 bytes). That gives 2666666667 nodes. Balanced tree will have 32 levels (log2(2666666667) ~= 31). When doing recursion you just need to keep pointer to the level, so this is 32 times pointer or just about 256 bytes of your cache.

Also, even if we were storing data in memory (and not in cache) this still is orders of magnitude less effort than accessing the memory for the large data structure.

<comment><USER23> > When doing recursion you just need to keep pointer to the level, so this is > 32 times pointer or just about 256 bytes of your cache.

What language do you use where a recursive function invocation only requires a call frame large enough to hold a single parameter? By what magic does the function know where to return? And what are you doing in that function that doesn't require any temporaries whatsoever?

At best you're looking at a couple of kilobytes. For all but the most constrained environments (toaster, a 4Kb kernel thread stack, etc), this isn't really worth worrying about. And compared to the pointer chasing, all those pushes and pops aren't much of a problem, either.

However, in a language like C without function closures, non-recursive traversal can keep code much more concise and clear. This is the real win, IMO.

<comment><USER24> You could simulate the recursion with an iteration, a depth byte and a stack- array of 32 pointers (64 for overkill). This removes the frame pointer and makes the temporaries clear too.

<comment><USER23> But that's not recursion, which pretty much by definition implies the use of repeated invocations of the same function(s). Explicitly building a stack is exactly what the article was discussing, though it's far more clever than the obvious approach. I've seen AVL implementations that use an explicit array for maintaining a stack in lieu of recursion or parent pointers.

In any event, you always need a stack of some sort. Recursion is one way to accomplish that, but saying that you're simulating recursion by explicitly building a stack reverses the categories.

</comment></comment></comment></comment></comment><comment><USER25> Honestly I'm quite surprised that although this uses both Perl and goto it gets upvoted. Sometimes I'm pleasantly surprised by HN community.

<comment><USER26> The goto's did get my nerves bristling a bit on first sight - but it's the right tool for the job (or at least gets the job done), so I was OK with it. I guess other languages may have better/cleaner ways to achieve it.

I love the discussion that it sparked here too. It's wonderful and educational to see so many smart people breaking down the problem space, discussing de/merits of this particular technique, presenting alternative approaches with analysis on memory use, performance..

</comment></comment><comment><USER27> Asking to write a tree iterator was our "FizzBuzz" question when interviewing people for the kernel dev position. Worked exceptionally well.

<comment><USER28> Are you still hiring?

<comment><USER27> No. It was over 10 years ago, sorry.

</comment></comment></comment><comment><USER29> I missed the introductory part where you mentioned that these are _binary trees only_ and not just trees. I think you should mention it more prominently.

<comment><USER30> There's an isomorphism between binary trees and general ordered trees, man. That's how we can have Lisp.

<comment><USER29> Yes, sure, there's an isomorphism. Using that fact, is it then so trivial to show how this FSM approach works out if you don't have a binary tree to start with? I think it would make an interesting follow-up blog post. But failing that, it would have been nice to call out at the top that these are binary trees.

<comment><USER30> It definitely doesn't work if you don't change your in-memory data representation, and it requires you to change it in a way that forecloses the possibility of FP-persistent tree structures or efficient DAGs. Compared to that, representing your ordered-tree nodes as cons chains is a barely- significant change.

It _does_ generalize, though, to ordered-tree nodes that aren't represented as cons chains. You just have to walk through the _n_ child pointers when you're moving back up the tree to search for the one you're returning from, adding an extra runtime cost factor proportional to _n_ but no extra memory. Moreover, if you devote three or four registers to an amnesic stack, you can avoid that cost for the bottom two or three levels of the tree, which will necessarily contain the vast majority of tree nodes.

</comment></comment></comment></comment><comment><USER31> right, traverse a maze by always turning left

</comment><comment><USER32> Brilliant, I need something like this for one of my projects (attempt to decrypt Beale cipher)!

</comment><comment><USER33> > then, when discussing how to turn recursive functions into iterators using > an explicit stack (which permits breadth-first searching);

How about a depth-first search?

</comment><comment><USER34> I thought everyone learns that in the university.

<comment><USER29> Not everyone went to university.

<comment><USER34> Just an observation, for some reason people take as elitism the observation of having a complete well rounded curriculum has certain advantages. And this article has a good point to illustrate that there is value in taking an engineer rather than a boot camp code fighter.

Data structures are a very specific subject of IT Engineering, every person studying subject should take the whole curriculum, including recursivity scanning data structures and how to transform these recursive procedures into iterative code (or FSM as the article says).

<comment><USER29> I'd take an English major who is self-taught in computer science over a CS grad any day. And I'd take a bootcamp kid who went on to study the theory themselves over a Stanford kid who only knows this shit because he was tested on it. I'm honestly, at this point, not sure why we even teach software engineering in schools, as we so manifestly fail at imparting anything to the kids, and the ones that come out as decent engineers would have been so anyway. Oh right, it's signalling.

<comment><USER34> Whatever comparison you do, you should compare a representative subset of both populations in similar situations in life, not just cherry pick the self selected ones, the successful and survivors, ones from one population against the potatos from the other.

There is no point of comparison in five dedicated years of highly academic study versus 6 months of touching the techs du jour. Will the boot campers have also graph theory background? And compilers and assembly or just webpack preprocessors? Database design or just picking stacks? and compilers and network protocols?

Are you sure you are not comparing a 35 year old responsible person who made a boot camp to sustain his family or an entitled post-grad IT engineer who is still a bit high from the graduation party? It would be hypocrite not to admit that a hungry serious battle tested individual will have the right attitude to build a role rather than the not serious entitled individual who just wants to see what he can get from there and does not make any effort in continuously improve.

Once in a plane I met a CEO who told me he would take any day an engineer for the decision making role or business related roles. Was he crazy or it was just his previous experience conditioning him? you can take whoever you want for your company, even someone who took gender studies and ended up taking a code camp hoping to pay its former debt and ends up making high impact blog posts about thyr experience in the digital IT world as person of XYZ gender.

<comment><USER29> > Whatever comparison you do, you should compare a representative subset of > both populations in similar situations in life

Well, yes and no.

If you're an 18 year old who wants to program computers and you read Turing and Dijkstra for fun, then go study math or physics, which you won't have much time to do later. Or pick a subject in the humanities if you like to read and write. Is it a great way to become a programmer? Maybe not, it would be better to just go and get an internship or a job, but (1) smart companies are hard to find and (2) you'd be missing out on a chance to get a university education at the age when most people do so.

On the other hand, if you're picking coworkers, go with the self-motivated one who had to learn everything they learned because they were interested in it, over the one who had to learn it to get a degree in a field that everyone knows is a ticket to a high-paying job.

> Once in a plane I met a CEO who told me he would take any day an engineer > for the decision making role or business related roles. Was he crazy

Sure, this is the same point: if the only thing you're good at is your specialization, then you're probably not going to be very good at it. Unless it's something like chess that really has minimal relation with anything else in society.

</comment></comment></comment></comment></comment></comment><comment><USER35> Congratulations you've just recreated CS Intro Data Structures 61 tree iterators? Constant space is the normal way with a parent pointer.</comment>


Update Regarding DDoS Event Against Dyn Managed DNS on Oct 21 - <AUTHOR> https://www.dynstatus.com/incidents/5r9mppc1kb77 <comment><USER2> I am sure the DDoS problem is something that the free market will sort out. The individual players will make it costly for the other players to send problems their way. I expect a chain of "charge the next node for resource usage" to evolve.

If this chain will go all the way to the end user, I don't know. If it will, then end users will probably start using routers that feature restrictions / monitoring / control of outbound traffic. So you know your coffee machine will not be able to use up too much resources. Just like you have a fuse for the power line.

<comment><USER3> I agree with you. IoT devices inside a SOHO should communicate externally through a proxy gateway device. IoT devices should only have communications in a p2p network in a LAN, and have strong restrictions or none access to WAN. Any type of updates should be given from a proxy device having proper hardening than a normal IoT device.

<comment><USER4> The router could provide password-protected web proxy to access the LAN IOT webserver. Then you've reduced the attack surface to the router.

It almost seems like we need some protocol extensions:

1) Standard auth protocol (not just web-based) for the router to protect the local computers. Some kind of user-and-software-friendly firewall. This could even extend to game servers and whatnot - what if the "shared password" for connecting to a hosted game server had the shared password implemented at router protocol level?

2) DHCP registration on a network should require a _name_ , one that the user was prompted to provide at some point. No more identifying devices on you router by IP or MAC. You already need to provide a name for SMB or DNS, just finish the job and name all DHCP clients. Possibly this should work with DNS in some way.

This way user-friendly logging information can be presented to the user. Without that, routers don't have the critical information needed to tell the user which device is screwing up.

Edit: Google tells me this is already a thing... Sadly, good conformance on providing meaningful DHCP client names won't happen unless the FCC et al start testing IP-enabled devices for it.

</comment></comment></comment><comment><USER5> The free market will solve this problem with an existing and efficient tool.

Tort law.

A few wins in court will do the trick. Here is how.

A victim of a DDoS attack sues manufacturers, distributors, and retailers of that product for selling a defective product--the IoT device used in the IoT attack.

As soon as there is a win, the product disappears. Distributors and retailers must now price in the externality of the risk created by the product. It is far easier to take the product off the shelf. Therefore the manufacturer must either make a better product or die.

We don't need government regulation. We just need time. The legal system will do what it is designed to do: assign economic consequences to the right parties.

<comment><USER6> You say this as if it is just one device used in these attacks. And as if it will be easy to stroll over to the owner's location and determine the supply chain of that one device. And as if it will be easy to collect from the Chinese manufacturer who probably folded last week and reopened under a different name for completely different reasons.

<comment><USER5> This is why tort lawyers sue distributors and retailers in the USA. They are here and they have insurance.

Once there is a court decision, a rational seller (oh, Amazon hypothetically) will understand that selling fly-by-night small manufacturer items is fraught with peril. The offending items disappear from the marketplace.

<comment><USER7> More likely the marketplace itself moves overseas. People use Amazon because they sell the stuff they want. If they stop selling it, the buyers go somewhere else. Amazon is a website. It could as easily be a website hosted out of China.

Also, it sounds like you don't mind if the effect of your proposal is to destroy things like Etsy. And eBay.

<comment><USER5> The legal system is less brain-dead than you imagine. :-) This is not the first time in history that a plague of imported items causes a problem.

Etsy and eBay will survive. Ford survived the Exploding Pinto. Firestone survived its tire debacle. And we are all the safer for it.

<comment><USER7> > Etsy and eBay will survive. Ford survived the Exploding Pinto. Firestone > survived its tire debacle. And we are all the safer for it.

Ford and Firestone aren't retailers, they're manufacturers, and they were held responsible for their own mistakes, not the mistakes of third parties.

There is no reasonable way for online retailers to evaluate product safety of millions of small batch third party products. Either they sell them without evaluating them or they don't sell them. Imposing liability on them is exactly how you get them to not sell them, but then we can't have Etsy or eBay.

> This is not the first time in history that a plague of imported items causes > a problem.

It seems like the first time the problem has happened in this particular way. Historically importing was a large-scale operation done in bulk with homogeneous products, so the importer knew what they were doing and had deep pockets. Today you can cost-effectively get a 99 cent piece of electronics shipped directly from a one-person shop in China. Either you shut down the entire _idea_ of that, and then things are going to cost a lot more than they do now, or we need a different approach.

</comment></comment></comment></comment></comment></comment><comment><USER8> Pardon my ignorance, but why don't companies run their own nameservers?

I get why you don't want to run email - it's highly reputation driven. But as far as I can tell, running nameservers is no harder than running webservers or DB servers. HA is potentially even easier, because the system was designed that way from day zero.

I'm not suggesting I'd run one for my personal website, but twitter and github are already managing distributed networks for this. What are the services Dyn and others provide that are so invaluable?

<comment><USER9> Getting good, consistent, well routed, fast and secure DNS Is harder than you'd think. Dyn typically sing speed as the main selling point for their DNS product, they do this through a large distribution of domain name servers geographically and anycast. Many hosts (like say, DigitalOcean) run their own DNS but use something like CloudFlare Virtual DNS on top. Personally I was surprised so many large sites trusted Dyn, Route 53 is a more robust product for production and scale. In the past, I've seen hosting providers switch to Dyn, give them load, cripple them, and have to scramble to revert away. I'm not at all surprised his happened, even given the uptick in botnet traffic globally.

<comment><USER10> Route 53 aint all that. We approached them about handling our customer's domains, and they said no way. They didn't have the capacity. Granted, this was 2 years ago, but Dyn has a much better reputation (still) than Route 53.

</comment></comment></comment><comment><USER11> This attack looks like another probing into critical internet infrastructure Bruce Schneier had talked about.

Who's next?

<comment><USER12> You suggest it's probing. Then how would the full scale attack look like? Also, how does probing help the perpetrators, doesn't it lead to better defenses in the future?

<comment><USER13> These attacks seem to be getting sophisticated faster than defenses are being thrown up. It might lead to better defenses in 5 or 10 years, but the Internet at large is built out of infrastructure is difficult and slow to upgrade. And is everyone scrambling for a solution? Not really... Most are taking the mentality "well sucks to by Dyn but it's not affecting me so I don't need to respond".

Especially because the vulnerabilities being exploited right now seem to by systemic. It's not like patching a zero day. "Uh... we've got 45,000,000 un- upgradeable IOT devices from dozens of different manufacturers executing a DoS attack". You can't fix that the same way you fix a privilege escalation bug in the linux kernel.

If the attackers end up finding the Internet's equivalent to a jugular, there might not be much we can do about it. BGP isn't going to be replaced in the next 10 years, it's here whether we like it or not. DNS is also not going to be replaced in the next 10 years. And neither are the major centralized internet exchange points. Any vulnerabilities an attacker can find related to the fundamental design of those things are going to remain vulnerabilities for many years. If the attacker can get good at exploiting them, we are in trouble.

</comment></comment></comment><comment><USER14> The movie Fantasia Sorcerers Apprentice was right! All the the household objects will rise and overthrow their masters!

</comment><comment><USER15> Is there any info on what volume (how much bandwidth) the DDoS attacks were?

<comment><USER16> Curious as well. I didnt see any in the status here nor the original.

</comment></comment><comment><USER17> DNS seems ripe for revolution!

Yesterday it felt obvious that we are treating DNS data as too ephemeral. I am not intimately familiar with the implementations in BIND and others, but it seems like when we hit the TTL we just throw away the data. Usually, that works fine. But, in the case of the origin servers not responding, yesterday I was wishing that it would just give me back the stale data rather than giving me nothing.

I'll admit the impact on me was somewhat limited. Around 10am Mountain I was trying to install some Atom.io modules and couldn't reach that site or a github download URL. I had some success with using 220.127.116.11 (184.108.40.206 was not answering the names).

Using a stale cached result probably wouldn't have helped for atom.io though, I hadn't been there in a while and this was querying my own local name servers. Do I want my name servers keeping weeks old stale data around? Probably not in RAM, but saving old names to disk sounds like it would require a lot of IOPS for a big provider. But I do know I'd visited the sites I was trying to hit within the last few weeks, since I couldn't reach the authoritative servers it'd be nice to have tried the last IP I had for them.

Of course, I ran into this about an hour after I rebooted my entire dev/staging infrastructure to fix the Linux kernel privilege escalation issue, so my caches were cold.

Sure would be nice if my server could "ask around" if it can't talk to an authoritative server. "Hey Google, hey Comcast, hey Level-3, do you know this name?" That's effectively what I did by changing my resolv.conf. But if you start asking around too widely, you ideally probably want to have some signature to verify the data you are getting.

Seems like the new norm might be listing authoritative DNS servers from multiple big providers (Dyn and Route53) and having to keep them in sync? Then you lose some of the advance features...

Funny aside: One of the sites I run uses Distil in front of it to protect against content scrapers. Months ago I was working with their support about getting a health checker set up in Route53 to test the full paper path through distil and fail over to our backup site if anything on the primary paper path didn't work. Distil assured me that their services were so resilient that we shouldn't worry about them being down. Guess what the only part of our infrastructure was that was impacted by this? :-)

</comment><comment><USER18> So is it retaliation for cutting Assange's internet, retaliation for threatening Russia with a cyber attack or both?

<comment><USER19> [https://twitter.com/wikileaks/status/789574436219449345](https://twitter.com/wikileaks/status/789574436219449345)

</comment></comment><comment><USER20> So I'm wondering, what are the implications of this ?

Someone is controlling a powerful enough botnet to do this.

How powerfull is it really ? What else can it do ? Was this just a message or a test ? Or both ?

What would happen if they would point it to google's nameservers ?

What should we expect next ?

<comment><USER9> It's complicated because the internet is a complicated(and I'm no expert)- but, it's not good, but it's not insurmountable. The big issues today are: lots of network attached compute, lots of types of traffic, highly highly distributed network. There are numerous ways of mitigating DDoS, today a lot of it is via BGP route announcement[1] - although we've seen folks using BGP in questionable ways to mitigate DDoS recently[2]. As more and more of the internet becomes software defined (like SD WAN sees large global rollout)[3] more and more granular but non-disruptive control will be enabled. To answer your question about google, their DNS is probably resilient enough to deal with a pretty huge attack but who knows really -- In my dream world we get some good neural networks built and deployed to the edge watching for unusual traffic patterns and disrupt them. It used to be biggest pipes win, I don't know that will continue to exclusively be the case.

[1] [http://www.enterprisenetworkingplanet.com/netsp/article.php/...](http://www.enterprisenetworkingplanet.com/netsp/article.php/3615896/Networking-101-Understanding- BGP-Routing.htm) [2] [https://www.youtube.com/watch?v=LFJzu0AFDpU](https://www.youtube.com/watch?v=LFJzu0AFDpU) (Dyn Engineer gave the talk) [3 ][http://www.rcrwireless.com/20160408/telecom- software/using-s...](http://www.rcrwireless.com/20160408/telecom- software/using-sd-wan-combat-ddos-aggressive-attacks-tag2)

<comment><USER20> > we get some good neural networks built and deployed to the edge watching for > unusual traffic patterns and disrupt them

I was thinking at this as well. It's probably under active development right now, if not already live for some.

</comment></comment></comment><comment><USER21> Did they mitigate the attack or attack has stopped? I am asking, because update is lacking the details about said mitigation.

</comment><comment><USER22> you can watch BGP routes changing (as we speak) here [https://stat.ripe.net/widget/bgplay#w.resource=126.96.36.199](https://stat.ripe.net/widget/bgplay#w.resource=184.108.40.206)

</comment><comment><USER23> I think it's about time they updated [http://dyn.com/ddos/](http://dyn.com/ddos/)

It was an unprecedented attack, sure, but I'm not sure how their sales guys are going to spin this.

</comment><comment><USER24> Did Dyn mitigate the attack, or did it stop by itself?

</comment><comment><USER25> I'm so looking forward at IPv6, the death of NAT, and billions of IoT devices with all ports exposed to the world :-)

<comment><USER26> Arent most IOT devices behind a router and thus unexposed directly to the internet (excepting routers)?

This part of these attacks confuses me.

<comment><AUTHOR> Compromised routers can be used to compromise devices behind it. Also many devices (like IP cameras) usually have port forwarding to allow the users to access it from outside.</comment></comment></comment>


Can Uber ever make money? - <AUTHOR> https://www.economist.com/business/2019/04/27/can-uber-ever-make-money <comment><USER2> There is literally no way for a mass-market taxi service to make money unless they have a monopoly, or some unexpected disruptive technology comes along that gives one company an unassailable advantage in lowering costs.

The Uber app was the original innovation, but it's not particularly innovative anymore. Besides Lyft, traditional taxi companies have similar apps in some markets.

Skirting regulation and good labor practices were also supposed to help, but that can't last once you become competitive with the establishment.

Self-driving cars were supposed to lower labor costs, but that's not going to be a thing for decades if it ever works. But even if it does, it won't be an exclusive technology. They won't be able to make any more profit from it. If their costs go down, so will everyone else's and they won't be able to undercut the competition without continuing to lose money.

<comment><USER3> But in a way, that's the beauty of disruptive use of technology. Without the innovation of the Uber app, the traditional taxi cartels would have no incentive to improve their services and the fact that they have been forced to do so in order to compete benefits the consumer. They're not quite there yet, and are still catching up. The next logical step is a lift aggregation service that creates a marketplace.

I want a lift from x to y to z. Bid on me.

<comment><USER4> As it stands the price you pay for a ride in an uber is unfairly low. Driving it even lower would be irresponsible.

<comment><USER5> Call me a snob, but Uber's criteria for what qualifies as an acceptable car has dramatically declined to the point where even Taxis are sometimes "nicer". Combine that with the fact that the drivers often don't know the neighborhood and rely exclusively on GPS (so tend to be worse drivers than a taxi).

I'm left feeling like the fare - strictly from the perspective of a consumer - is actually too high.

<comment><USER6> > Combine that with the fact that the drivers often don't know the > neighborhood

I used to travel to Houston quite a lot. Not sure if this is representative of taxis in general in the US, but the taxi drivers _never_ knew the neighbourhood. I couldn't say "I want to go to Acme Stuff in X district", I had to give a "Cross road" (a very US thing, that) and the driver would use satnav based on that. It did seem that a lot of taxi drivers were recent immigrants (invariably chatty), so that accounts for some, but not all.

At first, this was a surprise, as in my native UK I can literally give a street name within 50 miles of where I am, and the driver cocks their head for a second, gears whir, and yes! They know it!

<comment><USER7> That's probably because becoming a taxi driver in places like London is extremely hard ([https://www.theknowledgetaxi.co.uk](https://www.theknowledgetaxi.co.uk)), while becoming a taxi driver in Houston is probably as simple as "do you have a license?"

<comment><USER8> Yes, London is the whole UK.

EDIT: If it sets a standard then why has that standard not been adopted nationwide? Your argument is dead in the water and I can only assume you're speaking as a person who has never visited the UK.

<comment><USER7> It sets a standard.

The US has no such equivalent.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I can only see them making money in three ways :

\- They scale up massively and capture a huge fraction of the ride sharing market, thus allowing them to offer low cost / low margin rides, but making up for it by volumes : At least in India, they are in a good position, but they seem to have conceded the other markets with high potential for growth like SE Asia and China.

\- They have a premium offering that is actually quite a trend-setter : Revive the Uber Black service and take it up a notch, target richer clientele which will be willing to pay a premium to ride in a much better car and with trained chauffeurs.

\- They push heavily in the pooling space - this allows them to use their vehicular inventory better as well as drive up margins per ride : This doesn't seem to be a huge focus for them as of now, but it could change.

The other big change that could transform their fortunes is if they put out a viable self driving product. I do not think it'll be economically profitable, or even technologically viable in the near term.

<comment><USER10> Option 4: Uber uses the expertise, infrastructure, and brand it built in ridesharing to expand into the becoming the de facto standard for timely last mile delivery. This is why UberEats is such a major strategic focus for the company. It's proof of concept that they can do more than be a taxi.

An efficient platform for quickly and cheaply ferrying around stuff around town is a _way_ bigger market than ridesharing. We're not only talking about the obvious examples like residential packages, groceries, and laundry.

But entirely new business models that didn't previously exist, because there's no efficient last-mile delivery solution. Cloud kitchen are an example of a nascent industry that could only emerge in a post-Uber world.

<comment><USER11> Courier services have existed for years for last mile.

I have a friend who was (20 years ago) paid full time by one law office to get papers to/from the court room (he worked for a company that dedicated him to that job). Part of his job was knowing the proper way to walk into a trial in progress.

My dad more than once hired a taxi to take a package to the airport. I suspect post 9-11 this isn't legally allowed, but back then if you needed to get a bug fix to a customer your fastest option was put it on a tape, put the tape in a suitcase, buy a plane ticket for "Mr Package", and then check the suitcase in the name of Mr Package.

<comment><USER12> >My dad more than once hired a taxi to take a package to the airport. I suspect post 9-11 this isn't legally allowed, but back then if you needed to get a bug fix to a customer your fastest option was put it on a tape, put the tape in a suitcase, buy a plane ticket for "Mr Package", and then check the suitcase in the name of Mr Package

I worked for a company where we would hire a courier to fly with the package when needed.

We sent a few tapes that way, but also some important equipment when it was critical to a customer.

It was by far the fastest and best way to be sure a dude would be at the office door in X minutes, on the plane in Y, and at the customer site in Z with the needed thing(s).

It was expensive, but they were really reliable and professional.

</comment></comment></comment></comment><comment><USER13> Looking at Uber, WeWork, JustEat etc., I get the feeling there's a growing gap between what we were brought up to believe about entrepreneurship and innovation vs. what the reality is.

We're all familiar with the '>50% of small businesses fail' stat (not the real number) and that we should be toiling and not taking a salary for years until we break even, i.e. reach a point where we're no longer losing money on every project/sale. If you aren't able to achieve this, your pricing, product-market fit, etc. have failed, because the all-knowing, all-equilibrium-izing market has spoken.

Every politician out there shows reverence for the 'small businessperson' taking risks and bringing jobs and tax revenue into their communities. But once in office, the people they meet with are Travis Kalachian and others, people who cannot make a profit, but are excellent at convincing rich investors that they should make Travis a millionaire.

The reality is that a VC-backed company can simply outprice any competitor, and if it's not enough, can lobby city hall to change the laws to favour them. And this cash spigot can stay on for years, as long as you can show you're gaining customers every quarter. If you're a big enough play, you can move your whole HQ to a low-tax friendly country and administer local operations from there.

The jobs that are created by this are right around minimum-wage level, so how is this supposed to create wealth for anybody except the shareholders who got in early with cheap valuations?

<comment><USER14> The question (with Uber in particular) is that if this business model really covers up a huge ponzi scheme.

Before the last decade (or even 5 years?) the amount of money some of these companies have been raising would have been unheard of and required an IPO.

But if you can keep finding ever bigger investors to give you ever more money...

It's a really weird reality, because typically the worst kind of place to start a company is one with huge barriers to entry that requires a ton of money to get started. Yet that seems to have done quite well recently.

Squashing minimum wage service providers seems to be merely a side effect... these companies can't really make money EVEN WHEN they ignore regulations and exploit their not-technically-employees.

<comment><USER15> > with huge barriers to entry that requires a ton of money to get started

Just a note that the proof of concept stage was very cheaply done. Travis hired someone to make an app and the "dispatch" system was just a single php page. He then tested it out with his friends.

That said, I agree that ultimately to get to the position they are today required a ton of cash - just the getting started was pretty cheap.

Actually we have the blow-by-blow of how it all played out on our podcast since Jason was employee #7 and built out the original node dispatch system.

[http://techzinglive.com](http://techzinglive.com)

I'm not sure exactly what show he started working on it, might need to go back to 95. Full show list here:

[https://www.dropbox.com/sh/frnqzndys9j3vc1/P7jHgH0BQK](https://www.dropbox.com/sh/frnqzndys9j3vc1/P7jHgH0BQK)

<comment><USER16> Hustle recognize hustle but which podcast episode is it dude? There are 300+ eps at that link.

<comment><USER15> Well it's A LOT of them.

Because we spoke about the Uber stuff as Jason lived it in real time ;)

A fun one was when Jason turned down the CTO position and of course we had no idea at the time what that really meant.

I guess in the latest episode 321 (Quick & Dirty) Jason does give an overview of the Uber story and how he got involved if you want a TL;DR version.

</comment></comment></comment></comment></comment><comment><USER17> Keep in mind that this may not change when driverless cars become widely available. Maintaining a fleet thousands of driverless cars may prove to be more expensive than crowdsourcing a vehicle fleet.

<comment><USER10> There's either two possibilities for the self-driving future.

1) Hardly anybody owns cars anymore. When we want to get from point A to point B, we just call a driverless car from the network. But this type of service is a natural monopoly.

When people are ridesharing everywhere, the network with the biggest fleet, becomes the stochastically fastest and cheapest option. Plus add in the fact that they're collecting a lot more data to constantly improve algorithm. This is a winner-take-all market with fat monopoly margins for whoever wins. If Uber prevails, it's likely that their profits will be larger than the entire global auto market.

2) People still mostly own their own cars. They're driverless, but people like the convenience, customization and sense of personal ownership with having their own vehicle. Ridesharing is still an out-of-the-ordinary option for unconventional lifestyles or special occasions.

But in this world, Uber doesn't need to own their own fleet. The same crowdsourcing logic still works. Personal vehicle ownership means a huge number of idle vehicles sitting in people's driveways. Without the pain-in- the-ass factor of actually having to be the driver, expect a lot _more_ people to put their car on Uber's network.

<comment><USER18> Not following how this is a natural monopoly. There aren't barriers to multiple companies operating large fleets.

<comment><USER11> In fact the market is likely large enough that multiple large companies will.

</comment></comment></comment></comment><comment><USER19> I'm not quite sure everyone discussing this, including the economist article author, is in the loop with why Lyft and Uber are getting valued the way they are. Wall Street is not paying for a taxi company, they are betting that Uber and Lyft are the best poised to first rollout driverless ride sharing, i.e. one of them will own and monopolize patentable tech that the mass public wants to use. The existing experience is a commodity business where profits go to zero, that's well known and old news.

<comment><USER20> Driverless ride sharing isn't coming, in a meaningful way*, in the next ten years, I'm willing to bet my house on that.

Meaningful such that it actually delivers monetarily

<comment><USER21> Do you plan to short UBER on amount of your house cost?

<comment><USER22> That would be a bet on both rational markets and lack of driverless technology. That's a very different bet.

</comment></comment></comment></comment><comment><USER23> No, in its current state, they can't be profitable. Other transportation options will become more desirable if Uber had to truly charge what it costs to transport a customer so for now and the near future they have to subsidize the rides which kills the possibility of profit.

Autonomous cars will help but that won't happen for years, 5-10 maybe more. Additionally, if autonomy becomes a reality for Uber it will happen for other modes of transportation too so they will have to compete there too. That will be even harder since it's easier to get a bunch of cars working once the tech is available.

Also Uber will not survive as it stands now. Regulations and public pressure will eventually force it to conform to something similar to a very large cab company.

Every industry eventually gets hit by consolidation. A business of individual drivers can be even more profitable if it joins drivers into groups. They become more reliable, they cost less to maintain and they gain power that they can leverage against large industry actors. Maybe then, it will become profitable. But that won't happen until the industry matures.

<comment><USER24> I wonder if self-driving cars will be like Fusion, but instead of always being 20-40 years away, it's always 5-10 years away.

I do really feel like Waymo will prove successful in specific environments THIS year and increment slowly, and eventually our transportation habits will change to better utilize a cheaper, safer form of transportation.

But I wouldn't be surprised if I'm still saying that ten years from now.

</comment></comment><comment><USER25> The mouse changed the way we work, and it got commoditized. Just because a company was first to adopt and make a business out of a disruptive technology doesn't mean it's going to corner the market with it. In fact, history shows that it's not the best approach to be the first adopter of a technology (Kodak, Altavista, etc..)

</comment><comment><USER26> My observation - Uber is starting to have hard times competing on local market. The quality of local competitors - their apps, fleet and service overall is growing and Uber seems trying to reduce costs in this situation, which probably won't help in a way, so to speak.

</comment><comment><USER27> It seems like it it's the same with shared bikes. The only one making money are bikes manufacturer.

</comment><comment><USER28> Outline Link: [https://outline.com/endjbs](https://outline.com/endjbs)

<comment><USER29>
        If someone else might own the copyright to it, don't submit it. Outline is for reading pages that:

        1) you own the rights to,
        2) is in the public domain,
        3) constitutes fair use, or
        4) you have consent of the copyright holder.

</comment></comment><comment><USER30> Re Competition Taxis or Uber are not always the cheapest way. One of my friends used to do a lot of travelling to Tokyo on work assignments. The others from the company always hailed a cab and ended up paying $100 for a trip from the airport to the downtown hotels. My friend would jump onto the Tokyo subway and was always there at the hotel to greet the rest of them when their cabs would arrive. He usually paid just a few bucks for the subway token. Myself, I would sometimes get rides in the day from people I just met on the plane. But then it was a more trusting time. But maybe there is an app there that pairs people arriving in the same town. Those that have their cars in the airport parking lot and are willing to share a ride and those that are willing to pay a few bucks.

</comment><comment><USER31> With all of the ride sharing companies burning money it seems like the real bet here is being the last one standing. If Lyft folds, Uber will have much more leverage to play with the margins.

<comment><USER32> This is basically it.

I have a choice between 4 services in my city. I pick the cheapest one whenever i want to go somewhere (Thanks VC cash burn)

Last man standing gets the monopoly.

</comment></comment><comment><USER33> Theoretically, couldn't they just fire almost all non-maintenance engineers and continue running the app without building new features? Wouldn't that make them profitable instantly?

<comment><USER34> Stopping expanding would probably have a bigger impact. Are they really doing a lot of new features/incremental infrastructure development at this point? (Though I don't know how much they're still blowing on self-driving.)

</comment></comment><comment><USER35> This might be ignorant, but I've wondered about this for a while.

Isn't Uber essentially a software company? The drivers doesn't get anything besides their share of the fares?

According to the free part of the article they've lost 7.9 billion dollars since 2009.

How expensive can it possible be to develop and operate their software? According to some sources uber takes a cut from 20-25% (although some claim much higher fractions in practice)

[http://www.businessofapps.com/data/uber- statistics/#6](http://www.businessofapps.com/data/uber-statistics/#6) claims ~10B _quarterly_ revenue for 2018, with net loss at close to 1B. (EDIT: seem the revenue is counted as full fare price - not ubers cut, so that brings "actual" revenue to ~2B/Q)

How are they not making money..? Lots of normal taxi services have booking apps too. If they IPO it'll be interesting to see a more detailed breakdown of their costs.

EDIT2: found the IPO prospectus including some numbers: [https://www.sec.gov/Archives/edgar/data/1543151/000119312519...](https://www.sec.gov/Archives/edgar/data/1543151/000119312519103850/d647752ds1.htm#toc647752_9)

</comment><comment><USER36> I want to talk to someone who has invested in Lyft (or plans to do so for Uber) through the public markets and is in it for a long-term buy and hold. Like someone who really thinks the share price represents the net present value of all future cash streams for Lyft/Uber.

I think I'd have a hard time finding someone like this. You don't have to believe the company will make money, you just have to believe you can sell your piece of the company to the next guy for more.

<comment><USER37> > You don't have to believe the company will make money, you just have to > believe you can sell your piece of the company to the next guy for more.

This is why people invest in gold and dogecoins.

</comment></comment><comment><USER38> I always try to bring this up in re: Uber/Lyft et. al.:

[https://www.ianwelsh.net/the-market-fairy-will-not-solve- the...](https://www.ianwelsh.net/the-market-fairy-will-not-solve-the-problems- of-uber-and-lyft/)?

> Here is the thing about Uber and Lyft (and much of the "sharing economy").

> They don't pay the cost of their capital.

> The wages they pay to their drivers are less than the depreciation of the > cars and the expense of keeping the drivers fed, housed, and healthy. They > pay less than minimum wage in most markets, and, in most markets, that is > not enough to pay the costs of a car plus a human.

> These business models are ways of draining capital from the economy and > putting them into the hands of a few investors and executives. They prey on > desperate people who need money now, even if the money is insufficient to > pay their total costs. Drivers are draining their own reserves to get cash > now, but, hey, they gotta eat and pay the bills.

</comment><comment><USER39> Tesla is in the best position to win in this arena. As the article points out it's cutthroat so the one with the lowest costs, acquisition and operation will win.

Telsa makes it's own cars, the cars have super low operating costs: built for 1m miles, almost no ongoing maintenance, powered by electricity that is already much cheaper than gas and will continue to get cheaper. And finally no driver with self driving software.

No one else has this combination attributes for the lowest cost operation. Though it could be Tesla after it's bought by Google.

</comment><comment><USER40> Uber wants to be the Walmart of taxis where they start with super cheap pricing to hook in new customers and then turn the switch on once everyone is dependent on them.

It'll be very hard or impossible for Uber to be a monopoly and having more competitors means the prices will have to compete to remain attractive to people.

An unlikely idea for Uber becoming profitable is if they form a cartel with Lyft and other Uber-esque services to give all users no choice but to pay the high prices, but then they'll be competing with traditional taxi services once again.

</comment><comment><USER41> I think Uber can easily make money if they function more as a marketplace and less as an opaque transportation company.

Their job is to connect riders to potential taxi providers (which is what they are, since they're all 'contractors'). Contractors could have a rating that's viewable before accepting a ride offer from the driver.

Fire almost all of their staff, just need a skeleton crew to keep the webstore and ride-match stuff up. Stop wasting money on self driving stuff, it's never going to happen. Take a flat rate per-ride, done.

</comment><comment><USER42> It's so very hard to find any ride share profit Signal within all the Noise of monetary policy and VC weaponisation of capital.

Airlines at least have the possibility of temporarily defending margins on city pairs with mismatched supply/demand.

Perhaps surge pricing on ride share is an analog to that.

The accumulated debts of ride share players seems akin to picking up pennies on ground getting stickier and the steamrollers are accelerating.

It will be interesting to see what happens if/when full spectrum subscription ride share options become available.

</comment><comment><USER43> They have other lines of business. I think Uber Freight has a lot of potential to make money (and might already be doing so?). I'm a little skeptical of Uber Eats, though.

</comment><comment><USER44> What's the point of these speculative pieces that just list pros and cons? Give me actual projections, explore what it would take for them to be profitable. Otherwise it's just not that interesting

<comment><USER45> The people writing these pieces have neither the depth, nor the insights to generate those projections. If they did, they wouldn't be writing blog posts for ads clicks.

</comment></comment><comment><USER46> So UBER is just an more advanced cab company. You call a driver to drive you from A to B. It tries to under cut the traditional cab company significantly, and unfortunately that means a loss is needed until they can get rid of drivers using autonomous systems. At that time when it will also drive the cost down as they will Not be the only one with autonomous systems. It will look more like a commodity unless you are an monopoly

</comment><comment><USER47> The same way Amazon can make shareholder money back. Stay out of reach of the law forever. In ueber case it bets on cutting out the trad taxicompetition- one way or another.

Means ueber will want to be the market structure provider like Amazon for rides Haring and bets on becoming mandatory for all mobility related services.

It really jars the mind- all those billions betted against trust busting.

</comment><comment><USER48> Here's the full article: [https://outline.com/endjbs](https://outline.com/endjbs)

</comment><comment><USER49> Uber isn't in the taxi business, it's in the self-driving car business. Taxis with drivers are just a stopgap, just like DVD's were a stopgap for Netflix.

The Economist is right that there are few real network effects or moats around the taxi business. But Uber's valuation depends on it:

\- Developing the best driverless tech

\- Being able to roll driverless taxis out to an established customer base faster and at far larger scale than anyone else (main advantage over Waymo/Tesla)

\- Using a headstart in driverless tech, compared with huge data advantage from millions of Uber rides, to maintain the #1 position

Of course all this depends on Uber being _able_ to develop the best self- driving tech, and getting such a data+scale first-mover advantage to stay ahead of competitors.

Skeptics will say Waymo will have the best tech, and expand/scale its taxi service effortlessly, and thus that Uber's existing customer base doesn't provide any advantage at all -- everyone will just jump ship from Uber to Waymo.

<comment><USER50> As a customer I would just use whatever is cheap, safe, and fun. Tesla M3P, no chatty-but-dull driver = cheap, 5+ star safety rating = safe, and performance = fun.

</comment></comment><comment><USER51> Uber will eventually profit from long-term SAAS+Ops+robots contracts with municipalities and governments, assuming they can grow/operate for that long and out-lobby and out-execute Lyft, Ford, Tesla, Google et al.

</comment><comment><USER52> I would say.. no?

The optimal solution would be an exchange that provided order books with bid/asks for each passanger and driver.

Alternatively you could have a real-time auction to make it easier for the passanger.

The exchange could take a 5-10% cut on each ride.

</comment><comment><USER53> If I was a car manufacturer I would invest heavily on Uber. The fact that people need to have a new car to work for Uber is one of the main factors that keep the auto industry out of recession.

<comment><USER54> _people need to have a new car to work for Uber_

Quite the opposite. There have been complaints in this thread about how the cars are getting shittier and shittier.

For Lyft for Oregon the car needs to be 2004 or newer for most of the smaller cities, 2009 or newer for Portland. [https://www.lyft.com/driver-application- requirements/oregon](https://www.lyft.com/driver-application- requirements/oregon)

Uber is much the same. 15 years old in rural areas, 10 years old in Portland. [https://www.uber.com/drive/eastern-oregon/vehicle- requiremen...](https://www.uber.com/drive/eastern-oregon/vehicle- requirements/) [https://www.uber.com/drive/portland/vehicle- requirements/](https://www.uber.com/drive/portland/vehicle-requirements/)

Realistically, a 10 year old car is closer to "junkyard" than "new".

</comment></comment><comment><USER55> I feel like there is a hidden metaphor for OSS companies hiding in this article:

> Customers benefited, but no one else did

Customers being the users of OSS, the ones not benefiting being the companies making it

</comment><comment><USER56> Well, this may drive a crowd away, but maybe the best way for monetization is to make Uber a subscription based thing. Obviously, it would have to be cheap though.

<comment><USER57> Moviepass tried do so the subscription thing on a very large scale and that didn't really work. Netflix pulls it off since they own most of the content and some of the distribution network.

<comment><USER58> It didn't work for Moviepass because they were trying to sell a subscription at a price that was less than one movie a month.

AMC is doing it quite successfully. The difference?

\- they made a deal with the movie distributors to account for all shows to be sold for $8.99. AMC keeps 50-70%. Worse case, AMC pays $6.30 per movie. Movie Pass had to pay the full retail price.

\- AMC has crazy margins on concessions. Add to that the theatres with bars and full restaurants.

\- They make money on pre movie advertising.

</comment></comment></comment><comment><USER59> Is Uber yet selling the information on who took a ride, when, from where and to where? I expect them to monetize that. Privacy? Is that something edible?

</comment><comment><USER60> I would love an all you can ride monthly subscription. I know Uber/Lyft have done limited trials but nothing truly an unlimited subscription.

<comment><USER37> I would totally Moviepass the crap out of this. Road trips across Utah and the Colorado Rockies, Pacific Coast Highway, etc.

Amtrak used to have a similar pass arrangement.

<comment><USER60> There would be limits for this to work, like a geofence

</comment></comment></comment><comment><USER61> [http://archive.is/LP5BL](http://archive.is/LP5BL)

</comment><comment><USER62> Uber can't make money from taxi service. It's trying to mimic Amazon model where profit comes from subsidiary service. Uber is yet to find such thing. Without valid profit source I don't understand how SEC approves an IPO for a license to suck away average joe investment. Uber is walking a very thin line between being a hoax and a business.

</comment><comment><USER63> Uber is not making money? How? Why?

<comment><USER64> Their expenses are higher than their revenue.

<comment><USER65> If I bought 10 McDonald's franchises today, my expenses would be higher than my revenue for probably the entire first year if not more.

If things are going well, and I make another bet by buying 10 more, my expenses would continue being higher than my revenue (even if the first 10 stores become profitable).

If I keep doing this, I may not show a profit on my balance sheet for a long time.

This example illustrates why "expenses are higher than revenue" doesn't reveal anything and that such a simplistic picture of a business is useless.

<comment><USER58> What would it reveal if your marginal costs were greater than your marginal revenue? Has anyone done a cohort analysis?

</comment></comment></comment></comment><comment><USER66> If there is enough demand. So if people stop buying cars and rely on Uber then maybe.

</comment><comment><USER67> Uber's plan is to own every car once self-driving cars are realized.

</comment><comment><USER68> Uber might make sense as a public utility. If it's going to lose money no matter what, why not fund it like public transportation? Lots of taxes on the back end offset by user fees to take some of the sting off and mitigate overuse.

<comment><USER69> > If it's going to lose money no matter what, why not fund it like public > transportation?

Why would we as a society care if Uber exists enough to socialize the cost of it? I kind of like using Uber, but probably not enough to justify footing the full bill for it (thank you VCs).

</comment></comment><comment><USER70> Yes Uber can make money, once they have an EV that selfdrives.

</comment><comment><USER71> Yes, as soon as the car drives itself.

<comment><USER72> Tesla rents out their ai customer owned fleet taxi service to Uber and Lyft

</comment></comment><comment><USER73> Why use uber? Just take a bicycle...

</comment><comment><USER74> Uber prints money on non-pool rides in mature markets. The losses are all going to fund expansion and single-rider pools.

<comment><USER75> source?

<comment><USER74> [https://www.sec.gov/Archives/edgar/data/1543151/000119312519...](https://www.sec.gov/Archives/edgar/data/1543151/000119312519103850/d647752ds1.htm)

$940 million in Contribution Profit in 2018 and that does not net out new markets and non-pool rides.

So, to re-affirm more emphatically, Uber prints money on non-pool rides in mature markets.

</comment></comment></comment><comment><USER76> No. Relevant article by an expert:

      https://www.nakedcapitalism.com/2019/04/hubert-horan-can-uber-ever-deliver-part-nineteen-ubers-ipo-prospectus-overstates-its-2018-profit-improvement-by-5-billion.html

</comment><comment><USER77> What is the point of this article? If you're going to ask a question, do some work to actually find an answer.

</comment><comment><USER78> Betteridge's law of headlines:

      "Any headline that ends
       in a question mark can be
       answered by the word no."[a]

This opinion piece makes a compelling case that profitability will be hard to come by for Uber, Lyft, and their various smaller competitors.

[a] [https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...](https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines)

</comment><comment><USER79> Uber made 997 million in profit in 2018. Read the S1.  #fakenews

<comment><USER80> Where do you get this from? They lost $3B in 2018. Uber's S1 clearly states as one of their risk factors to maybe never make a profit.

</comment></comment><comment><USER81> Whether or not it makes money, Uber is making cities more polluted and less safe already.</comment>


Computing multiple hash values in parallel with AVX2 - <AUTHOR> https://www.snellman.net/blog/archive/2017-03-19-parallel-hashing-with-avx2/ <comment><USER2> Great post and he raises a very valid point that most high-performance hash functions are essentially optimized for high-end 64-bit superscalar microarchitectures. As a practical consequence, the only exploitable parallelism in the silicon is the multiple ALUs within a single core and the algorithms are designed to exploit that effectively. However, as noted in the article, this runs directly into the fact that vector pipelines only support a limited subset of the operations supported by the scalar pipelines, and for the purposes of high-performance hash functions these tend to be disjoint.

His prescription of a new high-performance hash function based on mul32/xor/shift is also exactly correct in my opinion. Not only does it allow you to scale the hash function down to limited microarchitectures but it also allows you to scale it up to the largest/fastest vector engines in most cases for extreme throughput on higher end hardware.

Coincidentally, I also started working on a new set of hash functions based on the mul32/xor/shift motifs precisely for the purpose of getting something that could be efficiently vectorized for performance while still able to run on, say, a 32-bit ARM core.

<comment><USER3> Thanks.

> Coincidentally, I also started working on a new set of hash functions based > on the mul32/xor/shift motifs precisely for the purpose of getting something > that could be efficiently vectorized for performance while still able to run > on, say, a 32-bit ARM core.

That's cool to hear. I'm happy to reimplement hash functions to fit my bizarre needs, but as per the post I really don't want to design them :)

What's your view on how switching from rotates to shift-xors affects quality? Is it mostly neutral as long as the constants are chosen correctly, or is there some quality loss?

<comment><USER2> The use of multiplication as an operator introduces a bit value bias that arguably makes replacing shift with rotate less of a problem than it would be for algorithms that use e.g. the add/rot/xor operators, since shift introduces a (non-symmetrical) counter-bias to multiply as it is basically a special case of division. Rotate is nice for bit diffusion but most of the necessary heavy lifting is in the high bits being shifted low.

I have no hard evidence but my intuition is that the loss of quality by moving from rotate to shift is below the noise floor of what is required to patch up the bit value bias introduced by multiply.

</comment></comment></comment><comment><USER4> Really good and well written article.

We did the similar trick with SSE. It's a good approach if a kind of non- portable hash can be used. In system I'm working on we introduced two APIs: hash() which maps to "fastest hash we have on this machine" and portable_hash() that's a predefined hash algorithm.

BTW, SSE4 introduced carry-less multiplication which can also be used to boost hashing[1]. However, it's good for large data; in our tests it outperformed other algorithms for inputs larger than 1kB.

[1] [https://github.com/lemire/StronglyUniversalStringHashing](https://github.com/lemire/StronglyUniversalStringHashing)

</comment><comment><USER5> Most proof-of-work cpu miners use this optimization. My own Cuckoo Cycle cpu solver supports 4-way, 8-way, and even 16-way siphash computation, where the latter use multiple sets of avx registers to allow for more interleaving of operations. On recent iCore i7, the 8-way performs best.

The Makefile at [https://github.com/tromp/cuckoo/blob/master/src/Makefile](https://github.com/tromp/cuckoo/blob/master/src/Makefile) has targets cuckoo28stx, cuckoo28stx4, cuckoo28stx8 and cuckoo28stx16 for these different levels of parallellization.

</comment><comment><USER6> I parallelized SHA-512 for a competition - the compression function runs 4 copies in AVX2.

[https://www.nayuki.io/page/lowest-sha512-value-by-brute- forc...](https://www.nayuki.io/page/lowest-sha512-value-by-brute-force)

[https://www.reddit.com/r/programming/comments/1y17al/hashcha...](https://www.reddit.com/r/programming/comments/1y17al/hashchallenge_can_you_find_the_lowest_value/)

</comment><comment><USER7> I'd just like to take a moment to acknowledge the author for their excellent "Don't look under the rug" section. Amazing.. I wish every article had information like that.

</comment><comment><USER8> > and given the way things are going, I wonder if a general purpose CPU using > AVX-512 will actually ever launch

Boy, I really hope you're wrong! Any further thoughts?

</comment><comment><USER9> That's wonderful. For non-cryptographic uses, this could speed-up a lot of things, e.g. even git, replacing the SHA1 with a e.g. a 128 bit hash made from a xxHash64(file_content)) plus the size of the file (64 + 64 bit). In fact, replacing SHA1 with xxHash64 + file size, and replacing zlib with LZ4, could allow git reduce "repack" times to the minimum. BTW, both xxHash and LZ4 are developments from Yann Collet, a very talented programmer (also author of Zstandard -most of the "magic" of both LZ4 an Zstandard comes from the hash function, which is a huge breakthrough-).

<comment><USER10> > For non-cryptographic uses, this could speed-up a lot of things, e.g. even > git

</comment></comment><comment><USER11> AMD XOP used to have a rotate instruction. Sadly it is dead in Ryzen. It seems that AVX-512 will have it though.

<comment><USER4> XOP had a few nice instructions, but was killed. AVX-512F has rotations, but only 32 and 64-bit elements. AVX-512BW will support rotation of bytes and words (8 and 16-bit words).</comment></comment>