Which hashing algorithm is best for uniqueness and speed? - <AUTHOR> http://programmers.stackexchange.com/a/145633 <comment><USER2> A more thorough test suite for hash function is SMHasher, by the same author of MurmurHash. It measures collisions, output uniformity, biased bits, etc... as well as speed, under several input distributions.

[https://code.google.com/p/smhasher/](https://code.google.com/p/smhasher/)

Here is the output for some popular hash functions:

[https://code.google.com/p/smhasher/wiki/SMHasherDump](https://code.google.com/p/smhasher/wiki/SMHasherDump)

<comment><USER3> If someone could run smhasher for the hash functions in the original article (and perhaps also MurmurHash 2/3) and summarize the result for the small key sizes I will give them a cookie.

<comment><USER4> I REALLY think we're all tired of random people giving us yet another cookie to store...

</comment></comment></comment><comment><USER5> It's worth remembering that when selecting things at random from a given pool, there is always a chance of collision. If you think of a hash as taking a value to a "random" result, with different objects going to results uniformly "at random", then you have to expect collisions. This is why the term "perfect hash" is defined so carefully.

For those who are interested, the probability of collision, then size of the hash destination, and the number of objects chosen are related by this formula:

        Choosing from N items, with N large, and wanting a
        probability T of having no collisions, the number of
        items you can choose at random with replacement is:

            k ~ sqrt( -2 N ln(T) )

So

            T ~ exp(-(k^2)/(2N))

Source: [http://www.solipsys.co.uk/new/TheBirthdayParadox.html?HN2013...](http://www.solipsys.co.uk/new/TheBirthdayParadox.html?HN20131014)

<comment><USER6> There's always a chance of a collision, but for a large enough pool relative to the number of objects the chance may be sufficiently smaller than the chance of hardware failure that it can be legitimately ignored. For something where the entire space must in some way be realized (eg. hashtable) this won't be the case, obviously.

<comment><USER7> Most of the time, that isn't true. Hardware failures are (barring quality control issues) independent, hash collisions are not.

For example, if you assume your hash will not have collisions, and design your hash table without any mechanism for coping with one, and it happens to have a clash on "Jones" and "Smith", your address book app will become unreliable for a rather large fraction of your users.

<comment><USER6> But at the scales we're talking about, it _won 't_ happen to have a clash on "Jones" and "Smith". This won't be a hash table - this will be things like a git commit hash, where generating 1/ns is likely to give _a_ collision in 3 years, before which the storage required _just to store hashes_ is in the yottabytes.

</comment></comment></comment></comment><comment><USER8> My research group at CMU uses a lot of hashing these days and we care mostly about performance. We've settled on three as our basics: \- CityHash as a nice, general purpose hash, particularly on newer machines with CRC32c support in hardware; \- SpookyHash for longer strings, particularly with no CRC32 support; \- CRC32c in hardware for a quick & dirty hash with possibly slightly more poor distribution.

It depends a lot on what you're hashing. For strings, both City and Spooky are great. You should benchmark both on your target architecture(s). If you're trying to take, e.g., integers to integers, we've found that CRC32c is a pretty darn good fallback _if_ you need insanely high performance.

If you're running a hash table exposed to _externally_ controlled input, then you also want to consider (whether you need) algorithms that provide decent resistance against "hash flooding". SipHash is the strongest contender there short of throwing your hands in the air and using MD5 with a local secret seed.

<comment><USER9> I think it's worth reiterating your last paragraph. If your hash table is exposed to external input, an effective DoS attack can be mounted by turning your hash table into an expensive linked list.

[http://events.ccc.de/congress/2011/Fahrplan/events/4680.en.h...](http://events.ccc.de/congress/2011/Fahrplan/events/4680.en.html)

</comment></comment><comment><USER10> One issue to be careful of is assuming that you don't need the guarantees a cryptographic hash provides (preimage resistance, extension resistance, etc.) in your application. If an adversary can force hash collisions they may, depending on your uses for the hashing algorithm, be able to cause degredation or denial of service, or perhaps trigger information disclosures if they're very smart. If there's any way in which an attacker might be able to have some control over the dataset you are hashing you should probably be looking at using a cryptographic hash anyway.

<comment><USER11> Is denial of service from an attacker who has your source something you actually consider when building a product?

I can see trying to prevent data breaches and such, but going so far as to be wary of denial of service attacks from people who have your source code might be wasted effort.

<comment><USER12> The prevalence of frameworks means attackers probably have your source, or at least enough of it to be interesting.

This specific denial of service attack hit a lot of systems in 2012, and all you needed to know was which framework they were using (or just try all of them, as the attack has incredible leverage)

It's as simple as crafting a query-string that results in an O(n²) hashtable building, and allows you to burn minutes of server CPU with each request.

<comment><USER11> Coming from the perspective of a startup that is building a product, future risk/estimated damage from DOS attacks is not worth any of today's efforts to prevent. I mean, weren't they all able to fix it _after the fact_ and not suffer much in the end?

These are problems you only run into if you've got a successful product (which happens very rarely) - and solving these problems before you have users might not be a good allocation of engineering resources for startups.

I liken it to making a system that scales _in advance_ to millions of concurrent users when you haven't even got one.

<comment><USER10> The thing is that anywhere in your code where you are doing something which involves hashing data to create a digest for some purpose you are almost certainly attempting to do something a bit _clever_ (I'm going to map the problem from the domain of entities to a domain of proxies for entities that are smaller and easier to deal with. I'll just deal with the special cases...). And if you're making a decision to use Murmur2 in that role, you're probably trying to be _very clever_. Whenever a programmer tries to do something clever, in my experience, there is something they haven't thought about. Often the something they haven't thought about leads to a security vulnerability.

The fact is it's often difficult to tell the difference between a place where the security properties of the digest algorithm you use are irrelevant, and a place where it might be critical. For example, a zip file uses a CRC32 checksum to validate each file; git uses SHA1 to uniquely identify objects; CouchDB shard mapping uses a strategy based on CRC32... if git had used murmur, would that cause problems for it? Is CouchDB being _clever_ by using a fast hash for sharding at the expense of being secure?

Now, quick, make a decision which algorithm to use to generate Etag headers for your web API HTTP response entities: are you going to go with SHA256 or Murmur2? Murmur2 is much faster, and it's certainly a _clever_ solution... but are you sure you didn't create a security hole?

I'm not saying "nobody ever got fired for choosing SHA256," but I am kind of saying that you have to be very sure you know the security model you're operating in before you build on a hash algorithm that doesn't provide strong extension and preimage resistance.

</comment></comment></comment></comment></comment><comment><USER13> >>FNV-1a collisions

>>altarage collides with zinke

>>altarages collides with zinkes

I found this most interesting. I have no idea how FNV-1a is constructed but I'm sure this is an interesting (mathematical?) quirk in an otherwise great algorithm since it had only 4 collisions for more than a couple of hundred thousand inputs.

<comment><USER14> Yeah, that seems to violate the goal of the avalanche effect.

[http://en.wikipedia.org/wiki/Avalanche_effect](http://en.wikipedia.org/wiki/Avalanche_effect)

<comment><USER15> That's a design goal for cryptographic hash functions, which these are not.

<comment><USER6> For something like English words, where you will have a _lot_ of common prefixes and suffixes, it seems a legitimate design goal as well (though it may not be a tradeoff worth making, to be sure).

</comment></comment></comment></comment><comment><USER16> For those that love this topic, xxhash[1] has a SMHasher score of 10[2] and is ~2x as fast as MurMur3 -- really interesting, incredibly fast, high quality hash function.

      [1] https://code.google.com/p/xxhash
      [2] https://code.google.com/p/xxhash/wiki/xxh32

<comment><USER17> I was going to say this as well. It has a Node.js port that I'm using for static file versioning, and the speed is great.

</comment></comment><comment><USER18> It doesn't mention SipHash: [https://131002.net/siphash/](https://131002.net/siphash/)

<comment><USER19> If you have the reputation points, please suggest it to be included in the test on Stack Exchange.

SipHash was developed to replace CityHash, Murmurhash as well as being a fast and secure PRF.

<comment><USER20> I added a comment to suggest it.

<comment><USER19> Thanks!

</comment></comment></comment></comment><comment><USER21> Nice answer. However in practice current Intel processors do CRC32 in hardware. This blows away the rest and has really nice results. So I use portable code that using Intel's slicing-by-8 CRC32 code for generic machines and then automatically uses SSE4.2 on x86's that most people are actually using.

</comment><comment><USER22> Nice analysis.

Since that post was submitted Murmurhash 3 and a number of other notable hash functions have been released. If you liked the linked post you should go read Aggregate Knowledge's series of posts on hash functions for a look at some newer algorithms: [http://blog.aggregateknowledge.com/2012/02/02/choosing-a- goo...](http://blog.aggregateknowledge.com/2012/02/02/choosing-a-good-hash- function-part-3/)

</comment><comment><USER6> The best hash algorithm for uniqueness and speed uses knowledge of the distribution of the incoming values and exploits whatever patterns are available. It's not one-size-fits-all. If I am being fed arbitrary binary blobs, taking the first 4 bytes might be as unique as any other approach (and will clearly be faster than anything else that actually depends on the data). If I am being fed English test, that is going to perform horribly on uniqueness.

If the source of the data might benefit (DoS attack being a common example) from deliberately causing collisions then there are additional constraints and looking at cryptographic hash functions may be warranted.

</comment><comment><USER12> I often see these "plot the output of the RNG on a 2d grid" things. I understand they're useful for demonstrating non-randomness, but if you're searching for patterns, do they reveal anything an FFT wouldn't?

<comment><USER23> isn't fft just a different way of looking at the data?

<comment><USER24> Essentially, yes, that was my understanding, too. I've only used in in signal processing - converting from the time domain to the frequency domain. Not sure how that would apply to this.

<comment><USER12> I was thinking specifically of this:

      When i look at the FNV-1a "number" map, i think i see subtle
      vertical patterns. With Murmur i see no patterns at all.
      What do you think?

Wouldn't that vertical repetition, and the other repeating patterns the other hashes form, show up as a nice spike in the frequency domain? Whereas I would expect uniform hashing to generate a flat, white-noise spectrum.

<comment><USER18> I find the avalanche maps used here easier to grasp: [http://home.comcast.net/~bretm/hash/5.html](http://home.comcast.net/~bretm/hash/5.html)

Here is FNV, quite a lot of reds: [http://home.comcast.net/~bretm/hash/6.html](http://home.comcast.net/~bretm/hash/6.html) And of course all green for SHA-1: [http://home.comcast.net/~bretm/hash/9.html](http://home.comcast.net/~bretm/hash/9.html)

</comment></comment></comment></comment></comment><comment><USER25> This seems to be very of date? My understanding was that when SipHash arrived it knocked out all of these? (Please correct me if I'm wrong.)

This may be an interesting weakness with the StackOverflow system; if there's an answer that used to be very good, it may take a lot to knock it out of the first position when reality changes.

<comment><USER9> I don't know of a performance comparison between SipHash and other secure hashes other than [1], which compares it to MD5, "Python's new randomized hash, with a 128-bit key and 64-bit output", and a couple of SHA's. It is significantly slower than DJB2, though.

[1] [http://bench.cr.yp.to/results-auth.html](http://bench.cr.yp.to/results- auth.html)

</comment></comment><comment><USER26> > Yes. I started writing my test program to see if hash > collisions actually > happen - and are not just a theoretical construct.

This should be no surprise. The birthday paradox, as Knuth points in in TAoCP, makes it clear why collisions happen. Given 23 people in a room, there is a greater that 50% chance that at least two have the same birthday.

Placing only 23 keys into a hash table with 365 entries with a perfectly random hash function will probably result in at least one collision.

<comment><USER5>
        Placing only 23 keys into a hash table with 365 entries with
        a perfectly random hash function will probably result in at
        least one collision.

No, it won't "probably" result in at least one collision, it will result in at least one collision with probability about 1/2\. As quoted elsewhere, placing _k_ objects in a hash table with _N_ entries will result in no collisions with probability about _exp(-(k^2) /(2N))_.

<comment><USER27> Interesting. I was going to complain, as I thought "probably" meant the probability of occurrence was greater than 50%.. That doesn't seem to be the case.

      Probably: almost certainly; as far as one knows or can tell

I would guess waynecochran thought something similar.

<comment><USER5> Curious. I've just taken a straw poll in the office. I asked "if someone said something was probably going to happen, what probability would you give it?"

The answers were:

* 80%

* 50% to 90%

* 90%

* > 80%

* 80%

* 80% to 90%

<comment><USER28> My guess would have been "exactly strictly higher than 50%".

<comment><USER29> While mathematically correct, this does not seem to be how people use with that phrase. "X is probably going to happen" implies there is some evidence in favor of X, while "X is probably not going to happen" implies evidence _against_. Now, in regular speech, people do not feel compelled to force themselves to choose one or the other, but are just as likely to say "I have no idea", implying no evidence in either direction.

From an Information Theory point of view, 50% probability means zero information, and 25%/75% probabilities mean one bit of information against/in- favor-of the event. This sounds roughly consistent with the results of the poll, with people assigning higher probabilities implying higher information density of the evidence.

</comment></comment></comment></comment></comment></comment><comment><USER30> I'm curious as to why the emphasis on speed here? I would think that hashes are mostly for either confirming a payload, or as a security measure against plain text passwords/keys.

I would think that at least in the latter, that less fast, more cpu/memory intensive would be more desirable. Something like SCRYPT may be overkill, especially if you use if for password hashes, under load for a lot of logins, but the comparison seems to fly in the face of that.

This also doesn't really mention random seed/salt values.

<comment><USER31> I'd think it's far more common to use hashes for various data structures (i.e. hash map) than for cryptographic or data integrity purposes.

<comment><USER32> Even for data structures, you generally want to use a cryptographic hash function, to prevent collision and other similar attacks from degrading the performance of a data structure.

For example, if I can sufficiently reverse your hashing process, I can degrade a hash tree in to a linked list for a particular set of data - something your firewalls and other security measures will have virtually no chance of stopping, but which will greatly increase the run time taken to parse (malicious) instructions about that data.

A common use of this is maliciously chosen inserts of new items, to cause the tree structure to become a list building on it.

<comment><USER3> Until recently I don't think any major hash table implementation (e.g. not Java, STL, Python, Ruby, etc) used a cryptographically secure hashing algorithm. They're just too slow. Slow enough that you might as well use a tree-based data structure, which doesn't suffer from hash-collision based attacks.

This is changing a bit now when SipHash is available. Unfortunately SipHash still is a bit too slow and forces an uncomfortable trade-off.

<comment><USER33> I would be really surprised to find that a tree is faster then a fast cryptographic hash function like SIP hash. Cache misses to main memory are worth 10s of instructions and that number is going up in many cases. Hash functions can be pretty efficient at allowing the processor to run multiple instructions in parallel.

If you know the tree will be cached and/or the keys you have to hash are large then sure a tree might be a win, but hashing a small key might only be the same as two or three cache misses. Comparing tree nodes is not instruction free either.

I think it is really a case by case sort of thing, but there is no substitute for measuring (and nailing down the comparison to specific hash functions).

</comment></comment></comment></comment></comment><comment><USER34> If you are hashing fixed-size keys, then tabulation hashing is simple and fast with good theoretical properties:

[http://en.wikipedia.org/wiki/Tabulation_hashing](http://en.wikipedia.org/wiki/Tabulation_hashing)

[http://rcoh.me/pyhash.pdf](http://rcoh.me/pyhash.pdf)

</comment><comment><USER35> It is worth mentioning that the libstdc++, in version 4.6.3 at least, it has implemented the Murmur and FNV hash function for user defined key types.

If you want to use Murmur you use std::_Hash_impl::hash and for FNV you use std::_Fnv_hash_impl::hash. (Include bits/functional_hash.h)

The implementation is at libsupc++/hash_bytes.cc

</comment><comment><USER36> Would the security of such algorithms improve dramatically if say 5 years from now we'd live in a world where everyone has a 64-bit device or machine, and the algorithms can be optimized for 64-bit? Are current algorithms limited by 32-bit a lot or not really?

<comment><USER37> Security of such algorithms now: none.

Security of such algorithms now where everyone has a 64-bit device: none.

Don't confuse hash functions like these and cryptographic hash functions.

</comment></comment><comment><USER38> gperf - [https://www.gnu.org/software/gperf/](https://www.gnu.org/software/gperf/) This is a "perfect" application of a "perfect hash" generator.

</comment><comment><USER39> seeing at least 1 collision the tests makes me wonder why the chance of a bitcoin address collision is almost nil. [https://en.bitcoin.it/wiki/Weaknesses#Generating_tons_of_add...](https://en.bitcoin.it/wiki/Weaknesses#Generating_tons_of_addresses)

<comment> vertex-four Bitcoin uses a cryptographic hash function, SHA256. There are, at present, no known collisions with SHA256.

This question is about non-cryptographic hash functions, as are commonly used in hash tables[1] which implement associative arrays, or sharding databases based on the hash of the record key.

Non-cryptographic hash functions are simply not in the same league as cryptographic ones; cryptographic hash functions are optimized to make it infeasible for an attacker to generate a different message with the same hash, while a non-cryptographic hash function generally simply has to have uniformly random output over its keyspace.

[1] [http://en.wikipedia.org/wiki/Hash_table](http://en.wikipedia.org/wiki/Hash_table)

<comment><USER40> All of which is true and well written. However it brings up the obvious point missed in the otherwise well written article that yes, if I use 32 bit hashes to shard into 2 to the power of 32 databases, some will have more collisions than others, aka the 32 MSB of a 32 bit hash has some "bad looking" variations. But I don't care about non-randomness in the LSBs because I'm infinitely more likely to shard into, perhaps, 2 to the power of 4 database machines.

The original article did not descend into that obvious area of research. I see no particular reason why a hash algo that has the worst randomness shoved into the least significant byte (which I simply don't care about) might be an inherent result of smooshing the best randomness into its most significant nibble, which I do care very much about. Given the likely use case for a sharding hash, a smart hash designer would make sure that most of his effort is put into smooth distribution in MSB and perhaps totally ignore the LSB for a given amount of latency / computation / electrical power. After all, the actual users are more likely to hash based on the first 4 bits than the first 24 bits. Although you'll always run into people who think its funny to pull their shard subset out of the hash using the LSB (why?) or some random byte in the middle (why?)

I think MSB for shard key comes out of the tradition in the really olden days of sharding based on raw unhashed data. Sometimes thats random enough such that the MSB of the data makes an excellent shard key and hashing would just slow things down for a minimal gain, even today.

<comment><USER12> Whoa, it's conventional to use the MSB? I usually just do

      bucket = hash(x) & mask

Should I expect non-cryptographic hashes and RNGs to produce better noise in the MSBs? That would explain "Sun Redefines Randomness"

<comment><USER40> Well, look on the bright side, at least you're not calculating

bucket = hash(x) % buckets

Unless you're stuck supporting non power of two number of shards in which case you need modulus. And shoving more randomness into the LSB is more important than the MSB.

I've thought about it some and I think anyone with a networking background is automatically going to "subnet" their data from MSB down out of pure raw habit. Of course you split top down, just like IP addrs.

I've also seen non-technical popular explanations of sharding which stereotypically use something like phone numbers and start at the most significant digit.

And a stereotypical test question is this does not work well with stuff like customer IDs or transaction IDs because they tend toward sequential-ish, unless you're talking about some kind of data mining thing not daily live transactions.

On the other hand it works well if you shard (sorta) by date if you use the right (usually non-native) format. So if you have dates like 2013-10-14 and shard by YYYYMM somehow, then it could be easy to wipe 201210 from the database because whichever shard its on, its probably not impacting the latency figures for 201310 today. Unless you wanted to speed up the delete by smoothing it across all shards in which case sharding by hash ends up being the smart idea.

Trying to do tricky things can turn into a mess when the tricky thing changes mid project, too.

<comment><USER33> This is actually an important topic for me. I am implementing a sharding distribution based on consistent hashing using MurmurHash3 as hash function.

I am taking the first 4-bytes of the hash function output and using that. I checked and MurmurHash3 mixes the first and last 8 bytes of hash output as a last step, but I am not sure how much differentiation there is in the first 4-bytes.

I guess it is something I should check.

</comment></comment></comment></comment></comment></comment><comment><USER41> >CRC32 collisions

>codding collides with gnu

accident?

</comment><comment><USER42> cityhash - [http://code.google.com/p/cityhash/](http://code.google.com/p/cityhash/)</comment>


See your code in action with a visualization engine - <AUTHOR> http://aesopacademy.com/landing

<comment><USER2> Visualization is neat. What I would like to see, more for debugging purposes than for learning (but I suppose it would be useful for both), is a debugger that makes it very easy to graph watched variables, function call frequencies, etc. Being able to quickly visualize data trends inside a program would be immensely useful.

Maybe something like this already exists. Am I missing out?

ETA: Make sure to click the "sandbox" link on the aesop academy page. There's more visualization and better controls available there. Neat stuff... I would have liked this when I was playing around in my "computer science" class in high school.

<comment><USER3> Sounds like the way electronics engineers simulate their designs. sounds like a great idea.

</comment></comment><comment><USER4> There is a similar one, for python, here:

[http://www.pythontutor.com/visualize.html#mode=edit](http://www.pythontutor.com/visualize.html#mode=edit)

On a side note, does anyone know of a way to "visualize" _possible_ code execution paths? Sort of like a control-flow graph, where one can easily count/see the cyclomatic complexity of a piece of code? A library that does it?

<comment><AUTHOR> Python Tutor is a great inspiration and benefits from a greater amount of detail, though I've found it not to be very accessible for teaching beginners due to its relative complexity. I was fortunate to have the opportunity to see Philip Guo (the creator) speak at Google, though - really inspiring, and clearly very passionate about what he does!

For anyone else interested in visualization who hasn't seen it, Bret Victor's Inventing on Principle is also a wonderful idea of how visualization could benefit programmers: [http://vimeo.com/36579366](http://vimeo.com/36579366)

</comment></comment><comment><USER5> This seems like an excellent way to teach Programming. I tried it out with some sorting problem. Pretty cool.</comment>


Scripting News: Social media bubble pop - <AUTHOR> http://scripting.com/stories/2011/05/15/socialMediaBubblePop.html <comment><USER2> What is this in reference to? Twitter itself?</comment>


How Fastly coded their own routing layer for scaling CDN - <AUTHOR> https://www.fastly.com/blog/building-and-scaling-fastly-network-part-1-fighting-fib <comment><USER2> TL;DR: Fastly needs full routing tables on all CDN nodes in order to determine which is the best transit path to push out content through. In order to save money, they used a programmable Arista switch instead of a traditional router. Their solution is to reflect BGP routes via the switch to the nodes and and fake direct connectivity between the nodes and the transit provides, so that nodes can directly push out content to whichever transit provider they determine is best on a per packet basis.

Please correct me if I'm wrong.

Maybe I'm just obtuse, but I found the blog post confusing about what it really is about and long winded, taking a very long time to come to the point. There is a severe lack of context in the beginning, it would have tremendously benefited from the what and the why of what they are trying to do.

<comment><USER3> I'm also _really_ confused as to why they insisted on taking full tables at their edge if they also built their own route optimization technology. It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP. You don't need full edge routes for that.

You need a single route reflector (or two for redundancy) to tell you what routes are in the table. Then probe the busiest prefixes (as determined by flow data) via each provider and take the best route. Localpref the best route higher in the route reflector and send it to the Linux hosts. No need for a huge FIB on the edge. This is basically what Noction IRP/Internap FCP platforms already do.

This reminds me of the scene in Primer where they ask if they're doing it for a reason or just showing off.

FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.

<comment><USER2> > It seems like it would be much simpler and faster if they just took a > default route from each provider and then did route probing on their top 30% > prefixes in each PoP.

Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet.

> FWIW, the Netflix networks just take a default route from a pair of transit > networks in each PoP. At their scale, they found it better to negotiate > better worldwide transit deals than optimize networking at the PoP level. > Different game than Fastly, but another data point.

Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing.

<comment><USER4> "Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet."

That's not how BGP works. There's no such thing as getting the best path from the first packet in terms of latency. There's nothing in BGP that tell you about the path in terms of latency or performance. The only "best" that BGP gives you is AS path length between you and the destination. It tells you nothing about congestion/packet loss upstream. The only way to get that is out of band probing, like taking latency measurements and pref'ing routes. There is no approach beyond "after the fact"

There is nothing in this article that talks about optimization of routes.

<comment><USER2> > That's not how BGP works.

Of course not. I only stated what Fastly needs, I did not make a statement about how BGP works.

By having full BGP feeds from all transit providers the nodes can use the AS paths as part of the heuristics to determine the best possible path (with the available information) from first packet. Obviously the heuristics will also use after the fact probe information to tune the model.

The point is that the solution proposed by the grandparent is not better than the one described in the article.

> There is nothing in this article that talks about optimization of routes.

No, they talk about route selection.

<comment><USER3> What you're talking about works OK in a single colo or datacenter model, but it doesn't map to the PoP model. It's the PoP model that's driving them to do this. In their PoPs they've got local peers and exchanges that will handle the bulk of their traffic. The leftover stuff is only local and probably minor. They can definitely find that with flow data and optimize across a pair of default routes (basically building their own routing table). It's a pretty common practice when all you deploy is PoPs. Why do you care about a full BGP feed when you're only handling 4-10% of the Internet in a given PoP?

AS path length doesn't tell you anything about a route's performance. It's a hint and only a hint.

<comment><USER2> > What you're talking about works OK in a single colo or datacenter model, but > it doesn't map to the PoP model. It's the PoP model that's driving them to > do this

Could you please explain how the PoP model in this case differs from the colo/DC model?

Outside the US Fastly's PoP cover multiple countries or whole continents.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> Interesting approach. So Fastly is offloading the full routing table to their carriers' router(s). That's because routers that can hold full BGP tables are expensive to purchase and maintain. But to retain some form of control, they're terminating eBGP at the switch and using iBGP to disseminate (inject) the providers' route (next hop).

I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts. So the cost is automatically spread out across their relatively massive edge nodes. A line showing a router plus support costing $100,000 looks bad in expenditures vs showing a server plus integrated routing showing $2500.

I'm curious about how this impacts Varnish considering how table look ups can be bus-expensive during odd route changes/flaps (storms). %sys must go through the roof as a result.

<comment><USER3> I didn't see him in the room, so that doesn't mean he wasn't there, but it sounds like Artur was paying attention to Dave Temkin's talk at NANOG (from Netlix).

Slides: [https://www.nanog.org/sites/default/files/wednesday.general....](https://www.nanog.org/sites/default/files/wednesday.general.temkin.panel.pdf)

Video: [https://www.youtube.com/watch?v=-05xWeYGn4A](https://www.youtube.com/watch?v=-05xWeYGn4A)

Titled: "Help! My Big Expensive Router Is Really Expensive!"

Netflix goes even cheaper/simpler and just uses default routes to a pair of transit providers. It may come as a shock to most of you, but no, Netflix is not 100% in AWS (compute, yes; network, oh hell no).

<comment><USER6> Considering how high are AWS' network transfer prices, I'm not surprised at all.

</comment></comment></comment><comment><USER7> Also note Spotify published their own stuff also on Arista hardware

[https://labs.spotify.com/2016/01/26/sdn-internet-router- part...](https://labs.spotify.com/2016/01/26/sdn-internet-router-part-1/)

Podcast where David Barroso talks about it:

[http://blog.ipspace.net/2015/01/sdn-router-spotify-on- softwa...](http://blog.ipspace.net/2015/01/sdn-router-spotify-on-software- gone-wild.html)

Arista blog post:

[https://eos.arista.com/spotifys-sdn-internet- router/](https://eos.arista.com/spotifys-sdn-internet-router/)

Disclaimer I work at Arista

<comment><USER8> Didn't they move to Google Cloud?

I was very surprised at the time to see their move shortly after reading David's articles.

<comment><USER9> I think they just moved event data processing.

</comment></comment></comment><comment><USER10> Great read, love the "hey what does it need?" approach rather than the "how is this done?" approach. Tut Systems had bought one of the first "hotel internet" companies back in the 90's which used a similar approach by subverting the ARP protocol, when you connected any thing you tried to ARP for it would respond "Yup, that's me! Send me your packets" and you would end up at the "Give us your credit card" signup.

The nice thing is that at this level networking is really simple. And if you can get access to the internals of switches to craft behaviors at that level, it is a pretty good way to go.

<comment><USER11> You mean proxy ARP?

<comment><USER10> Pretty much, but more like proxy ARP on steroids, sort of proxy DNS, proxy ARP, proxy everything.

<comment><USER8> Proxy ARP it's a thing and it is exactly what you've described :)

And if by proxy DNS you mean you'll subvert ARP to reach a DNS server... that's proxy ARP. DNS is a few layers above :)

Definition:

Proxy ARP is the technique in which one host, usually a router, answers ARP requests intended for another machine. By "faking" its identity, the router accepts responsibility for routing packets to the "real" destination.

<comment><USER12> And it is a super dirty network hack. Unfortunately can't put that genie back in the bottle.

</comment></comment></comment></comment></comment><comment><USER13> That's pretty impressive! As a side note, a writeup on BGP security: [https://security.stackexchange.com/questions/56069/what- secu...](https://security.stackexchange.com/questions/56069/what-security- mechanisms-are-used-in-bgp-and-why-do-they-fail)

</comment><comment><USER14> This is really great work. Do you have any plans to open source some (or all) of the code behind Silverton?

</comment><comment><USER15> If you like this, you'll like the ONS youtube channel.[1] In particular, the keynotes by Vahdat are pretty amazing.

[1] [https://www.youtube.com/channel/UCHo2uqQqpmE_Cg5b4qiUpUg](https://www.youtube.com/channel/UCHo2uqQqpmE_Cg5b4qiUpUg)

</comment><comment><USER16> Amazon, Google, etc. etc. all these companies building their own custom network devices, and so much of it coming back to both "It does too much, most of which we don't need" and "We want to do our own thing at that layer".

As James Hamilton noted at the AWS Re:Invent convention, not only is there they overhead and development expense of these unneeded components, just the sheer complexity of the application running is inevitably leading to bugs and unexpected behaviour. By simplifying the device to just do the few things you actually need it to do, you end up more performant, and more reliable.

I wonder if the entrenched network appliance providers will wake up?

</comment><comment><USER17> They'll want to be looking at using MPLS and EPE techniques now that there's support for it on their Arista platforms. This L2 technique is arcane, going to be painful to scale and reapply generally to other areas.

<comment><USER4> Huh? What would MPLS do? They don't operate a backbone. They're operate only at the edge like all CDNs. There's nothing arcane about this, just basic networking. Where is the scalability issue?

</comment></comment><comment><USER18> [https://www.youtube.com/watch?v=TLbzvbfWmfY](https://www.youtube.com/watch?v=TLbzvbfWmfY)

</comment><comment><USER19> I hope Arista pushes other vendors to open up their hardware and provide more APIs

<comment><USER20> That'd be great if they could. We started using Arista 5 or 6 years ago at a small startup I was working for because Arista was the _only_ vendor we could find who was comfortable giving us such an amazing amount of access to the inner workings of their high speed switches. Really enjoyed working with their gear and, when the occasional question came up, their engineers (top notch folks).

</comment></comment><comment><USER21> This is awesomely clever. Not surprised to see from the Fastly team.

</comment><comment><USER22> extremely cool stuff</comment>


Ask HN: Would people pay for this service? - <AUTHOR>

I realized the other day that I have an experience that not many people do: I've worked for five start-ups, founded two, raised money for four.  Of the five, one went public, one was an asset sale/close the door, one was bought out, one is alive and well, and one is new.
I also worked for the top VC firm (Accel Partners) and I've raised money from Greylock, USVP, Mayfield, Accel, ... and I've talked to all the others.
At Accel I sat through presentation after presentation by entrepreneurs. I've seen the good, the bad and the truly weird.
Now imagine the following service.  You are a start-up looking for a series A investment (something >$1m).  Suppose I consulted for you on how you present yourselves to VCs in exchange for a consulting fee in $ and a small equity position (perhaps as a stock option).  The fee would only be payable once series A closed.  No series A, no fee.  No cash up front.
Is this useful?  What help would people like? <comment><USER2> I've been running an HN meetup called Hackers and Founders meetup ( <http://hackersandfounders.com> ) in Silicon Valley for over a year and a half. I've seen hundreds of hackers/founders come and go in the last 20 months. I've also seen a number of people trying to sell consulting services to them.

My best guess is that it would be an 80/20 split. 80% of the members of Hackers and Founders would shy away from what you're offering. If you've been in the Valley for any length of time, and do basic networking, you're going to get to know people with ties to investors. Off the top of my head, I could think of 20 people that I'd go to for advice and introductions if I were looking for investor money right now, and they'd point me in the right direction. Most of my members might talk to you, but they'd be pretty leery of selling a stake in their corporation before they know you. I've found that the early stage founders I hang out with at Hackers and Founders tend to mistrust 2 types of people: 1) Pure business guys looking for technical co founders and 2) Money guys offering to help them out in exchange for a piece of the pie.

I could be wrong, but I'd think that the 20% of founders that would be most eager to take you up on your offer would be the most desperate of the bunch. I think that by pitching yourself primarily as a guy who helps people get through the funding pipeline, you're going to be selecting for the wrong type of founders. I can imagine that founders who aren't in the Valley might be more eager to take you up on your services, as well. There's a much bigger separation between founder and investor in other parts of the country and world, and those people need quite a bit more help in the funding department.

What has really helped the guys at Hackers and Founders are people who are available to be a friend throughout the process. Generally when the founders pursue the funding process, they turn to their friends who helped them early on, and if those friends have enough experience, those friends become board members (and at times get equity).

One of my members in particular has been phenomenal at that. He has a resume similar to yours, and he just hangs out with early stage founders for love of the game. He talks freely, shares his experiences and advice freely, takes people out to coffee all the time, and when founders are looking for fund raising advice, he's the first person they turn to.

What's in it for him, is that he gets to interact with hundreds of entrepreneurs and he can pick and choose the best of them to hook up with and work along side of. But, the teams that he signs up with trust him implicitly, because they've been getting advice from them for a while.

If you have other questions, feel free to ping me. I'd be happy to talk about it.

</comment><comment><USER3> Based on your posts here and your blog, if I look at what I would want to use that expertise for in my company it would be something different. You have an experience in tech and entrepreneurship that is one in a thousand, and that shouldn't be wasted on helping companies doing powerpoints. I would much rather want you as a boardmember/advisor. This would give a company two distinct advantages:

1) An extremely knowledgeable person who has been there before with which to talk on complicated issues. Experience in startups is rare as you point out. When it is coupled with deep tech knowledge it's even rarer.

2) Your name on the businessplan/presentation. Namedropping seems to be a major thing with investors. Unfortunately.

Also, companies that are out raising money often aren't good customers as patio11 points out. Besides there's no repeat business, meaning that you'll have to constantly be on the lookout for new customers. Equity payment is great, but it's a lottery ticket. I don't know your personal situation of course, but if you need bread on the table it's probably going to be hard.

But I'm sure lots of companies would pay in either equity/cash to have you onboard as an advisor. I know I would.

<comment><AUTHOR> I'd be happy to do those things for people if they really think I can help them.

</comment></comment><comment><USER4> I think the people who would most benefit from your help (startups who are likely to fund and want to improve their relative position) are going to see "small equity position" and think "Awww dang, that is going to cause us a million and one legal headaches and is an asterix we will have to revisit in every funding round from here to doomsday -- pass" and the people who are likely to be the clients from heck ("Our deck isn't quite ready yet, but we have a napkin!!") are going to see "FREE!" and jump at it. And then whine at you when they are, predictably, not successful at getting funded, despite "YOU HAVING AN EQUITY STAKE!!1"

Basically, it is the classic adverse selection problem that is eliminated by charging people money.

<comment><AUTHOR> I'm not concerned about your second point because my first step would be to walk away from people who are not likely to get funded at all.

The first one is interesting, my guess is that this could be handle just as a normal advisor position where a stock option is given. I've seen this done lots of times with external advisory boards and it doesn't normally cause trouble.

<comment><USER4> On reflection: _what am I doing_ shooting my mouth off on this. I know so little about stock options in the US I could probably misspell Series A. If your impression is that advisors getting them is normal practice over there, I trust your expertise.

<comment><USER5> No, you're not wrong. Exceptional equity grants are a pain in the ass, unless you've already set up the legal infrastructure to do them.

</comment></comment></comment></comment><comment><USER6> In Silicon Valley we'd politely refer to you as a Lamprey and at least at my events you'd be banned from attending.

I have two bits of advise that I tell all the startups I work with:

\- Never give equity for one-off work

\- Never work with fundraising assistants

Your background just isn't that unique here in the valley.

The Angels here give you money and will gladly help you with your pitch and intros when the time comes to do series A.

I'd suggest focusing on long term value. If you're valuable to a startup over the long haul (2+ years) then you can justify being an advisor and getting a .1-.25 percent grant. Throw in the pitch consulting for free. It's the least interesting part of building a company.

</comment><comment><USER7> I'm not sure about your monetization strategy, but I'm positive there would be customers for this.

In 1999 I spent a lot of time helping out some investors I knew by providing technical expertise after they'd gotten burnt by people proposing things that had a lot of ocean boiling numbers and hand wavy technologies. In doing that, I saw a lot of what you saw with regards to terrible and weird presentations, but I also saw some groups that looked like they had a ton of promise but were _terrible_ and communicating the point.

That being said, you may want to consider also providing services to some of the smaller funds out there as a bullshit detector. I think by and large investors are MUCH more technologically savvy these days, but I still get the feeling that there is a fair amount of money outside of SF/NYC/Boston/London that lacks confidence for some of the more interesting technology companies.

</comment><comment><USER8> This is making me wonder, could this work from the other end? Would a VC be willing to have you as a head hunter and pay you in equity for successful referrals?

It would still be your job to coach startups towards successful presentations but the money is coming from the VC end and not the entrepreneur.

</comment><comment><USER9> I've worked with someone offering a similar service in the UK. We didn't continue the engagement as the business was at the wrong stage of development for his involvement, but if we were in the right stage then I think his help would have been very valuable.

Maybe it'll be worth talking to him - <http://www.tecmentor.co.uk/> .

</comment><comment><USER10> I wouldn't do it, because living in the bay area, it's not too difficult to find people that have gone through the fundraising process and ask them for advice.

That being said there are a lot of desperate people out there that are willing to pay for this kind of information. You'll probably get the same kind of people who are willing to pay lots of money to pitch to investors.

<comment><USER11> From what I know about the startup culture in the UK, I think it could work well there.

</comment></comment><comment><USER12> Aren't there things like this already? I think more along the lines of consulting fees rather than equity, though, which makes them seem dubious to me. Equity for results seems fair to me.

Somehow though it doesn't seem very innovative... maybe there's something else that can be done in that space that's more creative (ala YC and its new model).

<comment><AUTHOR> It's not clear to me that innovation is important. What is important is stressing the things that matter to VCs in the presentation you are giving to them. These are often not the things that actually matter to the entrepreneur.

For example, technical people tend to get really excited about telling the VCs about the technical aspects of their idea/product, VCs often get more excited when you tell them why the team of people you have assembled are the best to solve the problem you are addressing.

<comment><USER12> > It's not clear to me that innovation is important.

May be - you surely know the space better than I do.

If you look up 'venture capital coaching' on Google, this is on the first page:

<http://myfirstround.com/>

Other sites are quite similar and look just as dubious. Maybe the people behind them actually know what they're doing, but to me it just looks like a market with a lot of ... "cruft" in it, which might lead to it being something of a market for lemons, with people thinking it wisest to steer clear of the whole thing.

By the way, as someone who did a lot of Tcl for a while, I'd be curious if you ever post any 'war story' type stuff from Electric Cloud.

<comment><AUTHOR> I worked with John Ousterhout at Scriptics/Ajuba and Electric Cloud. What sort of war story would you like to hear?

<comment><USER13> Any

</comment></comment></comment></comment></comment><comment><USER14> It sounds to me like a basic consulting service, albeit one in which you'd accept a small equity in place of part of the consulting fee.

So, I guess I'd say, yes, people do pay for consulting. The question is, what type of people/startups will you attract with the equity offer.

</comment><comment><USER15> Do you have an extraordinary ability to take a company on 1 trajectory and steepen and accelerate it upward?

Can you spot a winner, or spot something you can make a winner?

Go and approach some companies and people that appeal to You.

</comment><comment><USER16> I know I would. Can we talk? <http://beepl.com>

<comment><AUTHOR> Send me an email.

</comment></comment><comment><USER11> Do you do biotech?

<comment><AUTHOR> I'd be happy to help, but my real experience has been in software and hardware and although I think I could help I must admit to not being an expert in that area.</comment></comment>


Ask HN: Who do you respect, and why? - <AUTHOR>

People here seem to diverge from mainstream in how we value achievements -- in how we define what is truly 'prestigious' and what is merely faking / pathetic.
Google lets us make snap judgments on people dozens of times a day. When you google someone, what grabs your attention most? What do you consider to be prestigious, or worthy of respect? ...examples?
And the converse: What do you see people emphasizing about themselves over and over again, which actually signal they "just don't get it"? (What do people do in their online/public lives that you think are really stupid?) <comment><USER2> I really respect people who know their own shortcomings, and work hard to overcome them. In an effort to live up to my own ideal, I acknowledge that I often fail in this regard, and often catch myself dismissing something I don't immediately understand.

The converse of that is that it's always sad to me to meet someone who can't admit what they don't know, or can't acknowledge the achievements of others. All of the people I've known that I consider brilliant have also had an amazing amount of humility about what they didn't know, and have been fast to praise the achievements of others. It's a goal I strive for, but again, I know I often fall short.

<comment><AUTHOR> _"All of the people I've known that I consider brilliant have also had an amazing amount of humility about what they didn't know, and have been fast to praise the achievements of others."_

I agree. I think this has something to do with the fact that a "learning attitude" is actually _necessary_ to become brilliant in a field. If you haven't jigged your brain up to be really open and accepting to new ideas, you become too rigid quickly fall behind. By admitting what you don't know & praising achievements of others, you're not just doing the people around you a favour by being pleasant, you're also setting yourself up to become smarter.

I like how with this particular quality, "social" considerations (ie. being liked) overlap strongly with "mental/learning" considerations (ie. you become smarter). You win both ways.

I've definitely erred on this one too -- misjudged situations and thought they called for too much forcefulness/rigidity. Leadership and openness aren't mutually exclusive, but sometimes people make them out to be (eg/ "a good leader doesn't change his mind"). Probably because if you are in politics, you frequently have to abandon your intellectual ideals for pragmatic results.

(This is the reason why _Meditations_ by Marcus Aurelius is one of my favorite books. He rose to the greatest position of power in the world of his day, without losing his ability to be thoughtful and aware, at least with himself.)

</comment></comment><comment><USER3> I respect only one thing: competence.

In fact, that breaks down into 3 things: competence, knowledge, and experience, but only because the last two are often indicators of the first, so it would be unwise to dismiss them.

Achievements are a symptom of competence, so I do respect people who have achieved what I consider great things.

However, even where someone is knowledgeable, experienced, and having apparently achieved great things, if they display something which indicates a severe lack of competence, my respect for them will automatically decrease - not to zero, of course, but still, it can take a severe hit.

<comment><USER4> The irony is that people who have competence, knowledge, and experience usually got that way by taking risks and pushing themselves out of their comfort zone. And if you observe them taking a risk that doesn't pan out, it looks like incompetence.

Everyone was a novice once, and even experts are novices when they enter a domain outside of their expertise.

<comment><USER5> >And if you observe them taking a risk that doesn't pan out, it looks like incompetence.

Maybe... but everyone makes mistakes. There has to be a difference between making mistakes, and making the same mistakes over and over again. I definitely think part of competence is how someone handles failures.

<comment><USER4> You don't get a chance to observe most people over and over again - that's usually limited to friends and immediate coworkers (team members). Not coincidentally, those are the groups you should tap for startup cofounders.

The original question seemed to be about celebrity personalities and casual acquaintances - "when you Google someone..." etc.

<comment><USER3> We're not talking about "like" or even "can bear the company of" here, we're talking about respect. I'm friends with many people who I don't respect in a work sense.

Perhaps my mistake was not to append "in a work sense" to the word "respect".

Also, part of being competent at evaluating other people's competence is not to base your evaluation on a single sample.

</comment></comment></comment></comment></comment><comment><USER6> I think that what I respect is anyone who is unpretentiously humble. People who just do their thing, do it the best way they can, and do not constantly introspect about how good they are, or how they are in a better class that other people.

The difference is that if a humble person who has a lot of experience in a field talks to someone who is new to the field, once he realises that the knowledge of the person does not match his, and it would be impossible to have an interesting conversation, he just switches the topic to something else. But the non-humble person will insist on trying to teach the other person, or tell the other person how things should be done, even when his opinion is not solicited.

<comment><USER7> Isn't that a harmful effect of humbleness? Less teaching happens!

I can tell you, most of what my parents and teachers and mentors and peers and students have taught me was completely unsolicited. I didn't know they had something to teach me. But I sure am glad to have their wisdom with me now. Even in the dim light of memory, I've had things people taught me decades ago finally flower to utility in recent years.

<comment><USER6> That's an interesting point you bring up. There is nothing wrong with people giving you advice, but there is a fine line between people telling stuff because they want to show how much they know, and peope telling you stuff because they care for you and want you to progress in life.

<comment><USER7> I don't think there's a line between them, really. There's a gradient. People have egos, but what they do is not solely for self-aggrandizement, nor solely for altruistic motives.

</comment></comment></comment></comment><comment><USER8> > What do you see people emphasizing about themselves over and over again, > which actually signal they "just don't get it"?

Hard to answer, "it" has many values...

That the have _the_ or all the answers. That they _know_ and they have nothing to learn on a given topic.

Hubris, ego.

Also anyone who worships "personalities" like PG, joel, Jobs, Greenspan, Rand, etc.

In other words I care little about prestige, can't stand celebrity and value achievements lightly(which are often little more than being at right place at right time and not totally sucking). A person's character is vastly more important.

Character is something you can only discover up close. So, I personally know the people I respect and you have no reason to have ever heard of them.

</comment><comment><USER9> I respect people who, on at least one occasion, have been right when I'm wrong and think I'm right. The only way to really get my attention is to defeat me. Other qualities like kindness and honesty earn my regard and I'll do things to help that person, but that's not the same as hesitating to disagree with them.

Regardless of what this reveals about my personality, it's true.

<comment><AUTHOR> _"Regardless of what this reveals about my personality, it's true."_

Interesting. One thing I've seen in my personality -- something I didn't consciously choose to do, but I've observed enough times to know it's a part of me -- is that I am attracted to people who others typically view as intimidating. The more "intimidating" the person is -- the more attention in the room they command -- the more I seem to expect that I can learn something from them, and so I do the opposite of what most people do and address them directly and confidently.

</comment></comment><comment><USER10> I try to respect everyone, because I've found my own happiness seems to vary directly in proportion with how well I do this. In both directions.

I think you're also asking about who you admire, and what achievements you admire. Great questions, need some thought.

<comment><AUTHOR> Hmm. Do you try and respect everyone purely for personal happiness reasons, or do you really believe that everybody should be respected, no matter what kind of person they are?

<comment><USER10> It's not a "should", but a belief that everyone is the same, in some sense; an identification. By happiness, I mean an experience of freedom and clarity that tells me this is the right way: that to think less of another human being is to think less of this one.

 _No man is an island, entire of itself; every man is a piece of the continent, a part of the main. If a clod be washed away by the sea, Europe is the less, as well as if promontory were, as well as if a manor of thy friend's or of thine own were. Any man's death diminishes me, because I am involved in mankind; and therefore never send to know for whom the bell tolls; it tolls for thee._

<comment><AUTHOR> That's an interesting perspective.

So... if I think along the lines of "some _ideas and actions_ are much better than others" instead of "some _people_ are much better than others" ... I can retain my ability to differentiate from good/bad or better/worse, without heaping judgment on human beings themselves... I can retain the ability to empathize with others, even when how they act disagrees violently with my personal values? Makes quite a lot of sense to me.

It seems that we have our natural way of thinking working against us on this one... from _Fundamental attribution error_ on Wikipedia: _"People have an unjustified tendency to assume that a person's actions depend on what "kind" of person that person is rather than on the social and environmental forces influencing the person"_

</comment></comment></comment></comment><comment><USER11> Like everything else that we feel, I sense that we don't have a lot of control over who we feel great respect for... we only have the opportunity to justify it in hindsight.

I think that the people that I respect the most are people that I feel are honest and trustworthy. I have no patience for those who play politics or talk out of both sides of their mouths.

You could be the most talented coder or illustrator or kung-fu master or whatever fits the bill for the project that I'm working on... and if you're not honest and trustworthy, I really can't build any kind of working relationship with you.

</comment><comment><USER12> Well-roundedness. Programmers who can write, designers who can code, and so on.

</comment><comment><USER13> I respect people who can silence the the inner voice that fears embarrassment --people who can "dance like no one is watching." I've been working on this personally. I'm convinced it is one of life's great keys to success.

</comment><comment><USER7> More than anything, I respect awareness. I respect people who are self-aware, and cognizant of the world around them. I respect people who appreciate themselves and other people and the universe around them as the mystery they are, yet who still try to make inroads of understanding.

Compared to awareness, creativity, determination, and integrity, awareness stands alone.

<comment><AUTHOR> > _people who appreciate themselves and other people and the universe around > them as the mystery they are, yet who still try to make inroads of > understanding_

I like this sentence. Accepting and appreciating mystery seems to be a real key to unlocking openness and understanding... maybe because reason, as powerful as it is, often fails to capture all of what is meaningful and valuable in life? So many people seem to put 100% of their faith in a form of "reason" that is demonstrably incomplete... they think they're being rational by leaving out mystery, but I think it's the opposite.

I've learned that if I hold the belief that "reason is my only god, and I possess it entirely" I miss out on a huge chunk of what I actually care about. But when I accept that there's a vast swath of important stuff (relating to people and relationships and meaning-of-life in particular) that I currently have no way of adequately describing my thoughts on, or fully understanding, but that I want to experience and discover -- then I seem to be able to live in a much healthier way.

</comment></comment><comment> daniel-cussen The guy who stood up to those tanks in Tiananmen Square.

<comment><AUTHOR> Hmm. How do you draw a line between respect for standing up for morals vs. pragmatic action that drives results?

I'm not criticizing the Tiananmen Square students -- I just spent a year in Beijing and I sympathize with their plight as much as you can imagine. But sometimes I think people are too quick to justify any action so long as it is done for honourable/high purposes, even to the point of setting themselves up to lose. For awhile now I've been wrestling with this balance between standing up for rigid moral codes vs. pragmatic getting-things-done. If you make something big and important happen, almost any means can be justified in retrospect. And if you do something with a high moral purpose, almost any pragmatic failure can also be justified in retrospect. What's more acceptable: justifying poor performance as 'honourable loss', or justifying questionable means with 'I made something important happen'?

<comment> daniel-cussen He got the world's attention.

</comment></comment></comment><comment><USER14> I respect people who go above the call of duty. Sure you may be great at X, but if you only do whats assigned to you, it shows me that its just a job for you and not something you truly enjoy.

</comment><comment><USER15> Collectively, HN posters and commenters seem to 90% get it. Unbelievable for a website like this.

Reddit runs at about 50%

Metafilter runs at about 40%

Digg at 5%

<comment><USER16> Slashdot is into negative figures.

</comment></comment><comment><USER17> I think the people that I most respect are the people that inspire me to be better. So in no particular order:

Steve "Woz" Wozniak, Sir James Dyson, Yuri Gagarin & Fred Hollows are the ones that I could name off the top of my head.

</comment><comment><USER18> People who are diligent, intelligent and most importantly, compassionate.

</comment><comment><USER19> I respect those that have found a job they truly enjoy doing.

</comment><comment><USER20> I respect demonstrated performance.

Is is done? Does it work? Does it bring value to others?

Anything less is just posing.</comment>


Meet Nao, the first robot to learn emotions - <AUTHOR> http://news.techworld.com/personal-tech/3235829/meet-nao-the-first-robot-to-learn-emotions/?cmpid=sbredditrplant

<comment><USER2> At last, a loyal minion who will fear and respect me!!!! Mwuhahaha!</comment>


Alternative to OFX for your money transfers - <AUTHOR> https://www.remitr.com/blog/alternative-to-ofx-for-international-money-transfers/ <comment><USER2> currencyfair.com has been going for a long time and is also very good imo</comment>


Databricks lays production data pipelines - <AUTHOR> http://www.cio.com/article/3189806/analytics/databricks-lays-production-data-pipelines.html <comment><USER2> As much as Spark is great, the linked article seems to be buzzword laden marketspeak

<comment><USER3> PR piece

<comment><AUTHOR> I mean, it's CIO mag, so it's written for the audience. This is the first announcement I've seen for this topic, so maybe they'll follow-up with some engineering blog posts in the next few weeks.</comment></comment></comment>


HTML5 Weekly: A Free, Weekly HTML5 and Browser Tech Email Newsletter - <AUTHOR> http://html5weekly.com/ <comment><USER2> Thanks for posting this. Despite not taking off here, it did very well elsewhere and there are already just under 3000 subscribers before issue 1 has even gone out! :-)</comment>


Light Lane concept would protect cyclists, bring Tron to life - <AUTHOR> http://www.engadget.com/2009/01/16/light-lane-concept-would-protect-cyclists-bring-tron-to-life/

<comment><USER2> This is a repeat of

<http://news.ycombinator.com/item?id=437321>

where it already has many comments.</comment>


How The Bootstrapped Clicky Cranks Out Profits While Competing With Google - <AUTHOR> http://mixergy.com/noah-merritt-clicky-interview/

<comment><AUTHOR> Clicky <http://clicky.com> is an absolutely gem of a service (much better than Google Analytics - GA rocks on segmentation). And its interesting to know that it is run by just 2 people and they are doing 500K per year.</comment>


France rules out 'total lockdown' in case of coronavirus surge - <AUTHOR> https://news.yahoo.com/france-rules-total-lockdown-case-covid-surge-001848228.html <comment><AUTHOR> From the article:

>"My aim is to prepare France for a possible second wave while preserving our daily life, our economic and social life," new Prime Minister Jean Castex said in an interview on RTL television. "But we're not going to impose a lockdown like the one we did last March, because we've learned... that the economic and human consequences from a total lockdown are disastrous," he said.</comment>


Top DNC Official Wanted to Use Bernie Sanders's Religious Beliefs Against Him - <AUTHOR> https://theintercept.com/2016/07/22/new-leak-top-dnc-official-wanted-to-use-bernie-sanderss-religious-beliefs-against-him/ <comment><USER2> I think the best thing about these leaks is that it reveals how far divorced the reality is from the ideal of politics. No one had much optimism about American politics before this, I'm sure, but there was at least some hope that when one person yelled, "Marriage is one man, one woman!" and another yelled, "You're a closed-minded bigot!" that both of them, despite hating each other and engaging in a completely dysfunctional dialogue, actually believed what they were saying. This evidence lays bare what many suspected--not only is the whole thing a show, it's a show dictated by business and a bunch of people trying to figure out the best way to trick the American people into giving them power. I think even without Emailgate, Bernie and Trump would still have been popular, because even if they can't describe it exactly, people are starting to get a sense of the game behind the curtain. Now we see it laid bare.</comment>


Microsoft security chief: IE is not a browser, so stop using it as your default - <AUTHOR> https://www.zdnet.com/article/microsoft-security-chief-ie-is-not-a-browser-so-stop-using-it-as-your-default/ <comment><USER2> Then Microsoft should put its money where its mouth is. Every time IE launches, the default page should be a warning:

Internet Explorer is no longer suitable for general web browsing. It should only be used for legacy enterprise web portals that haven't been updated to modern web standards.</comment>


The Dark Silicon Problem and What It Means for CPU Designers (2013) - <AUTHOR> http://www.informit.com/articles/article.aspx?p=2142913 <comment><USER2> From the article "The heat generation per unit area of an integrated circuit passed the surface of a 100-watt light bulb in the mid 1990s, and now is somewhere between the inside of a nuclear reactor and the surface of a star. "

I can't tell if this is hyperbole or not, it amazes me but no amount of googling is coming up with a useful answer. Is anyone able to confirm or deny it for me?

<comment><USER3> After a quick napkin calculation+google it seems the sun has around 20kW of power per square centimeter. So not entirely unfeasible that a cooler star, or a nuclear reactor is closer to the typical 10-100W/cm2 of a modern cpu/gpu. Still some orders of magnitude off from our closest star. (Hope calculation is correct)

<comment><USER4> The temperature of the sun at its surface is ~5772 Kelvin. To get power per unit area, use Stefan-Boltzman: σ * T^4, σ * (5772 K)^4 ≈ 6294 W/cm^2. Dividing the sun's luminosity (power) by its surface area will also give a similar value.

A 815 mm^2, 250 W GPU will be 250 W / 8.15 cm^2 ≈ 31 W / cm^2.

<comment><USER5> Doesn't that only include radiated heat? If I touched it, it would be hotter.

<comment><USER6> It wouldn't be hotter (the temperature wouldn't change), but it would transfer more heat to you.

</comment></comment></comment></comment></comment><comment><USER7> The author states "For every watt of power the CPU consumes, it must dissipate a watt of heat." It was always my understanding that that in electrical devices, (with the exception of heaters) the amount of heat produced was inversely proportional to the efficiency of the device. So is it really true that all energy provided to the CPU or SOC is dissapated as heat?

<comment><USER8> The author's sentence is essentially a truism, because fiddling with information _as such_ and in theory doesn't use up any energy. As practically implemented in current CPU electronics, though, information needs to be communicated from point A to point B as a change in voltage. To convey that voltage change means having to move some amount of electrical charge into or out of the tiny capacitor that is a transistor's gate, via the non-perfect conductor that is the doped-silicon trace between the two points. Resistance saps part of the energy moving those electrons around and converts it to heat.

<comment><USER9> > fiddling with information _as such_ and in theory doesn't use up any energy.

that's not true at all. Any time you destroy information (for example an and gate can destroy information) you use energy.

<comment><USER8> Chances are you know more about this than I do, and I'd be interested to learn more.

If you're willing to explain, I wonder how an AND gate destroys information. Certainly the output of a 2-input AND gate carries less information than both inputs together. But unless the gate is destroying the input signals, which it isn't, I don't see how infomation is being destroyed.

<comment><USER9> [https://en.wikipedia.org/wiki/Landauer%27s_principle](https://en.wikipedia.org/wiki/Landauer%27s_principle)

</comment></comment></comment></comment></comment><comment><USER10> Why don't they start making CPUs 3 Dimensional like a cube with 6 "processors" each with multiple cores as its "sides" with the pins on the opposite sides of the cube wall. Seems to me more internal volume might allow for more cleaver head distribution channels

<comment><USER11> 1\. Heat dissipation is now ~n x m times worse where n is your transistor layer count. And where m is the increased thermal resistance*

2\. Power delivery is now ~n times worse where n is your transistor layer count. *

3\. Connections between chips are very slow, power hungry and expensive. Fabrication of "monolithic" 3D is temperature wise painful and usually results in crummier transistors.

With that being said, innovative 3D integration methods in specific applications can help a lot. Shameless plug: we at Vathys do this for deep learning chips.

* to a first order of approximation

<comment><USER12> The heat can be tackled in part by pumping water through holes in the CPU. I believe it was IBM that came up with this. Can't tell if it's feasible or not.

<comment><USER11> Yes but microfluidics limits how thin each die can be.

<comment><USER13> What about stacked heat vias ?

<comment><USER11> What are those? Do you mean something like thermal "dummy" vias?

<comment><USER13> yeah, something imprinted in all layers so you could evacuate heat

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> I had a couple of question about this bit of history mentioned in the article. I'm hoping someone could shed some light on this:

>"You can emulate floating-point arithmetic by using integer instructions—but taking 10–100 times as long."

Exactly how is/was floating point arithmetic emulated using only integers?

Why is that range given an order of magnitude? Is this dependent on the precision I'm guessing?

<comment><USER15> We had floating point when programming in BASIC on old 6502 8bit computers. There were software routines for doing the math. You know, multiply the mantissa, add the exponents... If someone gave you pointers to a couple 4-byte chunks of data and told you to write code to do floating point multiply on the contents using only C-char variables, what would you write? That's why it's 100 times slower than a nice modern fmul. It wasn't quite that bad, they could use the carry flag which isn't available in C.

<comment><USER16> Yep, here's the old Woz/Roy Rankin 6502 code for log, exp, conversions, and basic math: [http://www.6502.org/source/floats/wozfp1.txt](http://www.6502.org/source/floats/wozfp1.txt)

<comment><USER14> Wha a wonderful bit of history your link is. Thanks for sharing.

</comment></comment></comment></comment><comment><USER14> I had a question - the author states:

>"The most obvious is the instruction decoder, which is near the start of the pipeline, and is responsible (in the loosest possible terms) for passing the inputs to each of the execution units."

Why would it "in the loosest possible terms"? Isn't this "precisely" the job of the decoder?

<comment><USER17> IIRC in many chips the decoder stage translates native instructions into micro-ops, which are RISC-like and the main food for execution units. The translation is not necessarily a simple one (one native instruction is often more than one micro-op, and it's possible to collapse multiple native instructions -- especially stuff like prefixes -- into one or more micro-ops).

<comment><USER14> Ah OK that make sense. This is likely what the authors means here by "loosely." Cheers.

</comment></comment></comment><comment><USER18> We just need a clueless CIO to turn up and ask if 'Cloud' will solve the problem :D

<comment><USER19> I love the cynicism :-)

<comment><USER18> hehe - not everyone got it from the downvotes I got!</comment></comment></comment>


U.S. Will Require Drones to Be Registered - <AUTHOR> http://www.nbcnews.com/news/us-news/u-s-will-require-drones-be-registered-n446266 <comment><USER2> So my little 4" wide quad needs to be registered? What a joke.

I don't see why registration is necessary if they have laws in place around airspace and make the consequences of violating those very clear. Make a few examples and the problem will go away.

Registering doesn't solve anything as people will "illegally" build their own if they want to do anything malicious and this won't prevent that. It just adds another nuisance and more tracking.

</comment><comment><USER3> What about RC model planes? Do they have to be registered too now?</comment>


How Oulu became a thriving startup ecosystem in 3 years - <AUTHOR> http://www.arcticstartup.com/2014/03/13/how-oulu-became-a-thriving-startup-ecosystem-in-3-years

<comment><USER2> Check out how Oulu startup scene was built from the scratch to a globally renowned startup hub, attracting everyone from Wall Street Journal and Techcrunch to Samsung innovation scouts and StartUp Sauna warmups and Midnight Pitch Fest to Polar Bear Pitching.</comment>


First bug of 2011: iOS alarms not going off? - <AUTHOR> http://www.macnn.com/articles/10/12/31/reports.of.alarm.issues.continue.to.plague.ios.4/

<comment><USER2> This cost me a couple hours of work this morning, and put me in a generally bad mood. I need to wake up and be alert (driving 12 hours in inclement weather) at 5 tomorrow, and sleeping in a few hours messes that up royally.

These are the kind of bugs that screw with people's trust, along with messed up calendars and call-dropping. This is _not_ okay, and should not have happened.

<comment><USER3> But it does happen.

More complexity == likely more bugs.

</comment></comment><comment><USER4> "The workaround for this new bug is to set all standalone alarms to recurring after midnight. Tests done by 9to5Mac indicate that the bug will correct itself on January 3rd, but until then readers are advised not to solely rely on alarms set on devices using iOS 4.x."

What, "..the bug will correct itself.."? Color me confused but how do you end up with a "self healing 48 hour bug"? Anyone able to provide some clarity?

<comment><USER5> The Zune had a similar self-healing bug at the end of 2008. There was a fencepost error that resolved itself once the internal calendar code no longer thought it was the 366th day of the year. So, basically, a one day bug.

[http://www.zuneboards.com/forums/zune-news/38143-cause- zune-...](http://www.zuneboards.com/forums/zune-news/38143-cause- zune-30-leapyear-problem-isolated.html)

Obviously this isn't the same bug, but it's fun to see the broken code and speculate about similar sorts logic errors that could lead to this working like normal in a couple days.

</comment></comment><comment><USER6> It's really surprising to me that there is another alarm bug. This isn't exactly a tricky subject and should be easy testable.

<comment><USER7> If date and timezone related code seems simple/easy to you then you're using libraries! It's messy and easy to mess up!

<comment><USER6> And Apple isn't using libraries? Also, the time/date has been correct, just the alarm app failing to activate on time.

</comment></comment></comment><comment><USER8> It doesn't seem like it is just iOS. I noticed an issue in my conky display this morning.

You should be able to see the bug on a *nix box with:

> date '+%m-%d-%G' # shows up as 01-01-2010

where as the following works fine.

>date '+%m-%d-%y' # 01-01-2011

I guess it's not really a bug since %G is the year of ISO week number, but it would be easy for something like this to go unnoticed until the invalid date pops up.

</comment><comment><USER9> Android: We are all ready to win, just as we are born knowing only life. It is defeat that you must learn to prepare for.

iPhone: I don't waste my time with it. When it comes, I won't even notice.

Android: Oh? How so?

iPhone: I'll be too busy lookin' goood.

</comment><comment><USER10> Has anything like this happened on Android? I'm an iPhone user, but there are some really facepalm-stupid bugs on iOS that takes away from the platform's image as something polished and perfected.

<comment><USER11> Yep. Android 2.0 (I think) has a bug where alarms will be delayed while the phone's cpu is in sleep mode. This can mean they go off up to an hour late.</comment></comment>


Ask HN: Should I accept a job at a start up or a big company? - <AUTHOR> I have job offers for some interesting start ups and also some big companies. I currently work for a big company and sometimes get annoyed by the politics but I enjoy the security and benefits. The idea of working for a start up sounds cool to me in that I will get a lot of say over the tech stack,will learn a wide bunch of skills and there is probably a lot more room for growth. However, I am very aware a lot of start ups fail and I don't feel like I am strong enough to be their sole backend developer.
For others who have made the move from big company to start up or vice versa, what is your experience and what advice do you have for me?
Thanks! <comment><USER2> I began my career at startups (my office was the dining room!) and am now working for a Fortune 200 company. Startups are great at the beginning of your career because you play so many roles: developer, architect, product line manager, marketing manager, etc. The problem is this period in a startup's life is short-lived: traditional roles are quickly established and the defending of turf begins. So you're not going to avoid politics for long. On the upside by being small you can move very fast, that is until you're successful. Success tends to be an albatross - management becomes reticent to change things.

Big companies have some advantages people don't often think about. They have the financial wherewithal to invest in new technologies and methodologies. They can also afford to send you to conferences and interact with your colleagues. Yes they move slower, but success doesn't become an albatross - they keep moving.

It's the classic tortoise and the hare story.

Another thing you may want to consider is advancement. You tend to advance faster at startups and can attain more responsibility. That allows you to transfer in to a higher position in a big company later down the line. The fastest way to advance at a big company is to go to a smaller company, advance, and come back. Depending on your career goals you may or may not be interested in this aspect.</comment>


Data Masking in MySQL - <AUTHOR> https://mysqlserverteam.com/data-masking-in-mysql/ <comment><USER2> Interesting idea. I wonder if they would have that feature rolled into the community edition?</comment>


Test your IPv6 Connectivity - <AUTHOR> http://test-ipv6.com/

<comment><USER2> Failed! But then I knew that, I'm on comcast and I believe they've only recently rolled out IPv6 support in a single area (according to [http://www.networkworld.com/news/2011/020111-comcast- ipv6.ht...](http://www.networkworld.com/news/2011/020111-comcast-ipv6.html)). It's supposedly the first dual-stack DOCSIS system in the states, and it's in Colorado. I suppose it'll be awhile before it makes its appearance out here in the Bay Area.

That said, I'm looking forward to having access to my machines at home remotely without any special setup. Very tired of jumping through hoops for such a simple use case.

<comment><USER3> _I'm looking forward to having access to my machines at home remotely without any special setup._

If that's what you want, 6to4 (e.g. AirPort Extreme router with firewall disabled) can provide that today.

</comment></comment><comment><USER4> 7/10 for both, Teredo is easy to set up. I can help anyone who has questions with it.

</comment><comment><AUTHOR> My results: 7/10 for IPv4, 0/10 for IPv6</comment>


Several rejections from VCs? Depressing - <AUTHOR> Sorry for my English, English is not even my 3rd language. I pitched 3 startup ideas with simple prototypes to 3 different acceleration programs around my area. It was interesting for 21 years old at first exchanging thoughts with investors and trying to build a solid team and make a product until you get a cold rejection "Your application was not shortlisted"
apparently, the same 3 ideas I was working on had enough dedication and development somewhere in the world and went relatively big, all my ideas and pitch decks are all documented, I felt bad for making it first I'm the one to blame for not working harder enough, second from where I'm, Tech business is not yet a mainstream so the scene is almost dead still that's not a valid excuse. Me being childish contacting the same so-called investors who promised to invest and bailed in the last minute. I contacted them and let them know that the same concept was developed somewhere. Thier reply was so simple that they needed a working business plan like it works somewhere and they just apply to our market, They market themselves as investing in original ideas though.
A friend of mine said lesson learned and this should only boost your confidence that you're on the right path and you understand what solutions you can provide to the tech market. What else can I learn from my experience? <comment><USER2> It's hard to say what you might learn from this experience given your explanation. Some things that I would point out, is that there is no such thing as an original idea, however this is such thing as a differentiated idea. If you are building something that someone else already has, then they will win no matter how much money you raise. The other bit I would suggest is that investors like to believe that their investment will provide at least a 10x return. Unfortunately, it's really hard to tell which ideas will go huge and which won't, so they tend to work from a baseline set of metrics such as monthly recurring revenue, users, sales pipeline, go to market strategy, and also verifying that your goals are all reasonable. I'm not sure how much of this applies to you, but I would say that if you really believe that you have something special then keep going. Less than 1% of companies in the US get VC funded, I imagine that the number is even smaller in your area. Don't get discouraged.</comment>


Ask HN: Advice for avoiding being a bad participant in meetings - <AUTHOR> What negative patterns do you come across in meetings that lead to non productive outcomes and what advice would you give those participants responsible?<p>What mistakes did you make yourself in meetings that you have corrected as you progressed in your career? <comment><USER2> My experience of meetings and their players.

The Sheriff; my way or the highway.

The Bluffer; Yip, that's been done.

The Hard Worker; Always segways to their latest piece of work.

The Brain; Says little, has manufactured outcome of meeting before the meeting started.

The Joker; Makes everyone laugh, is heard the least.

I'm personally trying to be a little bit less the joker and the hard worker.

<comment><USER3> The Toll Collector; powerful, knows where things are, knows who to ping or where to look, but never offers help voluntarily--only offers it, barter style --for taking something off of their plate.

Also goes by "Brent"

</comment></comment><comment><USER3> Planning meetings because $thing has to be discussed, but failing to organize the meeting, set an agenda or scheduling meetings without any planned outcomes and takeaways.

The latter half of this isn't always possible, but lately in my career I've made it a point to never call a meeting that doesn't bring up at least 2-3 things that will be/need to be acted upon or looked into by someone (even if I'm that person) based on what gets discussed.

</comment><comment><USER4> I try and ask myself these questions every time I attend a meeting: \- Why am I here? \- How could I avoid being here next time?

If you aren't providing value or the meeting is not providing value to you then you shouldn't be there. Perhaps the meeting shouldn't even exist? If this could have been a slack message or email or perhaps part of another existing meeting then you also shouldn't be there.

</comment><comment><AUTHOR> Not so much being a bad participant, but something I've found are developers who propose a solution to a problem outside of a meeting, but when it's then thrown out to the group for suggestions they remain quiet. Hard to know what advice to give for building confidence.</comment>


Stories from the interview room - <AUTHOR> http://ayende.com/blog/162849/stories-from-the-interview-room

<comment><AUTHOR> Part II: [http://ayende.com/blog/163105/stories-from-the-interview- roo...](http://ayende.com/blog/163105/stories-from-the-interview-room-part-ii)

About the interview room series: [http://ayende.com/blog/163234/about-the- interview-room-serie...](http://ayende.com/blog/163234/about-the-interview- room-series)</comment>