What is the name for the comparison used in C's memcmp?
The question is about a three-valued function in a situation where a boolean is often used (eg &lt;=).  I personally refer to this as the sign or signum: the sign of the difference of the comparands.  However I have not seen a widely-used term; the best supportable answer I can give is that there is none.

In Quicksort, when a three-way partition (based on this "signum") is used instead of two-way, it's called ternary (and there are ternary search trees based on the same distinction), but that's about as specific as it gets.  

My reasoning behind my naming this quantity signum is that comparison is often implemented as subtraction; the CMP instruction is essentially a SUB instruction that discards the actual result but retains the sign and zero flags.

Strings are obviously not numbers, but they have a commonly-used embedding into [0,1) that makes lexicographic comparison equivalent to subtraction and signum extraction.

As to the idea of a two-valued (==, !=) predicate, there is some precedent for this idea in contexts where the signum may not be obtainable.  Fagerbergh, Pagh et al. construct what they call a "unordered trie" on hashed string values, where the original order of the strings is obscured due to hashing, but the position where they first disagree is still known.  This practice also has some use in applying collations, where only the branching character needs to be looked up to get the collated order (the equal prefix is skipped).