Fastest way to compare every element with every other in np array of strings
tl;dr
You don't want that.
Details
First let's note that you're actually building a triangular matrix: for the first element, compare it to the rest of the elements, then repeat recursively to the rest. You don't use the triangularity, though. You just cut off the diagonal (each element is always equal to itself) and merge the rows into one list in your example.
If you sort your source list, you won't need to compare each element to the rest of the elements, only to the next element. You'd have to keep the position with element using a tuple, to keep track of it after sorting.
You would sort the list of pairs in O(n log n) time, then scan it and find all the matches if O(n) time. Both sorting and finding the matches are simple and quick in your case.
After that, you'd have to create your 'bit vector', which is O(n^2) long. It would contain len(your vector) ** 2 elements, or 57600 million elements for a 240k-element vector. Even if you represented each element as one bit, it would take 53.6 Gbit, or 8.7 GBytes of memory.
Likely you don't want that. I suggest that you find a list of pairs in O(n log n) time, sort it by both first and second position in O(n log n) time, too, and recreate any portion of your desired bitmap by looking at that list of pairs; binary search would really help. Provided that you have much fewer matches than pairs of elements, the result may even fit in RAM.
I have a numpy array of strings, some duplicated, and I'd like to compare every element with every other element to produce a new vector of 1's and 0's indicating whether each pair (i,j) is the same or different.
e.g. ["a","b","a","c"] -> 12-element (4*3) vector [1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1]
Is there a way to do this quickly in numpy without a double loop through all pairs of elements? My array has ~240,000 elements, so it's taking a terribly long time to do it the naive way.
I'm aware of numpy.equal.outer, but apparently numpy.equal is not implemented on strings, so it seems like I'll need some more clever way to compare them.