Similarity of search results using Jaccard
I realised if I used the format

X = np.array([[5,4,2,3,1,0,0,0,0],[4,0,0,0,0,5,3,2,1]])


and a sparse matrix from the scipy module, I could use efficient matrix calculations. This ran in about 2 seconds for 7k samples and 400 features. 


from scipy.sparse import csr_matrix
import numpy as np


def jaccard_sim_matrix(X):
    """X is an integer array of features"""

    sparseX = csr_matrix(X)

    # make a binary version of the matrix
    binX = sparseX
    binX.data[:] = 1

    intersection = ((sparseX * binX.T) + (binX * sparseX.T))

    rowwise_sum = np.sum(sparseX, axis=1)
    union = np.repeat(rowwise_sum, intersection.shape[0], axis=1) + np.repeat(rowwise_sum.T, intersection.shape[0],
                                                                              axis=0)

    return intersection / union


Then of course I got the distance by 1 - similarity