Efficiently testing if strings are present in 2D numpy arrays
Question: I have a list of sentences that I want to convert into a Pandas MultiIndex
(but don't worry, this question can probably be answered purely using numpy).
For example, lets say the sentences are:

    
    
    sentences = ['she went', 'I went', 'she and I']
    

To make an index I first need to get a unique list of words in all the
sentences. Each word will become an index. The result of this should be:

    
    
    words = ['she', 'went', 'I', 'and']
    

Then to work out what values of the index each row has, I need a 2d array of
booleans. Making this array is the main problem as I want it to be as
efficient as possible, and hopefully without relying on python data
manipulation at all. This 2D array can be in _either one of_ two different
formats:

  * An array of tuples. Each tuple contains booleans to indicate the presence of the given word in the row. This will be passed to `pandas.MultiIndex.from_tuples()` For example:
    
        tuples = [
        #"She went" contains "she" and "went", but not "I" or "and"
        (True, True, False, False), 
        #"I went" contains "I" and "went", but not "she" or "and"
        (False, True, True, False),
        #"She and I" contains "she", "I" and "and", but not "went"
        (True, False, True, True),
    ]
    

  * An array of arrays, one inner array for each word. This will be passed to `pandas.MultiIndex.from_array()`. For example:
    
        arrays = [
        # 'she' is in the first and third sentences
        [True, False, True],
        # 'went' is in the first and second sentences
        [True, True, False],
        # 'I' is in the second and third sentences
        [True, False, True],
        # 'and' is in the first sentence only
        [True, False, False],
    ]
    

* * *

Ideally the solution will convert sentences to an np array and work with that
from then on. My naiive implementation is so far this. Unfortunately I'm not
sure how to do this with numpy without list comprehensions

    
    
    import pandas as pd
    
    sentences = ['she went', 'I went', 'she and I']
    
    # Can this be done using numpy?
    split_sentences = [sentence.split(" ") for sentence in sentences]
    
    words = list(set(sum(split_sentences, [])))
    
    # Is there a built in way of doing this with numpy, for example np.intersect?
    tuples = [
        [True if word in sentence_words else False for word in words]
        for sentence_words in split_sentences
    ]
    
    index = pd.MultiIndex.from_tuples(tuples, names=words)
    


Answer: Here's one approach vectorizing the crux of the problem of finding the
intersections -

    
    
    # Split setences
    split_sentences = [sentence.split(" ") for sentence in sentences]
    
    # Get group lengths
    grplens = np.array([len(item) for item in split_sentences])
    
    # ID each word
    unq_words,ID = np.unique(np.concatenate((split_sentences)),return_inverse=True)
    
    # Get lengths
    N = len(ID)
    Nunq = len(unq_words)
    Nsent = len(sentences)
    
    # Get ID shift positions and thus row shifts for final 2D array output
    row_shifts = np.zeros(N,dtype=int)
    row_shifts[grplens.cumsum()[:-1]] = 1
    
    # Finally get output boolean array using ID and row_shift IDs
    out = np.zeros((Nsent,Nunq),dtype=bool)
    out[row_shifts.cumsum(),ID] = 1
    

Sample run -

    
    
    In [494]: sentences
    Out[494]: ['she went', 'I went', 'she and I', 'with dog', 'dog and she']
    
    In [495]: unq_words
    Out[495]: 
    array(['I', 'and', 'dog', 'she', 'went', 'with'], 
          dtype='|S4')
    
    In [496]: out
    Out[496]: 
    array([[False, False, False,  True,  True, False],
           [ True, False, False, False,  True, False],
           [ True,  True, False,  True, False, False],
           [False, False,  True, False, False,  True],
           [False,  True,  True,  True, False, False]], dtype=bool)
    

