Title: Selecting rows based on boolean mask - why is there a difference in performance?
Tags: python;pandas;numpy
Question: I've been working with pandas DataFrame objects and selecting rows based on column values.
I've noticed that if you select rows using .values first then it is about twice as fast. Why is this?
And is there any reason why you should use the first example if it is slower?
df = pd.DataFrame(np.random.randint(0, high=10, size=(1000, 4)), columns=['A', 'B', 'C', 'D'])
%timeit df_test = df[df['A'] == 9]
The slowest run took 4.98 times longer than the fastest. This could mean that an intermediate result is being cached.
1000 loops, best of 3: 363 µs per loop
%timeit df_test = df[df['A'].values == 9]
1000 loops, best of 3: 181 µs per loop

Here is the accepted answer: Pandas masking returns a boolean mask in the form of series or DataFrame. Numpy masking returns a boolean mask in the form of an array.
To map boolean mask to series or dataframe takes a bit of time than returning an array of booleans.
In other words mapping mask to index and returning a series is where the additional time went when you do df['A'] == 9 or else both of them would be same.
To illustrate :
df['A'] == 9
0    False
1    False
2    False
3    False
4    False
Name: A, dtype: bool
type(df['A'] == 9)
pandas.core.series.Series
df['A'].values == 9
array([False, False, False, False, False], dtype=bool)
type(df['A'].values == 9)
numpy.ndarray
So why series masking when numpy masking is way faster  ?
Lets say you have dataframe with index sorted in a different way
df = pd.DataFrame(np.random.randint(0, high=10, size=(5, 4)), columns=list('ABCD'))
A  B  C  D
0  4  9  1  5
1  8  6  5  0
2  5  5  9  5
3  2  5  7  5
4  1  1  7  2
df2 = pd.DataFrame(np.random.randint(0, high=10, size=(5, 4)), columns=list('ABCD'),index=[4,3,2,1,0])
A  B  C  D
4  0  4  5  8
3  9  6  7  2
2  0  9  8  6
1  2  6  2  7
0  7  2  8  7
Now you want to select the row in df2 that has 4 in column A of df based on their index value
# If you do numpy masking
df2.loc[df['A'].values==4] # First index will be selected no matter what the actual index is
A  B  C  D
4  0  4  5  8
df2.loc[df['A']==4] # Row with that index will be selected
A  B  C  D
0  7  2  8  7
Not just this there are more cases where you want index and column data to work with so series masking is required. Hope that explains things bit better.
Comment for this answer: Yes you can use second approach if you want. But there are cases where you dont just want the plain array of booleans. Let me add an example
Comment for this answer: Ok, that makes sense. Why would you ever want to use the first scenario over the second if the second is faster? And they seem to do exactly the same thing from what I can tell


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2017:00:12/02/17', 'FavoriteCount': 1.0, 'PostTypeId': 1, 'Tags': '<python><pandas><numpy>', 'Title': 'Selecting rows based on boolean mask - why is there a difference in performance?', 'ViewCount': 5081, 'id': 47608339, 'user_AboutMe': '', 'user_AccountId': 3590905.0, 'user_DisplayName': 'bgrantham', 'user_Reputation': 261.0, 'user_Views': 17.0, 'user_id': 2996994.0}}