Improve Pandas dataframe filtering speed I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves.

I tried to split the original dataset into 3 sub-dataframes based on some simple rules. However, it takes a long time to execute the code. About 15-20 seconds just for the filtering.

Any alternative way that will improve the performance of the code?

import pandas as pd

#read dataset
df = pd.read_csv('myData.csv')

#create a dataframe with col1 10 and col2 &lt;= 15
df1 = df[(df.col1 == 10) &amp; (df.col2 &lt;= 15)]
df = df[~df.isin(df1)].dropna()

#create a dataframe with col3 7 and col4 &gt;= 4
df2 = df[(df.col3 == 7) &amp; (df.col4 &gt;= 4)]
df = df[~df.isin(df2)].dropna()


In the end, I have the df1, df2, df dataframes with the filtered data.
The concept to understand is that the conditional is actually a vector.  So, you can simply define the conditions, and then combine them logically, like:

condition1 = (df.col1 == 10) &amp; (df.col2 &lt;= 15)
condition2 = (df.col3 == 7) &amp; (df.col4 &gt;= 4)

# at this point, condition1 and condition2 are vectors of bools

df1 = df[condition1]
df2 = df[condition2 &amp; ~condition1]
df = df[~ (condition1 | condition2)]


This will be considerable faster as it only evaluates the conditional once.  Then it uses them to perform indexed lookup to create the new smaller dataframes.