Filtering a large Pandas DataFrame based on a list of strings in column names
You can drop columns from a dataframe by applying string contains with regular expression. Below is an example
df.drop(df.columns[df.columns.str.contains('^abc')], axis=1)
Stack Overflow Family,
I have recently started learning Python and am using Pandas to handle some factory data. The csv file is essentially a large dataframe (1621 rows × 5633 columns). While I need all the rows as these are data of each unit, I need to filter many unwanted columns. I have identified a list of strings in these column names that I can use to find only the wanted columns, however, I am not able to figure out what a good logic here would be or any built in python functions.
dropna is not an option for me as some of these wanted columns have NA as values (for example test limit)
dropna for columns with all NA is also not good enough as I will still end up with a large number of columns.
Looking for some guidance here. Thank you for your time.