Spreadsheet column membership test
Unfortunately I am not experienced with Pandas especially on large files. But I would have tried a more 'native' solution like VBA but it is because I have some experience with it. There is a chance that VBA (in the form of a macro) would perform faster.

Perhaps you can try a hybrid approach, for example add some temporary columns to the files you are processing in Python, and fill those columns with formulae like COUNTIF etc.
The aim is to let Excel do the computationally-intensive stuff. And then you let your script loop row by one using the precalculations made by Excel.

When you are done, you remove the temporary columns or simply close the file discarding changes.
This may not be the more elegant approach, but I am not surprised that an interpreted language like Python with all the middleware involved is slow for this kind of task.

Another option is to dump the sheets to a database, MS Access or SQLite, add some indexes where appropriate and use SQL to fill out the gaps and then re-import the result to Excel.

This is more a hack than a real solution. My considerations would be:


is this a one-shot operation or a process that you will have to repeat ? This is to answer the question: how much time and energy are you willing to invest in this ?
You have not mentioned the purpose of the Excel files, and you've not shared the structure or a data sample. 300K columns for an Excel file is very big. I am wondering if by any change you are on assignment with some corporate client where clerks use Excel like a database. This is a very common scenario unfortunately. Then I would advise them to use a better solution. The whole process and usage is flawed and the solution can hardly be elegant.


Probably there are solutions already available on the market, including free and open source software. From experience there is little added value trying to patch a flawed process.

Everybody uses Excel, so Excel files proliferate everywhere and the result is poorly-structured data that is hard to exploit.
Using Excel is often the problem and the solution is to move away from Excel.