Where does pandas store the DataFrame while the program is running?
Yes it is in memory, and yes when the dataset gets too large you have to use other tools.
Of course you can load data in chucks, process one chunk at a time and write down the results (and so free memory for the next chunk).
That works fine for some type of process like filtering and annotating while if you need sorting or grouping you need to use some other tool, personally I like bigquery from google cloud.
Is it in memory?
If so, then it doesn't matter if I import chunk by chunk or not because eventually, when I concatenate them, they'll all be stored in memory. 
Does that mean for a large data set, there is no way to use pandas?