How to create a big data frame in Python
You can use pandas.Dataframe.sparse.from_spmatrix.  It will create a Dataframe populated by pd.arrays.SparseArray from a scipy sparse matrix.
Pandas used to have explicit sparse dataframes, but in more modern versions there is no such concept.  Only normal pd.Dataframe populated by sparse data.