How to treat column with potentially meaningful NaNs
What you are asking about is called imputation - https://en.wikipedia.org/wiki/Imputation_(statistics)

Imputation basically means replacing NaNs with other data. There are lots of ways to do it and the method usually depends on the application. In this case you are the domain expert so you would be in the best position to decide what imputation method you should use.

Some common imputation methods are:

Mean (replace all NaNs with the average)

Median (replace all NaNs with the "middle" of the data)

MICE (Multiple Imputation by Chained Equations - probably overkill for your application, but worth looking into)

In your case I would go with mean imputation to start and then see what kind of results you get.