Dealing with NaN for predictive models
Going according to your problem, you should set the NaN values to 0. It follows logically as the number of minutes spent with the nurse or doctor will be zero if the patient hasn't visited them yet. And yes as you said filling with mean is not correct as it would not represent the problem correctly.

Also, if you fill the NaN values with big number such as 9999 they might be treated as outliers which is not the case. Hence, purely thinking from a logical point of view you should replace them with 0. But I also suggest you to run trial experiments with some other values such as large values. Machine Learning is a lot about hyperparameter tuning and in this case how you replace your NaN values behaves like a hyperparameter!