About 1M rows of data. Should I restrict myself to few columns as well? I'm trying to build a predictive model from about 1 million rows of data. My goal is to predict a certain numerical value.

I have the intuition that I should use very few numerical binary columns so I don't get data points that are too separated, a.k.a., the curse of dimensionality. Is this true? Besides, is it the same for numeric columns?
Did you try it?

In general, I believe the curse of dimensionality is way overrated. As a rule of thumb, the curse of dimensionality refers to having more columns than rows. I doubt that you have 1M columns...

A robust algorithm, such as Logistic Regression, can often deal with sparse data. I sometimes get the feeling that this algorithm doesn't get the credit it deserves. Especially in production (due to high bias and low variance), I prefer it whenever possible. Nothing worse than an algorithm that fails in production...