Why is Spark's LinearRegressionWithSGD very slow locally?
Spark is designed to be distributed across a cluster and use stochastic gradient descent (SGD) to optimize linear regression.
There is overhead for cluster infrastructure (even when the &quot;cluster&quot; is a single local node). Also, SGD is an iterative method that uses many batches to find a solution.
Given that your problem is 70k rows, it would be better to use a single node framework (e.g., scikit-learn) and ordinary least squares (OLS), a closed form solution, to optimize linear regression. Those two changes will greatly speed up training.