Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?
If &quot;variables&quot; refers to training examples:
You can use Stochastic Gradient Descent (SGD) where each iteration uses one training example.
Or you could use Mini-Batch Gradient Descent where each iteration uses a partition of the training set.  SGD is Mini-Batch Gradient Descent where the partition size is one training example.

If &quot;variables&quot; refers to features:
You should use dimensionality reduction to reduce your number of features.  For instance, you can use Principal Component Analysis (PCA) to reduce your feature vector size while maintaining high variance.  This would also help your models train significantly faster.