Why this calculation of weight vector in linear regression is only for small dataset?
$X$ is your design matrix (data), so can get large, both in number of data points and number of predictors. The issues are indeed:


Forming $XX^T$ may be computationally expensive
Computing it explicitly may be inaccurate due to floating-point roundoff, as you're adding lots of products
Inverting a matrix can be expensive here if the number of predictors is large as the Gramian is large
With lots of predictors and a big Gramian, it's more likely to be ill-conditioned and suffer from numerical instability