Big array and MemoryError: Unable to allocate memory (in Python) I am trying to create a predictive model using linear regression with a dataset that has 157673 entries.
The data (in a csv file) is in such format:
Timestamp,Signal_1,Signal_2,Signal_3,Signal_4,Signal_5
2021-04-13 11:03:13+02:00,3,3,3,12,12

My current code:
filename = 'test.csv'
df = pd.read_csv(filename , parse_dates=['Timestamp'], header=0)
df['Timestamp'] = pd.to_numeric(pd.to_datetime(df['Timestamp']))
u, v, w, x, y, z = df.values.T

X = np.asarray([v, w, x, y, z])
Y = np.asarray([u, u, u, u, u])
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

When printing out the shape of both X and Y it is (5, 157673) (after putting u 4 more times in the Y array, since it would otherwise give the error ValueError: Found input variables with inconsistent numbers of samples: [5, 1]).
However now I am running into the error MemoryError: Unable to allocate 185. GiB for an array with shape (157673, 157673) and data type float64.
Why is that? There must be a mistake somewhere and if not, why is it suddenly in the shape of (157673, 157673) instead of (6, 157673) ?
quick fix would be to change the data format - I can't see how your data looks like so my suggestion stay theoretical without example

float64 is the most expensive one. using float32 or float16 - depending how much precision point you need. If they are just integer depending on the range you can use even int8 (if less than 255) - use dtypes parameters in the pd.read_csv function. or convert them later on using astype.

Timestamp - decides what precision of time is needed - if you are fine with days - don't store hours, minutes and others

alternatively usse DataTable https://github.com/h2oai/datatable - Pandas is not the most efficient library when it comes to work with big data. however, I think the first 2 points will fix your issue. Because (157673, 157673) is not too big but 185. GiB sounds too much for it !