How to arrange the sets to predict y on x in time series? I'm performing my first NN with my own data and while I was already tuning the parameters I stumbled over an aspect which confuses me now such that I'm not sure what is right and what is wrong..

Given this (head of the) data df_train1_raw:

                     timestamp       A_phsA
2018-02-05 14:00:00 1.517839e+09    856.436487
2018-02-05 15:00:00 1.517843e+09    859.653339
2018-02-05 16:00:00 1.517846e+09    836.635463
2018-02-05 17:00:00 1.517850e+09    801.097284
2018-02-05 18:00:00 1.517854e+09    794.855960
                       (...)


The timestamp column is basically the index just converted into numeric so I can use these time information for the nn model.

Goal: Predict A_phsA on timestamp

First, I create the train and test sets:

# Prepare data
X_train1_raw = df_train1_raw.values
y_train1_raw = X_train1_raw

# Split data into appropriate sets
## Standardize and scale data

scaler = StandardScaler()
tscv = TimeSeriesSplit(n_splits = 5)
pyplot.figure(1)
index = 1

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();




This looks reasonable. When I plot the loss and val_loss values of the nn later it also looks reasonable.

But what struggles me actually is this line at the beginning:

y_train1_raw = X_train1_raw


I can't tell if it is plain stupid or if I can't get my head around it anymore. The reason is, when I look for example at KFold:

X =  list(range(10))
print (X)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
y = [x*x for x in X]
print (y)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

kf = KFold(n_splits=5)
X = np.array(X)
y = np.array(y)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print(“X_test: ", X_test)


they have different X and y. Which makes sense I would say.
But when I adjust my code accordingly

X_train1_raw = df_train1_raw.iloc[:, 1].values
y_train1_raw = df_train1_raw.iloc[:, 2].values


I get an error when performing

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();



  ValueError: Expected 2D array, got 1D array instead:
  array=[1.5178392e+09 1.5178428e+09 1.5178464e+09 1.5178500e+09
  1.5178536e+09


And I don't understand why. Which part is not correct?

edit: Is it scaler.fit_transform() ?
fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
     X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(X_train1_raw[test_index].reshape(-1,1))
     y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(y_train1_raw[test_index].reshape(-1,1))
     pyplot.subplot(510 + index)
     pyplot.plot(X_train1[:, 1])
     pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
     index +=1
pyplot.show();


This should fix your code and this happens because the Standard Scaler expects inputs as samples where each sample is an array of n_features to transform. The reshape(-1,1) does that for you if you have got a single feature to scale.