title: AttributeError: 'numpy.ndarray' object has no attribute 'index'
body: <p>I'm working on a Quantile Regression Neural Network (QRNN) that can act as a forecaster for wind power as well as a detector for false data injection attacks. </p>
  
  <p>I pretty much finished it but I am trying to quantify it's accuracy by taking the median quantile (tau = 0.5) and comparing it to the actual wind power by using an RMSE (while also trying to plot the actual wind power vs the median quantile). However, I keep running into issues with it. </p>
  
  <p>Because q_hat (the predicted median quantile) is a numpy array and y_test (actual wind power test data) is a pandas dataframe, I have to convert y_test to numpy, but it's giving me this error: </p>
  
  <p>"AttributeError: 'numpy.ndarray' object has no attribute 'index'"</p>
  
  <p><strong>Here is the pinball_loss.py file that is needed for this code:</strong></p>
  
  <pre><code>import numpy as np
  import tensorflow as tf
  from keras import backend as K
  K.set_floatx('float64')
  
  # pinball loss function with penalty
  def pinball_loss(y, q, tau, alpha = 0.001, kappa = 0, margin = 0):
      """
      :param y: target
      :param q: predicted quantile
      :param tau: coverage level
      :param alpha: smoothing parameter
      :param kappa: penalty term
      :param margin: margin for quantile cross-over
      :return: quantile loss
      """
  
      # calculate smooth pinball loss function
      error = (y - q)
      quantile_loss = K.mean(tau * error + alpha * K.softplus(-error / alpha) )
  
      # calculate smooth cross-over penalty
      diff = q[:, 1:] - q[:, :-1]
      penalty = kappa * K.mean(tf.square(K.maximum(tf.Variable(tf.zeros([1], dtype=tf.float64)), margin - diff)))
  
      return quantile_loss + penalty
  </code></pre>
  
  <p><strong>Here is my code in Jupyter Notebook (I separated my cells by dashed lines for convenience):</strong></p>
  
  <pre><code>import warnings
  warnings.filterwarnings("ignore")
  
  import pandas as pd
  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn.model_selection import train_test_split
  
  import tensorflow as tf
  from tensorflow.keras import regularizers
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.callbacks import EarlyStopping
  from tensorflow.keras.layers import Dense 
  from tensorflow.keras import backend as K
  ---------------------------------------------------------------------------------------------------------
  data = pd.read_csv('2018 - Lillgrund (Sweden) Offshore Wind Farm Data.csv', header = [0,1])
  data = pd.DataFrame(data, columns= [['local_time','output','windspeed'],['Europe/Copenhagen','kW','m/s']])
  
  #Combine local time with Europe/Copenhagen and output with kW as one header.
  data.columns = data.columns.map('_'.join)
  ---------------------------------------------------------------------------------------------------------
  #Use the date column as the index.
  data['local_time_Europe/Copenhagen'] = pd.to_datetime(data['local_time_Europe/Copenhagen'])
  
  #Two weeks of data from 1/1/18 to 2/1/18 indexed hourly (744 total data points).
  date_rng = (data['local_time_Europe/Copenhagen'] &gt;= '2018-1-01') &amp; (data['local_time_Europe/Copenhagen'] &lt;= '2018-2-1 00:00:00')
  data = data.loc[date_rng]
  
  #Sets array as dates instead of row number
  data.set_index('local_time_Europe/Copenhagen', inplace=True)
  
  #Check to see index
  # data.index
  ---------------------------------------------------------------------------------------------------------
  # create lagged time series features
  lags = range(1, 6) 
  df = pd.DataFrame(data= data['output_kW'], index=data.index)
  df = df.assign(**{'{} (t-{})'.format("Windspeed", t): data["windspeed_m/s"].shift(t) for t in lags}).dropna()
  
  #Windspeeds are taken from hour before. i.e. original windspeed @ 6am was 10.124 m/s, windspeed @ t-1 (5am) is 9.596 m/s.
  ---------------------------------------------------------------------------------------------------------
  # create training and testing data
  X = df.drop(columns=["output_kW"])
  Y = df['output_kW']
  X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, shuffle=False)
  ---------------------------------------------------------------------------------------------------------
  #tau = np.arange(0.05, 1.0, 0.05) Original tau with multiple quantiles that I've already plotted
  tau = 0.5 #Single point Quantile prediction
  #N_tau = len(tau) #tau starts from 0.05 to 1, incremented by 0.05, which is 19 points.
  N_tau = 1
  N_features = 5 #Windspeed is the feature for X.
  hidden_dim = 40 # number of hidden nodes
  Lambda = 0.001 # L2 regularization
  
  # loss function parameters (no need to modify)
  loss_param={
      'tau': tau,
      'alpha': 0.01,
  }
  ---------------------------------------------------------------------------------------------------------
  def pinball_loss(y, q, tau, alpha):
      """
      :param y: target
      :param q: predicted quantile
      :param tau: coverage level
      :param alpha: smoothing parameter
      :param kappa: penalty term
      :param margin: margin for quantile cross-over
      :return: quantile loss
      """
  
      # calculate smooth pinball loss function
      error = (y - q)
      quantile_loss = K.mean(tau * error + alpha * K.softplus(-error / alpha) )
  
      return quantile_loss 
  ---------------------------------------------------------------------------------------------------------
  # add first layer to a sequential model
  model = Sequential()
  model.add(Dense(hidden_dim,
                  input_dim=N_features,
                  kernel_regularizer=regularizers.l2(Lambda),
                  activation='relu'))
  
  # add output layer
  model.add(Dense(N_tau))
  
  # compile model
  model.compile(loss=lambda Y, Q: pinball_loss(y = Y, q = Q, **loss_param), optimizer='Adam')
  
  # fit the model
  history = model.fit(X_train, y_train, epochs=1000, verbose=0, batch_size=32, validation_split=0.1);
  ---------------------------------------------------------------------------------------------------------
  plt.plot(history.history['loss']);
  plt.plot(history.history['val_loss']);
  plt.legend(('Loss', 'Val Loss'))
  plt.ylabel('Loss');
  plt.xlabel('Epoch');
  plt.title("Training Loss");
  ---------------------------------------------------------------------------------------------------------
  # estimate quantiles of testing data
  q_hat = model.predict(X_test)
  q_hat = pd.DataFrame(q_hat, index = X_test.index)
  ---------------------------------------------------------------------------------------------------------
  N_PI = int(N_tau/2)
  #y3 = y_test.to_numpy() #THIS ISN'T WORKING EITHER   
  y3 = np.array(y_test)
  plt.figure(figsize=(12,6))
  plt.plot(y3, color='red')
  x = y3.index.values
  for i in range(N_PI):
      y1 = q_hat.iloc[:,i]
      y2 = q_hat.iloc[:,-1-i]
      plt.fill_between(x, y1, y2, color='blue', alpha=str(1/N_PI))
  plt.title('RMSE: %.2f'% np.sqrt(sum((q_hat-y3)**2)/len(y3)));
  plt.ylabel('Wind Power')
  plt.xlabel('Time (hour)');
  </code></pre>
  
  <p><strong>The final cell above is where I get the following error:</strong> </p>
  
  <pre><code>---------------------------------------------------------------------------
  AttributeError                            Traceback (most recent call last)
  &lt;ipython-input-20-338503ab24ea&gt; in &lt;module&gt;
        4 plt.figure(figsize=(12,6))
        5 plt.plot(y3, color='red')
  ----&gt; 6 x = y3.index.values
        7 for i in range(N_PI):
        8     y1 = q_hat.iloc[:,i]
  
  AttributeError: 'numpy.ndarray' object has no attribute 'index'
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/u0JJz.png" rel="nofollow noreferrer">Here is what the QRNN looks like with multiple quantiles vs the actual wind power (red line)</a></p>
  
  <p><a href="https://i.stack.imgur.com/H2TjB.png" rel="nofollow noreferrer">Here is what happens after the "Atribute Error" above when I try to plot the Median Quantile vs the Actual Wind Power (red line) with the RMSE</a></p>
  
  <p>I've also tried doing the following with the code where I made edits to the cell where I established tau and the last cell:</p>
  
  <pre><code>tau = np.arange(0.49, 0.51, 0.01) #Single point Quantile prediction
  N_tau = len(tau) #tau starts from 0.05 to 1, incremented by 0.05, which is 19 points.
  N_features = 5 #Windspeed is the feature for X.
  hidden_dim = 40 # number of hidden nodes
  Lambda = 0.001 # L2 regularization
  
  # loss function parameters (no need to modify)
  loss_param={
      'tau': tau,
      'alpha': 0.01,
  }
  ---------------------------------------------------------------------------------------------------------
  N_PI = int(N_tau/2)
  
  plt.figure(figsize=(12,6))
  plt.plot(y_test, color='red')
  x = y_test.index.values
  for i in range(N_PI):
      y1 = q_hat.iloc[:,i]
      y2 = q_hat.iloc[:,-1-i]
      plt.fill_between(x, y1, y2, color='blue', alpha=str(1/N_PI))
  y3 = (y2.values+y1.values)/2 
  #y3 is averaging the upper and lower quantiles, which is technically wrong since it isn't the median...
  plt.title('RMSE: %.2f'% np.sqrt(sum((y3-y_test)**2)/len(y_test)));
  plt.ylabel('Wind Power')
  plt.xlabel('Time (hour)');
  </code></pre>
  
  <p>To which I get the following plot:
  <a href="https://i.stack.imgur.com/CHoFM.png" rel="nofollow noreferrer">This is when I do tau = np.arange(0.49, 0.51, 0.01) and N_tau = len(tau)</a></p>
  
  <p>Although this looks correct and is pretty much what I want, there is another problem! Every time I restart the entire Kernel, the RMSE keeps changing (gets smaller by a little bit each time, i.e. from 15,000 to 14950 to 14830, etc.).</p>
  
  <p>I am so close to finishing this, I just want to be able to validate the accuracy of my QRNN by comparing the median quantile with the actual wind power, but this part (which is probably a simple fix) is giving me a lot of trouble. Please help!</p>