title: Getting array of arrays, need array of numbers. Ideas?
body: <p>I've been trying to process a dataset to feed into an LSTM model.
  Here is the code I've come up with to produce an array with specific timesteps forward and back for a times series dataset.</p>
  <pre><code>stepsForward = 3
  stepsBack = 7
  trainTestSplit = 0.75
  
  df_item = pd.DataFrame(df.Cannonball)
  
  minmax = MinMaxScaler().fit(df_item.values.reshape((-1,1)))
  df_normalize = minmax.transform(df_item.values.reshape((-1,1)))
  
  def walk_forward(X, stepsBack, stepsForward, trainTestSplit):
    n_records = len(X)
    split = int(n_records*trainTestSplit-stepsForward)
    x = []
    y = []
    for i in range(n_records):
      x.append(X[i : i + stepsBack])  
      y.append(X[i + stepsBack : i + stepsBack + stepsForward])
    
    xtrain = x[:split].copy()
    xtest = x[split + stepsForward : n_records - stepsForward].copy()
    ytrain = y[:split].copy()
    ytest = y[split + stepsForward : n_records - stepsForward].copy()
      
    print('train=%d, test=%d' % (len(xtrain), len(xtest))) 
    return np.asarray(xtrain), np.asarray(ytrain), np.asarray(xtest), np.asarray(ytest) 
  
  X_train, y_train, X_test, y_test = walk_forward(df_normalize, stepsBack, stepsForward, trainTestSplit)
  
  y_train_padded = pad_sequences(np.array(y_train), maxlen= stepsBack, 
                                 padding='post', dtype='float32')
  y_test_padded = pad_sequences(np.array(y_test), maxlen= stepsBack, 
                                 padding='post', dtype='float32')
  
  X_train_t = tf.convert_to_tensor(X_train, dtype = tf.float32)
  X_test_t = tf.convert_to_tensor(X_test, dtype = tf.float32)
  y_train_padded_t = tf.convert_to_tensor(y_train_padded, dtype = tf.float32)
  y_test_padded_t = tf.convert_to_tensor(y_test_padded, dtype = tf.float32)
  </code></pre>
  <p>At first glance the code should produce identical arrays for X_train and X_test. After running the code the arrays I get are:
  X_train[:3]</p>
  <pre><code>array([[[0.        ],
          [0.97938144],
          [0.97938144],
          [0.97938144],
          [0.98453608],
          [0.98453608],
          [0.98453608]],
  
         [[0.97938144],
          [0.97938144],
          [0.97938144],
          [0.98453608],
          [0.98453608],
          [0.98453608],
          [0.98453608]],
  
         [[0.97938144],
          [0.97938144],
          [0.98453608],
          [0.98453608],
          [0.98453608],
          [0.98453608],
          [0.98453608]]])
  </code></pre>
  <p>X_test</p>
  <pre><code>array([array([[0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289]]),
         array([[0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95360825]]),
         array([[0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95876289],
         [0.95360825],
         [0.95360825]])], dtype=object)
  </code></pre>
  <p>Does anybody know what's going on here?</p>
  <p>X_train has the correct format, X_test is an array of arrays instead of an array with deeper dimensions. I've tried several configurations and thought copying the datasets would fix it but no luck.</p>
  <p>Edit:
  Here is the error I am getting due to the X_test being an array of arrays when converting to a tensor.</p>
  <pre><code>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
    return array(a, dtype, copy=False, order=order)
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  TypeError: only size-1 arrays can be converted to Python scalars
  
  The above exception was the direct cause of the following exception:
  
  ValueError                                Traceback (most recent call last)
  &lt;ipython-input-8-4e51ce7906cd&gt; in &lt;module&gt;()
       33 
       34 X_train_t = tf.convert_to_tensor(X_train, dtype = tf.float32)
  ---&gt; 35 X_test_t = tf.convert_to_tensor(X_test, dtype = tf.float32)
       36 y_train_padded_t = tf.convert_to_tensor(y_train_padded, dtype = tf.float32)
       37 y_test_padded_t = tf.convert_to_tensor(y_test_padded, dtype = tf.float32)
  
  9 frames
  /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype)
      104       dtype = dtypes.as_dtype(dtype).as_datatype_enum
      105   ctx.ensure_initialized()
  --&gt; 106   return ops.EagerTensor(value, ctx.device_name, dtype)
      107 
      108 
  
  ValueError: setting an array element with a sequence.
  </code></pre>
  <p>Edit2:
  This is being ran in Google Colab.</p>
  <p>The following code produces an X_train that's an array of arrays like X_test. Still not sure what's causing this.</p>
  <pre><code>  x_copy = np.copy(x)
    x_copy2 = np.copy(x)
    y_copy = np.copy(y)
    y_copy2 = np.copy(y)
    xtrain = x_copy[:split]
    xtest = x_copy2[split + stepsForward : n_records - stepsForward]
    ytrain = y_copy[:split]
    ytest = y_copy2[split + stepsForward : n_records - stepsForward]
  </code></pre>
  <p>But this code produces the X_train and X_test just as in the format I first provided.</p>
  <pre><code>  x_copy = x.copy()
    x_copy2 = x.copy()
    y_copy = y.copy()
    y_copy2 = y.copy()
    xtrain = x_copy[:split]
    xtest = x_copy2[split + stepsForward : n_records - stepsForward]
    ytrain = y_copy[:split]
    ytest = y_copy2[split + stepsForward : n_records - stepsForward]
  </code></pre>