Q: Pandas KeyError: '[ 9000 31342 26296 25991 13449 1481 15566 15507 11247 18206] not in index' I'm trying a first neural network on Kaggle's digit recognizer dataset
In anaconda 3, as well as a direct Kaggle kernel, the classifier.fit call returns a key error I don't understand
The last call is as follows:
File "C:\Users\darkredhorse\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 1231, in _convert_to_indexer
raise KeyError('%s not in index' % objarr[mask])

KeyError: '[ 9000 31342 26296 25991 13449  1481 15566 15507 11247 18206] not in index'
#Import Pandas
import pandas as pd
import numpy as np

#Read csv dataset
dataset=pd.read_csv('Train.csv')

#X=independant variables=the image data
X=dataset.iloc[:,1:]

#y=dependant variable= the written digit label 0-9
y=dataset.iloc[:,:1]


#Grab the automatic data splitter from sklearn
from sklearn.model_selection import train_test_split

#Create the Training/Testing split for my cross validation
X_train,X_test,y_train,y_test = 
train_test_split(X,y,test_size=0.2,random_state=0)

#Feature scaling to make the ANN's life easier
from sklearn.preprocessing import StandardScaler
sc=StandardScaler()
sc.fit_transform(X_train)
sc.transform(X_train)
sc.transform(X_test)

#Import keras to create the sequential network structure
import keras
from keras.models import Sequential
from keras.layers import Dense

#Initialize the network
classifier = Sequential() 

classifier.add(Dense(units=400,kernel_initializer='uniform',activation='relu',input_dim=784))

#Add the first second hidden layer, I figure this requires ten for 10 digits
classifier.add(Dense(units=10,kernel_initializer='uniform',activation='relu'))

#Add the output layer, an analog digit value
classifier.add(Dense(units=1,kernel_initializer='uniform',activation='sigmoid'))

# Compiling the ANN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

# Fitting the ANN to the Training set
classifier.fit(X_train, y_train, batch_size = 10, epochs = 10)

# Part 3 - Making predictions and evaluating the model

# Predicting the Test set results
y_pred = classifier.predict(X_test)


A: Your CSV file seems to have a space/tab as a deimiter.
So try to parse it as follows:
#Read csv dataset
dataset=pd.read_csv('Train.csv', delim_whitespace=True)

if Train.csv doesn't have column names (labels):
dataset=pd.read_csv('Train.csv', header=None, delim_whitespace=True)


A: Try to convert pandas DataFrames into numpy arrays before feeding into fit(). This error occurs because the X_train[batch_idx] function call operates on columns instead of rows, unlike numpy arrays.
Also, it seems that you're using StandardScaler in a wrong way. StandardScaler.transform() does not modify data in place. You'll need to change the function calls into:
sc = StandardScaler()
sc.fit(X_train)
X_train = sc.transform(X_train)
X_test = sc.transform(X_test)

The output of StandardScaler.transform() will be a numpy array, so the error at fit() should also be fixed.
