title: Minor LSTM input alteration decimates learning
body: <p>The problem is as follows: I have an few toy networks which read in sentences in the form of a series of words codified as w2v(size 300) vectors. The aim of the game is to classify these sentences into one of two categories. Simple enough. The separation is not easy and the best of my networks managed around 60-65% accuracy. I then got hold of some (theoretically) usefull extra info relating to my end classification which will apply to many of the words in these sentences, it comes in the form of a small float. I took these floats, for the words to which they applied, and appended them to the w2v vectors of these words, I also appended zero to the vectors of words which didn't have this information. I then had vectors size 301 being fed in ... and now everything flatlines around 50%. What is going on?</p>
  
  <p>I could appreciate that the extra info may in fact be useless, or wrong, but as I see it, it either correlates(or anti corellates) with the outcome, or is noise (or so complex that the correlation hasn't been discovered so we just treat it as noise), whatever the outcome - the network should either learn to take advantage of this extra info, or disregard it. In my mind I can't see why I am no longer able to obtain my previous accuracy at least, particularly given that it is such a minor adjustment to the input data.</p>
  
  <p>As per a comment request I've attatched the relevent code - apologies for its hacky nature, this is all quite experimental at the moment</p>
  
  <p>Relevant code</p>
  
  <pre><code>    self.log('Beginning Training')
      # fix random seed for reproducibility
      numpy.random.seed(7)
      # load the dataset but only keep the top n words, zero the rest
      X,y = self.load_test_train_data(corpus_manager,sentiment_analyser)
  
  
      # truncate and pad input sequences
      max_review_length = 25
      self.log('Padding input vectors to length ', max_review_length)
      X = sequence.pad_sequences(X, maxlen=max_review_length)
      self.log('Padding complete.')
      # create the model
      embedding_vecor_length = 32
  
  
  
      self.log('Initialising model.')
      model = Sequential()
  
      model.add(LSTM(50,return_sequences=True,input_shape=(25, 301)))
      model.add(LSTM(50,return_sequences=True))
      model.add(LSTM(10))
      model.add(Dense(1, activation='sigmoid'))
  
      model.compile(loss='binary_crossentropy',optimizer='Adam', metrics=['accuracy'])
  
      self.log(model.summary())
      self.log('Training model.')
  
  
      avg_score = 0
  
  
      y = numpy.array(y)
  
  
      X_train = None
      X_test = None
  
      for train_index, test_index in self.kf.split(X):
          #for train_index, test_index in zip(self.train_indicies, self.test_indicies):
  
          X_train = X[train_index]
          X_test = X[test_index]
          y_train, y_test = y[train_index], y[test_index]
  
          self.log('Split into ', len(X_train), ' training and ', len(X_test),' testing events.')
  
  
          y_train = numpy.array(y_train)
          X_train = numpy.array(X_train)
  
          model.fit(X_train, y_train, epochs=250, batch_size=64,shuffle=True)#64
          # Final evaluation of the model
          scores = model.evaluate(X_test, y_test, verbose=1)
  
          avg_score += scores[1]
  
          self.model = model
  
          if scores[1] &gt; self.model_score:  
              #so we know which set the final model was actually tested and trained over
              self.optimum_X_train, self.optimum_X_test = X_train, X_test
              self.model_saccuracy = scores[1]
              self.model = self.model
              self.optimum_y_train , self.optimum_y_test = y_train, y_test
  </code></pre>
  
  <p>Also this function</p>
  
  <pre><code>def load_test_train_data(self, corpus_manager,sentiment_analyser,
          input_database='../data/processed_articles/ndaq_articles.csv',
          company_names_csv='../data/SP_500_company_list.csv',
          company_history_list='../data/financial_trading_history/sp_trading_history.pkl',
          previously_transformed_articles='../data/previously_transformed_articles.pkl'
          ):
      """
      Retrieve a set of training and testing data from the corpus manager
      in prepration for training or validation. 
      Parameters
      ----------
      corpus_manager: A CorpusManager class already fitted with processed articles
      Returns
      -------
          X_train, y_train: Training feature space and target space data points 
          X_test, y_test: Orthogonal test feature space and target space data points to the training set
      """
  
  
  
      article_manager = ArticleManager(input_database=input_database,
                                      verbosity=0)  
  
      data  = pandas.read_csv(input_database, encoding = "ISO-8859-1") 
      data = data[ data.title.str.contains("UPDATE") == False]
      data = data[ data.title.str.contains("DIARY") == False]
  
      '''
      for text in articles:
          get_text_feautre_vector(self, article_text)
          for each vector get its sentiment
          def evalaute_article(self, article):
          append it to end of feature vector
      ''' 
      sent_model_class = sentiment_analyser
  
      X = []
      if not os.path.isfile(previously_transformed_articles):
          self.log(" Transforming and storing articles.")
          with tqdm(total=len( list( data['title'] )) ) as pbar:
              for index, row in data.iterrows():
  
                  word_and_sent_vecs = corpus_manager.word2vec_and_sent(string = row['title'], sentiment_analyser = self.sentiment_analyser)
  
  
                  X.append(word_and_sent_vecs)
                  pbar.update(1)
  
          with open(previously_transformed_articles, 'wb') as handle:
              pickle.dump(X, handle, protocol=pickle.HIGHEST_PROTOCOL)
      else:
          with open(previously_transformed_articles, 'rb') as handle: 
              X = pickle.load(handle)
  
  
      hist = CompanyHistoryManager(article_manager.get_companies())
      hist.load_history(company_history_list)
      company_names_symbols  = pandas.read_csv(company_names_csv, encoding = "ISO-8859-1") 
      self.log(company_names_symbols)
      self.log(company_names_symbols.columns)
  
      Y=[] 
      if not os.path.isfile('stock_motion.pkl'):
          self.log(" Creating data on stock movements.")
  
          with tqdm(total=len( data.index )) as pbar:
              for _, row in data.iterrows():
  
                  #sym = company_names_symbols.loc[company_names_symbols['Symbol'] == row['symbol']]
                  stock_rose = hist.did_it_rise(row['date'], row['symbol'])
                  #print(stock_rose)
                  Y.append(stock_rose)
  
                  pbar.update(1)
  
          with open('stock_motion.pkl', 'wb') as handle:
              pickle.dump(Y, handle, protocol=pickle.HIGHEST_PROTOCOL)
  
      else:
          with open('stock_motion.pkl', 'rb') as handle: 
              Y = pickle.load(handle)
  
      #print(Y)
  
  
      #remove all the nones!!!!!!!!!!!!!!!!!!!!!! do this more efficiently
      self.log('Removing faulty data (Where we could not obtain the stock movement data)')
      X_purged = []
      Y_purged = []
      count = 0
      count1 = 0
      with tqdm(total=len(X)) as pbar:
          for i,j in zip(X,Y):
              if j == 'Invalid':
                  count += 1
  
              elif  j == 'Static':
                  count1 += 1  
              else:
                  X_purged.append(i)
                  Y_purged.append(j)
  
              pbar.update(1)
  
      self.log(100*(count/len(X)),'%% of the data was removed due to an inability to obtain the data.')
      self.log(100*(count1/len(X)),'%% of the data was removed due to the resultant stock movement being zero.')
      X = X_purged
      Y = Y_purged
  
      # Separate the articles into test and train - we used a fixed random state
      # so that our results are the same between runs
  
      #X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, random_state=42)
  
      self.kf = KFold(n_splits=5, random_state=42, shuffle=True)
  </code></pre>
  
  <p>and finally, the possibly erroneous funcion</p>
  
  <pre><code>def word2vec_and_sent(self, string, sentiment_analyser):
  
      w2v = self.w2v_model
      words = string.lower().split()
      vectors = []
      word_sent=[0]
  
      skip = False
      skip_count = 0
      for word in words:
          skip = False
          try:
              word_vec = w2v.wv[word]
          except:
              skip = True
              skip_count = skip_count + 1
          if not skip:
  
              if word in self.keywords:
                  bm25_vec = self.get_text_feautre_vector(word)
                  word_sent = sentiment_analyser.evalaute_article(np.array(bm25_vec).reshape(1, -1))
              else:
                  a=0
                  wordsent = [a]
  
              vec = word_vec.tolist()
  
  
              vec.append(word_sent[0])
  
  
  
              vectors.append(vec)
      return(vectors)
  </code></pre>