title: SciKit-Learn Text Classification from ODBC
body: <p>I'm trying to adapt <a href="http://scikit-learn.org/stable/auto_examples/document_classification_20newsgroups.html" rel="nofollow">this example</a> to some social media data I have in a SQL server database.</p>
  
  <p>I've intentionally forced both the training and test sets to only have social media posts that contain the word 'bunches'. Therefore I would expect an extremely high f-score for this word when I run it through all the algorithms. Instead I'm getting f-scores of around 2-4%. I have a feeling that I'm not feeding the data to the algorithms correctly.</p>
  
  <pre><code>from __future__ import print_function
  import numpy as np
  from time import time
  from sklearn.feature_extraction.text import TfidfVectorizer
  import pyodbc
  import pprint
  
  #local windows connection
  train = []
  db = pyodbc.connect('DRIVER={SQL Server Native Client 11.0};SERVER=SERVER_IP;DATABASE=DB_NAME;Trusted_Connection=Yes;')
  cursor = db.cursor()
  training_query = "SELECT top 2 percent postTitle FROM dbo.All_CH_Posts where monitorID ='1168136050' and postTitle like '%bunches%' ORDER BY NEWID()"
  trainquery = cursor.execute(training_query)
  traindata = cursor.fetchall()
  for row in traindata:
      train.extend(row)
  
  test = []
  test_query = "SELECT top 1 percent postTitle FROM dbo.All_CH_Posts where monitorID ='1168136050' and postTitle like '%bunches%' ORDER BY NEWID()"
  testquery = cursor.execute(test_query)
  testdata = cursor.fetchall()
  for row in testdata:
      test.extend(row)
  print('traindata')
  pp.pprint(traindata)
  print('testdata')
  pp.pprint(testdata)
  print('data loaded')
  
  # split a training set and a test set
  y_train = train
  y_test =test
  
  
  print("Extracting features from the training dataset using a sparse vectorizer")
  t0 = time()
  vectorizer = TfidfVectorizer(decode_error='ignore',sublinear_tf=True,
                               stop_words='english', lowercase=True, min_df=20)
  X_train = vectorizer.fit_transform(train)
  duration = time() - t0
  
  print("Extracting features from the test dataset using the same vectorizer")
  t0 = time()
  X_test = vectorizer.transform(test)
  duration = time() - t0
  print("n_samples: %d, n_features: %d" % X_test.shape)
  
  feature_names = np.asarray(vectorizer.get_feature_names())
  print(feature_names)
  </code></pre>
  
  <p>I intentionally set the min_df high to get a look at what words are in my sparse matrix:</p>
  
  <pre><code>n_samples: 237, n_features: 26
  ['almonds' 'amp' 'best' 'bowl' 'box' 'bunches' 'cereal' 'cheerios' 'crunch'
   'day' 'don' 'eat' 'eating' 'good' 'gt' 'honey' 'http' 'just' 'like' 'lol'
   'love' 'miss' 'morning' 'oats' 'rt' 'want']
  </code></pre>
  
  <p>So what am I doing wrong? Or am I thinking about this problem the wrong way/have a misconception of text classification?</p>
  
  <p><a href="https://drive.google.com/file/d/0ByAFICf5wlFqX00tMWpTem9oalE/view?usp=sharing" rel="nofollow">Here is my training set.</a></p>
  
  <p><a href="https://drive.google.com/file/d/0ByAFICf5wlFqOWp4YWFpc3RBN28/view?usp=sharing" rel="nofollow">Here is my test set.</a></p>