title: Is my code inefficient or is my task hard?
body: <p>hope someone can help here?</p>
  
  <p>I've written some code to perform a look-up task, and it's taking a long time to do it––far longer than it seems it should. For confidentiality reasons, I can't describe the exact task, but I can give one that's directly analogous.</p>
  
  <p>Let's say that I have a database containing a list of words and a set of values corresponding to these words. Say, colours and how much these colours are liked. </p>
  
  <p>Now, I have a text, and I want to compare all the words in this text to my database of colours, and where a match exists, I extract the 'liking' rating from the database and record it. When all words in the text have been matched (or not) with the database, the sum of the 'liking' ratings is my output. </p>
  
  <p>An equivalent to the code I've written is below, and it works fine for the toy example I've given. However, my actual problem contains a database of 40k entries, and the text typically has about 500 words; most of the words in the text will be in the database. And when I run it, it takes hours to execute. I understand that matching 500 words against a database of 40k entries means on the order of 20m comparisons. Still, <em>hours</em>? </p>
  
  <p>Can anyone suggest whether I'm doing a computationally intensive problem with limited hardware, or of my code is just massively inefficient?</p>
  
  <p>Thanks!</p>
  
  <pre><code>import pandas as pd
  import nltk as nltk
  
  ####### Creates toy data to test code on ###
  
  colour = ['red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet']
  values = [1,2,3,4,5,6,7]
  
  df1 = pd.DataFrame({'colour': ['red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet']})
  
  df2 = pd.DataFrame({'value': [1,2,3,4,5,6,7]})
  
  dfs = [df1, df2]
  
  ###### Code proper begins below
  
  data = pd.concat(dfs, axis = 1) ######## Dataframe
  
  
  #### Sample words 
  words = ['red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet', 'cyan', 'white', 'black', 'pink', 'grey', 'scarlet']
  
  
  #### Lists into which words are put post-analysis
  rating = 0
  rated = []
  unrated = []
  
  ####### Code
  
  for i in words:
      for j in range(len(data['colour'])):
          if i == data['colour'][j]: 
              rating = rating + data['value'][j]
              rated.append(i)
              break
          elif i not in unrated and i not in data.values: ### Ensures each unrated word is entered only once.
              unrated.append(i)
  </code></pre>